Googlebot

Hi all.

Not sure where the best place to post this question so I'll start here.....

I just got an email from Google stating that: "Over the last 24 hours, Googlebot encountered errors while attempting to access your robots.txt. To ensure that we didn't crawl any pages listed in that file, we postponed our crawl."

I went to the robots.txt file. I don't have much in there.

User-agent: *
Disallow: /cgi-bin/

I was under the impression that I wanted to keep the bot out of the cgi-bin area of a site...Is that true? Is my disallow to that area throwing this error?

Also, can someone suggest what SHOULD be in the robots.txt file beyond what I have...? Maybe post an example of a working robots.txt.....

Thank you for any help you can offer.....I've been fighting this bot issue for months and can't seem to get it right. I'm starting to think this bot hates me.... :laugh:

MM

Music Man:

I just got an email from Google stating that: "Over the last 24 hours, Googlebot encountered errors while attempting to access your robots.txt. To ensure that we didn't crawl any pages listed in that file, we postponed our crawl."

Point your webbrowser to http://yourdomain.com/robots.txt

Are there any errors? (You should see the contents of the file).

If you can't see the contents via the browser you either have a permissions problem, or something else is preventing it from being read (.htaccess perhaps)

Music Man:

I went to the robots.txt file. I don't have much in there.

User-agent: *
Disallow: /cgi-bin/

I was under the impression that I wanted to keep the bot out of the cgi-bin area of a site...Is that true?

Mostly correct. It doesn't prevent the crawlers/bots from accessing the contents of this folder (or any other folders listed). It merely instructs them to not index the contents. Not ALL bots honour this request though. The Google bot does.

Music Man:

Is my disallow to that area throwing this error?

No.

Music Man:

Also, can someone suggest what SHOULD be in the robots.txt file beyond what I have...? Maybe post an example of a working robots.txt.....

Typically:

User-agent: *
Disallow: /cgi-bin/
Disallow: /cache/
Disallow: /bmz_cache/
Disallow: /logs/
Disallow: /images/

Music Man:

Thank you for any help you can offer.....I've been fighting this bot issue for months and can't seem to get it right..

As mentioned, the robots.txt file is more of a polite request that the robots not index the contents the listed folders. There is generally no 'harm' done even if you don't have a robots.txt file at all. If you need to keep bots (and people) out of specific folders then this should be done via the .htaccess files. ZenCart has several of these scattered in various folders specifically for this reason.

Cheers
RodG

RodG:

Point your webbrowser to http://yourdomain.com/robots.txt

Are there any errors? (You should see the contents of the file).

If you can't see the contents via the browser you either have a permissions problem, or something else is preventing it from being read (.htaccess perhaps)

Mostly correct. It doesn't prevent the crawlers/bots from accessing the contents of this folder (or any other folders listed). It merely instructs them to not index the contents. Not ALL bots honour this request though. The Google bot does.

No.

Typically:

User-agent: *
Disallow: /cgi-bin/
Disallow: /cache/
Disallow: /bmz_cache/
Disallow: /logs/
Disallow: /images/

As mentioned, the robots.txt file is more of a polite request that the robots not index the contents the listed folders. There is generally no 'harm' done even if you don't have a robots.txt file at all. If you need to keep bots (and people) out of specific folders then this should be done via the .htaccess files. ZenCart has several of these scattered in various folders specifically for this reason.

Cheers
RodG

Thanks Rod...

If I point my browser at mydomain.robots.txt I see exactly what I posted above....clear as day. I don't understand why Googlebot is throwing the error.....would it help to add the other disallows you mention? I'm thinking it won't make any difference since the bot is stumbling over the 2 lines I currently have....

Thanks for the help....If you think of other things to try I would appreciate your insight....

Music Man:

If I point my browser at mydomain.robots.txt I see exactly what I posted above....clear as day. I don't understand why Googlebot is throwing the error.....would it help to add the other disallows you mention? I'm thinking it won't make any difference since the bot is stumbling over the 2 lines I currently have....

Thanks for the help....If you think of other things to try I would appreciate your insight....

No, adding the other dissallows won't make any difference.

Do you have a Google webmasters account? If not, you should create one. The set of tools it provides can give some valuable insights about your site. One of these tools is a robots.txt tester that will (attempt) to download and check the robots.txt files for any errors.

In your case, even though there aren't any errors to be seen, it may provide a clue as to why it can't access it.

Cheers
RodG.

RodG:

No, adding the other dissallows won't make any difference.

Do you have a Google webmasters account? If not, you should create one. The set of tools it provides can give some valuable insights about your site. One of these tools is a robots.txt tester that will (attempt) to download and check the robots.txt files for any errors.

In your case, even though there aren't any errors to be seen, it may provide a clue as to why it can't access it.

Cheers
RodG.

THanks Rod,

Yep I have a webmaster account....and yep I tried the robots.txt tool....it returned my file just fine.....just that bot..... I just did some cleanup of my files and changed locations....I'm curious to see if I might have cleared a better path.....who knows....

Don't go changing the file just because of a single notification message.

It's very possible that Google had a glitch ... believe it or not the mighty google encounters problems often, and the problems are theirs not yours.

If the problem were to happen every day for a week then it might be worth getting concerned.

Besides, if Google skips crawling your site today, what are they gonna miss that they didn't have yesterday? Is there some urgent new thing they need to see? If not, then it doesn't matter if their crawler can see your site anyway.

I'll also add that it could be something they're confused by with your recent changes you've been discussing here: I broke it!

DrByte:

Don't go changing the file just because of a single notification message.

It's very possible that Google had a glitch ... believe it or not the mighty google encounters problems often, and the problems are theirs not yours.

If the problem were to happen every day for a week then it might be worth getting concerned.

Yeah I thought of that the FIRST time I got a message....I think this was my third in as many months.... As I said in another thread "I Broke It"...I changed a bunch of stuff around. I asked Google to reindex....we'll see what happens. If I keep getting the error I suppose I can just take out the robots.txt file.....from what I'm reading that's sort of a semi voluntary measure anyhow that the 'bad bots' will ignore anyhow.....Google says it's not "necessary" to have it to index the site......

Music Man:

I think this was my third in as many months....

That's nothing to worry about. I'd put these down as a temporary glitch, and there are so many possible causes that you'll probably never be able to determine which one(s) are at play, and even if you could it would probably be out of your control to fix.

If you get several a week it may be worth investigating further as it could mean your site is being consistently overloaded.

If you get them daily then it will be worth investigating further as that could imply there is some sort of blocking (firewall?) at play which could also be blocking other (more important) pages .

Cheers
RodG