Google SitemapAdmin

[quantum]

Looking at the dates against where it is says the robots.txt file is unreachable, they are old dates.

I am not an expert on Google SiteMaps, but I assume 'Last Calculated' means the date these were last accessed?

If this is the case, then Google may be accessing your site.

Do you have a web stats package with your hosting plan? If so, take a look at it and see what stats show for the GoogleBot. This may give you an indication of whether Google is crawling your site.

Google has your site in its listings (only the home page at this stage), so that would indicate it has at least picked it up.

[rod44]

Sure! Since I didn't have the store under root directory, I submitted both www.bridgecampus.com and www.bridgecampus.com/catalog/ Both screen prints are attached. Thanks for your help!

Well, but looking at what you have posted, it appears that the last 'validation' was about 10 days ago, and correct me if I am wrong, you have changed the robots.txt file since. If you look at the last 2 pages dated the 18th it shows success. Seems that if they WOULD crawl your site again that it would probably work.

[quantum]

I also just noticed in your last two screen prints that it shows that the robots.txt file was succeffully downloaded on November 18th.

[stargre]

Well, but looking at what you have posted, it appears that the last 'validation' was about 10 days ago, and correct me if I am wrong, you have changed the robots.txt file since. If you look at the last 2 pages dated the 18th it shows success. Seems that if they WOULD crawl your site again that it would probably work.

That's true and that's also what I am confused about. It seems that they have seen the robots.txt change, but that didn't fix the crawl 'robots.txt unreachable' problem. So, modifying robots.txt might not be the correct resolution?

[rod44]

Sorry that I missed your question. But yeah, I do have googlesitemapadmin contribution installed. That's what you are talking about, right? I used to think that the google sitemapadmin was built to utilize google robots that sort of things. It seems I was wrong. They are totally different two routes of getting google notified? I don't see google sitemapadmin produce a lot of traffic for my site though.

Robots.txt and sitemaps are independent of eachother, and google actually PREFERS the SiteMap route as it doesn't require them to cralw links. What you will see if you use the User Tracking Mod (if you don't have it, this is an INCREDIBLE TOOL to see how visitors use your site) is 'hits' from BOTS and GoogleBot will be identified. Here are a couple of questions (and answers) to make sure you have everything in place:

1. Are you signed up WITH Google under sitemaps? If so, have you added their 'validation' page?

2. Have you created the sitemap through your admin and verified it is actually in your /catalog/ directory? You need to VIEW it through a browser or download and read in notepad or something to make sure it is right.

3. In the sitemap admin once you have verified the sitemap is there, use the ping button to let them know you are there.

If I sound like I am being elementary, I am not. I just want to see this start working for you so you can see some success with your site.

Good Luck!

[rod44]

That's true and that's also what I am confused about. It seems that they have seen the robots.txt change, but that didn't fix the crawl 'robots.txt unreachable' problem. So, modifying robots.txt might not be the correct resolution?

No, this is why you need to use the Google SiteMaps and relax. The Googlebot crawls through robots.txt SLOW, and you just have to wait for your 'number to be called' and once the recrawl is completed the page validations should change. This is one of those things you will have to wait a bit to see if it worked. But just so you know, my robots.txt looks like:

# robots.txt for http://www.hatnpatch.com/

User-agent: *
Disallow: /cgi-bin/ # these will soon disappear
Disallow: /logs/ # these will soon disappear

and they are crawling my .html pages off of that. Patience my friend,

[stargre]

Robots.txt and sitemaps are independent of eachother, and google actually PREFERS the SiteMap route as it doesn't require them to cralw links. What you will see if you use the User Tracking Mod (if you don't have it, this is an INCREDIBLE TOOL to see how visitors use your site) is 'hits' from BOTS and GoogleBot will be identified. Here are a couple of questions (and answers) to make sure you have everything in place:

1. Are you signed up WITH Google under sitemaps? If so, have you added their 'validation' page?

2. Have you created the sitemap through your admin and verified it is actually in your /catalog/ directory? You need to VIEW it through a browser or download and read in notepad or something to make sure it is right.

3. In the sitemap admin once you have verified the sitemap is there, use the ping button to let them know you are there.

If I sound like I am being elementary, I am not. I just want to see this start working for you so you can see some success with your site.

Good Luck!

1) Yes, I did signup and verified the site.
2) That's right. All three .xml files are in /catalog/ directory. They can be seen in web browser.
3) I did pinged them several times.

This seems so strange. I wish myself some good luck as well. But, any new ideas? Thanks!

[quantum]

Rod is correct, patience is what you need with the search engines (especially google) for crawling all of your site and indexing your pages.

Those screens are simply a historical view, so what is listed there will not update persay, just dissapear to be replaced with new results when google crawls those pages.

[stargre]

Rod is correct, patience is what you need with the search engines (especially google) for crawling all of your site and indexing your pages.

Those screens are simply a historical view, so what is listed there will not update persay, just dissapear to be replaced with new results when google crawls those pages.

I hope so. As far as the robots.txt file, since I don't have /cgi-bin/ and /log/ folders under my root directory, I didn't put it in the disallow list. But I don't have much idea about it. Are you suggesting I need to modify it? I just want to allow everything as suggested earlier, so that I can get indexed as soon as possible.

Yeah, you are right, I do need relax! Thanks all of you!

[stargre]

I just added 'Allow: /' to my robots.txt file. Is that all I need do?