still looking for this mod. where can i find it?
Printable View
still looking for this mod. where can i find it?
Hi! Follow up on the problem I reported yesterday. Google robot.txt analysis shows that they have detected the robots.txt file I have created according to the suggestions made earlier. However, web crawl results still didn't change and gave the same error: robots.txt unreachable.
So, in one word, the problem still exists. My first thought at this problem was that because of the fact my store was install under /catalog/ folder instead of root directory, google cannot reach it. Google told me:
This site is not located at the top level for the domain. A robots.txt file is only valid when located in the highest-level directory and applies to all directories within the domain. The robots.txt file that applies to your site (if one exists) is located at http://bridgecampus.com/robots.txt. This page provides information on that file.
Since it didn't work out quiet as expected by you folks, I am raising this again. Is that the possible reason? If so, how to resolve it? Or, am I missing something there.
Thanks so much!
Thank you for the link. :)
WOW! Well I just looked at the robots.txt and it's in the right place, however you seem to have your /catalog in Dasallow. What I would do if I was you is Allow ALL so you can get started being indexed and then Disallow directories as you start getting indexed. Our site http://www.hatnpatch.com is taking 1700 hits a day PLUS somewhere around 300+ per day (looking at the user tracking mod as well as server stats) from Googlebot, Inktomi and others. My robots.txt is set to ALLOW ALL and I used the GoogleSiteMap Contrib to make the .xml for them to index. I would caution you on trying to be 'too cautious' until your site starts clicking for you and fine tune as you go. Some may think this isn't a good way to go, but here's the question:
Would you like to have Google (and others) start indexing you today, or wait for another 2-3 weeks for this to happen?
The answer should be NOW. Here is what I would recommend (if you have not already):
1. Install User Tracking for verification of crawler and bots (visitors too);
2. Install Google SiteMaps, create the .xml file;
3. Signup for Google SiteMaps, complete their requirements; and
4. Ping Google Sitemaps to get it started.
Within 2 days you will start to see them indexing you and then you can 'tighten up' the other stuff later. :yes:
Hope that helps!
Rod
(former keywording and SEO Tech/Consultant)
Quote:
Originally Posted by rod44
Hi! Thanks for the suggestions. I tried to allow all. But still, my robots.txt is not accessible by google. Why is that happening? Can anyone take a better look at my site and give me some suggestions please? My store is installed under /catalog/ folder. Thanks so much!
Are you able to place a screen print up of the page in your Google Sitemaps that shows the error Google is giving you?
Sure! Since I didn't have the store under root directory, I submitted both www.bridgecampus.com and www.bridgecampus.com/catalog/ Both screen prints are attached. Thanks for your help!Quote:
Originally Posted by quantum
http://www.bridgecampus.com/images/1.bmp
http://www.bridgecampus.com/images/2.bmp
http://www.bridgecampus.com/images/4.bmp
http://www.bridgecampus.com/images/3.bmp
Maybe you should place a 'specific' allow and see if that will help. Try adding this:
User-Agent: Googlebot
Allow: /
Also, you still have not said if you have Google SiteMaps installed. Do you? If not, I would add it as it is the BEST way to get the attention of Google and get them to your site. Using the site map, as soon as you update your site you can just update the sitemap with the click of a button in your admin, then ping their server, and it's done. Might be better than getting too caughtup with the robots.txt? I am telling you, the sitemap mod is the best way to go.
One other thing, I use Firefox as a browser and your site comes up with the center page off to the right overlapping all the right boxes.
Hope that helps!
Sorry that I missed your question. But yeah, I do have googlesitemapadmin contribution installed. That's what you are talking about, right? I used to think that the google sitemapadmin was built to utilize google robots that sort of things. It seems I was wrong. They are totally different two routes of getting google notified? I don't see google sitemapadmin produce a lot of traffic for my site though.Quote:
Originally Posted by rod44