WOW! Well I just looked at the robots.txt and it's in the right place, however you seem to have your /catalog in Dasallow. What I would do if I was you is Allow ALL so you can get started being indexed and then Disallow directories as you start getting indexed. Our site http://www.hatnpatch.com is taking 1700 hits a day PLUS somewhere around 300+ per day (looking at the user tracking mod as well as server stats) from Googlebot, Inktomi and others. My robots.txt is set to ALLOW ALL and I used the GoogleSiteMap Contrib to make the .xml for them to index. I would caution you on trying to be 'too cautious' until your site starts clicking for you and fine tune as you go. Some may think this isn't a good way to go, but here's the question:
Would you like to have Google (and others) start indexing you today, or wait for another 2-3 weeks for this to happen?
The answer should be NOW. Here is what I would recommend (if you have not already):
1. Install User Tracking for verification of crawler and bots (visitors too);
2. Install Google SiteMaps, create the .xml file;
3. Signup for Google SiteMaps, complete their requirements; and
4. Ping Google Sitemaps to get it started.
Within 2 days you will start to see them indexing you and then you can 'tighten up' the other stuff later.![]()
Hope that helps!
Rod
(former keywording and SEO Tech/Consultant)


Reply With Quote
