Robots.txt File Help

[need_help]

I have the Robots txt file in the root of my directory which is just outside the zencart directory.

I have this within the file.

<Files *.txt>
Order Deny,Allow
Deny from all
</Files>
User-agent: *
Disallow: /index.php?main_page=login
Disallow: /admin
Disallow: /account.php
Disallow: /advanced_search.php
Disallow: /checkout_shipping.php
Disallow: /create_account.php
Disallow: /login.php
Disallow: /login.php
Disallow: /password_forgotten.php
Disallow: /popup_image.php
Disallow: /shopping_cart.php
Disallow: /product_reviews_write.php
Disallow: /cookie_usage.php
Disallow: /Googlebot-Image
Disallow: /default.ida
Disallow: /getout.php

Is this correct because i still see Google trying to index my login pages.

I am also a little confused with the spiders.

I saw a spider with the exact same IP address as a guest on my site and that confuses me especially as they were both looking at different items on the site.

How can a spider share the same IP as a guest or is the zencart who's online just reading this wrong ?

Also what do i actually put across from this request in the robot file.

<- Use if this brings up your ZENCARTS main page

What actually goes there ?

Help appreciated on the Robot.txt file.

[limelites]

If you use Google Webmaster Tools, you can create a robots text file there.

You should use supertracker mod to analyse traffic on site as this ignores spiders and bots if set to do so and it's probably the best mod for tracking site visitors... very user friendly and very accurate.

[need_help]

If you use Google Webmaster Tools, you can create a robots text file there.

You should use supertracker mod to analyse traffic on site as this ignores spiders and bots if set to do so and it's probably the best mod for tracking site visitors... very user friendly and very accurate.

I just found Google Webmasters which is great and i gave them my sitemap and i'll look into building the robots.txt file on their site.

[mprough]

<Files *.txt>
Order Deny,Allow
Deny from all
</Files>
User-agent: *
Disallow: /index.php?main_page=login
Disallow: /admin
Disallow: /account.php
Disallow: /advanced_search.php
Disallow: /checkout_shipping.php
Disallow: /create_account.php
Disallow: /login.php
Disallow: /login.php
Disallow: /password_forgotten.php
Disallow: /popup_image.php
Disallow: /shopping_cart.php
Disallow: /product_reviews_write.php
Disallow: /cookie_usage.php
Disallow: /Googlebot-Image
Disallow: /default.ida
Disallow: /getout.php

This does not belong there... it is server side coding. You would not wish to use it anyhow as then the search engines could not grab your robots.txt

<Files *.txt>
Order Deny,Allow
Deny from all
</Files>

Take the following out... as they are advertising your sensitive areas to hackers and would not be indexed anyhow.

Disallow: /admin
Disallow: /cookie_usage.php
Disallow: /default.ida
Disallow: /getout.php

If you are looking to disallow Google Imges, the above syntax will do nothing.... Goes like this...

User-agent: Googlebot-Image
Disallow: /

You have this in here twice

Disallow: /login.php

Bear in mind that a robots.txt DOES NOT prevent indexing, only crawling. To prevent indexing you must use a noindex,nofollow in the pages with your robots skip function.

~Melanie

[need_help]

Yes i changed it all using the Google webmasters but admin is still left in.

I will remove it and the other parts you mention.

They will get a shock if they try and access admin if they think it's my folder name as i changed the folder name and i also have an additional password protection for that directory and furthermore i have additional security on top of that.

[DrByte]

<Files *.txt>
Order Deny,Allow
Deny from all
</Files>
User-agent: *
Disallow: /index.php?main_page=login
Disallow: /admin
Disallow: /account.php
Disallow: /advanced_search.php
Disallow: /checkout_shipping.php
Disallow: /create_account.php
Disallow: /login.php
Disallow: /login.php
Disallow: /password_forgotten.php
Disallow: /popup_image.php
Disallow: /shopping_cart.php
Disallow: /product_reviews_write.php
Disallow: /cookie_usage.php
Disallow: /Googlebot-Image
Disallow: /default.ida
Disallow: /getout.php

As others have said, none of that is of any value in relation to Zen Cart, and all of it should be removed.
I don't know where you got it from ... but if someone recommended it, it was bad advice.

Zen Cart itself doesn't need ANYTHING in robots.txt

If you want to add a sitemap line into your own robots.txt, that's perfectly fine, and entirely up to you.

If you want to do something in robots.txt to stop images from being indexed, feel free.

because i still see Google trying to index my login pages.

If you're using Zen Cart v1.3.8 (or newer) and haven't customized anything that controls the automatic setting of "noindex,nofollow" on all the pages that don't require indexing, then it's still normal for Google to see the page, but when it sees the "noindex,nofollow" that Zen Cart automatically adds, it will go away and not actually index it, even though it visits the page.

[rjgonzalez]

Definition of the above robots.txt file:

User-agent: *
The asterisk (*) or wildcard represents a special value and means any robot.

Disallow:
The Disallow: line without a / (forward slash) tells the robots that they can index the entire site.

Any empty value, indicates that all URLs can be retrieved. At least one Disallow field needs to be present in a record without the / (forward slash) as shown above.

The presence of an empty "/robots.txt" file has no explicit associated semantics, it will be treated as if it was not present, i.e. all robots will consider themselves welcome.

The Disallow: line without the trailing slash (/) tells all robots to index everything. If you have a line that looks like this:

Disallow: /private/

It tells the robot that it cannot index the contents of that /private/ directory.
Summarizing the Robots Exclusion Protocol - robots.txt file

To allow all robots complete access:

User-agent: *
Disallow:

screen shot robots.txt fileThis is a screen shot of an empty robots.txt file Example of robots.txt File
Screen Shot - Robots Text File

To exclude all robots from the server:

User-agent: *
Disallow: /

To exclude all robots from parts of a server:

User-agent: *
Disallow: /private/
Disallow: /images-saved/
Disallow: /images-working/

To exclude a single robot from the server:

User-agent: Named Bot
Disallow: /

To exclude a single robot from parts of a server:

User-agent: Named Bot
Disallow: /private/
Disallow: /images-saved/
Disallow: /images-working/

Note: The asterisk (*) or wildcard in the User-agent field is a special value meaning "any Robot" and therefore is the only one needed until you fully understand how to set up different User-agents.

If you want to Disallow: a particular file within the directory, your Disallow: line might look like this one:

Disallow: /private/top-secret-stuff.htm

Keep in mind that using the above example excludes that specified page (top-secret-stuff.htm) but will not exclude the entire /private/ directory.

You should validate your robots.txt file. Enter the full URI to the robots.txt file on your server. The robots.txt file always resides at the root level of your web.

[need_help]

As others have said, none of that is of any value in relation to Zen Cart, and all of it should be removed.
I don't know where you got it from ... but if someone recommended it, it was bad advice.

Zen Cart itself doesn't need ANYTHING in robots.txt

If you want to add a sitemap line into your own robots.txt, that's perfectly fine, and entirely up to you.

If you want to do something in robots.txt to stop images from being indexed, feel free.



If you're using Zen Cart v1.3.8 and haven't customized anything that controls the automatic setting of "noindex,nofollow" on all the pages that don't require indexing, then it's still normal for Google to see the page, but when it sees the "noindex,nofollow" that Zen Cart automatically adds, it will go away and not actually index it, even though it visits the page.

I use a custom template and noindex nofollow doesn't seem to be in the header when i click view source on the main page.

What file do i put this in ?

[need_help]

Definition of the above robots.txt file:

User-agent: *
The asterisk (*) or wildcard represents a special value and means any robot.

Disallow:
The Disallow: line without a / (forward slash) tells the robots that they can index the entire site.

Any empty value, indicates that all URLs can be retrieved. At least one Disallow field needs to be present in a record without the / (forward slash) as shown above.

The presence of an empty "/robots.txt" file has no explicit associated semantics, it will be treated as if it was not present, i.e. all robots will consider themselves welcome.

The Disallow: line without the trailing slash (/) tells all robots to index everything. If you have a line that looks like this:

Disallow: /private/

It tells the robot that it cannot index the contents of that /private/ directory.
Summarizing the Robots Exclusion Protocol - robots.txt file

To allow all robots complete access:

User-agent: *
Disallow:

screen shot robots.txt fileThis is a screen shot of an empty robots.txt file Example of robots.txt File
Screen Shot - Robots Text File

To exclude all robots from the server:

User-agent: *
Disallow: /

To exclude all robots from parts of a server:

User-agent: *
Disallow: /private/
Disallow: /images-saved/
Disallow: /images-working/

To exclude a single robot from the server:

User-agent: Named Bot
Disallow: /

To exclude a single robot from parts of a server:

User-agent: Named Bot
Disallow: /private/
Disallow: /images-saved/
Disallow: /images-working/

Note: The asterisk (*) or wildcard in the User-agent field is a special value meaning "any Robot" and therefore is the only one needed until you fully understand how to set up different User-agents.

If you want to Disallow: a particular file within the directory, your Disallow: line might look like this one:

Disallow: /private/top-secret-stuff.htm

Keep in mind that using the above example excludes that specified page (top-secret-stuff.htm) but will not exclude the entire /private/ directory.

You should validate your robots.txt file. Enter the full URI to the robots.txt file on your server. The robots.txt file always resides at the root level of your web.

The robot file now has this in it.

User-Agent: *
Disallow: /disallow/catalog/account.php
Disallow: /disallow/checkout_shipping.php
Disallow: /disallow/catalog/shopping_cart.php
Disallow: /disallow/catalog/create_account.php
Disallow: /disallow/account.php
Disallow: /disallow/private
Disallow: /disallow/hidden
Disallow: /disallow/login.php
Disallow: /disallow/product_reviews_write.php
Disallow: /disallow/googlebot-image
Disallow: /disallow/shopping_cart.php
Disallow: /disallow/create_account.php
Disallow: /disallow/catalog/advanced_search.php
Disallow: /disallow/catalog/product_reviews_write.php
Disallow: /disallow/catalog/product_info.php
Disallow: /disallow/catalog/admin
Disallow: /disallow/catalog/popup_image.php
Disallow: /disallow/advanced_search.php
Disallow: /disallow/catalog/checkout_shipping.php
Disallow: /disallow/password_forgotten.php
Disallow: /disallow/popup_image.php
Disallow: /disallow/catalog/login.php
Disallow: /disallow/catalog/password_forgotten.php


Should i just remove it all and not have a robot file ?

[DrByte]

Given that everything you quoted in your last post is entirely invalid, yes, you should remove all of that.
(I'm not sure why you're putting everything in a "/disallow/" folder ... )