Robots.txt....to use or not to use?

[Music Man]

Hi all,

Just curious to get some feedback on using a robots.txt file or not. I understand the idea of blocking certain directories....and allowing the little bots to find my good stuff......but it seems to me a person intent on a hack will use the robots.txt file to know exactly where to start looking for your privates........

Thoughts?

[mc12345678]

Hi all,

Just curious to get some feedback on using a robots.txt file or not. I understand the idea of blocking certain directories....and allowing the little bots to find my good stuff......but it seems to me a person intent on a hack will use the robots.txt file to know exactly where to start looking for your privates........

Thoughts?

Thing is exactly that, the robots.txt file is not a "private" file, so things you truly want to remain hidden should not be listed in your robots.txt. The intent is for those items not to be indexed.

Indexing is different than accessed. For example, a bot is not going to be able to find your admin directory from any web page of your site, but if it becomes listed in the robots.txt file then others can see it (and whether it is a mean bot or a hacker, then the content of the robots.txt would be available.)

So as to whether to use or not use? Yeah kind of up to you, but my suggestion is don't use it to keep something "private", use it to keep something off of the index that would otherwise be fully known publicly.

[CandleMan]

just on the back of this, my hosting co say that most of my bandwidth usage is being caused by robots.
where is the text file, or where do i put the text file to stop them?
I cant find anything except a spiders text file in the includes folder and im sure this is for something else.
would it be better to use a .htaccess option?

[DrByte]

If you need a robots.txt file on your site, then you can simply add one. If your store is NOT in a subdirectory then the robots.txt goes in the same folder as your store, same place as where you see the ipn_main_handler.php etc.

Zen Cart normally includes a robots_example.txt file which you could rename to robots.txt and edit to suit your own needs.

[RodG]

just on the back of this, my hosting co say that most of my bandwidth usage is being caused by robots.
where is the text file, or where do i put the text file to stop them?
I cant find anything except a spiders text file in the includes folder and im sure this is for something else.
would it be better to use a .htaccess option?

The robots.txt doesn't exist by default, but there is an example provided that you can rename and and use (see reply from DrByte).

HOWEVER, please be advised the this file and contents are best considered as a 'map' of what you want (or don't want the spiders to indexed). It doesn't stop the spiders from crawling (accessing) your site, but a good spider won't crawl what you don't want indexed anyway, so it is only partially effective. The 'bad' bots will just ignore the file and crawl the site anyway.

There are several ways the bad bots can be blocked - The best method, being via a firewall. This isn't always possible though, so the more common method is by listing them in the .htaccess file(s).

There are many examples as to how to do this in these forums, as well as any one of a zillion other resources that you can find with a google search.

Bad bots, and the damaging effects they can have on a website/server isn't exclusive to ZenCart. It is a problem with all websites.

Cheers
RodG