Block ##########spider

[SethF]

Hi

Every day, about 40 BAI.DU spiders come on to, and index my site. I know that BAI.DU search is a legitimate Chinese search engine, but my website is not intended for China.

I have entered this into my robots.txt file:

#bai.duspider
User-agent: bai.duspider
Disallow: /


...but it doesn't seem to obey the file.

Is there any other way to block this spider without using htaccess and blocking each IP address? That would take a long time.


Thanks
S

[jc8125]

i had them going crazy on two different sites - added this to my robots: (Replace xxx with name of the bot which we dare not speak it's name)

#xxxspider
User-agent: xxxspider
Disallow: /

#xxx
User-agent: xxx
Disallow: /

#xxxAgain
User-agent: xxxspider/2.0
Disallow: /

#majestic started showing up 10min after i blocked xxx
User-agent: MJ12bot
Disallow: /

(last one added because as soon as the xxx stopped showing up, another very similar bot took it's place)

also added this to .htaccess:

RewriteCond %{HTTP_USER_AGENT} xxxspider
RewriteRule ^.*$ http://127.0.0.1 [R,L]

RewriteCond %{HTTP_USER_AGENT} xxx
RewriteRule ^.*$ http://127.0.0.1 [R,L] 

RewriteCond %{HTTP_USER_AGENT} xxx/2.0
RewriteRule ^.*$ http://127.0.0.1 [R,L]

can't remember for sure which one did the trick, but haven't seen "you know who" around for quite awhile..