How can I add a new spider to the no session id list?

5 Mar 2008, 12:56 AM

#1

Zen Follower

Join Date:: Apr 2007
Posts:: 106
Plugin Contributions:: 3

How can I add a new spider to the no session id list?

I noticed a new spider crawling my site and it turns out that its a vertical search engine. the problem is that it's giving that spider session id's which as we know are not good. How can i add this spider to the list of spider's that get no session id?

Thanks

Amish Furniture | site wide mark up | ezpage rel=nofollow SEO mod

5 Mar 2008, 1:14 AM

#2

Kim

Obaa-san

Join Date:: Jun 2003
Location:: West Coast, North America
Posts:: 26,684
Plugin Contributions:: 0

Re: How can I add a new spider to the no session id list?

Please post the details here and add it to your /includes/spiders.txt

Please do not PM for support issues: a private solution doesn't benefit the community.

Be careful with unsolicited advice via email or PM - Make sure the person you are talking to is a reliable source.

5 Mar 2008, 1:34 AM

#3

heimskr75

Zen Follower

Join Date:: Apr 2007
Posts:: 106
Plugin Contributions:: 3

Re: How can I add a new spider to the no session id list?

Thanks for the quick response!!

Like I said its a very niche specific SE that covers only Amish furniture. It goes by the name of amishbot.
I got a hold of the guy running it and it looks like adding it to the spiders.txt file worked fine. Looking at the who's online shows no session id for amishbot.
Somehow I only think this info (amishbot) will be useful to me but.... ya never know.
Thanks again!

Amish Furniture | site wide mark up | ezpage rel=nofollow SEO mod

11 Mar 2008, 3:44 PM

#4

shrimp_gumbo_mmmhhh

Totally Zenned

Join Date:: Jun 2007
Location:: Texas, USA
Posts:: 1,436
Plugin Contributions:: 0

Re: How can I add a new spider to the no session id list?

Here is a new spider that I have noticed:

Host: ec2-67-202-41-254.compute-1.amazonaws.com
User Agent: zermelo Mozilla/5.0 compatible; heritrix/1.12.1 (+http:// www. powerset.com [email:crawl @ powerset.com,email:paul @ page-store. com]