OK, so a quick background, I've always seen thefind.com crawling my site in whos_online.php but the ip's have never been marked as a spider, despite my best efforts to correctly add the UA to the spiders.txt file.

I have scoured the forums and found the latest spiders.txt file that Dr. Byte posted in the forum, which to my knowledge contains the header:
$Id: spiders.txt 18372 2011-02-08 21:21:42Z drbyte $

thefind.com's bot UA is FatBot 2.0, or more specifically: Mozilla/5.0 (compatible; FatBot 2.0; http://www.thefind.com/crawler)

Based on the above referenced spiders.txt file, the "Bot" part of "FatBot" should trigger the admin to identify it as a spider since the very first line after the header in spiders.txt is, in fact, "bot". Needless to say it doesn't. So of course, the next step would be to simply add "fatbot" on a new at line of the spiders.txt file - OK, did it, no dice. FatBot 2.0? - nope, no dice.

This really wouldn't be a problem for me, but at any given time I have a dozen or so different ip's from the find crawling my site, and all are being assigned a new session id, it's really starting to **** me off. What the heck do I need to do to prevent sessions for this dang thing?

BTW, applicable sessions settings are:
cookie domain: true
force cookie use: true (full ssl - not on shared ip)
check ssl session id: false
check user agent: false
check ip address: false
prevent spider sessions: true
recreate session: true
ip to hose conv status: true

Please note: ip address for true guest shown in SS are blurred intentionally to protect their anonymity, and full domain paths are blurred due to the adult nature of the website (I also don't feel the path is germane to the issue at hand). But as you'll see, each ip from the find has it's own unique session. googlebot is correctly pinned as a spider.. so why not fatbot??? :censored:

Those are caught on my site without any issues ...

Check your file:
/includes/spiders.txt

and make sure that the first bot under the ID is:
bot

litepockets:

Based on the above referenced spiders.txt file, the "Bot" part of "FatBot" should trigger the admin to identify it as a spider since the very first line after the header in spiders.txt is, in fact, "bot".

Hi Ajeh, I referenced that in op. Not sure what the heck the issue is.

litepockets,
$Id: spiders.txt 18372 2011-02-08 21:21:42Z drbyte $ - is 1.5

$Id: spiders.txt 16983 2010-07-25 17:40:59Z drbyte $ - is 1.3.9h

If using ZC v1.3.9h then why not try using the appropriate file?

Ajeh, I noticed this in 1.3.9h and 1.5 RC3, 'spiders.txt' file.

gulperbot
h￤m￤h￤kki
hämähäkki
hamahakki

Am I missing something or are those actual Robot names that will be recognized?

litepockets,
Perhaps your site's code isn't fully recognizing mixed-case names. So, "bot" won't match the "Bot" in "fatBot".
Workaround: since "thefind.com" is mentioned in the UA string and is all lowercase, you can add "thefind" to the end of your own spiders.txt file.

Website Rob:

litepockets,
$Id: spiders.txt 18372 2011-02-08 21:21:42Z drbyte $ - is 1.5

$Id: spiders.txt 16983 2010-07-25 17:40:59Z drbyte $ - is 1.3.9h

If using ZC v1.3.9h then why not try using the appropriate file?

Rob, I appreciate the input, but I downloaded the file per Dr. Byte's suggestion on this post: http://www.zen-cart.com/forum/showpost.php?p=992915&postcount=32 - as you can see, he stated to use that version, which is what was going to be used in v1.5 (turns out that the 1.5 version has been modified from this, but nonetheless it's still Dr. Bytes suggested version for 1.3.9h).

DrByte:

litepockets,
Perhaps your site's code isn't fully recognizing mixed-case names. So, "bot" won't match the "Bot" in "fatBot".
Workaround: since "thefind.com" is mentioned in the UA string and is all lowercase, you can add "thefind" to the end of your own spiders.txt file.

Dr. Byte, thanks, I'll give "thefind" a try. I had recently tried "thefind.com" but had no success. I'll report back :bigups:

Dr. Byte,

The addition of "thefind" to the end of the spiders.txt file seemed to bring about mixed results...

Good News: is that for the first time, I actually saw their dang bot correctly identified as a spider.

Bad News:

1. Correct identification isn't consistent. It reported 1 IP with the fatbot UA string as a spider, and another IP with an identical UA string as a guest (chronologically AFTER the one that was correctly reported).
2. Googlebot is suddenly not being reported as a spider.
3. It would appear that a true guest is being incorrectly tagged as a spider.

Screenshot:

Attached: my current spiders.txt

Also worth noting, I have already followed the suggestions given by Ajeh in these posts:
http://www.zen-cart.com/forum/showpost.php?p=983952&postcount=4
http://www.zen-cart.com/forum/showpost.php?p=917228&postcount=5

Sounds like something wrong with the many addons you've got.
I can't replicate your problem here.

Dr. Byte,

Thank you for trying. Any general idea of which file(s) to examine other than those outlined?

I'm having the same trouble the other users have experienced with this spider not being recognized as a spider. I have added thefind, thefind.com, fatlens, fatbot, FatBot, and any possible combination I can think of to my spiders.txt file, but no luck. At the moment, this spider has 25 separate pages/sessions running. Is there any other way I can get this properly classified?

Thanks!

Please click Reply below and answer all the questions in the Posting Tips section.

Hi Dr. Byte,

I'm hoping these are the questions you're referring to:

What version of Zen Cart?
v1.5

Is it a new install or an upgrade?
New install

What the addons/contributions you have installed?
Google Mapinator, Quick Updates, Ceon URI Mapping, USPS Shipping module, social sidebox, backup mysql, admin display product images, enhanced who's online, about us page, simple google analytics, instant search

What customizations have you made?
template modifications and more. no major changes to functions/classes that i'm aware of.

What type of server are you using?
Apache 2.2.21

What version of PHP is installed on the server?
5.2.17

What version of MYSQL is installed on the server?
5.0.95 community

Have you attempted to fix the problem?

What steps have you taken?

See prior post for additions to spiders.txt

DrByte:

Please click Reply below and answer all the questions in the Posting Tips section.

The "enhanced whos online" mod components are mostly built-in to v1.5.
Maybe what you installed has broken something?

Has anyone had any luck with with preventing User Agent: Mozilla/5.0 (compatible; FatBot 2.0; http://www.thefind.com/crawler) from starting a session and showing up as a guest instead of a spider?

firebug:

Has anyone had any luck with with preventing User Agent: Mozilla/5.0 (compatible; FatBot 2.0; http://www.thefind.com/crawler) from starting a session and showing up as a guest instead of a spider?
It already blocks it for me. I just tested it on v1.5.1.

DrByte:

It already blocks it for me. I just tested it on v1.5.1.

This doesn't block it for me either - Linux / Zen 1.5.1 / PHP 5.3.19 / Mysql 5.1.68

I believe the identification happens in the session code, so something there doesn't catch in the search!

Dan

I hate to start a new thread and it is not super important for me to have this fixed but it will be nice to see the true numbers rather than an inflated number of users.

My problem only when i turn "Force Cookie Use" to True then my User tracking start counts fatlens.com as a user, and when i turn the "Force Cookie Use" to False i start having a problem with

Cookie Usage

We have detected that your browser does not support cookies, or cookies have been disabled.

I am not planning on updating to 1.5 any time soon since i have all the bells and whistles i needed installed in my 1.3.9h (which maybe the reason for all this), and yes i can live with these issues but again it nice if someone figured this out and willing to share. Thanks.

TamyA:

I hate to start a new thread and it is not super important for me to have this fixed but it will be nice to see the true numbers rather than an inflated number of users.

My problem only when i turn "Force Cookie Use" to True then my User tracking start counts fatlens.com as a user, and when i turn the "Force Cookie Use" to False i start having a problem with
Cookie Usage

We have detected that your browser does not support cookies, or cookies have been disabled.

I am not planning on updating to 1.5 any time soon since i have all the bells and whistles i needed installed in my 1.3.9h (which maybe the reason for all this), and yes i can live with these issues but again it nice if someone figured this out and willing to share. Thanks.

Although I do think it is not a wise decision not to upgrade, even when you have a lot of custom work. You could try and use the spiders.txt from the Zen Cart 1.5.3 file-set

You could use

Bad Bot Blocker by DrByte

Design75:

Although I do think it is not a wise decision not to upgrade, even when you have a lot of custom work. You could try and use the spiders.txt from the Zen Cart 1.5.3 file-set

Yes i agree, let me put this way, i do not have the time to upgrade right now, i did start missing with that on my local server but most of the mods i tried did not give me the expected result.

Unfortunately the spiders.txt from the Zen Cart 1.5.3 file-set did not work, i even checked it and it has no reference to fatbot or fatlens or any thing that refer to the find unless i am missing something, i added whatever i though that might catch it like fatbot, Fatbot, FatBot, thefind, fatlens etc. but am still seeing it.