Hit Counter & Robots

[rehopper]

Can the hit counter be altered to omit hits by robots?

Several hundred robot hits are experienced occasionally and are very disappointing!

I find nothing in the ZC Manual or our forum.

[Ajeh]

Zen Cart v1.3.9 does not count spiders in the counts any longer ...

[rehopper]

Ajeh,
My admin page right now=01/17/2011 1268 - 1331

On StatCounter.com = Nice and Easy Direct 48

I'm in v139h, the counter.php file is the current one from the139h package. Any suggestions wher I should look to correct?

[Ajeh]

Check that your file:
/includes/application_top.php

is current and up to date:

if ($spider_flag == false) {
// counter and counter history
  require(DIR_WS_INCLUDES . 'counter.php');
}

Check that your file:
/includes/spiders.txt

is current and up to date ...

[rehopper]

My application_top.php appears to be OK but the spider.txt didn't include 2 bots crawling my site right now; googlebot and crawl.yahoo.
I added them but wonder how many more exist that I won't "catch in the act".
Is there an updated listing available to utilize?

[Ajeh]

Not sure how you lost those, but the file:
/includes/spiders.txt

that comes with v1.3.9h should be labeled at the top as:
$Id: spiders.txt 16983 2010-07-25 17:40:59Z drbyte $

[Ajeh]

Note: googlebot is referenced as google and crawl.yahoo is referenced as crawl ...

[rehopper]

that comes with v1.3.9h should be labeled at the top as:
$Id: spiders.txt 16983 2010-07-25 17:40:59Z drbyte $

Checks out OK!

It seems the googlebot is present more often than not & the counter isn't always bonkers. I'll attempt to monitor "who's online" closely for any strange spiders & report back with any findings.

My "who's online" right now has googlebot & 6 entries of the same IP address:

Who's Online
REFRESH LIST Legend: Active cart Inactive cart Active no cart Inactive no cart
Inactive is Last Click >= 180s || Inactive since arrival > 540s will be removed
Currently there are 7 customers online

Online ID Full Name
Asc Desc IP Address
Asc Desc Session
Asc Desc Entry Time
Asc Desc Last Click Time
Asc Desc Last URL Viewed
Asc Desc
03:22:29 0 ¥Spider 66.249.72.216 16:13:44 19:35:03
Time Since Clicked:
00:01:10 ago Session ID:
Host: crawl-66-249-72-216.googlebot.com
User Agent: Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
/139d/index.php?main_page=product_info&cPath=41_52_132&products_id=107600

00:08:45 0 ¥Guest 188.165.234.114 19:27:28 19:27:28
Time Since Clicked:
00:08:45 ago Session ID: 0180da289be3c49589f935caab26e646
Host: ns214021.ovh.net
User Agent:
/139d/index.php?main_page=product_info&products_id=102869

00:07:29 0 ¥Guest 188.165.234.114 19:28:44 19:28:44
Time Since Clicked:
00:07:29 ago Session ID: c6e043a1a9b30142482aeb74a0bd786f
Host: ns214021.ovh.net
User Agent:
/139d/index.php?main_page=product_info&products_id=102870

00:06:08 0 ¥Guest 188.165.234.114 19:30:05 19:30:05
Time Since Clicked:
00:06:08 ago Session ID: 4fd8c8862f183f3d2703c69331d35b37
Host: ns214021.ovh.net
User Agent:
/139d/index.php?main_page=product_info&products_id=102871

00:05:07 0 ¥Guest 188.165.234.114 19:31:06 19:31:06
Time Since Clicked:
00:05:07 ago Session ID: aacaab87110bf8048a01c2393cd8bac4
Host: ns214021.ovh.net
User Agent:
/139d/index.php?main_page=product_info&products_id=102872

00:03:51 0 ¥Guest 188.165.234.114 19:32:22 19:32:22
Time Since Clicked:
00:03:51 ago Session ID: 47fcb83e7cb2261daa93011bef2bb0b6
Host: ns214021.ovh.net
User Agent:
/139d/index.php?main_page=product_info&products_id=102873

00:01:30 0 ¥Guest 188.165.234.114 19:34:43 19:34:43
Time Since Clicked:
00:01:30 ago Session ID: 5d3384435e4d281278adcfa9f39d185b
Host: ns214021.ovh.net
User Agent:
/139d/index.php?main_page=product_info&products_id=102874

[rehopper]

Ajeh,
do you think it's possible my site is experiencing "web scraping"?
Here's my latest counts:
01/20/2011 2642 - 2668
01/19/2011 5868 - 5993
01/18/2011 544 - 634
The only spider found each check was googlebot, which is almost always present each check.
I found an IP (France origin} with 24 open pages yesterday but I wasn't smart enough to record that IP number.
I'm considering doing an .ht access file deny of such as this, hate to deny a bona fide client though!

[Ajeh]

Hard to say ... you would need to watch and see what you are finding going through your sites ...

Spiders are marked as Spiders and pink ...

Anything else in blue counts as a session and a hit for each page ...