Spyders get stuck at cookie_usage??

it seems that all the bots/spyders that come to the site get stuck at

/index.php?main_page=cookie_usage

Im not sure if this is normal or not, but all the bots are always on reviews or cookie usage.

Does it effect there travel through the site? maybe i should be using one of the SEOs?

any advice would be great

Please post the settings you are using in the Admin> Configuration> Sessions area.

Cookie Domain -- True
Force Cookie Use -- False
Check SSL Session ID -- True
Check User Agent -- False
Check IP Address -- False
Prevent Spider Sessions -- True
Recreate Session -- False
IP to Host Conversion Status --true

What version of Zen Cart are you using?

Kim:

What version of Zen Cart are you using?

ver 1.3 with the update

any ideas?

Did you happen to notice if these are inktomisearch.com vs the other bots more often than not?

inktomisearch.com do not appear all that bright ... :blink:

Ajeh:

Did you happen to notice if these are inktomisearch.com vs the other bots more often than not?

inktomisearch.com do not appear all that bright ... :blink:

Yes i have been watching that, and it seems that its will all bots, from google to slurp, every one. They all do the same thing practicly.

any help would be great.
thanks

I am having the same issue - but it seems that is the msn bot that is having the problem.

Cookie Domain = True
Force Cookie Use = False
Check SSL Session ID = False
Check User Agent = False
Check IP Address = False
Prevent Spider Sessions = True
Recreate Session False
IP to Host Conversion Status = true

thanks for any help

I had a thread running in the old forums and can't seem to find it now as the search function here does not seem to search the archives.

From memory it had something to do with the <noscript> tags on product pages which caused problems when cookies were disabled (like a spider). It was at least 6 to 8 months ago so forgive me for not remembering the problem.

I will try to find the posts again but no promises :cry:

To dig a relatively old topic up...

I experience the same problem;

00:04:10  	 0  	 ¥Spider  	207.46.98.112  	06:13:37  	06:13:37  	 
  Time Since Clicked:
    00:04:10 ago 	Session ID:
Host: msnbot.msn.com
User Agent: msnbot/1.0 (+http://search.msn.com/msnbot.htm)
/index.php?main_page=cookie_usage

Session Directory  	/home/xxx/xxx/cache  	 
Cookie Domain 	True 	Info 
Force Cookie Use 	False 	Info 
Check SSL Session ID 	False 	Info 
Check User Agent 	False 	Info 
Check IP Address 	False 	Info 
Prevent Spider Sessions 	True 	Info 
Recreate Session 	False 	Info 
IP to Host Conversion Status 	true

Version: 1.3.0.5 (yes the newest)

duwane:

I had a thread running in the old forums and can't seem to find it now as the search function here does not seem to search the archives.The search function does search the archives, but it seems like the topic isn't in the archive anymore...

The google cache still has it online though: http://66.249.93.104/search?q=cache:pIqLAvsCcxQJ:www.zen-cart.com/modules/ipb/index.php%3Fact%3DPrint%26client%3Dwordr%26f%3D4%26t%3D15495+gigablast+duwane&hl=nl&gl=nl&ct=clnk&cd=2&client=firefox-a

:wink2:

I haven't read the whole thread, but I think that thread is mostly about the cookie usage message on "normal" pages. But SE's hitting main_page=cookie_usage is also discussed there.

Thanx, the quoted robots.txt file looks useful.

edit: only a few minutes later;

00:03:07 0 ¥Spider 38.113.234.181 01:51:44 01:51:44
Time Since Clicked:
00:03:07 ago Session ID:
Host: crawl2.cosmixcorp.com
User Agent: voyager/1.0
/index.php?main_page=cookie_usage

:(

Not all bots listen to robots.txt files, only the nice ones do.

(And from what I have heard they only read robots.txt once in while, but I am not sure about that.)

have this problem too right now theres 7 of them online all inktomisearch.com and at the cookie_usage.html

I have seen this mainly with inktomisearch on the whos_online ...

And I have seen them all over the same site ...

The spiders getting stuck on the /index.php?main_page=cookie_usage does not seem to prevent the spidering of the rest of the site ...

You could always adjust the robots.txt to tell them no ... :smile:

Granted reading yahoo on the way their spiders work it almost sounds like yes we do listen to robots.txt but we might stick the link in there anyway ... :blink:

cookie_usage.htmlI assume you understand that when using SEO url's you need a different robots.txt file CnTGifts

I am having the same thing happen to us...it seems like it tries to go through our website and then ends up like this and LEAVES the website...and then only to return a few minutes later and repeats the process...

00:07:530 ¥Spider 74.6.87.1412:24:0712:24:07 Time Since Clicked:
00:07:53 agoSession ID:
Host: ct501054.inktomisearch.com
User Agent: Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)
/index.php?main_page=popup_image&pID=422 00:07:420 ¥Spider 74.6.86.20012:24:1812:24:57 Time Since Clicked:
00:07:03 agoSession ID:
Host: ct501091.inktomisearch.com
User Agent: Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)
/index.php?main_page=cookie_usage 00:07:330 ¥Spider 74.6.86.16112:24:2712:25:07 Time Since Clicked:
00:06:53 agoSession ID:
Host: ct501140.inktomisearch.com
User Agent: Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)
/index.php?main_page=cookie_usage

I noticed this from another thread that is no longer in zencart but was in the google catch as mentioned before...it was recommended to put the following in the robots.txt file...

Disallow: /index.php?main_page=cookie_usage

and it seemed to work for Duwane and Woodymon

but I do not want to do anything that would hender the website being indexed, what do you recommend?

Trying hard to get indexed here....thanks in advance to all for any suggestions on this cookie_usage issue...:no:

Also, if you have any suggestions for my website if you could PM me about I would appreciate it, as I am getting a minimum of 300 sessions a day from people, but only about 1 or 2 orders a day right now...

I have the same problem with MSN spider. 1 thing I suspect: I have a small javascript (provided by MSN), sort of GoogleAnalytics to watch my page stat, it may be causing the problem.

I have a related issue. Google has been good to me and quickly I may add. There are about 6 pages in google's index from my site that are listed as Cookie Usage: xxxx

These pages are pages like About us, write review, etc. The index listing looks pretty silly.

Any Ideas other than the url rewrite for now??

Thanks