Spyders get stuck at cookie_usage??

[nl2dav]

To dig a relatively old topic up...


I experience the same problem;

00:04:10  	 0  	 ¥Spider  	207.46.98.112  	06:13:37  	06:13:37  	 
  Time Since Clicked:
    00:04:10 ago 	Session ID:
Host: msnbot.msn.com
User Agent: msnbot/1.0 (+http://search.msn.com/msnbot.htm)
/index.php?main_page=cookie_usage

Session Directory  	/home/xxx/xxx/cache  	 
Cookie Domain 	True 	Info 
Force Cookie Use 	False 	Info 
Check SSL Session ID 	False 	Info 
Check User Agent 	False 	Info 
Check IP Address 	False 	Info 
Prevent Spider Sessions 	True 	Info 
Recreate Session 	False 	Info 
IP to Host Conversion Status 	true

Version: 1.3.0.5 (yes the newest)

[paulm]

I had a thread running in the old forums and can't seem to find it now as the search function here does not seem to search the archives.

The search function does search the archives, but it seems like the topic isn't in the archive anymore...

The google cache still has it online though: http://66.249.93.104/search?q=cache:...ient=firefox-a



I haven't read the whole thread, but I think that thread is mostly about the cookie usage message on "normal" pages. But SE's hitting main_page=cookie_usage is also discussed there.

[nl2dav]

Thanx, the quoted robots.txt file looks useful.

edit: only a few minutes later;

00:03:07 0 ¥Spider 38.113.234.181 01:51:44 01:51:44
Time Since Clicked:
00:03:07 ago Session ID:
Host: crawl2.cosmixcorp.com
User Agent: voyager/1.0
/index.php?main_page=cookie_usage

[paulm]

Not all bots listen to robots.txt files, only the nice ones do.

(And from what I have heard they only read robots.txt once in while, but I am not sure about that.)

[CnTGifts]

have this problem too right now theres 7 of them online all inktomisearch.com and at the cookie_usage.html

[Ajeh]

I have seen this mainly with inktomisearch on the whos_online ...

And I have seen them all over the same site ...

The spiders getting stuck on the /index.php?main_page=cookie_usage does not seem to prevent the spidering of the rest of the site ...

You could always adjust the robots.txt to tell them no ...

Granted reading yahoo on the way their spiders work it almost sounds like yes we do listen to robots.txt but we might stick the link in there anyway ...

[paulm]

cookie_usage.html

I assume you understand that when using SEO url's you need a different robots.txt file CnTGifts

[Scorpiowoman]

I am having the same thing happen to us...it seems like it tries to go through our website and then ends up like this and LEAVES the website...and then only to return a few minutes later and repeats the process...

00:07:530 ¥Spider 74.6.87.1412:24:0712:24:07 Time Since Clicked:
00:07:53 agoSession ID:
Host: ct501054.inktomisearch.com
User Agent: Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)
/index.php?main_page=popup_image&pID=422 00:07:420 ¥Spider 74.6.86.20012:24:1812:24:57 Time Since Clicked:
00:07:03 agoSession ID:
Host: ct501091.inktomisearch.com
User Agent: Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)
/index.php?main_page=cookie_usage 00:07:330 ¥Spider 74.6.86.16112:24:2712:25:07 Time Since Clicked:
00:06:53 agoSession ID:
Host: ct501140.inktomisearch.com
User Agent: Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)
/index.php?main_page=cookie_usage

I noticed this from another thread that is no longer in zencart but was in the google catch as mentioned before...it was recommended to put the following in the robots.txt file...

[FONT=Arial]Disallow: /index.php?main_page=cookie_usage[/FONT]

and it seemed to work for Duwane and Woodymon

but I do not want to do anything that would hender the website being indexed, what do you recommend?

Trying hard to get indexed here....thanks in advance to all for any suggestions on this cookie_usage issue...

Also, if you have any suggestions for my website if you could PM me about I would appreciate it, as I am getting a minimum of 300 sessions a day from people, but only about 1 or 2 orders a day right now...

[yellow1912]

I have the same problem with MSN spider. 1 thing I suspect: I have a small javascript (provided by MSN), sort of GoogleAnalytics to watch my page stat, it may be causing the problem.

[studeo]

I have a related issue. Google has been good to me and quickly I may add. There are about 6 pages in google's index from my site that are listed as Cookie Usage: xxxx

These pages are pages like About us, write review, etc. The index listing looks pretty silly.

Any Ideas other than the url rewrite for now??

Thanks