search. in the spider list

15 Nov 2011, 6:32 PM

Zen Follower

Join Date:: Oct 2011
Location:: AZ
Posts:: 383
Plugin Contributions:: 0

search. in the spider list

Based on some posts in this thread:
http://www.zen-cart.com/forum/showthread.php?t=163007&page=5

I downloaded the new spiders.txt and made sure that search. wasn't in there.

So now the msnbot: search.msn.com is checking in as guest instead of indexer. That will mean indexed results with session ids in the url. (At least the # of msn searchers compared to google is very low.)

What am I supposed to do to ensure that msn can spider as a spider, but that some of my customers don't get the cookie/login errors?

1.3.9h, linux server. no addons anymore.

16 Nov 2011, 4:53 AM

drbyte

Sensei

Join Date:: Jan 2004
Posts:: 63,511
Plugin Contributions:: 176

Re: search. in the spider list

What is the EXACT user-agent you're seeing on these msn spiders?

MSN traditionally sets its user-agent to "msnbot".
Zen Cart filters spiders based on user-agent. NOT ON IP ADDRESS OR HOST NAME like 22.33.44.55 or search.msn.com

.
Zen Cart - putting the dream of business ownership within reach of anyone!
Donate to: DrByte directly or to the Zen Cart team as a whole

Remember: Any code suggestions you see here are merely suggestions. You assume full responsibility for your use of any such suggestions, including any impact ANY alterations you make to your site may have on your PCI compliance.
Furthermore, any advice you see here about PCI matters is merely an opinion, and should not be relied upon as "official". Official PCI information should be obtained from the PCI Security Council directly or from one of their authorized Assessors.