Yahoo Ignoring 'ROBOTS_PAGES_TO_SKIP'

[Tech-E]

This is probably not a problem with Zen Cart, unless the ROBOTS_PAGES_TO_SKIP list is not generating a noindex meta tag. My main business is SEO and I've been doing it for over 10 years.

Several spiders do not respond quickly to changes in either meta tags or the robots.txt file. The "noindex" meta tag is more effective than the robots.txt file. Regardless of what Google or Yahoo say in their documentation, they do not visit the robots.txt file regularly and only randomly scan the file.

I've always found that Yahoo is the most stubborn and it can sometimes take a year for them to recognize changes to the robots.txt file.

The robots.txt file works best when it is in place when the site first goes up. Once a spider indexes a site, changes to the file take longer to have an effect.

The ROBOTS_PAGES_TO_SKIP list should generate a noindex meta tag on each of the pages in the list. Have you checked to see if that is working correctly?