spider71.yandex.ru

gonewild:

Would there be a part of the yandex spider that blocks other users?
No, but if it's visiting pages on your site too quickly then it could be making your site too busy for other visitors to stick around.
Same goes with any other spider, or even customers for that matter.

Thanks for that explanation. It was sometimes staying for 35 hours and had several sessions open with different ips.

Before adding the fix, Yandex was listed 5 times in the who's online, along with the googlebot, though it didn't stay very long and then Yandex opened up several more sessions.

Simultaneously I had a browsing customer (not yet a member) and a member who was logged in at the time with Yandex and Googlebot sessions.

I would have to agree with Dr. Byte that Yandex could and probably does max out each site while it is indexing them and whatever other evils it is doing or attempting to do to our sites, causing others to be blocked or getting the page busy error stuff, which in my opinion, adds to the evilness of this bot.

I'm grateful for this fix! Thank you very much for your hard work and efforts!

Like others, I have been hit by the yandex spider in December and it actually took me off the net because it caused my site to exceed its bandwidth. My webmaster has been trying to block it and now that I have found Dr. Bytes' Dec 10th posting we will add that php coding.

However, there is a site that I came across in Dec that I would like you to look at and it is russianmarketer.com/russian-search-engine-spiders-user-agents . This guy says:

"Inserting the following user-agent strings into the User Agent FireFox Plugin would allow you to browse the web as Yandex or Rambler."What does this mean? Could this be part of the problem, too? Would like to know if you could explain what this guy is suggesting.

TIA
Anita

I just did a 'spoofing yandex spider' google search and came across this forum (webmasterworld.com/search_engine_spiders/4011987.htm) at WebmasterWorld.com. Could this be what we are seeing in our stats? 'Baddies' spoofing the yandex search bots?

Anita

Maybe. But unlikely. And, I'm not sure I'd be bothered about people attempting to emulate a particular spider ... it serves them no purpose, and does nothing bad to your store.

Thanks, Sensei!

Always good to hear from Dr. Byte.

Happy New Year to you. Here's hoping you have an easier winter back there in Ontario compared to last year.

Anita
(from beautiful, wet and windy BC)

P.S. My webmaster added the php file this morning to block yandex - will keep an eye on the stats to see how it goes.

skookumnarrows:

This guy says:

"Inserting the following user-agent strings into the User Agent FireFox Plugin would allow you to browse the web as Yandex or Rambler."

What does this mean? Could this be part of the problem, too? Would like to know if you could explain what this guy is suggesting.

TIA
Anita

There are several reasons why a developer person may want to spoof a spider. For example, if you are looking for a way to block the spider you need a method to check if the blocking is working. You can either wait until the next 'attack', or you can 'fake' an attack by setting the User Agent of your own browser. it should be obvious that being able to 'fake it' is a lot more reliable than waiting around for an unknown period in the hopes that you don't see the thing being blocked :-)

OK, I guess that is ONE reason ... In this instance I can't think of any others. However, the ability to fake User Agent ID's does have other uses, for example, with some sites that insist I use MsExplorer rather than Firefox I can usually gain access simply by changing the UAiD rather than the browser itself.

Cheers
Rod

Should this code be ok if our ZC is a subdomain ?

DrByte:

USE AT OWN RISK: Implementing this wrong could cause your site to malfunction or not be indexed at all.

Create a new file: /includes/extra_configures/block_yandex_spider.php ... containing ONLY the following code:```

taogem:

Should this code be ok if our ZC is a subdomain ?

Yeah. It should be fine.

Cheers
Rod

I just implemented Dr. Byte's fix, i.e.: /includes/extra_configures/block_yandex_spider.php

But when I checked "who's Online" the yandex spiders were still there (around 6 of them). Can I expect a lapse of time of some sort before it starts working? And just to add my own suspicion concerning this Very-Bad bot: Prior to when it showed up a few weeks ago, I had almost NO spam for my store email account. Now I get several per day (pharmacy junk). Coincidence? I think not.

The real Yandex bots supposedly obey robots.txt directives, but Yandex is an enigma for some webmasters at: http://www.webmasterworld.com/search_engine_spiders/4126929.htm

speedcreed:

I just implemented Dr. Byte's fix, i.e.: /includes/extra_configures/block_yandex_spider.php

But when I checked "who's Online" the yandex spiders were still there (around 6 of them). Can I expect a lapse of time of some sort before it starts working?
It "works" immediately. But any existing connections will have to timeout before they disappear from your Who's Online list. That's normally 20 minutes.

Thank you, Dr. Byte.

I have a new zencart and am just learning. When I checked this morning, I had a yandex spider camped out all night.

Googled your solution. Bye bye spider.

Thanks for being there.

Seeing that no-one has complained that the 'block_yandex_spider.php' did any harm, I've just added it to one of my Zen Cart sites and all seems good so far.

Thank you for this solution, :cool:
let's hope that this bandwidth thief keeps away.