Spiders being banned from my site

I've had this as my spiders.txt since yesterday, as ########## is the bulk of what clogs up my who's online:
$Id: spiders.txt 2014-06-17 drbyte $
##########spider

I am able to index my site using Screaming Frog SEO Spider while spiders.txt is like this. When I use the default contents of spiders.txt, I get error 500 when trying to index. $spiders = null ; I am able to index just fine (with the above mentioned spiders.txt containing just two lines)

Zedez512:

I've had this as my spiders.txt since yesterday, as ########## is the bulk of what clogs up my who's online:
$Id: spiders.txt 2014-06-17 drbyte $
##########spider

I am able to index my site using Screaming Frog SEO Spider while spiders.txt is like this. When I use the default contents of spiders.txt, I get error 500 when trying to index. $spiders = null ; I am able to index just fine (with the above mentioned spiders.txt containing just two lines)

Shame that the forum software obfuscated the text, but I'm convinced it didn't read 'HelloWorld' :-)

No need to keep repeating yourself. No need to introduce yet another factor (Screaming Frog).

I will repeat my request. (Slightly differently, as you haven't given me the next clue I'm seeking)

What happens with the /includes/init_includes/init_sessions.php unchanged and the spiders.txt containing a SINGLE LINE that reads 'HelloWorld'.

I'm 99% sure I've already identified the cause, BUT, without knowing the result of the 1st test there is still that 1% doubt that I'd like to eliminate.

Thanks,
RodG

spiders.txt containing only HelloWorld, I'm able to index the same as if it were empty or just had those 2 lines in it.

Zedez512:

spiders.txt containing only HelloWorld, I'm able to index the same as if it were empty or just had those 2 lines in it.

EXCELLENT. Mystery solved. It is something in the default contents of the file that is triggering a security setting.

If I assume (not a good thing) that it is something matching a mod_security rule, you can possibly rectify the problem by following the instructions found here here:
http://www.faqforge.com/linux/how-to-disable-apache-mod_security-for-a-website-in-ispconfig-3/

However, I can't guarantee that this will resolve the problem because your server could be using something other than mod_security. The 'fix' here can also be difficult for some people to apply due to the inability to determine where to place the disable directives (which is bad practice anyway).

Another approach would be to view the server error logs (available via *ispconfig 3 ) *and looking/searching for the '500' error, as this line will tell you exactly which entry is triggering the security rule. (Note: APACHE SERVER LOGS, not the ZENCART logs)

My approach however would be the one I suggested earlier (especially as you/we have now positively confirmed the source of the problem), and that you repopulate the spiders.txt file with the default data - but to only add 50-100 lines at a time. If you get the 500 error with the first batch, just delete lines until the problem goes away (keeping in mind that it may even be the very 1st entry that is the culprit).

Basically, a simple process of elimination. I'm suggesting 50-100 lines at a time because it is going to be quicker to eliminate by 'chunks' rather than one line at a time.

Bottom line though is that the cause has now been positively confirmed. You now need to find the actual culprit. No need to look anywhere else.

After you have identified the line (or lines) causing the problem you'll have two options.

1. just leave them deleted, and hopefully it isn't one of the bots that is clogging up your whos online.
2. Make changes to the mod_security rules so that the culprit text no longer triggers the security rules. (You will probably need to Google how to do this because I'm not familiar enough with ispconfig to give any further guidance).

I anxiously await your findings.

Cheers
RodG

Just by adding "spider" to it I get error 500 when indexing with screaming frog.
My user agent when using this indexing tool is "User Agent: Screaming Frog SEO Spider/3.3" so when I add "screaming" to robots.txt I of course get error 500.

Which just makes me think the same thing as in my first post: anything identified as a spider gets error 500

Zedez512:

Just by adding "spider" to it I get error 500 when indexing with screaming frog.
My user agent when using this indexing tool is "User Agent: Screaming Frog SEO Spider/3.3" so when I add "screaming" to robots.txt I of course get error 500.

Which just makes me think the same thing as in my first post: anything identified as a spider gets error 500

1. why are you adding stuff to the robots.txt? I thought we'd identified spiders.txt as being the source of the problem.
2. there is nothing 'obvious' about an entry in the robots.txt as being the cause of any server 500 errors.
3. At this stage your thoughts about it being anything identified as being a bot is unfounded. It could be valid, but how about a simple test to prove/disprove it? What happens if 'screaming' is the one and only line in the spiders.txt file?
4. Be very careful when using Screaming frog, this crawler can/does really hammer a site with its crawling, so much so that it will either overload the server, or possibly trigger different security rule that places limits on the speed/frequency permitted by any given host. Either of which will also create a server 500 error. (SF has options to limit the rate of the crawl)
   Note: if Screaming frog can crawl even a single page before the server 500 error shows then the problem will NOT be 'anything identified as a spider'.
5. It could be the text 'spider' that is triggering the security rules.
   Back to you.....

I meant spiders.txt not robots.txt, I'm very tired right now.

I just added mozilla to spiders.txt then went to my website and it showed me as a spider in the who's online list, I didn't have any error when browsing my site!
I made google fetch my site with nothing in spiders.txt except "google", it was unreachable
I removed it from spiders.txt and it was able to reach my site.

I will try to have someone look into mod_security today.

Zedez512:

I meant spiders.txt not robots.txt, I'm very tired right now.

I just added mozilla to spiders.txt then went to my website and it showed me as a spider in the who's online list, I didn't have any error when browsing my site!
I made google fetch my site with nothing in spiders.txt except "google", it was unreachable
I removed it from spiders.txt and it was able to reach my site.

I will try to have someone look into mod_security today.

Ok, so adding Mozilla as a spider into the spiders.txt file shows it as a spider with no server 500 errors. Conclusion, your assumption that anything flagged as a spider is therefore incorrect. We are once again back on track with my diagnosis that it is something in the contents of this file that is causing the 500 error.

I'm a little bit concerned about the same experiment using 'google' as the text entry gave an 'unreachable' error, because this is nothing like the server 500 error that we are trying to pinpoint. This could be yet a different problem again.

I'm somewhat peeved that we've got this far and you are once again changing horses mid race by getting "someone to look at mod_security, because quite frankly until the cause of the trigger has been identified all you will end up doing is having someone making semi random changes that has just as much chance of causing a new problem in addition to the one that we are currently trying to resolve. Don't do it.

I've already given you two valid methods that will have the exact same end results without messing with the security rules. The final solution could very well come back to this, but you aren't at that stage yet and the last thing you want is another 'expert' that is going to compound things. Even with my decades of experience this isn't an area that I'd jump into lightly, and never at all until I've identified the exact trigger so that I'd know what the heck it is I'd be needing to change. Furthermore, as I've also stated previously that this is definitely a security related issue that is probably related to mod_security, it also has a good chance that it is an entirely different security mechanism that is causing the problem, so once again just jumping on this bandwagon has more chance of breaking things in other ways and compounding the issue.
Is that what you really want?

Is it really so hard to follow my instructions? Seriously, a few messages ago we were basically one step away from finding the cause (or what I believe to be the cause), but since then you've taken two steps backwards of your own accord, leaving me to provide additional information that has now demonstrated what I've been suspecting all along (that it isn't 'anything identified as a bot).
We could spend days changing things to demonstrate everything that isn't the cause, OR you can save us both a lot of time by following my instructions to the letter and find the cause (and final solution in just a couple more message exchanges.. I place where we would be now if you had already done this instead of going off on these weird tangents.

Anyway, I'm done for the day. Please try not to screw things up for yourself any further. OTOH, why should I care, you seem to ignore half of what I've written anyway :-(