Spiders being banned from my site

About two years ago I decided to hire a freelancer to make a custom skin and few mods for me, at the time I was not interested in SEO so I wouldn't have noticed, but I suspect he banned bots from indexing my site. I have also hired another freelancer within that time to do things for me so it could have been him instead.

None the less, when I tried my hand at some SEO, some 9 months ago, I noticed that bots were unable to index my site. I eventually got help from a sysadmin friend who found that one of the files had banned anything from spiders.txt from indexing my site. We argued about whether this was a normal function of zen cart and then his fix was to simply empty my spiders.txt file and I foolishly did not note down what file he had found the offending code in. I can't get a hold of him to ask him to take a look and find the file again and I have downloaded all of my site files and searched a few keywords (such as spider) to try finding the code again without any luck. What can I do?

My who's online is bogged down by bots and extremely laggy.

Zedez512:

About two years ago I decided to hire a freelancer to make a custom skin and few mods for me, at the time I was not interested in SEO so I wouldn't have noticed, but I suspect he banned bots from indexing my site. I have also hired another freelancer within that time to do things for me so it could have been him instead.

None the less, when I tried my hand at some SEO, some 9 months ago, I noticed that bots were unable to index my site. I eventually got help from a sysadmin friend who found that one of the files had banned anything from spiders.txt from indexing my site. We argued about whether this was a normal function of zen cart and then his fix was to simply empty my spiders.txt file and I foolishly did not note down what file he had found the offending code in. I can't get a hold of him to ask him to take a look and find the file again and I have downloaded all of my site files and searched a few keywords (such as spider) to try finding the code again without any luck. What can I do?

My who's online is bogged down by bots and extremely laggy.

Geeez, I hate it when someone makes unauthorized changes to a client's site!!! Seems like none of your 'helpers' has the slightest clue about the workings of ZenCart!

What is your ZC version?
Which mods are installed?

Zedez512:

I eventually got help from a sysadmin friend who found that one of the files had banned anything from spiders.txt from indexing my site. We argued about whether this was a normal function of zen cart and then his fix was to simply empty my spiders.txt file and I foolishly did not note down what file he had found the offending code in.

Curious. Although theoretically possible, the mind boggles at what was needed to make the spiders.txt file act as though it were a robots.txt file.

The spiders.txt file is a zencart thingy that prevents known spiders from creating SESSIONS. It doesn't (or isn't supposed to) prevent bots from crawling a site. That is what the robots.txt is for

Zedez512:

I can't get a hold of him to ask him to take a look and find the file again and I have downloaded all of my site files and searched a few keywords (such as spider) to try finding the code again without any luck. What can I do?.

IF the freelancer did somehow manage to make the spiders.txt file act like a robots.txt file s/he is both very clever and very foolish.
I couldn't. Why would anyone do that????

I wouldn't know where to start looking for this - So I won't even try. I'm hoping that the freelancer didn't make any actual code changes to pull this feat off.

What I can do is tell you what you should have.

... but first.....

Zedez512:

My who's online is bogged down by bots and extremely laggy.

As a general rule, bots are a good thing. Without them your site is going to be near impossible to find. Be very careful you don't block them all.

OK, so there are basically two files for you to check and consider

/includes/spiders.txt
This file contains a simple list of known bots. One per line. This file is part of all Zencart releases - it isn't version specific, so check the contents of this file, if it is empty (or near empty) just replace it with a copy from the ZenCart distribution files. It will contain over 500 lines/entries. Not something to be edited by hand.

This will take care a lot of the 'whos online' bot activity - but it doesn't prevent them.

The other file is called
robots.txt from crawling the site. This is located in the root folder of your store. It quite a small file, that would typically read like:

User-agent: *
Disallow: /cgi-bin/
Disallow: /cache/
Disallow: /logs/

This is telling all 'good' bots to not crawl the folders that are disallowed.

To Disallow a specific bot from crawling anything on the site, you'll need to add something like"

User-Agent: badbot
Disallow: /

You can find more examples here http://www.robotstxt.org/robotstxt.html

Restoring the spiders.txt file and judicially editing the robots.txt file should get things back to how they should be again. If not, you are going to have to dig really deep into the zencart code to see what the freelancer changed to make the spiders.txt file act like a robots.txt file. It probably won't come to that though 'cos it is giving the freelancer more credit than they deserve. ;-)

Important: Not all bots will 'honor' the contents of the robots.txt file - (Google, Bing, and most do). For those that don't, they need to be block via other means - generally via the .htaccess file(s), but there are other methods (such as using a firewall).

Hopefully this helps get you back on track.

Cheers
RodG

My robots.txt and spiders.txt are fine, I'm 100% sure.

What happens is that anything inside spiders.txt or I think identified as a spider, is being denied access to my website. So the current "fix" to allow spiders to index my site is to have an empty spiders.txt (I have a copy with entries though).

Basically the issue is that if I have entries in spiders.txt, those bots are banned from indexing my site and when I don't have those entries, the who's online is completely clogged up with 2,000 bots so I can't actually see who is online with an active cart.

I really wish that I had noted down the file which my friend showed me. I remember clearly seeing a couple of lines which were specifically banning spiders. I'm not sure what keywords I can search to track down this code within my files since "spider" doesn't seem to be showing it. I have already searched "deny" and "die" and didn't see anything out of the ordinary there.

I guess I am asking for words/code which I can search to try finding this so I don't have to spend hundreds of dollars having someone dig through every file for me.

Zedez512:

My robots.txt and spiders.txt are fine, I'm 100% sure.

What happens is that anything inside spiders.txt or I think identified as a spider, is being denied access to my website. So the current "fix" to allow spiders to index my site is to have an empty spiders.txt (I have a copy with entries though).

Basically the issue is that if I have entries in spiders.txt, those bots are banned from indexing my site and when I don't have those entries, the who's online is completely clogged up with 2,000 bots so I can't actually see who is online with an active cart.

I really wish that I had noted down the file which my friend showed me. I remember clearly seeing a couple of lines which were specifically banning spiders. I'm not sure what keywords I can search to track down this code within my files since "spider" doesn't seem to be showing it. I have already searched "deny" and "die" and didn't see anything out of the ordinary there.

I guess I am asking for words/code which I can search to try finding this so I don't have to spend hundreds of dollars having someone dig through every file for me.

You could do a comparison of your file system with a stock ZC file system of the same version. Use BeyondCompare or WinMerge or similar. Tedious but it will show the changes your guys did to the cart.

Zedez512:

My robots.txt and spiders.txt are fine, I'm 100% sure.

I'd be happier if you were 200% sure :-)

Zedez512:

I really wish that I had noted down the file which my friend showed me. I remember clearly seeing a couple of lines which were specifically banning spiders.

If I were going to do a hack like this (and being a 'lazy' programmer) I'd probably hook into where the spiders.txt is already being read, which is the file

/includes/init_includes/init_sessions.php

My suggestion would be to compare this file with a ZenCart original.

Zedez512:

I'm not sure what keywords I can search to track down this code within my files since "spider" doesn't seem to be showing it. I have already searched "deny" and "die" and didn't see anything out of the ordinary there.

The 'obvious' thing to search for would be 'spiders.txt' - You should find ONE and only one file that contains this (the one I've mentioned above).

If you didn't even find this one with your search then if there are others, you won't find them either, and that there is something amiss with your search (I used the developers toolkit. What did you use?)

Zedez512:

I guess I am asking for words/code which I can search to try finding this so I don't have to spend hundreds of dollars having someone dig through every file for me.

The way I see it, although the code could have been hacked to behave as described in a number of places, it could only work as described with some reference to the spiders.txt file itself - So if the hack hasn't been performed on the init_sessions.php file there must be another file that also contains 'spiders.txt'.

Accept this as an indisputable fact, and take it from there. :-)

Cheers
RodG

The only reference to spiders.txt is in fact that file and it's no different to the original :(

When spiders.txt is empty, my site can be indexed, but we don't want spiders to have sessions do we?
When spiders.txt has entries like normal, all bots get internal server error 500

I just did some testing, changing:
$spiders = file(DIR_WS_INCLUDES . 'newfile.txt');
and changed $spider to something else and they still get error 500 in both cases.

Did you look at my suggestion in post #5 of this thread?

frank18:

Did you look at my suggestion in post #5 of this thread?I will have a go at this but what if it is inside the new template files? The problem has persisted across two servers and the site was indexed just fine (under a different domain, I went for a new look and a new name) before I had any work done.

Now that I think about it, I did have him make it so that customers were always logged in when returning to the site (or never logged out, unless they manually log out) so that they wouldn't lose their cart session, as most were unaware they would lose it when inactive and not logged in. Is it possible that he has done something to sessions which is causing a server error when bots are not given a customer session id?

Zedez512:

I will have a go at this but what if it is inside the new template files? The problem has persisted across two servers and the site was indexed just fine (under a different domain, I went for a new look and a new name) before I had any work done.

Now that I think about it, I did have him make it so that customers were always logged in when returning to the site (or never logged out, unless they manually log out) so that they wouldn't lose their cart session, as most were unaware they would lose it when inactive and not logged in. Is it possible that he has done something to sessions which is causing a server error when bots are not given a customer session id?

Hard to say without knowing what he has actually done.

What do you see in the error logs ( /logs folder )?

An extreme amount of errors, so much so that I can't open the directory (ftp times out every time, winzip crashes when trying to open it locally) so I had to rename it and create a new one just to view these:
PHP Warning: Division by zero in includes/modules/theme203/footer.php on line 28
PHP Warning: Invalid argument supplied for foreach() in includes/templates/theme203/common/tpl_footer.php on line 42
PHP Fatal error: Call to a member function get_products() on a non-object in includes/templates/theme203/common/tpl_header.php on line 64
PHP Fatal error: Call to a member function set_snapshot() on a non-object in includes/modules/pages/product_info/header_php.php on line 35

footer.php: http://pastebin.com/g2uhYWH6
tpl_footer.php: http://pastebin.com/sTDpFnAJ
tpl_header.php: http://pastebin.com/pDi8hSBB
header_php.php: http://pastebin.com/9KNAdKUh

Zedez512:

An extreme amount of errors, so much so that I can't open the directory (ftp times out every time, winzip crashes when trying to open it locally) so I had to rename it and create a new one just to view these:
PHP Warning: Division by zero in includes/modules/theme203/footer.php on line 28
PHP Warning: Invalid argument supplied for foreach() in includes/templates/theme203/common/tpl_footer.php on line 42
PHP Fatal error: Call to a member function get_products() on a non-object in includes/templates/theme203/common/tpl_header.php on line 64
PHP Fatal error: Call to a member function set_snapshot() on a non-object in includes/modules/pages/product_info/header_php.php on line 35

footer.php: http://pastebin.com/g2uhYWH6
tpl_footer.php: http://pastebin.com/sTDpFnAJ
tpl_header.php: http://pastebin.com/pDi8hSBB
header_php.php: http://pastebin.com/9KNAdKUh

What happens if you switch to the classic template? Do you still get errors?

Also noted theme203 - is that a template from Template Monster?

Zedez512:

The only reference to spiders.txt is in fact that file and it's no different to the original :(

When spiders.txt is empty, my site can be indexed, but we don't want spiders to have sessions do we?
When spiders.txt has entries like normal, all bots get internal server error 500

I just did some testing, changing:
$spiders = file(DIR_WS_INCLUDES . 'newfile.txt');
and changed $spider to something else and they still get error 500 in both cases.

Ahah. We've been barking up the wrong tree. The bots aren't being blocked at all, well, not in the traditional meaning of being blocked.
The giveaway being the server 500 error.
You won't find the cause as being anything to do with any php code. The so called blocking is unintentional.

What's happening here is that the server is "dying". This will never be by intent.

Server 500 errors can often be very difficult to diagnose, BUT, In this case you've already narrowed things down a lot.

There are two possible causes. (ok, more than two, but at this stage only 2 worth considering)
Try the easy one first. File permissions.
Set the file permissions of the spiders.txt file to "644".
Does the problem persist? (if the previous permissions were "777" this change is possibly all that is needed.

The other cause will be related to a mod_securty rule that your webhost has in place, and that one (or more) of the entries in the file is triggering the rule.
The easiest way to tackle this is to seek help from your host (be sure to tell them that it is a server 500 error (nothing else will be important for them to know). They will have log files not available to you that will detail the exact cause of the error and they will either fix the problem for you (by adjusting their security rules) or they'll tell you what the actual 'trigger' is, and you will be able to remove just that entry from the spiders.txt file.
Alternatively you can try to determine this yourself by deleting "chunks" from the spiders.txt file until the problem goes away (re adding them as each "chunk" is eliminated) IOW, a bit of trial and error to find the specific line causing the error.
Note: it could also be the "file" command itself that has been disallowed by your host, but this is a little less likely because I would imagine that if this was the case you'd get the server 500 error even if the spiders.txt file were empty, but your comments so far suggest that that isn't the case.

Anyway, point is, you/we have been looking in the wrong places, the server 500 error is a real game changer.

Cheers
RodG

frank18:

What do you see in the error logs ( /logs folder )?

I can answer this. Nothing, nada, zilch. (pertaining to the 500 error)
Server 500 errors aren't logged anywhere that is accessible to the user. Just as the server 500 errors themselves are usually security related, making this information about the cause itself is also a security issue. This is the very reason why the error message itself lacks any details.

Cheers
RodG

Zedez512:

An extreme amount of errors, so much so that I can't open the directory (ftp times out every time, winzip crashes when trying to open it locally) so I had to rename it and create a new one just to view these:
PHP Warning: Division by zero in includes/modules/theme203/footer.php on line 28
PHP Warning: Invalid argument supplied for foreach() in includes/templates/theme203/common/tpl_footer.php on line 42
PHP Fatal error: Call to a member function get_products() on a non-object in includes/templates/theme203/common/tpl_header.php on line 64
PHP Fatal error: Call to a member function set_snapshot() on a non-object in includes/modules/pages/product_info/header_php.php on line 35

footer.php: http://pastebin.com/g2uhYWH6
tpl_footer.php: http://pastebin.com/sTDpFnAJ
tpl_header.php: http://pastebin.com/pDi8hSBB
header_php.php: http://pastebin.com/9KNAdKUh

Although these are errors that need to be sorted out, how about we keep our eye on the prize? These header and footer files and related log entries would be applicable for almost every page load on the site (that's why you have so many error logs), but, the problem being discussed, the server 500 error when reading (or attempting to read) the spiders.txt file is pretty darn specific.

If you plan on following this path of trying to solve every single error logged In the interim then I'm out of here, at least until you've taken care of all the PHP errors before finally coming back to the server 500 (security) error that this discussion is based upon.

Cheers
RodG

frank18:

What happens if you switch to the classic template? Do you still get errors?

Yes, the spiders.txt / server 500 error will still persist. Why wouldn't it? When has any template ever needed to make changes to the spiders.txt file or the init file that loads it?
I'm having a really difficult time trying to follow your logic here Frank, and I genuinely believe that you are leading the OP the long path to finding the cause of a very specific problem.

Not saying that your suggestions are wrong, and indeed there are many times that taking this 'global' approach of elimination is the right (and sometimes only) approach, but this really isn't one of those times, at least not in my opinion.

For the benefit of the OP though, I have to acknowledge that reverting back to the classic template is almost guaranteed to solve all the logged PHP errors relating to the header and footer files, so I'm guessing that this is the logic behind this suggestion.
As I wrote a short while ago if the OP wishes to follow the path of fixing the PHP errors before coming back to the server 500 security error then I'm outa here, he's all yours <g>

Cheers
RodG

Zedez512:

The only reference to spiders.txt is in fact that file and it's no different to the original :(

When spiders.txt is empty, my site can be indexed, but we don't want spiders to have sessions do we?
When spiders.txt has entries like normal, all bots get internal server error 500

I just did some testing, changing:
$spiders = file(DIR_WS_INCLUDES . 'newfile.txt');
and changed $spider to something else and they still get error 500 in both cases.

Whilst error 500 never shows in the /logs (every T,D&H knows this) one may still have entries in the /logs folder, sometimes the cause of a logged error may trigger the 500 server error. I have seen this twice this year on client's sites when I have been asked to fix their faults.

Some non-core file in your installation causes this problem hence my suggestion to swap templates. If the error still persists then it is obviously not caused by a file in a template folder.

So then let's move on and do a comparison of the remaining files to a core ZC file system. Any file which was added to the live file system will show up instantly.

You may want to pay special attention to your includes/init_includes/overrides folder to see if anything has been added there - might be a modified init_sessions.php file ....

Rod, I have a dedicated server running ispconfig 3. My sysadmin friend apparently fixed everything up from a server admin point of view and he didn't report anything wrong from there. I'll link him to your post and see if I can pay him to help me (again, 200 euros the first time..)

Oh and spiders.txt was 644

frank18:

Some non-core file in your installation causes this problem hence my suggestion to swap templates.

This is a wild assumption.

frank18:

So then let's move on and do a comparison of the remaining files to a core ZC file system.

Egads man! Remind me to never seek your help faultfinding a problem. You're like the proverbial taxi driver that takes passengers the long way to their intended destination. This approach (for me) is always a last ditch measure when there are no other clues to go by.

We HAVE the clues needed. The problem is, and always has been identified as being related to the spiders.txt file (even the OP had managed to determine that , and further identified as being security related.

If it isn't the permissions or contents of this file itself then it must be related to another file that is referencing this one. A simple search for spiders.txt will identify the latter (which I'm betting won't be the case), so it's back to the permissions or contents of the file itself. (or perhaps the 'file' directive being administratively disallowed)

This is a security related issue. plain and simple.

If the OP wishes to take your approach I wish them the best of luck. It will just be a matter of time before they come back to the direction I'm leading them.

Even the OP had managed to narrow things down closer than you've gotten.

Zedez512:

What happens is that anything inside spiders.txt or I think identified as a spider, is being denied access to my website. So the current "fix" to allow spiders to index my site is to have an empty spiders.txt

The only thing missing to this part of the puzzle is the server 500 error, which the OP mistakenly interpreted as being a 'denied access').

Kudo's to the OP for taking the diagnostic approach.

If I'm wrong with my further diagnosis of the problem I'm going to hang my hat in shame and you'll never hear from me again.
I'm that confident.

Cheers
RodG

Zedez512:

Rod, I have a dedicated server running ispconfig 3. My sysadmin friend apparently fixed everything up from a server admin point of view and he didn't report anything wrong from there. I'll link him to your post and see if I can pay him to help me (again, 200 euros the first time..)

Oh and spiders.txt was 644

Could you please do me (and yourself) a favour. Please edit your spiders.txt file so that it contains just a single line:

"HelloWorld"

If the Server 500 error goes away, the problem is related to the contents of the file. If the error doesn't go away, please edit the file /includes/init_includes/init_sessions.php and change

 $spiders = file(DIR_WS_INCLUDES . 'spiders.txt');

to

$spiders = null ;

Please report your results for the next step(s).

Cheers
RodG