Internal server error (500), yet pages of store load normally (...)

[Sovjohn]

Hello,

Our Technical Dept. has tried everything, and our last resort is Zen Cart (and the control panel vendor, CPanel) to resolve this.

We have searched all documentation, as well as forums and online resources, and could not trace a problem similar to ours. We have set permissions (755/644/et al) as recommended by Zen Cart documentation for SUPHP installations, as well.

A customer has specified a very weird error on his website, namely he is facing problems with search engine crawlers / bots.

Upon investigation, we found out a very peculiar outcome:

The customer's website ( http://www.futurehouse.gr , based on the last stable version of Zen Cart) loads normally to every human visitor using any browser, such as Firefox, IE, et al.

However, attempting to access this website either as a bot, or even using the 'wget' command, gives out an HTTP 500 Internal Server Error message.

As can be easily understood, crawlers / bots give up after encountering an HTTP 500 error, while for some reason unknown to us, human visitors using conventional browsers can view the website normally, despite the HTTP response, the website loads as normal.

We have found no information at all for this error message in our error logs, but I thought that you might be able to point us to the right direction.

The server runs a relatively hardened PHP installation, under a SUPHP/Suhosin/ et al environment.

The customer, rightly so, complains that his website is totally inaccessible to search engines. Yet it works...with humans, or at least browsers.

Any kind of feedback would be most welcome. We will also contact the control panel vendor about this issue, in case they can point us to a new direction.

[Website Rob]

Welcome to the Forum.

It's not uncommon to have 'wget' disabled but it is strange that SE Bots would be denied access. Suggest you check your .htaccess file and chat with your Hoster as they may know more about the Server settings that could cause the Bot problem.

[Sovjohn]

I'm the hoster

Customer does not know why the problem appears, and we didn't find any information anywhere which would relate to the problem.

I have never seen something open up normally through browsers and yet deny access through wget or bots/scripts, so if it's not config-related, I don't know what it is. A bug, I'd reckon.

[BlindSide]

The server runs a relatively hardened PHP installation, under a SUPHP/Suhosin/ et al environment.

As an IT guy, this is a red flag for me. Without removing the hardening from the equation, we cannot rule out that your server is getting in the way. Would it be possible for you to carbon copy the entire site (files & mysql database) to another location without the PHP hardening? That would definitively pinpoint the issue as residing with the software or the server.

You said you're running "the last stable version of Zen Cart." What version specifically are you running?

[BlindSide]

More technical detail:
Human-readable web browsers are fairly flexible. Even if they receive an error code, they will still display any page content returned along with the error code response. They only fall back on their default error view if no content is provided.

The wget command and search engine crawlers are far more strict, they will bail out as soon as they see the error code response.

This is definitely a problem with your server configuration.

[Website Rob]

Let us not confuse 'wget' with 'GET' as they are two different commands.

Although I'm not sure about 'wget', the GET command is working just fine as evidenced by your pages showing. Although there is one 500 error code, it is at the beginning of the page and I suspect is has to do with JS being used. Try removing the "Script Source: CodeLifter.com" code block and see what happens. Otherwise, there could be a clash between the umlauts used within the language and SUPHP which could require a lot more work to correct.

As to the SE Bots being blocked, not sure how you know they are being blocked nor what type error msgs. are being generated. What is different between the main site and the 'test' site; since the 'test' site is being indexed.

[DrByte]

I can't seem to replicate the problem on that site when emulating wget or msnbot, or other common spider engines.

What's the permission level set on /includes/spiders.txt? Is that file damaged in any way?
What Zen Cart version is in use?
What addons are installed? How is this installation different from a brand new clean installation without any addons or customizations using the latest version of Zen Cart?

[DrByte]

Further investigation, which you can do yourself, reveals that your server *is* actually sending a 500 response.

Try this:
- Use Firefox
- Enable the LiveHTTPheaders extension
- Visit the site

You'll see that the first response is indeed a 500 Internal Server error response.

So, wget is legitimately saying what's really going on. Human browser viewing is apparently ignoring that in favor of the rest of the data that follows it.
As the host, you'll need to fix whatever's triggering that from your server.

http://www.futurehouse.gr/



GET / HTTP/1.1

Host: www.futurehouse.gr

User-Agent: Mozilla/5.0 ... Gecko/20100101 Firefox/6.0.1

Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8

Accept-Language: en-us,en;q=0.5

Accept-Encoding: gzip, deflate

Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7

DNT: 1

Connection: keep-alive

Cache-Control: max-age=0



HTTP/1.1 500 Internal Server Error

Date: Tue, 06 Sep 2011 18:19:26 GMT

Server: Apache

X-Powered-By: PHP/5.2.17

Expires: Thu, 19 Nov 1981 08:52:00 GMT

Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0

Pragma: no-cache

Content-Encoding: gzip

Vary: Accept-Encoding

Connection: close

Transfer-Encoding: chunked

Content-Type: text/html; charset=utf-8