blank page due to spider scans overloading the server ?

Hi,

since I 've let search engines scan my store by commenting out the line in robots.txt, I am experiencing blank pages. Do you think it's related ?

additional info :

• I set a 302 redirection in the root .htaccess for the whole /store/ folder to a "store offline" page, and when I removed it after 30 min or so, the store was accessible again, but only for a time. Then, blank page again.

• google and yahoo bots are almost constantly listed in the admin "who's online" even when I get blank pages

• with Safari, I get "connection dropped.... server too busy" message

• debugging options for blank pages produced no logs, and there is no error in the server logs either

• the last upload I did produced no blank page so I rule out a PHP or an file upload error

• the section of my site outside the store does not experience any problem.

• blank pages never occur for the admin section, which work fine all the time

What's happening ??? I am puzzled.
Any opinion appreciated.
Thanks

Denis
http://viaterra.net

You need to speak to your webhost about this.... The spiders are crawling normally.... this is a good thing. Your hosting is running out of resources, and since crawlers make the same http access as people.... This is a problem!

Might look at your image optimization.... 50K images are a genuine resource problem.

~Melanie

Thanks for the comments.

I am checking if changing the crawlers access rate has any effect.
Parameters are now as follows :

YAHOO (set in robots.txt)
User-agent: Slurp
Crawl-delay: 0.5

GOOGLE (set in webmaster tools)
250 s between requests

The problem is intermittent so it's hard to judge.
If it's related to server load, I don't understand why admin pages and non-store pages remain accessible, and only the store itself is affected.

Denis

the problem remains and is intermittent.
I noticed :

• IE8 : when I can reach my store, the status bar keeps saying "1 item remaining"

• Chrome : when I cannot connect :
  Error 324 (net::ERR_EMPTY_RESPONSE) : Unknown Error

Any idea ? You can try for yourself :
http://viaterra.net/store

Thanks in advance for any help.

Denis

I may have found the problem : extra </div> at the very end of the page. To find it, I copied the source code from the browser into an HTML editor and it highlighted the invalide code. Here's the section of code :

<!--bof- site copyright display -->

</div>

Now I have to find where that extra </div> comes from, comment it, and test ....

Denis

found the extra </div> in tpl_main_page.php
commented it : now no invalid code but :

... still blank page ... and still get the "1 remaining item" ...

Also, I noticed that IE8 sometimes displays the site with the "1 remaining item" in the status bar, while at the same time, FF gives a blank page.

I am stuck...

found the cause of the "1 remaining item" :
http://www.addthis.com/forum/viewtopic.php?f=2&t=8177&st=0&sk=t&sd=a&sid=d3b74b34c9f6703518affb6f2971336d

Blank page not related, problem persists...

checking again today : no problem of blank pages.
So could have caused the problem ?

-1- server load ? not sure, since :

• spiders weren't always crawling when the problem occurred
• spiders are on my pages as I write this and there is no blank page
• the blank page could occur on FF and not on IE at the same time
• pages outside the store, and the admin pages were always accessible

-2- temporarily unstable internet connection causing page time-out ? not sure either, since :

• other websites and the admin pages loaded normally

-3- PHP errors ? definitely not, since :

• problem was intermittent, an it worked before and it works today
• no error in blank page debug logs

-4- the extra </div> ? not the cause since :

• it was always there from installation
• removing it did not solve the problem

-5- the "1 remaining item" due to the AddThis button ? not the cause since :

• resolving that did not solve the blank pages at the time

Conclusion : I don't think I can explain why I was getting these blank pages. Hopefully it won't occur again, it's a bit worrying when a store just goes live and this sort of thing occurs !!!

I hope this thread can help someone some day.
Denis