blank page due to spider scans overloading the server ?

[dlg_ie]

Hi,

since I 've let search engines scan my store by commenting out the line in robots.txt, I am experiencing blank pages. Do you think it's related ?

additional info :

- I set a 302 redirection in the root .htaccess for the whole /store/ folder to a "store offline" page, and when I removed it after 30 min or so, the store was accessible again, but only for a time. Then, blank page again.

- google and yahoo bots are almost constantly listed in the admin "who's online" even when I get blank pages

- with Safari, I get "connection dropped.... server too busy" message

- debugging options for blank pages produced no logs, and there is no error in the server logs either

- the last upload I did produced no blank page so I rule out a PHP or an file upload error

- the section of my site outside the store does not experience any problem.

- blank pages never occur for the admin section, which work fine all the time

What's happening ??? I am puzzled.
Any opinion appreciated.
Thanks

Denis
http://viaterra.net

[mprough]

You need to speak to your webhost about this.... The spiders are crawling normally.... this is a good thing. Your hosting is running out of resources, and since crawlers make the same http access as people.... This is a problem!

Might look at your image optimization.... 50K images are a genuine resource problem.

~Melanie

[dlg_ie]

Thanks for the comments.

I am checking if changing the crawlers access rate has any effect.
Parameters are now as follows :

YAHOO (set in robots.txt)
User-agent: Slurp
Crawl-delay: 0.5

GOOGLE (set in webmaster tools)
250 s between requests

The problem is intermittent so it's hard to judge.
If it's related to server load, I don't understand why admin pages and non-store pages remain accessible, and only the store itself is affected.

Denis

[dlg_ie]

the problem remains and is intermittent.
I noticed :

- IE8 : when I can reach my store, the status bar keeps saying "1 item remaining"

- Chrome : when I cannot connect :
Error 324 (net::ERR_EMPTY_RESPONSE) : Unknown Error

Any idea ? You can try for yourself :
http://viaterra.net/store

Thanks in advance for any help.

Denis

[dlg_ie]

I may have found the problem : extra </div> at the very end of the page. To find it, I copied the source code from the browser into an HTML editor and it highlighted the invalide code. Here's the section of code :

[FONT="Courier New"]<!--bof- site copyright display -->
<div id="siteinfoLegal" class="legalCopyright">Design &amp; Copyright &copy; 2009 <a href="http://viaterra.net/store/index.php?main_page=index" target="_blank">viaterra.net</a>. Powered by <a href="http://www.zen-cart.com" target="_blank">Zen Cart</a></div>
<!--eof- site copyright display -->


</div>
<!--bof- parse time display -->
<!--eof- parse time display -->
<!--bof- banner #6 display -->
<!--eof- banner #6 display -->
</body></html>[/FONT]

Now I have to find where that extra </div> comes from, comment it, and test ....

Denis

[dlg_ie]

found the extra </div> in tpl_main_page.php
commented it : now no invalid code but :

... still blank page ... and still get the "1 remaining item" ...

Also, I noticed that IE8 sometimes displays the site with the "1 remaining item" in the status bar, while at the same time, FF gives a blank page.

I am stuck...

[dlg_ie]

found the cause of the "1 remaining item" :
http://www.addthis.com/forum/viewtop...affb6f2971336d

Blank page not related, problem persists...

[dlg_ie]

checking again today : no problem of blank pages.
So could have caused the problem ?

-1- server load ? not sure, since :

- spiders weren't always crawling when the problem occurred
- spiders are on my pages as I write this and there is no blank page
- the blank page could occur on FF and not on IE at the same time
- pages outside the store, and the admin pages were always accessible

-2- temporarily unstable internet connection causing page time-out ? not sure either, since :

- other websites and the admin pages loaded normally

-3- PHP errors ? definitely not, since :

- problem was intermittent, an it worked before and it works today
- no error in blank page debug logs

-4- the extra </div> ? not the cause since :

- it was always there from installation
- removing it did not solve the problem

-5- the "1 remaining item" due to the AddThis button ? not the cause since :

- resolving that did not solve the blank pages at the time

Conclusion : I don't think I can explain why I was getting these blank pages. Hopefully it won't occur again, it's a bit worrying when a store just goes live and this sort of thing occurs !!!

I hope this thread can help someone some day.
Denis