Suddenly Sloooooooow Server

1.3.8.a with security patches (sever info below) mySql running on localhost. Bluehost "Pro" service with dedicated ip and ssl

We have had several sites running on this server for over two years. 16 days ago they did some maintenance on the server and rebooted it. Every since then all server responses have been sloooooow. I call and complain and they tell me nothings wrong, then it gets better for a few hours or days, than back to the same old slooooooow.

They asked if I optimized and repaired databases, and I do, through phpMyadmin, I select all tables and choose "Optimize" and just ran the repair feature to no avail.

I checked and there has been no serious throttling reported and the server specs are below. I have seen the load at 300+ and when I call they say it's running fine.

http://www.stuff4toys.com/ and other sites on my tag line are taking anywhere from 40 seconds to a minute and a half. This site had been taking 10,000 - 14,000 requests per day, now it is down to 5,000.

There are alot of entries in the tmp folder on the server under mysql_slow_queries there are many log file entries. This may be the issue, but why now? All of a sudden?

Thanks
JOhn ><>

Server OS: Linux 2.6.32-29.1.BHsmp Database: MySQL 5.1.56-community-log Server Date: 06/23/2011 10:40:54 Database Date: 06/23/2011 10:40:54 Server Up Time: 10:40:54 up 16 days, 16:24, 1 user, load average: 23.85, 26.46, 25.74 HTTP Server: Apache PHP Version: 5.2.17 (Zend: 2.2.0) PHP Memory Limit: 128M PHP Safe Mode: Off PHP File Uploads: On Max Size: 10M POST Max Size: 8M Database Data Size: 48,426 kB Database Index Size: 2,968 kB

I thought it might be something the site is running as an add-on. Annalytics, Carfty Syntax or something, but the admin is slow too.

and then every now and then, it loads in under 5 seconds. What's up with that?

The server loads that you posted above show that the server is being consistently overloaded over a long period of time.

That data needs to be read with the number of processor, but if it were eight (as a fairly typical number), in addition to the eight processes being processed, there are on average three queued for every one of those processors.

kuroi:

The server loads that you posted above show that the server is being consistently overloaded over a long period of time.

That data needs to be read with the number of processor, but if it were eight (as a fairly typical number), in addition to the eight processes being processed, there are on average three queued for every one of those processors.
How do you interpret the server load? When I call them and tell them these numbers, they just go, huh? I am not a server guru, how do I explain the their techs what is happening?

JOhn ><>

If the support people at the company managing the server don't understand server load statistics, I wouldn't bother explaining it to them. I'd move to company that knows what it's doing.

Can someone look at the mysql_slow_queries and tell me if something weird is going on?

You've got a number of sites together on that server. Are they all suffering, or is it just this one? (would go to whether it's a server configuration problem, or a problem with this specific site).

We've seen cases where server "upgrades" resulted in default settings being re-introduced, unintentionally throttling performance. Bluehost also intentionally uses CPU throttling to share CPU time between users. Is it possible that you have too much traffic within your Pro Account limit.

What entries are showing the worst times in your slow query log and how frequent are they?

It's every site on the server and even cpanel runs sloooooow, so I doubt it's a domain issue.

I will look at the log files, they are quite complex and show every database request, so it's may be hard to determine. Most of the files are over 150k so I will zip some up and attach or link to them.

Thanks
JOhn ><>

Here's today's logs so far http://www.stuff4toys.com/mysql_slow_queries.rar

thanks for taking the time to look.
JOhn ><>

:clap:

Your most common slow query is a simple query on the banners table. This would normally be very fast, so I wonder whether you've built up a big history of them that's being trawled each page load.

That still shouldn't be a problem, as there's an table index specifically to speed that query. I'd check that you sites have it. It's called "idx_status_group_zen" and combines the status and banner_groups fields.

However, I don't think that this is your problem.

My hypothesis is that BlueHost have gotten around to applying CPU throttling to your account. This would limit your access to the CPU to 12% of that available on the server. Or on an 8 processor server, the equivalent of 1 CPU (that's a bit over simplistic, but is close enough for this discussion).

You have what 13, 14 heavyweight sites operating in a shared server environment? I suspect that you were consuming a disproportionate share of its resources and have been pegged back.

Whereas most of the time in this forum we advise store owners to resist unwarranted pressure to upgrade to a VPS or dedicated server, I think you should be seriously considering this.