mysqli_connect(): (HY000/2002): Can't connect to local MySQL server through socket

myDEBUG logs began last night @ 11:03pm. Below is the contents of the first myDEBUG file. Many, many others. slightly different filesizes.
This is way over my head, so any suggestings greatly appreciated.

I am guessing something changed by host since I was in bed at that time.

First indication from host is "Mysql is up and running fine in your server"

ZC1.5.4
upgraded from zc1.5.3 several weeks ago

[01-Aug-2015 23:03:46 America/Los_Angeles] #1  mysqli_connect(localhost, DBnameREMOVED, DBpasswordREMOVED) called at [/home/chainwea/public_html/includes/classes/db/mysql/query_factory.php:62]
#2  queryFactory->connect(localhost, DBnameREMOVED, DBpasswordREMOVED, DBnameREMOVED, false, ) called at [/home/chainwea/public_html/includes/init_includes/init_database.php:23]
#3  require(/home/chainwea/public_html/includes/init_includes/init_database.php) called at [/home/chainwea/public_html/includes/autoload_func.php:48]
#4  require(/home/chainwea/public_html/includes/autoload_func.php) called at [/home/chainwea/public_html/includes/application_top.php:158]
#5  require(/home/chainwea/public_html/includes/application_top.php) called at [/home/chainwea/public_html/index.php:26]

[01-Aug-2015 23:03:46 America/Los_Angeles] PHP Warning:  mysqli_connect(): (HY000/2002): Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2) in /home/chainwea/public_html/includes/classes/db/mysql/query_factory.php on line 62
[01-Aug-2015 23:03:46 America/Los_Angeles] #1  mysqli_connect(localhost, DBnameREMOVED, DBpasswordREMOVED) called at [/home/chainwea/public_html/includes/classes/db/mysql/query_factory.php:62]
#2  queryFactory->connect(localhost, DBnameREMOVED, DBpasswordREMOVED, DBnameREMOVED, false, ) called at [/home/chainwea/public_html/includes/init_includes/init_database.php:23]
#3  require(/home/chainwea/public_html/includes/init_includes/init_database.php) called at [/home/chainwea/public_html/includes/autoload_func.php:48]
#4  require(/home/chainwea/public_html/includes/autoload_func.php) called at [/home/chainwea/public_html/includes/application_top.php:158]
#5  require(/home/chainwea/public_html/includes/application_top.php) called at [/home/chainwea/public_html/index.php:26]

[01-Aug-2015 23:03:46 America/Los_Angeles] PHP Warning:  mysqli_connect(): (HY000/2002): Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2) in /home/chainwea/public_html/includes/classes/db/mysql/query_factory.php on line 62
[01-Aug-2015 23:03:46 America/Los_Angeles] #1  mysqli_connect(localhost, DBnameREMOVED, DBpasswordREMOVED) called at [/home/chainwea/public_html/includes/classes/db/mysql/query_factory.php:62]
#2  queryFactory->connect(localhost, DBnameREMOVED, DBpasswordREMOVED, DBnameREMOVED, false, ) called at [/home/chainwea/public_html/includes/init_includes/init_database.php:23]
#3  require(/home/chainwea/public_html/includes/init_includes/init_database.php) called at [/home/chainwea/public_html/includes/autoload_func.php:48]
#4  require(/home/chainwea/public_html/includes/autoload_func.php) called at [/home/chainwea/public_html/includes/application_top.php:158]
#5  require(/home/chainwea/public_html/includes/application_top.php) called at [/home/chainwea/public_html/index.php:26]

[01-Aug-2015 23:03:46 America/Los_Angeles] PHP Warning:  mysqli_connect(): (HY000/2002): Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2) in /home/chainwea/public_html/includes/classes/db/mysql/query_factory.php on line 62
[01-Aug-2015 23:03:46 America/Los_Angeles] #1  mysqli_connect(localhost, DBnameREMOVED, DBpasswordREMOVED) called at [/home/chainwea/public_html/includes/classes/db/mysql/query_factory.php:62]
#2  queryFactory->connect(localhost, DBnameREMOVED, DBpasswordREMOVED, DBnameREMOVED, false, ) called at [/home/chainwea/public_html/includes/init_includes/init_database.php:23]
#3  require(/home/chainwea/public_html/includes/init_includes/init_database.php) called at [/home/chainwea/public_html/includes/autoload_func.php:48]
#4  require(/home/chainwea/public_html/includes/autoload_func.php) called at [/home/chainwea/public_html/includes/application_top.php:158]
#5  require(/home/chainwea/public_html/includes/application_top.php) called at [/home/chainwea/public_html/index.php:26]

[01-Aug-2015 23:03:46 America/Los_Angeles] PHP Warning:  mysqli_connect(): (HY000/2002): Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2) in /home/chainwea/public_html/includes/classes/db/mysql/query_factory.php on line 62
[01-Aug-2015 23:03:46 America/Los_Angeles] #1  mysqli_connect(localhost, DBnameREMOVED, DBpasswordREMOVED) called at [/home/chainwea/public_html/includes/classes/db/mysql/query_factory.php:62]
#2  queryFactory->connect(localhost, DBnameREMOVED, DBpasswordREMOVED, DBnameREMOVED, false, ) called at [/home/chainwea/public_html/includes/init_includes/init_database.php:23]
#3  require(/home/chainwea/public_html/includes/init_includes/init_database.php) called at [/home/chainwea/public_html/includes/autoload_func.php:48]
#4  require(/home/chainwea/public_html/includes/autoload_func.php) called at [/home/chainwea/public_html/includes/application_top.php:158]
#5  require(/home/chainwea/public_html/includes/application_top.php) called at [/home/chainwea/public_html/index.php:26]

[01-Aug-2015 23:03:46 America/Los_Angeles] PHP Warning:  mysqli_connect(): (HY000/2002): Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2) in /home/chainwea/public_html/includes/classes/db/mysql/query_factory.php on line 62
[01-Aug-2015 23:03:46 America/Los_Angeles] #1  mysqli_connect(localhost, DBnameREMOVED, DBpasswordREMOVED) called at [/home/chainwea/public_html/includes/classes/db/mysql/query_factory.php:62]
#2  queryFactory->connect(localhost, DBnameREMOVED, DBpasswordREMOVED, DBnameREMOVED, false, ) called at [/home/chainwea/public_html/includes/init_includes/init_database.php:23]
#3  require(/home/chainwea/public_html/includes/init_includes/init_database.php) called at [/home/chainwea/public_html/includes/autoload_func.php:48]
#4  require(/home/chainwea/public_html/includes/autoload_func.php) called at [/home/chainwea/public_html/includes/application_top.php:158]
#5  require(/home/chainwea/public_html/includes/application_top.php) called at [/home/chainwea/public_html/index.php:26]

[01-Aug-2015 23:03:46 America/Los_Angeles] PHP Warning:  mysqli_connect(): (HY000/2002): Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2) in /home/chainwea/public_html/includes/classes/db/mysql/query_factory.php on line 62
[01-Aug-2015 23:03:46 America/Los_Angeles] #1  mysqli_connect(localhost, DBnameREMOVED, DBpasswordREMOVED) called at [/home/chainwea/public_html/includes/classes/db/mysql/query_factory.php:62]
#2  queryFactory->connect(localhost, DBnameREMOVED, DBpasswordREMOVED, DBnameREMOVED, false, ) called at [/home/chainwea/public_html/includes/init_includes/init_database.php:23]
#3  require(/home/chainwea/public_html/includes/init_includes/init_database.php) called at [/home/chainwea/public_html/includes/autoload_func.php:48]
#4  require(/home/chainwea/public_html/includes/autoload_func.php) called at [/home/chainwea/public_html/includes/application_top.php:158]
#5  require(/home/chainwea/public_html/includes/application_top.php) called at [/home/chainwea/public_html/index.php:26]

[01-Aug-2015 23:03:46 America/Los_Angeles] PHP Warning:  mysqli_connect(): (HY000/2002): Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2) in /home/chainwea/public_html/includes/classes/db/mysql/query_factory.php on line 62
[01-Aug-2015 23:03:46 America/Los_Angeles] #1  mysqli_connect(localhost, DBnameREMOVED, DBpasswordREMOVED) called at [/home/chainwea/public_html/includes/classes/db/mysql/query_factory.php:62]
#2  queryFactory->connect(localhost, DBnameREMOVED, DBpasswordREMOVED, DBnameREMOVED, false, ) called at [/home/chainwea/public_html/includes/init_includes/init_database.php:23]
#3  require(/home/chainwea/public_html/includes/init_includes/init_database.php) called at [/home/chainwea/public_html/includes/autoload_func.php:48]
#4  require(/home/chainwea/public_html/includes/autoload_func.php) called at [/home/chainwea/public_html/includes/application_top.php:158]
#5  require(/home/chainwea/public_html/includes/application_top.php) called at [/home/chainwea/public_html/index.php:26]

[01-Aug-2015 23:03:46 America/Los_Angeles] PHP Warning:  mysqli_connect(): (HY000/2002): Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2) in /home/chainwea/public_html/includes/classes/db/mysql/query_factory.php on line 62
[01-Aug-2015 23:03:46 America/Los_Angeles] #1  mysqli_connect(localhost, DBnameREMOVED, DBpasswordREMOVED) called at [/home/chainwea/public_html/includes/classes/db/mysql/query_factory.php:62]
#2  queryFactory->connect(localhost, DBnameREMOVED, DBpasswordREMOVED, DBnameREMOVED, false, ) called at [/home/chainwea/public_html/includes/init_includes/init_database.php:23]
#3  require(/home/chainwea/public_html/includes/init_includes/init_database.php) called at [/home/chainwea/public_html/includes/autoload_func.php:48]
#4  require(/home/chainwea/public_html/includes/autoload_func.php) called at [/home/chainwea/public_html/includes/application_top.php:158]
#5  require(/home/chainwea/public_html/includes/application_top.php) called at [/home/chainwea/public_html/index.php:26]

[01-Aug-2015 23:03:46 America/Los_Angeles] PHP Warning:  mysqli_connect(): (HY000/2002): Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2) in /home/chainwea/public_html/includes/classes/db/mysql/query_factory.php on line 62
[01-Aug-2015 23:03:46 America/Los_Angeles] #1  mysqli_connect(localhost, DBnameREMOVED, DBpasswordREMOVED) called at [/home/chainwea/public_html/includes/classes/db/mysql/query_factory.php:62]
#2  queryFactory->connect(localhost, DBnameREMOVED, DBpasswordREMOVED, DBnameREMOVED, false, ) called at [/home/chainwea/public_html/includes/init_includes/init_database.php:23]
#3  require(/home/chainwea/public_html/includes/init_includes/init_database.php) called at [/home/chainwea/public_html/includes/autoload_func.php:48]
#4  require(/home/chainwea/public_html/includes/autoload_func.php) called at [/home/chainwea/public_html/includes/application_top.php:158]
#5  require(/home/chainwea/public_html/includes/application_top.php) called at [/home/chainwea/public_html/index.php:26]

[01-Aug-2015 23:03:46 America/Los_Angeles] PHP Warning:  mysqli_connect(): (HY000/2002): Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2) in /home/chainwea/public_html/includes/classes/db/mysql/query_factory.php on line 62

Not sure what caused the problem but it magically fixed itself instantly when host 'repaired' the database.
The issue began surprisingly at the exact time the host restarted the mySQL server last nite for an undisclosed reason.

Only 1000+ myDEBUG files later..... and a plethora of grumpy customer emails this morning.

Darn, it sure would have been "interesting" to see the before and after databases! Do you have any idea what "repair" was performed?

The repair was a one-click "Repair" action from within cpanel.

• I am waiting to see if the host has a DB backup from between 11:03pm PDT last night and the one-click repair.
• I have a backup made shortly after the repair.
• I also have a backup made yesterday morning.

Coincidently, the mySQL server was restarted last night at 11:03pm, the exact time that the myDEBUG logs began. The log above was the first one. The files afterwards are slightly different, smaller in size but I didn't bother looking at the details other than I saw references to whosonline.php in them. I don't mind looking if you're interested.

The only thing presented to website visitors was a white screen containing ....
**WARNING: An Error occurred, please refresh the page and try again.
**

If your host can find such a backup and you're willing to share the before/after database contents, let me know via PM and I'll run a "diff" between the two versions and report back.

I think I have before and after.

Darn Windows10 is not being cooperative. Seems to be hiding files that I know exist and that can be seen by my ftp program. I'll let you know when/if I find them.

RixStix:

Coincidently, the mySQL server was restarted last night at 11:03pm, the exact time that the myDEBUG logs began. The log above was the first one.
The timing is revealing. It could easily be that when the MySQL server was restarted it corrupted some data somehow. Perhaps a consequence of the hosting company doing software upgrades that went badly. If they're willing to admit it I suspect they probably had to "repair" everybody else's database on the same server too.

That's what I suspected. They have been working to upgrade all their VPS for a while and inspite of testing, they are still working minor bugs.

I didn't get to grab a DB backup from the interim timeslot. Not that it would make much difference.

Just happened AGAIN. Repair DB using cPanel command. All returned to normal.
What was different this time. Both Admin and Public down 500 error.

When is NDDBC.html supposed to be triggered?

ZC1.5.4 upgraded from 1.5.3 from 1.3.9h from 1.3.8a
Upgraded by using v1.5.4 changed file list and DBupgrade
PHP 5.5.25
mySQL 5.5.42

The myDEBUG is greek to me, so a layman's interpretation/suggestions would be appreciated.

The last server logfile entry before webserver found to be down was:

157.55.39.16 - - [24/Aug/2015:10:16:36 -0700] "GET /index.php?pID=517 HTTP/1.1" 200 7638 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"

This the following server logfile entry when we found the server down:

MyStaticIPaddress - - [24/Aug/2015:11:34:40 -0700] "GET /MyZenCartAdminFolder/login.php?zenAdminID=5bc2585d7afd249f46543422847c46fd HTTP/1.1" 401 381 "-" "Mozilla/5.0 (Windows NT 10.0; WOW64; rv:40.0) Gecko/20100101 Firefox/40.0"

This is the myDEBUG file generated at the same instant of our attempt to login to admin:
( I think that I have renamed all admin/login credentials )

[24-Aug-2015 11:34:53 America/Los_Angeles] #1  trigger_error(126:Incorrect key file for table './MyDBname/sessions.MYI'; try to repair it :: delete from sessions where expiry < 1440441292 ==> (as called by) /home/MycPanelLogin/public_html/includes/functions/sessions.php on line 86 <==, 256) called at [/home/MycPanelLogin/public_html/includes/classes/db/mysql/query_factory.php:155]
#2  queryFactory->show_error() called at [/home/MycPanelLogin/public_html/includes/classes/db/mysql/query_factory.php:132]
#3  queryFactory->set_error(126, Incorrect key file for table './MyDBname/sessions.MYI'; try to repair it, 1) called at [/home/MycPanelLogin/public_html/includes/classes/db/mysql/query_factory.php:262]
#4  queryFactory->Execute(delete from sessions where expiry < 1440441292) called at [/home/MycPanelLogin/public_html/includes/functions/sessions.php:86]
#5  _sess_gc(900)
#6  session_start() called at [/home/MycPanelLogin/public_html/includes/functions/sessions.php:109]
#7  zen_session_start() called at [/home/MycPanelLogin/public_html/MyAdminFolder/includes/init_includes/init_sessions.php:39]
#8  require(/home/MycPanelLogin/public_html/MyAdminFolder/includes/init_includes/init_sessions.php) called at [/home/MycPanelLogin/public_html/includes/autoload_func.php:48]
#9  require(/home/MycPanelLogin/public_html/includes/autoload_func.php) called at [/home/MycPanelLogin/public_html/MyAdminFolder/includes/application_top.php:156]
#10 require(/home/MycPanelLogin/public_html/MyAdminFolder/includes/application_top.php) called at [/home/MycPanelLogin/public_html/MyAdminFolder/login.php:16]

[24-Aug-2015 11:34:53 America/Los_Angeles] PHP Fatal error:  126:Incorrect key file for table './MyDBname/sessions.MYI'; try to repair it :: delete from sessions where expiry < 1440441292 ==> (as called by) /home/MycPanelLogin/public_html/includes/functions/sessions.php on line 86 <== in /home/MycPanelLogin/public_html/includes/classes/db/mysql/query_factory.php on line 155
[24-Aug-2015 11:34:53 America/Los_Angeles] PHP Fatal error:  126:Incorrect key file for table './MyDBname/sessions.MYI'; try to repair it :: insert IGNORE into sessions
           values ('64977228d464bcbeae19f3a8cffbf858', '1440442193', '') ==> (as called by) /home/MycPanelLogin/public_html/includes/functions/sessions.php on line 65 <== in /home/MycPanelLogin/public_html/includes/classes/db/mysql/query_factory.php on line 155

This is the 2nd myDEBUG file. I was surprised to see the DBname and DBpassword in that file. Both names have been edited. File attached instead of inserting CODE tags due to screen length of file.

Simply put, the server crashed, and in doing so it corrupted one of the database tables (specifically the 'sessions' table).

There is no way to tell what caused the server to crash (from the log files provided).

It could be a hardware problem, it could be a software problem (runaway process consuming all available memory). Only your host will be able to provide a definitive answer to that.

Although not really common, it isn't unusual (or surprising) that it is/was the sessions table that suffered the corruption because of all the database tables this is the one with the most active 'writes', in fact it is being written to pretty much all of the time which makes it prone to corruption when the server fails.

Putting this another way, the database corruption is just a symptom of a more serious server related issue. I'd be taking this up with the host if I were you. Two server failures in as many weeks is not good.

Cheers
RodG

RixStix:

When is NDDBC.html supposed to be triggered?

Either when the site is put into maintenance mode or when the query_factory.php code detects a database query error.

There is a bit of a caveat here though, because if there is a problem with the database itself, the whole site can 'fail' long before any SQL commands are sent to the query_factory. This is when you'll get the server 500 errors, blank screen, or whatever. IOW, the database server itself needs to be functional before it is able to detect query related errors.

Cheers
RodG

TNX RodG,

After I posted, I did see that the DB server was restarted within a minute before the error logs based upon timestamps.

Host says my DB is finicky and zencart should do something to fix database performance issues, such as:

• There are too many joins without indexes.This means that joins are doing full table scans. Adding indexes for the columns being used in the join conditions will greatly speed up table joins
• The rate of reading data from a fixed position is high.This indicates that many queries need to sort results and/or do a full table scan, including join queries that do not use indexes. Add indexes where applicable.
• The rate of reading the next table row is high.This indicates that many queries are doing full table scans. Add indexes where applicable.

Those issues have always been in the database advisor screen, so don't put much merit in the comments.
This has only occurred twice since 2009. Both times in the past 24 days. Possibly coincidence but server VPS software was upgraded/updated in July 2015 and there were several hiccups during that update/upgrade process that didn't go as planned.

But then, the update from zc v1.5.3 to v1.5.4 was on July 11.

So VPS server updates and zc update have been within the last 6 weeks. First DB crash 3 weeks after zc update. 2nd DB crash 3 weeks later. That is a pattern too.

Hard to imagine that the DB server just crashed sometime between 10:15 and 11:30 AM just sitting idle.

RixStix:

Those issues have always been in the database advisor screen, so don't put much merit in the comments.

I don't. I was going to tell you the same thing :-)

RixStix:

Hard to imagine that the DB server just crashed sometime between 10:15 and 11:30 AM just sitting idle.

According to the apache logs that you posted, the server wasn't 'sitting idle' - It was being crawled by the bingbot at the time.

The lack of further apache logging until 11:34 is telling me that this service also crashed. Whether this was an effect of the SQL server eating up all resources, or whether it was apache eating up all the server resources (causing the DB server to crash), or whether it was something else entirely causing both to crash is something that I can't determine based on the information provided.

The fact that both times you've had this problem occured at 'about the same time' could suggest that it is being triggered by a cronjob - Do you have an automatic backup, stats collection, log rotation, etc that are being activated at around that time? If so, it could be one of those that is the culprit. If you have several cronjobs set to run at the same time you may be able to alleviate the problem by staggering them. If staggering them doesn't alleviate the problem, and assuming that it is one of them that is the cause, the next time it happens will coincide with the cronjob causing the problem. (I hope that makes sense).

Cheers
RodG

Just out of curiousity, how much memory does your VPS have?

Anything less than 1Gb is not enough, and could well be the cause (especially if the server is being under heavy load by a bot).
Cheers
RodG

RodG:

The fact that both times you've had this problem occured at 'about the same time' could suggest that it is being triggered by a cronjob - Do you have an automatic backup, stats collection, log rotation, etc that are being activated at around that time? If so, it could be one of those that is the culprit. If you have several cronjobs set to run at the same time you may be able to alleviate the problem by staggering them. If staggering them doesn't alleviate the problem, and assuming that it is one of them that is the cause, the next time it happens will coincide with the cronjob causing the problem. (I hope that makes sense).

No cronjobs. No autobackup, stats, etc. 1st problem was 11PM. Today's problem was 11:30AM local time.

Server virus scan happens around 5am local time and takes 146 seconds.

I wouldn't consider the server to be heavily loaded. < 500 sessions per day except on hacking days.

10:15 was the last hit by the BingBot. Nothing in the apache log after that until 11:34.
Just realized, I did not look at the SSL logfile for details.

zencart update to 1.5.4
3 weeks later: DB crash
3 weeks later: DB crash

VPS software update Mid July and there were unexpected hiccups.

Prior to that, DB server runs for months between restarts and never required to repair tables since 2009

RodG:

Just out of curiousity, how much memory does your VPS have?

Anything less than 1Gb is not enough, and could well be the cause (especially if the server is being under heavy load by a bot).
Cheers
RodG

1.5 Gb

DB server monitor screen never seems to indicate a bottleneck even with bot load.

Last week, when a server in Israel had just over 9000 sessions going, the memory didn't seem to be a problem, though the CPU load was at 100% and page updates were taking about 15 seconds. That is the highest load that I have ever seen.

Check the kernel logs - that should provide a clue or two

/var/log/kern.log (at a guess).

Cheers
RodG

Don't have access to anything like that within cPanel. I stepped though every hidden folder. Could be due to managed VPS to keep people's fingers where they don't belong :-)

RixStix:

Don't have access to anything like that within cPanel. I stepped though every hidden folder. Could be due to managed VPS to keep people's fingers where they don't belong :-)

No SSH access? What's the point of a VPS without root access in one way or another? ;-)

Cheers
RodD

VPS SSH access is available. I've never needed it (that I knew of), so never asked. So I wouldn't know it if it hit me 'tween the eyes or how to access. :huh:

It only made sense to me that if cPanel restarted the database server, there should be some record of "WHY" that happened. Haven't heard anything yet from that comment to host.