mysqli_connect(): (HY000/2002): Can't connect to local MySQL server through socket

[RixStix]

myDEBUG logs began last night @ 11:03pm. Below is the contents of the first myDEBUG file. Many, many others. slightly different filesizes.
This is way over my head, so any suggestings greatly appreciated.

I am guessing something changed by host since I was in bed at that time.

First indication from host is "Mysql is up and running fine in your server"

ZC1.5.4
upgraded from zc1.5.3 several weeks ago

[01-Aug-2015 23:03:46 America/Los_Angeles] #1  mysqli_connect(localhost, DBnameREMOVED, DBpasswordREMOVED) called at [/home/chainwea/public_html/includes/classes/db/mysql/query_factory.php:62]
#2  queryFactory->connect(localhost, DBnameREMOVED, DBpasswordREMOVED, DBnameREMOVED, false, ) called at [/home/chainwea/public_html/includes/init_includes/init_database.php:23]
#3  require(/home/chainwea/public_html/includes/init_includes/init_database.php) called at [/home/chainwea/public_html/includes/autoload_func.php:48]
#4  require(/home/chainwea/public_html/includes/autoload_func.php) called at [/home/chainwea/public_html/includes/application_top.php:158]
#5  require(/home/chainwea/public_html/includes/application_top.php) called at [/home/chainwea/public_html/index.php:26]

[01-Aug-2015 23:03:46 America/Los_Angeles] PHP Warning:  mysqli_connect(): (HY000/2002): Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2) in /home/chainwea/public_html/includes/classes/db/mysql/query_factory.php on line 62
[01-Aug-2015 23:03:46 America/Los_Angeles] #1  mysqli_connect(localhost, DBnameREMOVED, DBpasswordREMOVED) called at [/home/chainwea/public_html/includes/classes/db/mysql/query_factory.php:62]
#2  queryFactory->connect(localhost, DBnameREMOVED, DBpasswordREMOVED, DBnameREMOVED, false, ) called at [/home/chainwea/public_html/includes/init_includes/init_database.php:23]
#3  require(/home/chainwea/public_html/includes/init_includes/init_database.php) called at [/home/chainwea/public_html/includes/autoload_func.php:48]
#4  require(/home/chainwea/public_html/includes/autoload_func.php) called at [/home/chainwea/public_html/includes/application_top.php:158]
#5  require(/home/chainwea/public_html/includes/application_top.php) called at [/home/chainwea/public_html/index.php:26]

[01-Aug-2015 23:03:46 America/Los_Angeles] PHP Warning:  mysqli_connect(): (HY000/2002): Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2) in /home/chainwea/public_html/includes/classes/db/mysql/query_factory.php on line 62
[01-Aug-2015 23:03:46 America/Los_Angeles] #1  mysqli_connect(localhost, DBnameREMOVED, DBpasswordREMOVED) called at [/home/chainwea/public_html/includes/classes/db/mysql/query_factory.php:62]
#2  queryFactory->connect(localhost, DBnameREMOVED, DBpasswordREMOVED, DBnameREMOVED, false, ) called at [/home/chainwea/public_html/includes/init_includes/init_database.php:23]
#3  require(/home/chainwea/public_html/includes/init_includes/init_database.php) called at [/home/chainwea/public_html/includes/autoload_func.php:48]
#4  require(/home/chainwea/public_html/includes/autoload_func.php) called at [/home/chainwea/public_html/includes/application_top.php:158]
#5  require(/home/chainwea/public_html/includes/application_top.php) called at [/home/chainwea/public_html/index.php:26]

[01-Aug-2015 23:03:46 America/Los_Angeles] PHP Warning:  mysqli_connect(): (HY000/2002): Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2) in /home/chainwea/public_html/includes/classes/db/mysql/query_factory.php on line 62
[01-Aug-2015 23:03:46 America/Los_Angeles] #1  mysqli_connect(localhost, DBnameREMOVED, DBpasswordREMOVED) called at [/home/chainwea/public_html/includes/classes/db/mysql/query_factory.php:62]
#2  queryFactory->connect(localhost, DBnameREMOVED, DBpasswordREMOVED, DBnameREMOVED, false, ) called at [/home/chainwea/public_html/includes/init_includes/init_database.php:23]
#3  require(/home/chainwea/public_html/includes/init_includes/init_database.php) called at [/home/chainwea/public_html/includes/autoload_func.php:48]
#4  require(/home/chainwea/public_html/includes/autoload_func.php) called at [/home/chainwea/public_html/includes/application_top.php:158]
#5  require(/home/chainwea/public_html/includes/application_top.php) called at [/home/chainwea/public_html/index.php:26]

[01-Aug-2015 23:03:46 America/Los_Angeles] PHP Warning:  mysqli_connect(): (HY000/2002): Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2) in /home/chainwea/public_html/includes/classes/db/mysql/query_factory.php on line 62
[01-Aug-2015 23:03:46 America/Los_Angeles] #1  mysqli_connect(localhost, DBnameREMOVED, DBpasswordREMOVED) called at [/home/chainwea/public_html/includes/classes/db/mysql/query_factory.php:62]
#2  queryFactory->connect(localhost, DBnameREMOVED, DBpasswordREMOVED, DBnameREMOVED, false, ) called at [/home/chainwea/public_html/includes/init_includes/init_database.php:23]
#3  require(/home/chainwea/public_html/includes/init_includes/init_database.php) called at [/home/chainwea/public_html/includes/autoload_func.php:48]
#4  require(/home/chainwea/public_html/includes/autoload_func.php) called at [/home/chainwea/public_html/includes/application_top.php:158]
#5  require(/home/chainwea/public_html/includes/application_top.php) called at [/home/chainwea/public_html/index.php:26]

[01-Aug-2015 23:03:46 America/Los_Angeles] PHP Warning:  mysqli_connect(): (HY000/2002): Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2) in /home/chainwea/public_html/includes/classes/db/mysql/query_factory.php on line 62
[01-Aug-2015 23:03:46 America/Los_Angeles] #1  mysqli_connect(localhost, DBnameREMOVED, DBpasswordREMOVED) called at [/home/chainwea/public_html/includes/classes/db/mysql/query_factory.php:62]
#2  queryFactory->connect(localhost, DBnameREMOVED, DBpasswordREMOVED, DBnameREMOVED, false, ) called at [/home/chainwea/public_html/includes/init_includes/init_database.php:23]
#3  require(/home/chainwea/public_html/includes/init_includes/init_database.php) called at [/home/chainwea/public_html/includes/autoload_func.php:48]
#4  require(/home/chainwea/public_html/includes/autoload_func.php) called at [/home/chainwea/public_html/includes/application_top.php:158]
#5  require(/home/chainwea/public_html/includes/application_top.php) called at [/home/chainwea/public_html/index.php:26]

[01-Aug-2015 23:03:46 America/Los_Angeles] PHP Warning:  mysqli_connect(): (HY000/2002): Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2) in /home/chainwea/public_html/includes/classes/db/mysql/query_factory.php on line 62
[01-Aug-2015 23:03:46 America/Los_Angeles] #1  mysqli_connect(localhost, DBnameREMOVED, DBpasswordREMOVED) called at [/home/chainwea/public_html/includes/classes/db/mysql/query_factory.php:62]
#2  queryFactory->connect(localhost, DBnameREMOVED, DBpasswordREMOVED, DBnameREMOVED, false, ) called at [/home/chainwea/public_html/includes/init_includes/init_database.php:23]
#3  require(/home/chainwea/public_html/includes/init_includes/init_database.php) called at [/home/chainwea/public_html/includes/autoload_func.php:48]
#4  require(/home/chainwea/public_html/includes/autoload_func.php) called at [/home/chainwea/public_html/includes/application_top.php:158]
#5  require(/home/chainwea/public_html/includes/application_top.php) called at [/home/chainwea/public_html/index.php:26]

[01-Aug-2015 23:03:46 America/Los_Angeles] PHP Warning:  mysqli_connect(): (HY000/2002): Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2) in /home/chainwea/public_html/includes/classes/db/mysql/query_factory.php on line 62
[01-Aug-2015 23:03:46 America/Los_Angeles] #1  mysqli_connect(localhost, DBnameREMOVED, DBpasswordREMOVED) called at [/home/chainwea/public_html/includes/classes/db/mysql/query_factory.php:62]
#2  queryFactory->connect(localhost, DBnameREMOVED, DBpasswordREMOVED, DBnameREMOVED, false, ) called at [/home/chainwea/public_html/includes/init_includes/init_database.php:23]
#3  require(/home/chainwea/public_html/includes/init_includes/init_database.php) called at [/home/chainwea/public_html/includes/autoload_func.php:48]
#4  require(/home/chainwea/public_html/includes/autoload_func.php) called at [/home/chainwea/public_html/includes/application_top.php:158]
#5  require(/home/chainwea/public_html/includes/application_top.php) called at [/home/chainwea/public_html/index.php:26]

[01-Aug-2015 23:03:46 America/Los_Angeles] PHP Warning:  mysqli_connect(): (HY000/2002): Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2) in /home/chainwea/public_html/includes/classes/db/mysql/query_factory.php on line 62
[01-Aug-2015 23:03:46 America/Los_Angeles] #1  mysqli_connect(localhost, DBnameREMOVED, DBpasswordREMOVED) called at [/home/chainwea/public_html/includes/classes/db/mysql/query_factory.php:62]
#2  queryFactory->connect(localhost, DBnameREMOVED, DBpasswordREMOVED, DBnameREMOVED, false, ) called at [/home/chainwea/public_html/includes/init_includes/init_database.php:23]
#3  require(/home/chainwea/public_html/includes/init_includes/init_database.php) called at [/home/chainwea/public_html/includes/autoload_func.php:48]
#4  require(/home/chainwea/public_html/includes/autoload_func.php) called at [/home/chainwea/public_html/includes/application_top.php:158]
#5  require(/home/chainwea/public_html/includes/application_top.php) called at [/home/chainwea/public_html/index.php:26]

[01-Aug-2015 23:03:46 America/Los_Angeles] PHP Warning:  mysqli_connect(): (HY000/2002): Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2) in /home/chainwea/public_html/includes/classes/db/mysql/query_factory.php on line 62
[01-Aug-2015 23:03:46 America/Los_Angeles] #1  mysqli_connect(localhost, DBnameREMOVED, DBpasswordREMOVED) called at [/home/chainwea/public_html/includes/classes/db/mysql/query_factory.php:62]
#2  queryFactory->connect(localhost, DBnameREMOVED, DBpasswordREMOVED, DBnameREMOVED, false, ) called at [/home/chainwea/public_html/includes/init_includes/init_database.php:23]
#3  require(/home/chainwea/public_html/includes/init_includes/init_database.php) called at [/home/chainwea/public_html/includes/autoload_func.php:48]
#4  require(/home/chainwea/public_html/includes/autoload_func.php) called at [/home/chainwea/public_html/includes/application_top.php:158]
#5  require(/home/chainwea/public_html/includes/application_top.php) called at [/home/chainwea/public_html/index.php:26]

[01-Aug-2015 23:03:46 America/Los_Angeles] PHP Warning:  mysqli_connect(): (HY000/2002): Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2) in /home/chainwea/public_html/includes/classes/db/mysql/query_factory.php on line 62

[RixStix]

Not sure what caused the problem but it magically fixed itself instantly when host 'repaired' the database.
The issue began surprisingly at the exact time the host restarted the mySQL server last nite for an undisclosed reason.

Only 1000+ myDEBUG files later..... and a plethora of grumpy customer emails this morning.

[lat9]

Darn, it sure would have been "interesting" to see the before and after databases! Do you have any idea what "repair" was performed?

[RixStix]

The repair was a one-click "Repair" action from within cpanel.

• I am waiting to see if the host has a DB backup from between 11:03pm PDT last night and the one-click repair.
• I have a backup made shortly after the repair.
• I also have a backup made yesterday morning.

Coincidently, the mySQL server was restarted last night at 11:03pm, the exact time that the myDEBUG logs began. The log above was the first one. The files afterwards are slightly different, smaller in size but I didn't bother looking at the details other than I saw references to whosonline.php in them. I don't mind looking if you're interested.

The only thing presented to website visitors was a white screen containing ....
WARNING: An Error occurred, please refresh the page and try again.

[lat9]

If your host can find such a backup and you're willing to share the before/after database contents, let me know via PM and I'll run a "diff" between the two versions and report back.

[RixStix]

I think I have before and after.

Darn Windows10 is not being cooperative. Seems to be hiding files that I know exist and that can be seen by my ftp program. I'll let you know when/if I find them.

[DrByte]

Coincidently, the mySQL server was restarted last night at 11:03pm, the exact time that the myDEBUG logs began. The log above was the first one.

The timing is revealing. It could easily be that when the MySQL server was restarted it corrupted some data somehow. Perhaps a consequence of the hosting company doing software upgrades that went badly. If they're willing to admit it I suspect they probably had to "repair" everybody else's database on the same server too.

[RixStix]

That's what I suspected. They have been working to upgrade all their VPS for a while and inspite of testing, they are still working minor bugs.

I didn't get to grab a DB backup from the interim timeslot. Not that it would make much difference.

[RixStix]

Just happened AGAIN. Repair DB using cPanel command. All returned to normal.
What was different this time. Both Admin and Public down 500 error.

When is NDDBC.html supposed to be triggered?

ZC1.5.4 upgraded from 1.5.3 from 1.3.9h from 1.3.8a
Upgraded by using v1.5.4 changed file list and DBupgrade
PHP 5.5.25
mySQL 5.5.42


The myDEBUG is greek to me, so a layman's interpretation/suggestions would be appreciated.

The last server logfile entry before webserver found to be down was:

157.55.39.16 - - [24/Aug/2015:10:16:36 -0700] "GET /index.php?pID=517 HTTP/1.1" 200 7638 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"

This the following server logfile entry when we found the server down:

MyStaticIPaddress - - [24/Aug/2015:11:34:40 -0700] "GET /MyZenCartAdminFolder/login.php?zenAdminID=5bc2585d7afd249f46543422847c46fd HTTP/1.1" 401 381 "-" "Mozilla/5.0 (Windows NT 10.0; WOW64; rv:40.0) Gecko/20100101 Firefox/40.0"

This is the myDEBUG file generated at the same instant of our attempt to login to admin:
( I think that I have renamed all admin/login credentials )

[24-Aug-2015 11:34:53 America/Los_Angeles] #1  trigger_error(126:Incorrect key file for table './MyDBname/sessions.MYI'; try to repair it :: delete from sessions where expiry < 1440441292 ==> (as called by) /home/MycPanelLogin/public_html/includes/functions/sessions.php on line 86 <==, 256) called at [/home/MycPanelLogin/public_html/includes/classes/db/mysql/query_factory.php:155]
#2  queryFactory->show_error() called at [/home/MycPanelLogin/public_html/includes/classes/db/mysql/query_factory.php:132]
#3  queryFactory->set_error(126, Incorrect key file for table './MyDBname/sessions.MYI'; try to repair it, 1) called at [/home/MycPanelLogin/public_html/includes/classes/db/mysql/query_factory.php:262]
#4  queryFactory->Execute(delete from sessions where expiry < 1440441292) called at [/home/MycPanelLogin/public_html/includes/functions/sessions.php:86]
#5  _sess_gc(900)
#6  session_start() called at [/home/MycPanelLogin/public_html/includes/functions/sessions.php:109]
#7  zen_session_start() called at [/home/MycPanelLogin/public_html/MyAdminFolder/includes/init_includes/init_sessions.php:39]
#8  require(/home/MycPanelLogin/public_html/MyAdminFolder/includes/init_includes/init_sessions.php) called at [/home/MycPanelLogin/public_html/includes/autoload_func.php:48]
#9  require(/home/MycPanelLogin/public_html/includes/autoload_func.php) called at [/home/MycPanelLogin/public_html/MyAdminFolder/includes/application_top.php:156]
#10 require(/home/MycPanelLogin/public_html/MyAdminFolder/includes/application_top.php) called at [/home/MycPanelLogin/public_html/MyAdminFolder/login.php:16]

[24-Aug-2015 11:34:53 America/Los_Angeles] PHP Fatal error:  126:Incorrect key file for table './MyDBname/sessions.MYI'; try to repair it :: delete from sessions where expiry < 1440441292 ==> (as called by) /home/MycPanelLogin/public_html/includes/functions/sessions.php on line 86 <== in /home/MycPanelLogin/public_html/includes/classes/db/mysql/query_factory.php on line 155
[24-Aug-2015 11:34:53 America/Los_Angeles] PHP Fatal error:  126:Incorrect key file for table './MyDBname/sessions.MYI'; try to repair it :: insert IGNORE into sessions
           values ('64977228d464bcbeae19f3a8cffbf858', '1440442193', '') ==> (as called by) /home/MycPanelLogin/public_html/includes/functions/sessions.php on line 65 <== in /home/MycPanelLogin/public_html/includes/classes/db/mysql/query_factory.php on line 155

This is the 2nd myDEBUG file. I was surprised to see the DBname and DBpassword in that file. Both names have been edited. File attached instead of inserting CODE tags due to screen length of file.

[RodG]

Simply put, the server crashed, and in doing so it corrupted one of the database tables (specifically the 'sessions' table).

There is no way to tell what caused the server to crash (from the log files provided).

It could be a hardware problem, it could be a software problem (runaway process consuming all available memory). Only your host will be able to provide a definitive answer to that.

Although not really common, it isn't unusual (or surprising) that it is/was the sessions table that suffered the corruption because of all the database tables this is the one with the most active 'writes', in fact it is being written to pretty much all of the time which makes it prone to corruption when the server fails.

Putting this another way, the database corruption is just a symptom of a more serious server related issue. I'd be taking this up with the host if I were you. Two server failures in as many weeks is not good.

Cheers
RodG