Forums / General Questions / MySQL bottlneck - getting mysql errors

MySQL bottlneck - getting mysql errors

Locked

Views: 4,594

Results 1 to 20 of 26
This thread is locked. New replies are disabled.
8 Apr 2008, 7:18 PM
#1
ksoup avatar

ksoup

Zen Follower

Join Date:
Jan 2004
Posts:
413
Plugin Contributions:
1

MySQL bottlneck - getting mysql errors

1.3.7 (latest) not 1.3.8

Our database appears to be 500 MiB... and we are having mysql errors by the boatload.

Is there someone who can help ... the store will continue to grow, so we need to learn what to do now with such a busy database.

Thank you in advance.

9 Apr 2008, 12:54 AM
#2
ksoup avatar

ksoup

Zen Follower

Join Date:
Jan 2004
Posts:
413
Plugin Contributions:
1

Re: MySQL bottlneck - getting mysql errors

Our loads are pretty high. How can we pinpoint what is using the resources?

ksoup:

1.3.7 (latest) not 1.3.8

Our database appears to be 500 MiB... and we are having mysql errors by the boatload.

Is there someone who can help ... the store will continue to grow, so we need to learn what to do now with such a busy database.

Thank you in advance.

9 Apr 2008, 1:14 AM
#3
Kim avatar

Kim

Obaa-san

Join Date:
Jun 2003
Location:
West Coast, North America
Posts:
26,684
Plugin Contributions:
0

Re: MySQL bottlneck - getting mysql errors

Need to know more - exactly what are the errors? What version of MYSQL? Etc.

Please do not PM for support issues: a private solution doesn't benefit the community.

Be careful with unsolicited advice via email or PM - Make sure the person you are talking to is a reliable source.

9 Apr 2008, 2:22 AM
#4
ksoup avatar

ksoup

Zen Follower

Join Date:
Jan 2004
Posts:
413
Plugin Contributions:
1

Re: MySQL bottlneck - getting mysql errors

Thank you, Kim....

The webhost tells us our server loads are high by a huge margin!
Database: MySQL 5.0.45
load average: 12.67, 11.41, 10.83
HTTP Server: Apache/1.3.39 (Unix) PHP/5.2.5

Here are some of the errors received (but mainly sluggggiiiisssshhhh store!!)

~ MySql Server has gone away

~ 1064 error

~ Common Customer Complaint
I am doing my VERY best to buy a ton of stuff from your
store while it is on sale....however....it is taking FOREVER to load.

~ 2006 MySQL server has gone away
in: [select code, title, symbol_left, symbol_right, decimal_point,
thousands_point, decimal_places, value from zen_currencies]

~ 1062 Duplicate entry 'zc_6ec65decb95a7741ada473af0fde76e1' for key 1
in: [db_cache table]

After clicking add item to cart

~ 2008 MySQL client ran out of memory

We have repaired our tables... the webhost has done a deeper level database repair and they are intact.

Any help you can give will be most appreciated! :)

Kim:

Need to know more - exactly what are the errors? What version of MYSQL? Etc.

9 Apr 2008, 4:11 AM
#5
drbyte avatar

drbyte

Sensei

Join Date:
Jan 2004
Posts:
63,515
Plugin Contributions:
176

Re: MySQL bottlneck - getting mysql errors

Do all your tables have the correct indexes on them? (To find out, you'll need to export your table structures and compare them against the master mysql_zencart.sql file in the zc_install/sql folder)

.
Zen Cart - putting the dream of business ownership within reach of anyone!
Donate to: DrByte directly or to the Zen Cart team as a whole

Remember: Any code suggestions you see here are merely suggestions. You assume full responsibility for your use of any such suggestions, including any impact ANY alterations you make to your site may have on your PCI compliance.
Furthermore, any advice you see here about PCI matters is merely an opinion, and should not be relied upon as "official". Official PCI information should be obtained from the PCI Security Council directly or from one of their authorized Assessors.

9 Apr 2008, 1:50 PM
#6
ksoup avatar

ksoup

Zen Follower

Join Date:
Jan 2004
Posts:
413
Plugin Contributions:
1

Re: MySQL bottlneck - getting mysql errors

I have started comparing... and I see the following commonly... the _zen is missing from the keynames listed in phpmyadmin.... Is that a problem?

Master SQL
KEY idx_page_accessed_zen (page_accessed),
KEY idx_access_date_zen (access_date),
KEY idx_ip_zen (ip_address)

Index shown in PHPmyadmin
keyname type cardinality field
PRIMARY PRIMARY log_id
page_accessed INDEX page_accessed
access_date INDEX access_date
idx_ip_zen INDEX ip_address

DrByte:

Do all your tables have the correct indexes on them? (To find out, you'll need to export your table structures and compare them against the master mysql_zencart.sql file in the zc_install/sql folder)

9 Apr 2008, 2:26 PM
#7
drbyte avatar

drbyte

Sensei

Join Date:
Jan 2004
Posts:
63,515
Plugin Contributions:
176

Re: MySQL bottlneck - getting mysql errors

The "name" of the indexes is not important.
The existence of the indexes, along with having the correct fields defined to them, is the important part.

.
Zen Cart - putting the dream of business ownership within reach of anyone!
Donate to: DrByte directly or to the Zen Cart team as a whole

Remember: Any code suggestions you see here are merely suggestions. You assume full responsibility for your use of any such suggestions, including any impact ANY alterations you make to your site may have on your PCI compliance.
Furthermore, any advice you see here about PCI matters is merely an opinion, and should not be relied upon as "official". Official PCI information should be obtained from the PCI Security Council directly or from one of their authorized Assessors.

9 Apr 2008, 4:18 PM
#8
ksoup avatar

ksoup

Zen Follower

Join Date:
Jan 2004
Posts:
413
Plugin Contributions:
1

Re: MySQL bottlneck - getting mysql errors

I am still checking the database...

The host wants to upgrade apache... Can you tell us if that is a good idea? We experienced issues with it the last time they tried due to incompatibilities within PHP and MySQL

Currently we are at ...
Kernel version 2.6.9-55.0.9.ELsmp
Apache version 1.3.39 (Unix)
PHP version 5.2.5
MySQL version 5.0.45-community

DrByte:

The "name" of the indexes is not important.
The existence of the indexes, along with having the correct fields defined to them, is the important part.

9 Apr 2008, 4:35 PM
#9
drbyte avatar

drbyte

Sensei

Join Date:
Jan 2004
Posts:
63,515
Plugin Contributions:
176

Re: MySQL bottlneck - getting mysql errors

I don't see that upgrading apache would "help" your specific situation.

However, Zen Cart itself doesn't care whether you're using Apache 2.x or 1.x.

.
Zen Cart - putting the dream of business ownership within reach of anyone!
Donate to: DrByte directly or to the Zen Cart team as a whole

Remember: Any code suggestions you see here are merely suggestions. You assume full responsibility for your use of any such suggestions, including any impact ANY alterations you make to your site may have on your PCI compliance.
Furthermore, any advice you see here about PCI matters is merely an opinion, and should not be relied upon as "official". Official PCI information should be obtained from the PCI Security Council directly or from one of their authorized Assessors.

9 Apr 2008, 4:53 PM
#10
website_rob avatar

website_rob

Inactive

Join Date:
Oct 2006
Location:
Alberta, Canada
Posts:
4,572
Plugin Contributions:
0

Re: MySQL bottlneck - getting mysql errors

ksoup:

Our database appears to be 500 MiB... and we are having mysql errors by the boatload.

The webhost tells us our server loads are high by a huge margin!
Database: MySQL 5.0.45
load average: 12.67, 11.41, 10.83
HTTP Server: Apache/1.3.39 (Unix) PHP/5.2.5

When you say "our Server loads" does this mean you have your own Server?

Yes - then more / better optimization needs to be done at the Server level

No - then probably time to get your own Server

A 500 MB database is HUGE and, if the above load stats are from a busy Shared Server and are continually at that level, your site will never get past the problems it has now.

The learning is in the doing.

Potent Products

9 Apr 2008, 5:55 PM
#11
ksoup avatar

ksoup

Zen Follower

Join Date:
Jan 2004
Posts:
413
Plugin Contributions:
1

Re: MySQL bottlneck - getting mysql errors

i certainly appreciate your responses and help!!

i understand we are on a dedicated server (does that share resources with the masses?). I do know, however, we share the resources with our forum and gallery.

Website Rob:

When you say "our Server loads" does this mean you have your own Server?

Yes - then more / better optimization needs to be done at the Server level

No - then probably time to get your own Server

A 500 MB database is HUGE and, if the above load stats are from a busy Shared Server and are continually at that level, your site will never get past the problems it has now.

9 Apr 2008, 6:08 PM
#12
ksoup avatar

ksoup

Zen Follower

Join Date:
Jan 2004
Posts:
413
Plugin Contributions:
1

Re: MySQL bottlneck - getting mysql errors

crud... forgot to ask this, too...

Do you have suggestions for better optimization for ZenCart.

And, are there any versions of mySQL and/or PHP that they recommend NOT using?

9 Apr 2008, 6:48 PM
#13
kobra avatar

kobra

Black Belt

Join Date:
Aug 2005
Location:
Arizona
Posts:
31,500
Plugin Contributions:
4

Re: MySQL bottlneck - getting mysql errors

crud... forgot to ask this, too...

Do you have suggestions for better optimization for ZenCart.

And, are there any versions of mySQL and/or PHP that they recommend NOT using?
Not better optimization of Zen Cart - more to optimizing your server LAMP settings and these are not something many will share as hours upon hours are spent in finetuning these settings.

Question: DB is 500MB?? How many products do you have?

Zen-Venom Get Bitten

9 Apr 2008, 8:16 PM
#14
ksoup avatar

ksoup

Zen Follower

Join Date:
Jan 2004
Posts:
413
Plugin Contributions:
1

Re: MySQL bottlneck - getting mysql errors

about 7400... mainly downloadables (which means at least one attribute on each one)

has anyone seen a zen cart database larger than this? how far can it be pushed!

kobra:

Not better optimization of Zen Cart - more to optimizing your server LAMP settings and these are not something many will share as hours upon hours are spent in finetuning these settings.

Question: DB is 500MB?? How many products do you have?

9 Apr 2008, 8:44 PM
#15
drbyte avatar

drbyte

Sensei

Join Date:
Jan 2004
Posts:
63,515
Plugin Contributions:
176

Re: MySQL bottlneck - getting mysql errors

Have you enabled Email Archiving? If so, why? You should probably trim it down if you plan to keep it enabled.
Do you clear your Admin Activity Log regularly?
Naturally, make backups before you go deleting any information.

I've seen several databases larger than 500 MB, and they operate just fine.

Your performance issues are more likely related to your server's general configuration than to the size of the database. Having such high sustained load-averages is going to slow everything down.

Tuning your MySQL settings may help, but your load averages should be dealt with first.

.
Zen Cart - putting the dream of business ownership within reach of anyone!
Donate to: DrByte directly or to the Zen Cart team as a whole

Remember: Any code suggestions you see here are merely suggestions. You assume full responsibility for your use of any such suggestions, including any impact ANY alterations you make to your site may have on your PCI compliance.
Furthermore, any advice you see here about PCI matters is merely an opinion, and should not be relied upon as "official". Official PCI information should be obtained from the PCI Security Council directly or from one of their authorized Assessors.

9 Apr 2008, 8:47 PM
#16
ksoup avatar

ksoup

Zen Follower

Join Date:
Jan 2004
Posts:
413
Plugin Contributions:
1

Re: MySQL bottlneck - getting mysql errors

The email archive table is empty .. and we clean admin when the site flags us at the top in the admin area... so both good things.

DrByte:

Have you enabled Email Archiving? If so, why? You should probably trim it down if you plan to keep it enabled.
Do you clear your Admin Activity Log regularly?
Naturally, make backups before you go deleting any information.

I've seen several databases larger than 500 MB, and they operate just fine.

Your performance issues are more likely related to your server's general configuration than to the size of the database. Having such high sustained load-averages is going to slow everything down.

Tuning your MySQL settings may help, but your load averages should be dealt with first.

11 Apr 2008, 4:25 PM
#17
ksoup avatar

ksoup

Zen Follower

Join Date:
Jan 2004
Posts:
413
Plugin Contributions:
1

Re: MySQL bottlneck - getting mysql errors

Here is a common error we are getting... along with others... but i have seen this one several times or more

2006 MySQL server has gone away in:
[select code, title, symbol_left, symbol_right, decimal_point, thousands_point, decimal_places, value from zen_currencies]

Thoughts?

We are looking at a server upgrade and will do that soon. But, why do we continue to get errors??

11 Apr 2008, 5:44 PM
#18
drbyte avatar

drbyte

Sensei

Join Date:
Jan 2004
Posts:
63,515
Plugin Contributions:
176

Re: MySQL bottlneck - getting mysql errors

ksoup:

2006 MySQL server has gone awayThis is a result of several things:

  1. A database connection was opened, but not closed, because there is intention to use it again.
  2. Then the program goes off and does some processing or talks to other servers, etc.
  3. Then it comes back and attempts to access the database again, but the processing in the previous step has taken longer than the timeout allowed by the server's configuration settings. Thus, since the server has closed the database connection, the program is not able to access it (since it had a reasonable expectation to be able to reach it again).

Some timeout problems can occur when attempting to open the Checkout-Shipping page and perhaps the real-time quote query from USPS takes to long (such as their servers are taking a long time to respond, are down, etc), and when the program attempts to process the rest of the page logic, the database is no longer disconnected.

But generally speaking, unless there is some sort of attempt to talk to an external resource or you have some sort of intense processing with custom code, there should be no reason for any database timeouts ... unless the problem is related to your server/database-server configuration.

.
Zen Cart - putting the dream of business ownership within reach of anyone!
Donate to: DrByte directly or to the Zen Cart team as a whole

Remember: Any code suggestions you see here are merely suggestions. You assume full responsibility for your use of any such suggestions, including any impact ANY alterations you make to your site may have on your PCI compliance.
Furthermore, any advice you see here about PCI matters is merely an opinion, and should not be relied upon as "official". Official PCI information should be obtained from the PCI Security Council directly or from one of their authorized Assessors.

11 Apr 2008, 11:50 PM
#19
website_rob avatar

website_rob

Inactive

Join Date:
Oct 2006
Location:
Alberta, Canada
Posts:
4,572
Plugin Contributions:
0

Re: MySQL bottlneck - getting mysql errors

Depending upon your Server Hardware specs, optimizing your current setup may get you better results.

For example, in the Server 'php.ini' file the following should be used:

[MySQL]
; Allow or prevent persistent links.
mysql.allow_persistent = OFF

That will help to prevent the situation as described by DrByte; which is also known as an "Orphan process".

The learning is in the doing.

Potent Products

1 May 2008, 3:10 PM
#20
joelmmcc avatar

joelmmcc

New Zenner

Join Date:
Jan 2008
Posts:
7
Plugin Contributions:
0

Re: MySQL bottlneck - getting mysql errors

Don’t overlook the possibility of just plain bad SQL coding, especially if you’ve customized anything. Zen Cart’s SQL through 1.3.x is pretty darned inefficient with its SQL anyway (they’re fixing much of that in 1.4.x, currently in beta — around 700 queries removed in the generation of the Home Page alone!). When customizing ad-hoc SQL queries pieced together from variables and named constants and string constants and expressions and the like, a typo can easily be overlooked.

We just had this problem with constantly getting “2008 MySQL client ran out of memory” errors on the products listings pages (“?main_page=index…”). After reading this thread and worrying about indices and MySQL and Apache config. parameters and our shared hosting service not being able to handle the load, etc., I finally simply pasted the query as shown on the error page into a phpMySQLadmin SQL window and executed it.

It returned well over seven million records, from one of the smaller categories! This, mind you, for a store with only about 1,600 items, only about 20 of which were in that category!

In analyzing the, I found some embarrassing typos on my part: in my customized “{store root}/includes/index_filters/default.filter.php” file, I had used the same alias on both sides of the “=” in my custom “LEFT JOIN … ON … =”s! I had, for instance (bad table alias capitalized here to call attention to it, but not of course capitalized in the original PHP code):

$listing_sql = "select … left join " . TABLE_PRODUCTS_MYTYPE_EXTRA . " x on p.products_id = x.products_id
    left join " . TABLE_MY_FILTER . " f on x.myfilter_id = X.myfilter_id,"
```when it *should*’ve been```php
$listing_sql = "select … left join " . TABLE_PRODUCTS_MYTYPE_EXTRA . " x on p.products_id = x.products_id
    left join " . TABLE_MY_FILTER . " f on x.myfilter_id = F.myfilter_id,"
``` in all four times that that query appears (in various variants) in default_filter.php! Since I had more than one such custom filter, and all had that same typo (gah!), well, you can imagine why it returned so many records!

Fixing that brought the results down to a far more manageable mere several thousand, which no longer crashed, but which still returned *way* too many records and was showing duplicate products in the listing — better than an error message, perhaps, but still not good.

This time, the fault seemed to be in part of the original SQL query of the default_filter.php file! Maybe my custom product types and filters showed it up, but it seems to have been bad code all along, even in the default!

It has to do with the Manufacturers portion of the query (so if your store doesn’t use Manufacturers, this probably wouldn’t affect you, but you should still consider fixing it in case you do start using Manufacturers in the future).

Again, look for where the variable “$listing_sql” is assigned (it’s assigned four times, one of which will be executed based on various conditionals). In all four places, the table whose name is stored in the constant “TABLE_MANUFACTURERS” with alias “m” is listed as a cross join rather than a left or inner join! In all but the last one, this is mitigated by adding this conditional in the WHERE clause: “…and p.manufacturers_id = m.manufacturers.id” and so shouldn’t actually cause that problem unless the last one is executed.

It’s referenced right after “TABLE_PRODUCTS_DESCRIPTION” which is also listed as a cross join, but that one with good reason: the “languages_id” field of the “zen_products_description” table means that that is a many-to-one relation with the “zen_products” table and so needs to be filtered separately for current language.

I went ahead and reworded the query to make the TABLE_MANUFACTURERS referenced as a “LEFT JOIN ON p.manufacturers_id = m.manufacturers.id” in all four variants of the query, and removed the now-redundant “…and p.manufacturers_id = m.manufacturers.id” from the WHERE clause. To make the syntax cleaner, I moved it above the TABLE_PRODUCTS_DESCRIPTION reference.

Now my products index listing pages return no out-of-memory errors, and return just what they’re supposed to! And, they’re quite a bit faster, to boot!