UTF-8 Zen-Cart Tutorial for v1.3.x

DrByte:

I agree -- that script was probably the closest I've seen to reliably handling most of the steps involved in properly preserving data while converting character sets and collation to UTF8.

I've made some alterations to overcome several of the limitations I see in it, and contributed the result here:
http://www.zen-cart.com/index.php?main_page=product_contrib_info&products_id=1937

Feedback welcome.

DrByte, your instruction said> e) Update your Zen Cart configure.php settings to use UTF8 for DB_CHARSET I do not see the option for this change in either the catalog or admin configure.php. Is this setting located somewhere else?

Thanks!

Post 92, earlier in this thread, has a link to how to handle this in 1.3.9: http://www.zen-cart.com/forum/showthread.php?t=153582

DrByte:

Post 92, earlier in this thread, has a link to how to handle this in 1.3.9: http://www.zen-cart.com/forum/showthread.php?t=153582

Thanks DrByte! Do I need to touch these two lines of code in english.php for both catalog or admin? Just curious.

setlocale(LC_TIME, 'en_US.ISO_8859-1');
define('CHARSET', 'iso-8859-1');

Hmmm ... ya, probably a good idea. :)

DrByte:

Hmmm ... ya, probably a good idea. :)

Okay, this is getting more complicated than I thought. There are actually 8 files or so in Zen Cart that reference iso-8859-1 and I am not sure of only english.php files needs the changes or if I have to touch more files.

When you get a chance to give this some thought it will probably be a good idea to provide more information on the best way to approach this.

Originally, this was going to be my plan of attack.

1. run your script

2. create /includes/extra_configures/my_db_charset.php and /adminFOLDER/includes/extra_configures/my_db_charset.php

3 Add this code to the above two files:

<?php define('DB_CHARSET', 'utf8');

Update english php file in the catalog and admin to:

change this line from php setlocale(LC_TIME, 'en_US.ISO_8859-1'); to ```php
setlocale(LC_TIME, 'en_US.utf8');

change this line from ```php
define('CHARSET', 'iso-8859-1');
``` to ```php
define('CHARSET', 'utf-8');

Also, I am not sure what the difference between utf8 and utf-8 is just simply follow the instruction provided. A clarification between the two if any will help as well.

Keep in mind that the script itself is only intended to deal with the database conversion.

You are doing this on a TEST site, right?

If you're going to edit any language files, you need to be sure to save them encoded as UTF8-without-BOM. Otherwise you'll have a BOM header at the top of each file, which will cause you headaches with blank pages and Headers Already Sent errors.

DB_CHARSET would be 'utf8'
CHARSET in the language files would be 'UTF-8'
and LC_TIME would be 'en_US.utf8'

The script worked flawlessly without any errors and it was done applied in production but I had like 4 different backup of the database.

Thanks and I am glad I didn't destroyed our site in the process of converting the DB to utf8. That was a very nifty script!

:clap: :)

Just so I am perfectly clear, there's absolutely no reason to touch any of these files even after the database has been converted to utf-8?

CATALOG/nddbc.html
Line #4 : <meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1" />

ADMINFOLDER/alert_page.php
Line #22 : <meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1" />

includes/classes/class.phpmailer.php
Line #43 : var $CharSet = "iso-8859-1";

/includes/functions/functions_email.php
Line #171 : $mail->CharSet = (defined('CHARSET')) ? CHARSET : "iso-8859-1";

/includes/functions/functions_general.php
Line #1582 : if (function_exists('iconv')) $string = iconv("Windows-1252", "ISO-8859-1//IGNORE", $string);

/includes/templates/template_default/templates/tpl_zc_install_suggested_default.php
Line #19 : <meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1" />

Correct.
Most of those files are standalone and don't need touching. The others only mention 8859 as a fallback if the rest of the system is sorely misconfigured.

DrByte:

Correct.
Most of those files are standalone and don't need touching. The others only mention 8859 as a fallback if the rest of the system is sorely misconfigured.

Thank you for the clarification!

DrByte:

I agree -- that script was probably the closest I've seen to reliably handling most of the steps involved in properly preserving data while converting character sets and collation to UTF8.

I've made some alterations to overcome several of the limitations I see in it, and contributed the result here:
http://www.zen-cart.com/index.php?main_page=product_contrib_info&products_id=1937

NOTES:
MAKE A DATABASE BACKUP FIRST!
And take your store Down For Maintenance while converting.
Watch out for any warnings - they will be shown in bold. Every error should receive appropriate attention before putting your site back online.

Feedback welcome.

The script has been updated so that it now properly handles NULL and DEFAULT values. The original from which it was adapted didn't handle those, and simply used defaults, thus losing all NULL or DEFAULT value settings. If you've used the previous edition of this script, you will have lost those values from your database. You'll have to manually add them back by comparing your database structure with the original Zen Cart database structure.

DrByte:

The script has been updated so that it now properly handles NULL and DEFAULT values. The original from which it was adapted didn't handle those, and simply used defaults, thus losing all NULL or DEFAULT value settings. If you've used the previous edition of this script, you will have lost those values from your database. You'll have to manually add them back by comparing your database structure with the original Zen Cart database structure.

Is it possible to rerun the updated script? Will that fix the NULL or DEFAULT values or the best approach simply to manually change the NULL and DEFAULT values in the database?

Thanks for updating the script.

The script cannot fix the values it left off the last time you ran it. You'll have to fix them by hand. Sorry. There are lots :(

DrByte:

The script cannot fix the values it left off the last time you ran it. You'll have to fix them by hand. Sorry. There are lots :(

Thanks DrByte! I know there are lots of them which is why I wanted to ask before fixing them. I will fix those and then post back something else I observed if it's not gone after fixing that.

Thanks again for your help on this.

Hi
I have some problems with this, maybe someone can help?

I am still using ZC 1.3.8

I followed the instructions from srw2d.com (which is now in a pdf file)

Since I already have a large databse with thousands of customers i had to convert it.

What i did was backup my database first (using the drop command - actually my version of phpmyadmin just calls it the DROP command and does NOT MENTION DROP TABLE/VIEW/PROCEDURE/FUNCTION

I then use textwrangler to find and replace where it said latin1 and changed it to utf8 (exactly like that find"latin1" replace "utf8") what it said before was CHARSET=latin1 and it now says CHARSET=utf8

There were 99 replacements made which is the same number as there are tables in the database.. so it looks right

Then what i did was make a new database on the same server.. go to phpyadmin and change it to utf8 then drop my old (modified) database tables on top (i kept the old database incase i had to switch back)

I change both my admin configure and configure files with setting for new database

Then i made all the changes from srw2d.com and saved the modified files.

Site works.. i check in firefox in admin and main areas encoding is set to utf8 according to firefox

However in view source the HEADER says

So there is something wrong there?

I have searched the database iso-8859-1 does not appear anywhere.

I have searched the files using developers tool kit and found the following instances of it

/home/xxxx/public_html/nddbc.html

Line #4 : <meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1" />

/home/xxxx/public_html/includes/classes/class.phpmailer.php

Line #43 : var $CharSet = "iso-8859-1";

/home/xxxx/public_html/includes/functions/functions_email.php

Line #163 : $mail->CharSet = (defined('CHARSET')) ? CHARSET : "iso-8859-1";

/home/xxxx/public_html/includes/languages/pure_blue/english.php

Line #53 : define('CHARSET', 'iso-8859-1');

/home/xxxx/public_html/includes/templates/template_default/templates/tpl_zc_install_suggested_default.php

Line #19 : <meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1" />

What should i do? The character encoding shown in the header is not right?

At the moment i only have english characters on my website.

Thanks

In the admin section header it is saying text/html; charset=utf-8">

Thats good
But im looking at one of my old orders.. which was placed before when the database was set to latin

It is in Thai script and in the area where customers can enter their special comments for order section the customer typed in Thai language. However it appears like this

in a string of acii character code symbols. .. i cant paste them in this forum because this webpage somehow translate them into Thai script as they should look on MY site!!

Obviously that is not Thai script. The point of this excersise was to get these Thai language orders showing in Thai script!

At the moment customers just see this gobeldygook before they submit their order (the stuff they entered in Thai language is turned into this ascii code stuff)

Ive tried putting some Thai text ดอกไม้ไทย on my index page
but it still displays as ดà¸##กไม้ไทย
Any ideas?

Al

Just to confirm in the database i entered utf8 and the files define('CHARSET', 'utf-8'); setlocale(LC_TIME, 'en_US.utf8'); etc

You said your admin html shows utf-8, but your storefront does not.
Thus (assuming you're using english) you need to update your /includes/languages/english.php -- define CHARSET as utf-8