Upgrading from 1.3.x with latin1, needing to have utf-8 from now on

[ellivir]

So,
it is explained here in the link below what to do when you do NOT want to upgrade the latin1 to utf-8, but what to do when you NEED to have utf-8 for several different languages used in the webshop?
http://www.zen-cart.com/content.php?313

Our upgrade is still in a temporary folder and the old 1.3.9h webshop is alive untouched, but the upgrade webshop folder shows wrong accented or special Scandinavian characters. When I go to see in the database it is still in latin1 in each table. Should the upgrade process change the collation, tables to utf-8?

I have searched and can't find what to do when people would like to properly upgrade into utf-8. Making a fresh utf-8 webshop would be easier but that's not the case here.

[kobra]

Try this read instructions

http://www.zen-cart.com/downloads.php?do=file&id=1318

[ellivir]

Ok, tried and got the database nicely as utf-8. But for example the customer names etc are still full of question marks though after that.
Tested with the Localization -> Orders Status settings and if I write those again it saves them correctly without garbled characters.

[shinypenguin]

There is no way to convert the existing data to the correct UTF8 characters, since the extra character encoding was lost when it was converted to latin1 initially.

Just came across this thread looking to convert from utf8_general_ci to utf8_unicode_ci, which the script was able to do with a few small edits. From what I've read, unicode is the recommended collation, since it has improved accuracy at a very small processing cost (http://stackoverflow.com/questions/7...tf8-unicode-ci)

[ellivir]

Well, there is few thousand customer names and there are over ten thousand products, more than half of them have Scandinavian characters. There is no warning or guides in the upgrade process how to do the upgrade with these. The situation is very normal, there is no special add-on conditions altering the result. How has it been PLANNED to proceed in this kind of situation?
All the language related packs are already in UTF-8, from that point of view things are ready and fresh install with new products and customers work perfectly (has been tested and changed in the process quite a few things in the language files with another shop).

[ellivir]

If I edit indeed in the admin order status now and the new word saved has "ä" included, then I look in the database there is "ä". In the admin on the hand it looks all ok but surely this shouldn't be like this in the db.

[lat9]

Converting the database is the "hard" part. Next, you need to make sure that your configuration files match up, too:

1. In /YOUR_ADMIN/includes/configure.php and /includes/configure.php, make sure that there is a define('DB_CHARSET', 'utf8'); in the database configuration area.
2. Make sure that the CHARSET in /YOUR_ADMIN/includes/languages/english.php and /includes/languages/YOUR_TEMPLATE/english.php is defined as UTF-8: define('CHARSET', 'utf-8'); and that the locale is set for utf-8: @setlocale(LC_TIME, 'en_US.UTF-8');. Make sure that you repeat this for all of your primary language files, too, with the appropriate locale code for those non-English languages.
3. Each of the files in your languages directories (both store and admin) must be saved using 'UTF-8 without BOM' encoding. This is a pain in the , but when it's done it's done.

[ellivir]

Converting the database is the "hard" part. Next, you need to make sure that your configuration files match up, too:

1. In /YOUR_ADMIN/includes/configure.php and /includes/configure.php, make sure that there is a define('DB_CHARSET', 'utf8'); in the database configuration area.

Thanks a million, I compared the configure files with the other webshop that all is functioning and it had these definitions for the CHARSET, and then I realized our admin configuration didn't have it all! So, the upgrade process doesn't "understand" to add it.

1. Make sure that the CHARSET in /YOUR_ADMIN/includes/languages/english.php and /includes/languages/YOUR_TEMPLATE/english.php is defined as UTF-8: define('CHARSET', 'utf-8'); and that the locale is set for utf-8: @setlocale(LC_TIME, 'en_US.UTF-8');. Make sure that you repeat this for all of your primary language files, too, with the appropriate locale code for those non-English languages.
2. Each of the files in your languages directories (both store and admin) must be saved using 'UTF-8 without BOM' encoding. This is a pain in the , but when it's done it's done.

All this other really laborious work has been done already due to the other webshop where is the same language, so this part was fine. But this is a great list for somebody else now that is getting bothered and not finding a solution. Adding the definition was like a magic trick (database was utf8, files, language definitions etc but only this was the "key" missing!).

Thanks, thanks and thanks!