I have a two language cart, the primary language being English with Japanese language packs for the cart and admin areas installed.

After upgrading, in the cart, in the following areas the Japanese characters are not appearing correctly (mojibake): In the Search Box, both bread crumb areas, Who's Online, the word "Manufactured by" that appears below each product… (The names of the items and descriptions are readable!)

In the Admin area it is the opposite. The names of each of the products, the descriptions, the Japanese names of customers, the status of each order are all unreadable!

I have been trying to find a fix and found this thread http://www.zen-cart.com/showthread.php?186747-Japanese-stopped-displaying-in-cart-still-displays-in-Admin for version 1.3.9:
...To get the utf-8 to work properly, you need to insert a one-line php file into the the includes/extra_configures folder. The name can be anything, but the contents are the single line:

What are your database tables??
UTF8 or latin1_swedish_ci

In your /includes/languages/english.php what is defined for

// charset for web pages and emails
  define('CHARSET', '?????');

In your /includes/languages/japanese.php what is defined for

// charset for web pages and emails
  define('CHARSET', '?????');

The database character set is: UTF8-general_ci

For /includes/languages/english.php it is: 'utf-8'

For /includes/languages/japanese.php it is: 'utf-8'

I guess I am going to have to convert the database to just UTF8. Anyway, I await your instructions.

BTW, it seems that pre-upgrade database was: latin1_swedish_ci

Thank you!

The database character set is: UTF8-general_ci
Using phpmyadmin and loading your database - are all tables UTF-8?

You could also try in your japanese.php setting the character set to

define('CHARSET', 'iso-8859-1');

Save the original on your PC in case it does not work

You could also try

define('CHARSET', 'iso-2022-jp');

Using phpmyadmin and loading your database - are all tables UTF-8?

No! See the attached is a screen shot showing some variation. All the tables below the screen shot are encoded with:

latin1_swedish_ci

You could also try in your japanese.php setting the character set to

define('CHARSET', 'iso-8859-1');

> Save the original on your PC in case it does not work

Will try...

Try running this
http://www.zen-cart.com/downloads.php?do=file&id=1318

Follow instructions

Back at work on getting this sorted out...

I have narrowed things down to this:

In the japanese/header.php file in this section:

// charset for web pages and emails
define('CHARSET', 'UTF-8');

using UTF-8 will give me a readable words "home" and "login" (in Japanese) in the breadcrumb area, but the body of the page is unreadable.

If I change UTF-8 to EUC-JP the body becomes readable but the breadcrumb area gets messed up...

There appears to be something else messing things up. There seem to be conflicting charsets...

I would appreciate any suggestions...

Peace Freak:

...
In the japanese/header.php file in this section:

// charset for web pages and emails
define('CHARSET', 'UTF-8');

using UTF-8 will give me a readable words "home" and "login" (in Japanese) in the breadcrumb area, but the body of the page is unreadable.

If I change UTF-8 to EUC-JP the body becomes readable but the breadcrumb area gets messed up...
disclaimer In the last few years I've only needed to dabble with character sets and character encoding in a small number of cases... My last real in-depth excursion in conversion and special handling was back when I was doing programming for IRC...

Sounds indeed as if you have conflicting character sets... Sounds like part of your database storage is using a mix of latin1_swedish_ci and UTF8... And sounds like the data being sent to / from the database is in a mix of latin1_swedish_ci, EUC-JP, and UTF-8... And the website (and PHP) uses CHARSET (based upon language) to determine which encoding to communicate with the browser...

So basically you have the DB Storage CHARSET, the DB Communication CHARSET, the PHP / (X)HTML / CSS CHARSET, and the WebServer / Browser negotiated CHARSET for communications... Sound complicated? It can be... To make things simpler most modern applications just use UTF8 for everything as 99% of systems support UTF8 and most character sets can be encoded using UTF8... Zen Cart 1.5 out of box (not upgrade) has switched to this same convention.

In the long term, scrubbing your data to convert everything possible to use the same Character Encoding is probably the best route to go - especially for multilingual sites. Going to be a little bit of work to straighten everything out (and convert the stored data)... And I would not recommend working on the live site while converting the data... I suspect you will need to convert the English and Japanese data sets separately as they use a different encoding...

I believe the tool posted earlier assumes the data stored in the database is using the same character encoding as the database table (which works for English / iso-latin-1 conversion)... For the EUC-JP I suspect you would need to pull out the data, convert the data to UTF8, reset the DB table + communication to UTF8, then save the converted data back into the table...

In the meantime, try changing "define('DB_CHARSET', 'utf8');" to "define('DB_CHARSET', 'iso-latin-1');" in both configure.php files (tells Zen Cart to talk to the database using iso-latin-1 instead of UTF8) and setting the correct charsets for the browsers in the language files using define('CHARSET', 'xxxxxx')" where xxxxxx is the correct charset previously used before the upgrade (guessing 'latin1' and 'EUC-JP'). See: Should I Convert My Store From iso-8859-1 To utf8.

lhungil:

[SIZE=3][COLOR="#FF0000"]

...In the long term, scrubbing your data to convert everything possible to use the same Character Encoding is probably the best route to go - especially for multilingual sites. Going to be a little bit of work to straighten everything out (and convert the stored data)... And I would not recommend working on the live site while converting the data... I suspect you will need to convert the English and Japanese data sets separately as they use a different encoding...

I believe the tool posted earlier assumes the data stored in the database is using the same character encoding as the database table (which works for English / iso-latin-1 conversion)... For the EUC-JP I suspect you would need to pull out the data, convert the data to UTF8, reset the DB table + communication to UTF8, then save the converted data back into the table...[/URL].

Thank you very much for your detailed response! As the store is currently offline, I think it would be best to fix it once and for all with your long term solution, but to do so, I will need some more assistance with the procedure...

After I ran Convert db2utf8:
http://www.zen-cart.com/downloads.php?do=file&id=1318
I checked phpMyAdmin and also checked with my hosting company, and they say the database is now UTF8-general_ci.

If this is the correct encoding, then isn't all the data (Japanese & English) in the database already UTF-8?

If it is not the correct encoding, what should the encoding be?

Peace Freak:

... I checked phpMyAdmin and also checked with my hosting company, and they say the database is now UTF8-general_ci. ... If this is the correct encoding, then isn't all the data (Japanese & English) in the database already UTF-8?
I've not analyzed "Convert db2utf8" to see if it:

1. Just uses DB_CHARSET and database table storage format.
2. Uses DB_CHARSET, the database table storage format, and runs queries per language (converting row by row based upon the language's CHARSET).
   If it does the first - you may run into problems if your original data used two different CHARSETs (it sounds like yours did). If the script does the second - things probably will be just fine. Quickest way to check is after running the script set both languages to use CHARSET UTF8 and DB_CHARSET UTF8 and see what happens :)

What does this mean?
Take the following example of the issues (imperfect, very basic, but hopefully helpful)...
Starting with the current state of things with multiple file types (language CHARSETs), one communication format (DB_CHARSET), and one storage type (database table encoding).

1. Take a zip file, send it to a server (communicating using gzip), and store the file into a zip file.
2. Take a rar file, send it to a server (communicating using gzip), and store the file into a zip file.

Now if we run a script which only knows to talk to the server using gzip and the server stores the file into a zip file... And then tell the server to change the storage from a zip file to a gzip file... And at the same time says "convert the data from zip to gzip"....

Case 1 above will turn out just fine as the data (zip) and the underlying storage (zip) were the same. However Case 2 will take the data (rar) and convert using (zip -> gzip) as it does not know the original data is a rar file (and assumes it is the same as the way the data is stored on the server). This will probably result in corrupted data.

Now if the script also knows there are multiple types of original files (zip and rar)... The script can then start by selectively pulling all the rows with rar files and converting them to zip files. Afterwards the bulk conversion from zip to gzip will work just fine for both Case 1 and Case 2.

Thank you! The test you suggested is a bit beyond me so I have sent a message to the author of Convert db2utf8 and asked him your questions... Will let you know what I discover.

lhungil:

Quickest way to check is after running the script set both languages to use CHARSET UTF8 and DB_CHARSET UTF8 and see what happens :)

I haven't heard from the author of Convert db2utf8 yet, but in the meantime I would like to try your "quick" solution.

Could you be more specific as to exactly what files I should change to CHARSET UTF8 and DB_CHARSET UTF8?

Thanks!

Peace Freak:

Could you be more specific as to exactly what files I should change to CHARSET UTF8 and DB_CHARSET UTF8?
CHARSET is set in the language files. You will need to do this for all languages installed. For example inside "/includes/languages/english.php" and "/admin/includes/languages/english.php":

define('CHARSET', 'utf-8');

DB_CHARSET is set in the store configuration. For example inside "/includes/configure.php" and "/admin/includes/configure.php":

define('DB_CHARSET', 'utf8');

Thanks.

I noticed in the Japanese language file that there are other places that the encoding can be changed. For example:

mb_language('Japanese');
mb_internal_encoding("iso-8859-1");
mb_http_output("iso-8859-1");

// EMAIL config
define('EMAIL_CHARSET', 'iso-8859-1');

// charset for web pages and emails
define('CHARSET', 'iso-8859-1');

Should all of these be changed to UTF-8?

I found that I only have a configure.php file in the includes folder. I do not have one in the admin folder! I checked in my version 1.3 backup and there is no configure.php file in that admin folder either. If there should be one, where do I get it?

In includes/configure.php I do not see any line of code for:

define('DB_CHARSET',

I guess it should be added to it, but I am not sure where it should go. Can you advise?

Thanks in advance!

I have now changed everything that I could change to UTF-8, except for the "missing" items as noted in my previous post.

Basically, all the menus now appear correct, including the breadcrumb area, the text in the Search Box, Who's Online etc.

However, in the store (what the customer sees) the text for each of the products (titles and descriptions) is unreadable. If I change the browser encoding to Japanese(EUC) everything becomes readable!

So I went to the admin area and tried doing the same. It doesn't matter what browser encoding I use (UTF-8, Japanese(EUC) etc), none of the Japanese is readable!

But then I noticed that ONE single item WAS READABLE which struck me as really strange. I checked if that was readable in the store/customer area and it wasn't. I tried all the different encoding options in my browser but none worked for that particular item!

As a test, in the admin area, I deleted the unreadable title and description of another product and replaced it with Japanese and saved it. It became readable in the admin area but in the store/customer area it became unreadable regardless of the browser encoding I chose...

I am not sure what to make of this...

It seems that the data is encoded in Japanese (EUC) and as the admin area is set to UTF-8 it cannot be read.

If I add new items in the admin area, they become readable in the admin area, but in the browser side, they cannot be read regardless of the browers encoding.

I feel like I am really close to getting this solved, but I really don't know enough to decide what I need to do.

I am looking forward to hearing any ideas on what I might try next.

Thanks...

Well, I have heard from the author of Convert db2utf8. Below are his comments. I am wondering if this makes sense to you lhungil.

What do you or anyone recommend I do?

=====
This script works very simply.

It analyzes the Collation of all tables with the specified prefix. If Collation is not equal to utf8_general_ci, it converts the table using the sql-query
Code:
ALTER TABLE table_name CONVERT TO CHARACTER SET utf8 COLLATE utf8_general_ci;
and then sets the default table charset and collation to utf-8:
Code:
ALTER TABLE table_name DEFAULT CHARACTER SET utf8 COLLATE utf8_general_ci;
All data conversion made by mySQL. You can read more about the conversion on the mySQL site:
http://dev.mysql.com/doc/refman/5.5/en/charset-conversion.html

Peace Freak:

I have now changed everything that I could change to UTF-8, except for the "missing" items as noted in my previous post.

Basically, all the menus now appear correct, including the breadcrumb area, the text in the Search Box, Who's Online etc.

However, in the store (what the customer sees) the text for each of the products (titles and descriptions) is unreadable. If I change the browser encoding to Japanese(EUC) everything becomes readable!

So I went to the admin area and tried doing the same. It doesn't matter what browser encoding I use (UTF-8, Japanese(EUC) etc), none of the Japanese is readable!

But then I noticed that ONE single item WAS READABLE which struck me as really strange. I checked if that was readable in the store/customer area and it wasn't. I tried all the different encoding options in my browser but none worked for that particular item!

As a test, in the admin area, I deleted the unreadable title and description of another product and replaced it with Japanese and saved it. It became readable in the admin area but in the store/customer area it became unreadable regardless of the browser encoding I chose...

I am not sure what to make of this...

It seems that the data is encoded in Japanese (EUC) and as the admin area is set to UTF-8 it cannot be read.

If I add new items in the admin area, they become readable in the admin area, but in the browser side, they cannot be read regardless of the browers encoding.

I feel like I am really close to getting this solved, but I really don't know enough to decide what I need to do.

I am looking forward to hearing any ideas on what I might try next.

Thanks...

I am really stuck on this and need to get things working. In a nutshell:

1/ In the backend of the store, using UTF-8 Text Encoding allows me to see only newly added items correctly. None of the other items (added previously) be read regardless of the text encoding setting in my browser.

2/ In the store itself, the newly added items do not appear readable regardless of the text encoding of my browser, however existing items in the store can be read, if the text encoding is set to Japanese EUC, yet these same items are unreadable in the backend!

My logic tells me that the backend is now set to UTF-8 as I can now read a couple of items in there with my browser set to UTF-8, which is actually the default setting of my browser.

As that same content is not readable in the frontend of the store (with my browser set to UTF-8) there must be some setting that I have missed that I need to change.

Which files control the display of product data in the frontend of the store? It seems the setting is stuck on Japanese EUC.

Thank you in advance!

Have been doing further testing to try and get this fixed. Here is what I have discovered...

In the backend, I have Japanese titles written for book categories. If I click on the green edit button for one of the category names, I see this string of Unicode characters:

& # 1 2 3 8 1 ; & # 1 2 3 9 8 ; & # 1 2 4 1 1 ; & # 1 2 3 6 3 ; (I had to put spaces in between the characters so they wouldn't display as Japanese in this message.)

This string of unicode displays correctly in both the front and backend correctly like this:

そのほか

If I go back and edit the category and replace:

& # 1 2 3 8 1 ; & # 1 2 3 9 8 ; & # 1 2 4 1 1 ; & # 1 2 3 6 3 ;
with
そのほか

then save, in the backend it appears correctly (そのほか) but in the front end it appears as: ???? (Four question marks!)

So this tells me the backend can handle both formats of the characters, but the front end can only display the Japanese correctly if I use the Unicode format: & # 1 2 3 8 1 ; & # 1 2 3 9 8 ; & # 1 2 4 1 1 ; & # 1 2 3 6 3 ;

As the text displays correctly in the backend, there must be some setting that I have missed that is not allowing the front end to display things properly. How do I change the front end from Unicode???

Or is there something else I am missing? Any help or advice would be greatly appreciated!

Peace Freak:

...

If I go back and edit the category and replace ... with そのほか then save, in the backend it appears correctly (そのほか) but in the front end it appears as: ???? (Four question marks!)

...

As the text displays correctly in the backend, there must be some setting that I have missed that is not allowing the front end to display things properly.
What do you for CHARSET and local (LC_) settings in "/includes/languages/japanese.php" and "/admin/includes/languages/japanese.php"? And do they match? Do you have a template override of the language file in "/includes/languages/my_template_name/japanese.php"? If so, do the settings match the ones from the admin language files? Are you using copy / paste or typing in the characters directly to the admin? Are you using CKeditor? Does your website template include the meta tag to indicate content is utf8?