Language Display Problems...

[Peace Freak]

I have a two language cart, the primary language being English with Japanese language packs for the cart and admin areas installed.

After upgrading, in the cart, in the following areas the Japanese characters are not appearing correctly (mojibake): In the Search Box, both bread crumb areas, Who's Online, the word "Manufactured by" that appears below each product… (The names of the items and descriptions are readable!)

In the Admin area it is the opposite. The names of each of the products, the descriptions, the Japanese names of customers, the status of each order are all unreadable!

I have been trying to find a fix and found this thread http://www.zen-cart.com/showthread.p...plays-in-Admin for version 1.3.9:
...To get the utf-8 to work properly, you need to insert a one-line php file into the the includes/extra_configures folder. The name can be anything, but the contents are the single line:
<?php define('DB_CHARSET', 'utf8');...

I tried putting the file in both the admin and cart areas. In the cart it caused the correctly displaying Japanese to get messed up! In the admin area it did change the format of the characters, but the characters remained unreadable. Anyway, both have been removed.

I am wondering what needs to be done for things to work in 1.5.1. Do individual files need to have their charsets adjusted or is there some setting I need to adjust to get thing right?

Thank you!

[kobra]

What are your database tables??
UTF8 or latin1_swedish_ci

In your /includes/languages/english.php what is defined for

// charset for web pages and emails
  define('CHARSET', '?????');

In your /includes/languages/japanese.php what is defined for

// charset for web pages and emails
  define('CHARSET', '?????');

[Peace Freak]

The database character set is: UTF8-general_ci

For /includes/languages/english.php it is: 'utf-8'

For /includes/languages/japanese.php it is: 'utf-8'

I guess I am going to have to convert the database to just UTF8. Anyway, I await your instructions.

BTW, it seems that pre-upgrade database was: latin1_swedish_ci

Thank you!

[kobra]

The database character set is: UTF8-general_ci

Using phpmyadmin and loading your database - are all tables UTF-8?

You could also try in your japanese.php setting the character set to

define('CHARSET', 'iso-8859-1');

Save the original on your PC in case it does not work

[kobra]

You could also try

define('CHARSET', 'iso-2022-jp');

[Peace Freak]

Using phpmyadmin and loading your database - are all tables UTF-8?

No! See the attached is a screen shot showing some variation. All the tables below the screen shot are encoded with:

latin1_swedish_ci

You could also try in your japanese.php setting the character set to

define('CHARSET', 'iso-8859-1');

Save the original on your PC in case it does not work

Will try...

[kobra]

Try running this
http://www.zen-cart.com/downloads.php?do=file&id=1318

Follow instructions

[Peace Freak]

Back at work on getting this sorted out...

I have narrowed things down to this:

In the japanese/header.php file in this section:

// charset for web pages and emails
define('CHARSET', 'UTF-8');

using UTF-8 will give me a readable words "home" and "login" (in Japanese) in the breadcrumb area, but the body of the page is unreadable.

If I change UTF-8 to EUC-JP the body becomes readable but the breadcrumb area gets messed up...

There appears to be something else messing things up. There seem to be conflicting charsets...

I would appreciate any suggestions...

[lhungil]

...
In the japanese/header.php file in this section:

// charset for web pages and emails
define('CHARSET', 'UTF-8');

using UTF-8 will give me a readable words "home" and "login" (in Japanese) in the breadcrumb area, but the body of the page is unreadable.

If I change UTF-8 to EUC-JP the body becomes readable but the breadcrumb area gets messed up...

**disclaimer** In the last few years I've only needed to dabble with character sets and character encoding in a small number of cases... My last real in-depth excursion in conversion and special handling was back when I was doing programming for IRC...

Sounds indeed as if you have conflicting character sets... Sounds like part of your database storage is using a mix of latin1_swedish_ci and UTF8... And sounds like the data being sent to / from the database is in a mix of latin1_swedish_ci, EUC-JP, and UTF-8... And the website (and PHP) uses CHARSET (based upon language) to determine which encoding to communicate with the browser...

So basically you have the DB Storage CHARSET, the DB Communication CHARSET, the PHP / (X)HTML / CSS CHARSET, and the WebServer / Browser negotiated CHARSET for communications... Sound complicated? It can be... To make things simpler most modern applications just use UTF8 for everything as 99% of systems support UTF8 and most character sets can be encoded using UTF8... Zen Cart 1.5 out of box (not upgrade) has switched to this same convention.

In the long term, scrubbing your data to convert everything possible to use the same Character Encoding is probably the best route to go - especially for multilingual sites. Going to be a little bit of work to straighten everything out (and convert the stored data)... And I would not recommend working on the live site while converting the data... I suspect you will need to convert the English and Japanese data sets separately as they use a different encoding...

I believe the tool posted earlier assumes the data stored in the database is using the same character encoding as the database table (which works for English / iso-latin-1 conversion)... For the EUC-JP I suspect you would need to pull out the data, convert the data to UTF8, reset the DB table + communication to UTF8, then save the converted data back into the table...

In the meantime, try changing "define('DB_CHARSET', 'utf8');" to "define('DB_CHARSET', 'iso-latin-1');" in both configure.php files (tells Zen Cart to talk to the database using iso-latin-1 instead of UTF8) and setting the correct charsets for the browsers in the language files using define('CHARSET', 'xxxxxx')" where xxxxxx is the correct charset previously used before the upgrade (guessing 'latin1' and 'EUC-JP'). See: Should I Convert My Store From iso-8859-1 To utf8.

[Peace Freak]

[SIZE=3][COLOR="#FF0000"]

...In the long term, scrubbing your data to convert everything possible to use the same Character Encoding is probably the best route to go - especially for multilingual sites. Going to be a little bit of work to straighten everything out (and convert the stored data)... And I would not recommend working on the live site while converting the data... I suspect you will need to convert the English and Japanese data sets separately as they use a different encoding...

I believe the tool posted earlier assumes the data stored in the database is using the same character encoding as the database table (which works for English / iso-latin-1 conversion)... For the EUC-JP I suspect you would need to pull out the data, convert the data to UTF8, reset the DB table + communication to UTF8, then save the converted data back into the table...[/URL].

Thank you very much for your detailed response! As the store is currently offline, I think it would be best to fix it once and for all with your long term solution, but to do so, I will need some more assistance with the procedure...

After I ran Convert db2utf8:
http://www.zen-cart.com/downloads.php?do=file&id=1318
I checked phpMyAdmin and also checked with my hosting company, and they say the database is now UTF8-general_ci.

If this is the correct encoding, then isn't all the data (Japanese & English) in the database already UTF-8?

If it is not the correct encoding, what should the encoding be?