Language Display Problems...

[lhungil]

... I checked phpMyAdmin and also checked with my hosting company, and they say the database is now UTF8-general_ci. ... If this is the correct encoding, then isn't all the data (Japanese & English) in the database already UTF-8?

I've not analyzed "Convert db2utf8" to see if it:

1. Just uses DB_CHARSET and database table storage format.
2. Uses DB_CHARSET, the database table storage format, and runs queries per language (converting row by row based upon the language's CHARSET).

If it does the first - you may run into problems if your original data used two different CHARSETs (it sounds like yours did). If the script does the second - things probably will be just fine. Quickest way to check is after running the script set both languages to use CHARSET UTF8 and DB_CHARSET UTF8 and see what happens :)

What does this mean?
Take the following example of the issues (imperfect, very basic, but hopefully helpful)...
Starting with the current state of things with multiple file types (language CHARSETs), one communication format (DB_CHARSET), and one storage type (database table encoding).

1. Take a zip file, send it to a server (communicating using gzip), and store the file into a zip file.
2. Take a rar file, send it to a server (communicating using gzip), and store the file into a zip file.

Now if we run a script which only knows to talk to the server using gzip and the server stores the file into a zip file... And then tell the server to change the storage from a zip file to a gzip file... And at the same time says "convert the data from zip to gzip"....

Case 1 above will turn out just fine as the data (zip) and the underlying storage (zip) were the same. However Case 2 will take the data (rar) and convert using (zip -> gzip) as it does not know the original data is a rar file (and assumes it is the same as the way the data is stored on the server). This will probably result in corrupted data.

Now if the script also knows there are multiple types of original files (zip and rar)... The script can then start by selectively pulling all the rows with rar files and converting them to zip files. Afterwards the bulk conversion from zip to gzip will work just fine for both Case 1 and Case 2.

[Peace Freak]

Thank you! The test you suggested is a bit beyond me so I have sent a message to the author of Convert db2utf8 and asked him your questions... Will let you know what I discover.

[Peace Freak]

Quickest way to check is after running the script set both languages to use CHARSET UTF8 and DB_CHARSET UTF8 and see what happens :)

I haven't heard from the author of Convert db2utf8 yet, but in the meantime I would like to try your "quick" solution.

Could you be more specific as to exactly what files I should change to CHARSET UTF8 and DB_CHARSET UTF8?

Thanks!

[lhungil]

Could you be more specific as to exactly what files I should change to CHARSET UTF8 and DB_CHARSET UTF8?

CHARSET is set in the language files. You will need to do this for all languages installed. For example inside "/includes/languages/english.php" and "/admin/includes/languages/english.php":

define('CHARSET', 'utf-8');

DB_CHARSET is set in the store configuration. For example inside "/includes/configure.php" and "/admin/includes/configure.php":

define('DB_CHARSET', 'utf8');

[Peace Freak]

Thanks.

I noticed in the Japanese language file that there are other places that the encoding can be changed. For example:
------
mb_language('Japanese');
mb_internal_encoding("iso-8859-1");
mb_http_output("iso-8859-1");

// EMAIL config
define('EMAIL_CHARSET', 'iso-8859-1');

// charset for web pages and emails
define('CHARSET', 'iso-8859-1');
------

Should all of these be changed to UTF-8?

I found that I only have a configure.php file in the includes folder. I do not have one in the admin folder! I checked in my version 1.3 backup and there is no configure.php file in that admin folder either. If there should be one, where do I get it?

In includes/configure.php I do not see any line of code for:

define('DB_CHARSET',

I guess it should be added to it, but I am not sure where it should go. Can you advise?

Thanks in advance!

[Peace Freak]

I have now changed everything that I could change to UTF-8, except for the "missing" items as noted in my previous post.

Basically, all the menus now appear correct, including the breadcrumb area, the text in the Search Box, Who's Online etc.

However, in the store (what the customer sees) the text for each of the products (titles and descriptions) is unreadable. If I change the browser encoding to Japanese(EUC) everything becomes readable!

So I went to the admin area and tried doing the same. It doesn't matter what browser encoding I use (UTF-8, Japanese(EUC) etc), none of the Japanese is readable!

But then I noticed that ONE single item WAS READABLE which struck me as really strange. I checked if that was readable in the store/customer area and it wasn't. I tried all the different encoding options in my browser but none worked for that particular item!

As a test, in the admin area, I deleted the unreadable title and description of another product and replaced it with Japanese and saved it. It became readable in the admin area but in the store/customer area it became unreadable regardless of the browser encoding I chose...

I am not sure what to make of this...

It seems that the data is encoded in Japanese (EUC) and as the admin area is set to UTF-8 it cannot be read.

If I add new items in the admin area, they become readable in the admin area, but in the browser side, they cannot be read regardless of the browers encoding.

I feel like I am really close to getting this solved, but I really don't know enough to decide what I need to do.

I am looking forward to hearing any ideas on what I might try next.

Thanks...

[Peace Freak]

Well, I have heard from the author of Convert db2utf8. Below are his comments. I am wondering if this makes sense to you lhungil.

What do you or anyone recommend I do?

=====
This script works very simply.

It analyzes the Collation of all tables with the specified prefix. If Collation is not equal to utf8_general_ci, it converts the table using the sql-query
Code:
ALTER TABLE `table_name` CONVERT TO CHARACTER SET utf8 COLLATE utf8_general_ci;
and then sets the default table charset and collation to utf-8:
Code:
ALTER TABLE `table_name` DEFAULT CHARACTER SET utf8 COLLATE utf8_general_ci;
All data conversion made by mySQL. You can read more about the conversion on the mySQL site:
http://dev.mysql.com/doc/refman/5.5/...onversion.html
=====

[Peace Freak]

I have now changed everything that I could change to UTF-8, except for the "missing" items as noted in my previous post.

Basically, all the menus now appear correct, including the breadcrumb area, the text in the Search Box, Who's Online etc.

However, in the store (what the customer sees) the text for each of the products (titles and descriptions) is unreadable. If I change the browser encoding to Japanese(EUC) everything becomes readable!

So I went to the admin area and tried doing the same. It doesn't matter what browser encoding I use (UTF-8, Japanese(EUC) etc), none of the Japanese is readable!

But then I noticed that ONE single item WAS READABLE which struck me as really strange. I checked if that was readable in the store/customer area and it wasn't. I tried all the different encoding options in my browser but none worked for that particular item!

As a test, in the admin area, I deleted the unreadable title and description of another product and replaced it with Japanese and saved it. It became readable in the admin area but in the store/customer area it became unreadable regardless of the browser encoding I chose...

I am not sure what to make of this...

It seems that the data is encoded in Japanese (EUC) and as the admin area is set to UTF-8 it cannot be read.

If I add new items in the admin area, they become readable in the admin area, but in the browser side, they cannot be read regardless of the browers encoding.

I feel like I am really close to getting this solved, but I really don't know enough to decide what I need to do.

I am looking forward to hearing any ideas on what I might try next.

Thanks...

I am really stuck on this and need to get things working. In a nutshell:

1/ In the backend of the store, using UTF-8 Text Encoding allows me to see only newly added items correctly. None of the other items (added previously) be read regardless of the text encoding setting in my browser.

2/ In the store itself, the newly added items do not appear readable regardless of the text encoding of my browser, however existing items in the store can be read, if the text encoding is set to Japanese EUC, yet these same items are unreadable in the backend!

My logic tells me that the backend is now set to UTF-8 as I can now read a couple of items in there with my browser set to UTF-8, which is actually the default setting of my browser.

As that same content is not readable in the frontend of the store (with my browser set to UTF-8) there must be some setting that I have missed that I need to change.

Which files control the display of product data in the frontend of the store? It seems the setting is stuck on Japanese EUC.

Thank you in advance!

[Peace Freak]

Have been doing further testing to try and get this fixed. Here is what I have discovered...

In the backend, I have Japanese titles written for book categories. If I click on the green edit button for one of the category names, I see this string of Unicode characters:

& # 1 2 3 8 1 ; & # 1 2 3 9 8 ; & # 1 2 4 1 1 ; & # 1 2 3 6 3 ; (I had to put spaces in between the characters so they wouldn't display as Japanese in this message.)

This string of unicode displays correctly in both the front and backend correctly like this:

そのほか

If I go back and edit the category and replace:

& # 1 2 3 8 1 ; & # 1 2 3 9 8 ; & # 1 2 4 1 1 ; & # 1 2 3 6 3 ;
with
そのほか

then save, in the backend it appears correctly (そのほか) but in the front end it appears as: ???? (Four question marks!)

So this tells me the backend can handle both formats of the characters, but the front end can only display the Japanese correctly if I use the Unicode format: & # 1 2 3 8 1 ; & # 1 2 3 9 8 ; & # 1 2 4 1 1 ; & # 1 2 3 6 3 ;

As the text displays correctly in the backend, there must be some setting that I have missed that is not allowing the front end to display things properly. How do I change the front end from Unicode???

Or is there something else I am missing? Any help or advice would be greatly appreciated!

[lhungil]

...

If I go back and edit the category and replace ... with そのほか then save, in the backend it appears correctly (そのほか) but in the front end it appears as: ???? (Four question marks!)

...

As the text displays correctly in the backend, there must be some setting that I have missed that is not allowing the front end to display things properly.

What do you for CHARSET and local (LC_) settings in "/includes/languages/japanese.php" and "/admin/includes/languages/japanese.php"? And do they match? Do you have a template override of the language file in "/includes/languages/my_template_name/japanese.php"? If so, do the settings match the ones from the admin language files? Are you using copy / paste or typing in the characters directly to the admin? Are you using CKeditor? Does your website template include the meta tag to indicate content is utf8?