Special characters messed up in ZC Admin

I recently installed a fresh copy of 1.3.0.1 on a new webhost (linux virtual server) The special danish characters (å, ø, æ) are not displaying correctly in ZC admin. They are replaced by ? and gobbledy gook.

I was plagued with similar problems in my database and on the webpages since installing, due to not using the right collation when creating the database. Over the last few days, I've managed to clear that up (with Absolute's kind help) but the problem in ZC Admin remains.

If the info in the database is "Tilbehør", it'll display correctly on the webpage, but in ZC admin, it'll read "Tilbeh?>" If I edit that category and then save without touching the name, it'll then display on my website as "Tilbeh?>". If I fix the name in admin, it'll be incorrect on the webpage and in the database.

I've tried to change the locale and charsets to everything I can think of, but nothing's worked.

Here's some info on my new setup:
PHP version: 4.4.2
MySQL Version Reported = 4.1.18-standard
HTTP_ACCEPT_LANGUAGE= "en-us,en;q=0.5"
HTTP_ACCEPT_CHARSET= "ISO-8859-1,utf-8;q=0.7,*;q=0.7"
SERVER_SOFTWARE= "Apache/2.0.51 (Fedora)"

Please, anybody with some help or suggestions -- I would really appreciate it.

Nobody has any ideas here? Any suggestions would be appreciated.

I realized you guys could use more information. Here's what I have in the main language files:

admin/includes/languages/danish.php:
setlocale(LC_TIME, 'da_DK.ISO_8859-1');
define('CHARSET', 'iso-8859-1');

includes/languages/danish.php:
@setlocale(LC_TIME, 'da_DK.ISO_8859-1');
define('CHARSET', 'iso-8859-1');

Let me know if any other info will help. Thanks, MP

I'm sorry to bump but I'm really desperate for some kind of direction on this problem. I'm willing to pay to have this fixed, because I'm not sure if all the work I'm continuing to do on the site won't be undermined by some initial error in encoding, collating, etc. - It'd be disastrous if I had to start from scratch now, even worse later.

Does anyone have any clue what's going on? Suggestions on things to check, places to look, rumors or innuendos? Anything...

Are the characters displaying properly in the catalog?

How about a guess?

I haven't had this problem with ZC, but other PHP scripts - if a .php file is saved with the wrong encoding/charset, it can mess up the special chars... Since you say it's working in the shop, but not admin, maybe a .php file in admin/ is saved w/the wrong character set?

Second guess - is there anything different in your web server's config between yourstore/ and yourstore/admin. .htaccess, etc?

Sorry - I'm not much help!
Chris

Yes the characters display correctly in the catalog if it's set correctly in the database. The admin is messed up. But, I can correct it in the admin, and then the database and catalog are messed up.

Chris, no -- your guess is helpful! It gets me looking in the right places. I'll check out what you recommend and get back with the answers.

I replaced my admin folder with a fresh 1.3.0.1 copy -- merged in my configure settings. The problem's still there.

I don't have any htaccess files in my admin. What else should I check?

I noticed in my database that some fields (category_description, etc) were collated as utf-8_danish_ci and some other fields were latin1_danish_ci. So, I created a new database as utf-8, exported my database, changed all the fields to utf-8_danish_ci in a text editor. I ran this exported file into my new database. It worked fine, my site is back up but... Nothing changed! The characters are still displaying incorrectly.

Is it okay to just switch the collation of these "fields" (what are they called??) in a text editor?

And, of course... anything else I should try to narrow it down?

Quote:

---

Originally Posted by magicpants

So, I created a new database as utf-8

---

How are you creating the database - in cpanel, etc?

What options for character set (collation?) are you given when you make a new database?

Is it possible to install software on your host? I use phpMyAdmin to access the mysql databases directly - if you can install it (or it's already installed), it might give you a clue to what character sets are available.

Chris

Yeah, I'm using phpmyadmin already -- that's how i have been creating the databases, etc. I'm using phpMyAdmin 2.6.4-pl4, and MySQL 4.1.18-standard.

When I create a database, I can set the collation -- I've tried both latin1_danish_ci and utf8_danish_ci with it set appropriately in the zen cart files. But, ALSO, on the right hand side of phpmyadmin, there's a few more settings: I can set the "language" (currently en-utf-8) and the "mySQL connection collation" (currently set to utf8_danish_ci). Both of these can be set by me. However, a third setting : "MySQL charset" can not be changed by me and it's currently set to UTF-8 Unicode (utf8). These are all the settings I see on this page, and I've tried many different combinations to no avail.

Further, I noticed that when I create a new database with a collation, let's say latin1_danish_ci, then export my current database, fix all the weird characters and change all the references to latin1_danish_ci in a text editor, then import that sql file into the new database, it looks okay. But, then if I immediately export to a text file, the sql file has the characters all screwed up again! HOWEVER, if I export the file WITHOUT SAVING A FILE -- just opening it on the screen -- it's fine! This could be an unrelated quirk of mySQL or phpmyadmin, I don't know...

I went through my php info and extracted all lines that had something to do with language, charsets, collations, etc.... I'll list them below. Maybe it'll shed some light on what's going on? Thanks so far, Chris...

Apache Environment
------------------
HTTP_ACCEPT_LANGUAGE en-us
HTTP_REFERER http://MYSITE.COM/pma/main.php?lang=...utf8_danish_ci
HTTP_COOKIE pma_theme=original; pma_collation_connection=utf8_danish_ci; pma_lang=en-utf-8; pma_charset=iso-8859-1
QUERY_STRING lang=en-utf-8&server=1&collation_connection=utf8_danish_ci
REQUEST_URI /pma/phpinfo.php?lang=en-utf-8&server=1&collation_connection=utf8_danish_ci

HTTP HEADERS INFORMATION
------------------------
Referer http://MYSITE.COM/pma/main.php?lang=...utf8_danish_ci
Accept-Language en-us
Cookie pma_theme=original; pma_collation_connection=utf8_danish_ci; pma_lang=en-utf-8; pma_charset=iso-8859-1
Content-Type text/html; charset=UTF-8

PHP Variables
-------------
_REQUEST["collation_connection"] utf8_danish_ci
_REQUEST["pma_collation_connection"] utf8_danish_ci
_REQUEST["pma_collation_connection"] utf8_danish_ci
_REQUEST["pma_lang"] en-utf-8
_REQUEST["pma_charset"] iso-8859-1
_GET["lang"] en-utf-8
_GET["collation_connection"] utf8_danish_ci
_COOKIE["pma_collation_connection"] utf8_danish_ci
_COOKIE["pma_lang"] en-utf-8
_COOKIE["pma_charset"] iso-8859-1
_SERVER["HTTP_REFERER"] http://MYSITE.COM/pma/main.php?lang=...utf8_danish_ci
_SERVER["HTTP_ACCEPT_LANGUAGE"] en-us
_SERVER["HTTP_COOKIE"] pma_theme=original; pma_collation_connection=utf8_danish_ci; pma_lang=en-utf-8; pma_charset=iso-8859-1
_SERVER["QUERY_STRING"] lang=en-utf-8&server=1&collation_connection=utf8_danish_ci
_SERVER["REQUEST_URI"] /pma/phpinfo.php?lang=en-utf-8&server=1&collation_connection=utf8_danish_ci
_SERVER["argv"] Array([0] => lang=en-utf-8&server=1&collation_connection=utf8_danish_ci)

I feel your pain - I had similar issues with a db-driven spanish site...

I'm running out of suggestions, but if you could export a couple of entries with the problem into a sql file, i'd be happy to take a look at it and see if I can reproduce/fix the problem...

I'll PM you my email addr.

Chris

Further testing this issue, I just installed a fresh 1.3.0.1 cart, separate from my real one. Chose latin1_danish_ci for the database collation and MySQL connection collation. (Of course, the MySQL charset was set to UTF-8 Unicode, because I can't change that setting)

I manually added one product with danish characters through the admin - and it shows up wrong in the database and on the website.

It must be a problem with the server or my php version or mysql version or who knows! I'd appreciate it if someone from the zen cart team could chime in here. Because I suspect this is a compatibility issue and it's already pushed us way off schedule.

Chris, thanks again -- i'll contact you by email...

Hi,

Is it possible for you to post a full phpinfo for your site. If you do not want to do this publically, then please pm me or drbyte.

I'm also looking at the possibility this may be an mbstring problem.

BTW wher in admin are you entering this value Tilbehør,

Is it as a product name, a category name or something else. ??

Just the results of my initial fiddlings

Initially I was having problems where admin worked fine with Tilbehør, used as a ategory name. It would display OK in admin, and I could repeatedly edit/save it without any problems.

However it always dislplayed incorrectly in Catalog.

I set character encoding in catalog and admin english.php to UTF-8 and everything works fine.

Note this is a default mysql5 install using the detalt latin1_swedish collation

Okay, I finally got my webhost to clear up the mbstring problem. It's installed and I thought this would finally solve the problem for us. But, it didn't.

After resolving mbstring -- I exported the database AGAIN, fixed the errors in Win32Pad, imported into a new database with latin1_danish_ci. Tried both iso-8859-1 and utf-8 in the language files, and the EXACT SAME PROBLEM.

I'm going to send another PM to wilt and dr. byte right now -- giving you guys full access to our system. I'm about two days away from throwing in the towel and looking into organized crime as a career option. It is %¤#%¤&@$ crazy how many hours I have wasted from my life trying to investigate this, and not even moving one small step toward solving it.

The man of the hour is Wilt! He fixed it up all nice and easy.

For anybody having this kind of problem in the future, it was because our server was somehow overriding the charset set by Zen Cart.

In our case, it was solved by adding one line to admin/includes/init_includes/init_sessions.php.

Find:

Code:

---

if (!defined('IS_ADMIN_FLAG')) {
  die('Illegal Access');
}

---

Add this line under it:

Code:

---

header("Content-Type: text/html;charset=ISO-8859-1");

---

Just want to say, that this represents a bug in zen-cart.

The problem is this, Zen cart does send out correct html and sets in both admin and catalog

meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1" />

or whatever the charset maybe.

The problem is that Apache also sends out headers setting the charset and if these are different to the html charset, your browser will listen to apache and not ZC.

We do already send override headers in the catalog, but forgot to put them in admin. :(

Needless to say, thye will be there in the next release. (albeit, in a different file than you mentioned above)

However, glad to have helped :)

I am having a similar problem.
I just moved a store from one host to another and now my spanish special characters aren't displaying correctly in the admin and catalog.

When you are running the backup of the database into the new blank database via phpMyAdmin make sure to select Latin 1 as the collation.

Your new host is probably using utf8 as the default character set - and that's why Spanish language special characters are becoming corrupted.

utf8 is what everyone will be using eventually, but it's creating havoc with databases that were first compiled in Latin1 or en-iso-8859 (same thing).

Vger

i had originally done utf8 but re imported with latin1 and that fixed my issue.
thanks :)

en-iso-8859 sounds familiar. i think thats what it was set to in my old database.

Ok I found a reference to iso-8859-1 on my spanish.php language file.

// charset for web pages and emails
define('CHARSET', 'iso-8859-1');

Can I change it to latin1?

iso-8859-1 and Latin1 are the same - no need to change anything.

Vger

Just for the benefit of others, the reference above to

Quote:

---

It will be in the bottom of admin/includes/init_templates.php in the next release.

---

by DrByte should probably be
/admin/includes/init_includes/init_templates.php, as that's where I found it.

CHARSET appears to be set in includes/languages/english.php and /admin/includes/languages/english.php:

Code:

---

// charset for web pages and emails
define('CHARSET', 'iso-8859-1');

---

I'm having a similar issue. I am migrating hosting companies and upgrading from 1.2.7 to 1.3.6 all at the same time. The shop is primarily English with some Japanese, and I prefer to use UTF for encoding.

i've set the CHARSET define in both admin and catalog areaa to UTF-8. And the pages seem to be loading as such. I've uploaded the DB from the old hosting company to the new in UTF-8. Japanese characters show up in phpmyadmin correctly in new hosting location.

However, I get the customary ????? in both admin and catalog areas.

Stats on new install:

Database: MySQL 5.0.18-standard-log
Server Date: 12/28/2006 11:56:15
Database Date: 12/28/2006 11:56:15

PHP Version: 4.4.4 (Zend: 1.3.0)
HTTP Server: Apache/2.0.54 (Unix) PHP/4.4.2 mod_ssl/2.0.54 OpenSSL/0.9.7e mod_fastcgi/2.4.2 DAV/2 SVN/1.1.4

I have a short window to do this upgrade and transfer over the new year's holiday, so any help would be appreciated. Thanks in advance!

What's the MySQL and phpMyAdmin Default Character Set?

Vger

Default Char set for MySQL and phpMyAdmin are utf8_unicode_ci.

I've also tried utf8_bin, but that didn't seem to have an effect.

I've entered a new test product on the new server with Japanese and English as a test. The Japanese shows up great in Admin and Catalog views, but is jumbled in phpMyAdmin. So clearly there is a mismatch of encoding somehow.

Am I going to have to re-enter all of the Japanese on my site?

I got this figured out for me. It doesn't make sense, but it works for now.

I saved the old DB and uploaded it to the new servers in Latin1. JPN doesn't display in phpMyAdmin, but works great on the catalog and in the admin. So I'm going with it for now.

Hi there

I am having trouble with the trademark special character.

It shows up perfectly on my local machine in admin and catalog views. But when I try and import/export to my live server I don't get to see the special characters. After the import to the live database, if I check the name and descriptions, I can see the correct trademark special character.

Can anybody help? I've read through this thread but can't make out what exactly it is I need to do.

I am using version 1.3.7 of ZC.

Thanks

This is most likely a problem with the MySQL Character Collation being utf8 on your online server, and the sql file you uploaded being en-iso-8859-1 collation.

This is a very tricky subject and not easily answered in a forum post. Your best bet is to Google for "database problems with utf8 and en-iso-8859-1" or similar.

Vger

Quote:

---

Originally Posted by Vger

This is most likely a problem with the MySQL Character Collation being utf8 on your online server, and the sql file you uploaded being en-iso-8859-1 collation.

This is a very tricky subject and not easily answered in a forum post. Your best bet is to Google for "database problems with utf8 and en-iso-8859-1" or similar.

Vger

---

First of all, what should the collation be? And is it possible for me to change the collation of the online database via phpMyAdmin?

I will try to explain the problems reported in this post as well as I possibly can. Not being a PHP expert, or MySQL expert, or Zen Cart expert, it is quite possible that I will get something in this explanation wrong. If I do, I ask the true experts to feel free to correct me (I will not be offended).

I must admit, that despite my love of Zen Cart, I find it incredibly disappointing that support for unicode and international characters still has not been properly implemented by the wonderful Zen team.

I have come across this same problem before, with other php scripts, trying to implement appropriate support for UTF-8 (many of my sites are in Esperanto, the international language, and I need to be able to type accented characters in several different languages). I need to be able to display these character appropriately both in the functional side of the site, as well as on the admin side. Being an administrator, it is also important for me to be able to use the phpMyAdmin interface to transfer data into and out of my databases from time to time. So, my characters need to show well in ALL interfaces.

I guess the first thing people have to understand, is that there are 3 separate components which are involved:

1) INTERFACE: the html interface which presents text to, and gathers text from the user, in their browser

2) DATABASE: the MySQL database that stores the text

3) CONTROLLER: the php script which moves the text between the database and the interface.

-------------------------
SOLUTION
-------------------------
What we all want, is to change the 3 parts of the system so that the text encoding used throughout is 'utf-8' (which includes just about every alphabet and character set in the world). Changing your entire system to using unicode (utf-8) is better than choosing a specific character set (such as 'english', 'chinese', 'japanese', 'cyrillic' or 'icelandic'), because utf-8 includes *all* these characters and more. This means that your shop would be able to natively support all these languages - should you choose to add any of them in the future!

1) INTERFACE:
When you install Zen Cart, by default all its pages, both in the catalog and in the admin side, are encoded as "English", "iso-8859-1". In order to change the interface to 'utf-8', you must change the /includes/languages/english.php file. Specifically, you have to change the line that says:

define('CHARSET', 'iso-8859-1');

to

define('CHARSET', 'utf-8');


If your shop is already using other languages, you have to make this same change for all installed languages, not just for the default 'english.php'.

This changes the default interface language of the catalogue. But to change the interface of the admin, you have to make exactly the same change in the equivalent admin file:

/admin/includes/languages/english.php (and also in any other language files you have installed there, too).

2) DATABASE:
This is a *really* troublesome step - it is long, time-consuming, tiresome, and requires a lot of patience.

When Zen Cart installed itself, it created default tables in the MySQL database you had specified. These tables, however, by default are not expecting to be holding utf-8 text. So, if you send utf-8 text into these tables, the tables will not store the text 'properly' (ie., it will not display appropriately in phpMyAdmin), because the database itself thinks the text is something else.

In MySQL, you can separately specify the encoding for the *entire database*, for *each table*, as well as for *each text field* inside a table. The encoding, in MySQL jargon, is called the 'collation'.

So, in order to modify your entire table, so that it is all utf-8, you have to use phpMyAdmin to:

a) open each table in the database, and change the collation of each text field in each table to 'utf8_generic_ci'
b) change the collation of each *table* to "utf8_generic_ci'

>>Request to the Zen Car team: it would be nice for someone to create an sql script to do this automatically for us. In future versions of the installer, it would be nice to automatically do every CREATE TABLE with "DEFAULT CHARACTER SET utf8" at the end, if the user has MySQL >= 4.1.

3) CONTROLLER:
So, your interface is now fully utf-8, and your database is fully utf-8. So everything must be right... Unfortunately, there is a third component in our system, which is the php script which transfer text (via sql queries) from/to the user's browser and the database.

The php script does not know that the queries you are making use utf-8 text. So, when you type something international into your utf-8 interface, the script grabs it, and then sends it *with the wrong encoding* to the database. The database accepts what the script is sending, and re-encodes it in utf-8. So, the result is, that when you look at your data in phpMyAdmin, it all looks garbled.

When you do a search in your catalogue, the script retrieves information from the database, and sends it to the browser. The browser, however, displays the text it is receiving as 'utf-8' (which it actually is), so it actually displays correctly (even though your database data is a mess).

The only way to fix this is to explicitly TELL the script to use 'utf-8' when talking to the database. To do this, open the following file:

/includes/classes/db/mysql/query_factory.php

Around line 37 you will see:

if (@mysql_select_db($zf_database, $this->link)) {
$this->db_connected = true;
return true;

Add the following text in between the first and second lines, so that you end up with:

if (@mysql_select_db($zf_database, $this->link)) {
if (version_compare(mysql_get_server_info(), '4.1.0', '>=')) {
mysql_query('SET NAMES "utf8"', $this->link);}
$this->db_connected = true;
return true;

>>Request to Zen Cart team: further down in the same document, there is another function ('selectdb()'), which uses 'mysql_select_db'. I searched through every code file in my shop, however, and could not find any calls to this function at all. Is this old code that should be removed? If not, then the 'SET NAMES' query shown above may need to be added here, too!

---------------------------------------------
DISCLAIMER
---------------------------------------------
You must back up your entire setup before attempting to change any of the files in your site.

I've tested the following patches in my own shop, and it works perfectly. This does not, however, means that it would work fine for yours.

Once again I emphasize, that I am not an expert, and this is a reasonably simple solution I've found, which works with every single language I've tested. It does not mean, however, that it is the BEST or most efficient solution. If you are an expert, and you can improve on these instructions, please do so!

---------------------------------------------
REQUEST TO ZEN CART TEAM
---------------------------------------------
International character support is a very critical issue for many of us using Zen Cart. I suspect that there will be a lot of people looking for the solution I described here, and so I ask you to please post it elsewhere, or include it in an FAQ or manual, as you see fit.

Last of all, I ask you to include the changes described here in the next main release of Zen Cart. As you can see, providing a fully compatible level of utf-8 support is not difficult, and I'm sure you can improve the code, and make it even more backwards compatible (if need be!).

Once again, thank you so much for such a fabulous product. I hope this little contribution will help Zen Cart continue to progress and thrive.

All that you need to do to make the mysql work as utf-8 is to open it in a programme set to utf-8 by default (e.g. PS Pad), and once it is opened click 'Save'.

In addition to setting the default encoding to utf-8 in the language files you also need to do the same as above for all files, because, normally, whenever you open them in a text editor of some sort they are opened as 'Windows' type files in your browser - assuming you use a Windows operating system.
Then phpMyAdmin has to be set to utf-8, ditto php.ini, ditto MySQL on the server.

Actually en-iso-8859-1 covers most accented characters, because it is a European-wide language. Admittedly it does not cover Cyrilic, Japanese or Chinese characters etc. but then neither does utf-8.

Vger

@Vger: your information may not be correct.

Japanese, Chinese, Greek, Cyrillic and most other writing systems of the world are covered by UTF-8.

The default ISO encoding you mention does *not* include most European characters. It does not include even most accented latin characters - European characters such as the Esperanto characters, are excluded.

Last of all, the solution you suggest does *not* enable full-use of international characters. Sure, if all someone wants is for their shop to 'look ok', then depending on your backend server configuration, the simple solution found in the FAQ (just changing the charset) will be sufficient. However, there are many of us who need the product to *really* support international characters. In my case, for instance, I need to be able to see/export/import my data using phpMyAdmin. With the previously provided solution, the information in phpMyAdmin was unusable, even if correct.

By providing full support to UTF-8, the Zen Cart team will not be excluding English or English speakers, but will indeed be making the software even more usable to a wider audience around the world.

The reason that "Esperanto" characters are not covered by European character encoding is because it is not a real language. It was a good idea which never took off.

I'm not knocking utf8 - it's also a very good idea. The problem is not with its implementation for new websites. The problem is with the trillions of websites on servers that formerly used en-iso-8859-1 (Latin1) which have upgraded MySQL and PHP to a level where utf8 is now the only option.

I've spent dozens of hours on this and there is no easy fix for it. Presently I am working on a site coming from a server using en-iso-8859-1 to a server using utf8 - with one site language being in Arabic. It works fine on en-iso-8859-1 and is totally screwed on utf8.

I just wish that the nerds who thought this great idea up had a small thought along the way for the implications of their wonderful idea.

Vger

Vger, the explanation I've given above might account for the problems you have in your new server with Arabic pages. If you read the instructions, you will see that changing everything over to utf-8 is actually not a big hassle - the code changes are actually *minimal*.

I recently had to convert all my sites from an old server (without utf-8), to a new server (utf-8). I had to apply the patch I described above (setting all script queries to use "SET NAMES 'utf-8"), and then usually, all I had to do was download an sql dump of the database in question, open it in a text processor, convert the text to utf-8, resave it, and reload it. I was able to change ALL of my sites over in just a couple of days, and never lost any data. Mind you, I'm not a professional programmer, or expert, and that is all it took me to convert ALL of my scripts.

Apart from Zen Cart, I use other open-source scripts (like the excellent "Vanilla" forum, or ExtCalendar, for a community calendar application). All the scripts suffered from the same lack of *real* support to utf-8. This lack of support comes not from any real difficulty in implementing the solution (as you can see, the solution is simple enough, that even I can implement it), but rather from lack of understanding on the programmer's behalf of internationalisation issues, and utf-8.

Using utf-8 is of GREAT benefit, as once it is implemented, you basically do not have to worry any longer about what language your users are using. In my forums, for instance, people can type in japanese, chinese, arabic, cyrillic, or any other language they choose, and the characters display perfectly well not only on the interface, but also in the database (via phpMyAdmin). As an administrator, this gives me fabulous freedom, to oversee and manage my data directly on the database, as I see fit.

As you said yourself, utf-8 is the way everything is going - for reasons that are obvious for those of us who use it. I thought that the Zen Cart team hadn't implemented it in full yet, because it might be too hard to do - but now I see that this may not be the case.

Last of all: Esperanto is indeed a living language, spoken in over 100 countries, with active associations all over the world, and an active community of speakers. I should know, because I've help put together a few web sites for the Australian Esperanto Association:

http://www.esperanto.org.au

http://forumoj.esperanto.org.au

http://aesk.esperanto.org.au

http://kalendaro.esperanto.org.au

http://libroservo.esperanto.org.au - using Zen Cart, but currently under renovations! Come back in a couple of weeks to see the new improved site!

You can find out more general information about Esperanto at the multi-lingual Esperanto information centre:

http://www.esperanto.net

Or at the site of the Universal Esperanto Association:

http://www.uea.org

Or try doing a quick (and free) fun online course at:

http://www.lernu.net

And good luck at implementing UTF-8 in your current sites. Even if it takes you a while to work it all out, from my own experience I can tell you that it will be worth it in the end, as once it's done, it will be the end of all your charset worries.

I knew when I said Esperanto wasn't a real language that you wouldn't be best pleased. But I'll stick by that all the same. It remains a 'hobby' language for a few dedicated followers, as far as myself and most of the rest of the world is concerned.

The whole idea of it was to unify the world by getting everyone to use Esperanto as a common second language. That did not happen, and never will.

Furthermore, there's no need to have Esperanto as a common second language for the world - because the world now has one, it's called English. Children in China, Japan and Russia learn English as their second language. Children in India and Pakistan who get an education learn English as a second language, because of their cultural heritage. Put those together with all those countries which use English as their first language and you've got most of the world's population covered. As for the French - well, they pass laws to try and stop the spread of the use of English. C est la vie!

Back on topic.

There's no doubt that utf8 is good for new websites and new servers. But it does create problems when transferring websites from older servers which use en-iso-8859-1 to new servers which use utf8. Web hosts like ourselves just have to deal with it on a day to day basis. I do actually look forward to the day when all sites are utf8 - it will make our life a whole lot simpler. It's the transition that's a real pain in the ****.

Vger

Vger, glad to see that you've come around to utf-8. Good for you.

As for your off-topic views on Esperanto, I just hope that eventually you will drop your English-centred view of the world. As an English speaker, it may seem to you as if the world already speaks English, but that is certainly not the case. Consider, for instance, that I've seen Brazilians who visited Australia, and commented to me that they should have never bothered to try and learn English, because "everyone in Australia speaks Portuguese". These same Brazilians tell me that Portuguese is now being taught in places as far as Universities in Korea and China (which apparently is true). Similarly, there are many Chinese living in Australia for years, who have never learnt any English, because everyone they relate to - even in shops - speaks Mandarin, Cantonese or even Hokkien. Indeed, I've heard more than one person say to me, that the most useful and international language they'd ever learnt was Mandarin, because it did not matter where they went in the world, there was *always* someone there who could speak it!

There are studies that confirm that now there are more speakers of Esperanto than many other national languages (like Icelandic, unfortunately). Esperanto is a great choice, not just because it has a true neutral, international culture, but mostly because it is easier to learn than any other language - you can be absolutely fluent in a year.

Esperanto also and has a helpful community of speakers all around the globe - which you don't get by learning any other language. Having myself through Esperanto been able to relate with Germans, Americans, Japanese, Koreans, British, Brazilians, Indonesians, Chinese, Zimbabwans, Australians, Slovakians, Dutch, Russians, Italians, and many other nationalities, I must tell you that the experience of developing a relationship with them in Esperanto is a lot different to doing it in English (or struggling to try and use their national language). Esperanto allows us to relate in an equal level, and to help each other equally cross the cultural divide - rather than expecting that the other will do all the effort.

The Esperanto community is wide, and vibrant, and has a fascinating history and culture. By making derogatory comments about the language, you may unwillingly be offending the many Esperanto musicians, authors, scientists and journalists who help each other around the world.

Take some time, and have a look at some of the links about Esperanto in my previous post. They are a good starting point for you, and may open doors for you which you didn't even know were there to be opened!

Hello there.

Thanks to icouto for the excellent post explaining how to move the whole shop to UTF8

I managed to do it thanks to that guide, then it is my turn to give something back to the community,..
Get the file from darthnet40.com/zencart1.3.7-UTF8.zip

This file is ONLY FOR A NEW INSTALLATION (It creates the correct UTF8 Collation and character sets in the new tables).

[U]To use it, follow the steps:
1) Copy the whole Zencart distribution on your server
2) Put the files i provide on your server, overwriting the default ZC ones
3) Run setup as usual
4) Enjoy

I have this some problem when I have to move a shop from one to another server. A lot of charaters are messed up.

The sql file Moullas posted takes only care of a new installation. How can I import an database form a running shop and not run into trouble. Is there a sql file for to run over an existing imported database?

I hope you will understand my question, my native speech is Dutch.

Ok, i don't know if this helps.
The command you have to run is
ALTER TABLE TableName DEFAULT CHARACTER SET utf8 COLLATE utf8_general_ci;


This changes the collation of the Table. However, i think the fields will stay to their former values. So you'll have to get all your table names and run the command from PHPMyAdmin to change the Table collation (again, time consuming).

This page looks like it could help a bit more, however my PHP and MySQL skills are non-existent, i don't know if someone could make a script out of this to convert an existing structure

http://www.phpwact.org/php/i18n/utf-8/mysql

:clap:WOW! icouto, I owe you BIG TIME! Your solution WORKS PERFECTLY! (see icouto's solution below):clap:

Background on my issue, which was special characters not displaying properly:

I have been running a Zen Cart installation at http://www.GermanFoodatJosies.com for just over 2 years now. We set everything up in English originally thinking it would save time and resources. The store works great and has been all this time, but lately I've really been putting a lot of extra time into it in an effort to take advantage of a lot more of the features such as Manufacturers, and now Languages.

All of our product descriptions were in English for the most part, and we have been going through the products and trying to add the German description in below the English one. The umlauts and special characters always worked for the English site.

I downloaded and installed the German Language pack yesterday and everything worked great right away, for the most part. I had to go in and redo page 1, page 2, and page 3, and also translated my privacy policy, shipping & other information in the info box because they weren't set up at all for German. I'm not sure why the installation required me to recreate those pages, but it wasn't too much trouble with Google Translate.

After all those were set up, I also had to then manually put in the category names in German, because they didn't translate either. No big deal, just a little time consuming.

Now all of the Sidebox headings translated over fine, just like the home and login links.

Everything is working great now at this point, except for one more thing, and this is where icouto saves the day for me.

I spent hours researching this everywhere, btw, and somehow finally found this thread just when I was about to give up. I will go back to the other threads and post icouto's answer so they're all informed as well.

Ok, so back to the final problem, and that was the product descriptions. As I said, after a few manual translations, everything for the most part was working great but the product descriptions - for some reason, the special characters worked fine on the English side that also had German language in it, but on the German side, the special characters were inverted question marks (not working).

I followed icouto's steps and now everything is working perfectly. Actually, I did one thing differently which saved a TON of time - since only my product descriptions needed fixed, I only edited the "products description" table and the text fields within it using pHpMyAdmin. Once inside there, under "products description", I edited the collation value to utf8_general_ci. There was no need for me to go in and edit all of the other tables.

Also, make sure you do step 3, because 1 & 2 will not do it alone.

Thanks again, icouto.

***please note, icouto's instructions say to change it to utf8_generic_ci - I did not see a "generic" and assumed that he meant "general", which is working fine for me.

OK. I followed all the instructions to have everything with utf-8. All my input in admin and in website with special characters registers fine and is accessed fine from DB but now all data requested from browser to php files like "privacy notice" comes out garbled when it came out fine with iso-8859-1.

Can anyone help?

Thanks a lot to icouto!!!

I have done exactly how you described and my product and admin pages now show German characters perfectly! German special characters are also saved correctly in the database.

But there are still 3 issues I have been trying to solve for days now (I am using ZC v1.3.8):

----------------------------------------------------------------------------
1) PROBLEM WITH DATABASE BACKUP / RESTORE

When I produce a backup of my database and restore the backup using phpMyAdmin, I see via phpMyAdmin that special characters are now saved wrongly in the database (e.g. in the initial database I had 'Zürich' but in the restored db it is written 'Zürich') and therefore special characters are also displayed wrongly in the shop interface. Any idea how I can solve this issue?

----------------------------------------------------------------------------
2) PROBLEM HANDING OVER SPECIAL CHARACTERS TO PAYPAL IPN

And this is now a very mean thing where I am completely lost: I want my clients to use PayPal IPN - Website Payments Standard (my shop uses the preinstalled module). But the thing is that whenever a client clicks on the confirmation button and is redirected to the PayPal site, the special characters of the client's address in the prefilled PayPal form are displayed wrongly (this time, 'Zürich' is not shown as 'Zürich' but instead the 'ü' is replaced by a small questionmark in a black square).

I checked that in the PayPal Admin panel, profile->language encoding->more options->encoding is set to UTF-8. So I would think that throughout Zen Cart and PayPal I am using UTF-8.

I next deleted the code line $s = preg_replace ('/&([a-zA-Z])(uml|acute|elig|grave|circ|tilde|cedil|ring|quest|slash|caron);/', '$1', $s); from the function replace_accents($s) in the file includes/functions/functions_general.php.

This actually seemed to solve my problem: Now 'Zürich' was displayed correctly as 'Zürich' in the PayPal address form.

----------------------------------------------------------------------------
3) UNSUPPORTED CHARACTERS MESSAGE IN PAYPAL

When I thought that I had solved issue 2), issue 3) poped up immediately: Now PayPal is displaying everything nicely and correctly and clients are also able to process the payment with PayPal, but unfortunately they are displayed the following message in red letters: "You have entered unsupported characters for this field. Current available language character types are: European, Chinese, Korean, Japanese, and Thai. Please try again".
I am now completely lost. I have read that this issue should have been solved years ago in an earlier version of Zen Cart!?!?! Furthermore, PayPal support told me that I am their only client facing such an issue and they would not be able to help me.
-> Does anyone have a clue how I could end my misery????


I am using the following settings in the PayPal IPN module:
Enable PayPal Module
True

Business ID
[email protected]

PDT Token (Payment Data Transfer)


Transaction Currency
Selected Currency

Payment Zone
--none--

Set Pending Notification Status
Pending [1]

Set Order Status
Processing [2]

Set Refund Order Status
Pending [1]

Sort order of display.
1

Page Style
iSupply

Mode for PayPal web services

Default:
www.paypal.com/cgi-bin/webscr
or
www.paypal.com/us/cgi-bin/webscr
or for the UK,
www.paypal.com/uk/cgi-bin/webscr
www.paypal.com/cgi-bin/webscr

Debug Mode
Off

Yesssss!!!!!!!!!!!!! I have got the solution!!!

I have tested the following browsers and found out that only Chrome shows the error message! Seems that Chrome has a bug here.

Crome 5.0 Error
Firefox 3.6 OK
IE 8.0, 7.0 OK
Safari 4.0 OK
iPhone/Safari OK


I am actually a little bit surprised that no one else seems to have spotted this bug. Could anyone of you shorty test your PayPal IPN integration with Chrome and let us know whether you are experiencing the same problems?

Quote:

---

Originally Posted by ivanbr

OK. I followed all the instructions to have everything with utf-8. All my input in admin and in website with special characters registers fine and is accessed fine from DB but now all data requested from browser to php files like "privacy notice" comes out garbled when it came out fine with iso-8859-1.

Can anyone help?

---

Open /includes/languages/YOUR_TEMPLATE/english.php

Around lines 56-57 you find

Quote:

---

// charset for web pages and emails
define('CHARSET', 'utf-8');

---

set to

Quote:

---

// charset for web pages and emails
define('CHARSET', 'iso-8859-1');

---

Now do the same in admin/includes/languages/english.php around lines 65-66

No more 'funny' characters :P