A few UTF8 questions

Hello,

I'm in the process of upgrading from 1.3.7 to 1.5.0. The method I choose is to build a brand new site on the files side and keeping database on the other.
Running the install script did ask me to upgrade the database, to be more secure I've made it step by step, i.e. 1.3.7 to 1.3.8 then 1.3.8 to 1.3.9 then 1.3.9 to 1.5.0.

I get a few error messages but I don't think they're important as they're all SKIPPED messages.

My site is offering four languages, english, spanish, french and italian. Of course the brand new 1.5.0 file set doesn't include three of them so the admin site is warning me about the lack of language files.

1st ) as far as my text editor tells me, all the 1.5.0 files are UTF-8. Does all the files have to be UTF-8 encoded ?
2) is the 1.5.0 database UTF-8 ?
I saw a convert_db2utf8 script somewhere, do I have to run it ? Is it a good idea to run it ? Is it better to not do that or even is the 1.5.0 converted database already UTF-8 converted ?
3) Does a 1.5.0 file set (UTF-8 encoded) have to run against a UTF-8 database ? If it's not te case am I at risk ?
4) spanish, french and italian file sets are not 1.5.0 compatible. Can I still install the 1.3.9 version of these file set (waiting for the final versions) ?

If the language file sets are very very different from 1.3.9 to 1.5.0 is there any Unix tool able to build a glossary from the previous version comparison and use this glossary to help us make a new file set. I mean some kind of software able to
1-learn translations from the difference between 1.3.9 english and 1.3.9 foreign file sets,
2-use this knowledge to build a foreign 1.5.0 file set from the 1.5.0 english originals

Thanks for reading and answering

regards

Hubert

Remember your site html pages are created from the data in your database + the languages files, and headed with a line that says to the browser "the text in this page is utf8".
So the browser will expect everything to be utf8. Anything that is not, and has an accent, will be garbled.

1. as far as my text editor tells me, all the 1.5.0 files are UTF-8. Does all the files have to be UTF-8 encoded ?In theory only the language files as the encoding bit only comes into play for the data it manipulates that includes extended ascii characters (ones with accents). So .php files no.
   If you have some hardcoded text somewhere in a .php file, change it to use a constant.

2. is the 1.5.0 database UTF-8 ?
   I saw a convert_db2utf8 script somewhere, do I have to run it ? Is it a good idea to run it ? Is it better to not do that or even is the 1.5.0 converted database already UTF-8 converted ?I don't think the upgrade script auto converts (although it should give the option IMHO).
   So yes you have to convert.
   Until you convert you will see garbled chars in the displayed database data.

3. Does a 1.5.0 file set (UTF-8 encoded) have to run against a UTF-8 database ? If it's not te case am I at risk ?1.5 by default will read and write to the database using utf-8. If you let it do this for a while and then convert, I think the newly entered data will get garbled as it gets converted unnecessarily, but I could be wrong.

4. spanish, french and italian file sets are not 1.5.0 compatible.Spanish is done.

Can I still install the 1.3.9 version of these file set (waiting for the final versions) ?Yes, but you will have to bulk convert them to UTF8 for the reasons I mentioned and then hunt down the missing constants.

In addition the database collation should also be set to UTF8-general.

https://www.zen-cart.com/tutorials/index.php?article=414

Quoting the referenced tutorial "To do this, you'll need to edit your upgraded english.php file to use the old CHARSET and setlocale settings, as well as edit your configure.php file to set DB_CHARSET to 'latin1' instead of 'utf8'.
That is, KEEP the old references to iso-8859-1 INSTEAD of utf8."

The english.php is straight forward. I can't find any reference to DB_CHARSET in the configure.php files anywhere. The forums keep referencign the configuration.php files and set DB_CHARSET to "latin1".

I haven't been able to find this reference. What am I missing?

bwtharp:

The english.php is straight forward. I can't find any reference to DB_CHARSET in the configure.php files anywhere. The forums keep referencign the configuration.php files and set DB_CHARSET to "latin1".

I haven't been able to find this reference. What am I missing?
Evidently you're using an older version of Zen Cart that doesn't have those references in those files.

... which begs the question: why are you fussing with charsets at all. As a general rule, they shouldn't be touched on a live site.

DrByte:

Evidently you're using an older version of Zen Cart that doesn't have those references in those files.
Just out of curiosity I checked my clean, untouched ZC 1.5.0 files. There is no DB_CHARSET in either of the dist-configure.php files. Is it inserted during the install process?

Rob

The dist-configure.php files are only example templates. They're not actually the files that are used by the install process. They're basically just placeholders with dummy data because if we gave empty files everyone would be confused and complain that the files are empty.
The correct way to generate configure.php files is using zc_install via the official installation procedure as described in all the documentation including the FAQs, the /docs/ and the Implementation Guide (also found in the /docs/ folder).

I'm using a new install of 1.5, and did use the authorized install method. I'm fooling with the files because the upgrade resulted in odd characters appearing and my admin page showing a blank description file when updating product.

Again, as stated earlier in this discussion: https://www.zen-cart.com/tutorials/index.php?article=414