descriptions truncating on degree symbol while importing from csv

[makenoiz]

We are importing our products, not using an editor so text is going directly to db from a csv file. Our descriptions are getting truncated at the degree symbol. Is this a character set incompatibility. If so, which character set should I be on for 1.5.1? And is there a tutorial for converting the entire site over to one character set or another.

Basically 3 questions here..

1) Why truncations on the degree symbol.
2) Which char set for 1.5.1
3) Is there tutorial for converting character set completely incl db.

Thanks

[lhungil]

... 2) Which char set for 1.5.1 ...

Default (not upgraded) uses what was specified when the system was installed... Upgrades may have mixed character sets in the database...

For example with a clean install if UTF8 was selected during the install for the database... Zen Cart uses UTF8 for the database tables... UTF8 for the database communication... and UTF8 for the English language.

You can find what the database tables are using by browsing with PhpMyAdmin...

You can find what your Zen Cart installation is using by looking at a couple main locations... DB_CHARSET in "/includes/configure.php" should contain the database communication (and hopefully match your database tables). CHARSET in "/includes/languages/English.php" should contain the character set used for the generated website pages.

... 3) Is there tutorial for converting character set completely incl db. ...

Not exactly what you are looking for... But there is a tool in the plugins section for converting the database tables (and data) from ISO-Latin-1 to UTF8... You might also find some of the information in this post useful.

[torvista]

what are you using to create the csv?..I only use Libre/Openoffice as you can select the encoding when you open and save csv (ie easy conversions).
I don't think you can be sure with excel so you may be introducing an unknown there.

[makenoiz]

Thank you both for your relies. Greatly appreciated. I will look at the links
lhungil sent . And torvista, I tried openoffice last year. its was soooo slow that I had to get rid of it. My patience could not take it. Maybe Ill try it again. I know excel has issues in that area but I have yet to find a better alternative. Thanks for the reminder about open office... Ill give it another shot...

[makenoiz]

Im going to update this thread as I proceed in the event it can help any one else.

My config files do show UTF8 and my Lang shows UTF-8

admin/includes/configure.php = define('DB_CHARSET', 'utf8');
/includes/configure.php = define('DB_CHARSET', 'utf8');

as does languages/english.php but its spelled differently not sure if this is important or not.
includes/languages/english.php = define('CHARSET', 'utf-8');

DB tables are mixed with utf8 and latin1_general, latin1_sweedish and only 4 tables (the admin tables) are utf8_general.

I will proceed by running Dr. Bytes DB Char set conversion tool here http://www.zen-cart.com/downloads.php?do=file&id=1280 on my development test site and see what happens! Ill post back to let you know results......... stay tuned............

[makenoiz]

Ran the conversion program and all tables and db are now utf8 but if I add degree symbol directly into the database products_descriptions I get this: ° instead of this ° for the degree symbol.

so while the db shows 360° turn the front end browser shows 360° turn

.... ideas?

Thanks

[makenoiz]

Ok found the solution to the browser display problems. I have an override english.php file that still was using define('CHARSET', 'iso-8859-1'); so I changed to
define('CHARSET', 'utf-8');

Browser shows degree sign now IF I add the degree sign manually into the database. it appears that my problem is with my importer truncating on the degree sign when Im importing into the database in the first place. Im going to dig a little deeper.

[kobra]

When entering the degree code - switch to the plain text editor if you are not using it currently
and use the html escape for degree (remove spaces)
& d e g ;

See if this helps