Some products' descriptions disappeared after migration and upgrade from 1.3.9 to 1.5

[delia]

There doesn't seem to be any rhyme or reason. Just some of the old products are affected and only in admin. They show fine on the site itself. I went thru the description of one of them and found the curly apostrophes instead of the regular ones and changed them just in case but it's made no difference. It just has to be something in the description that's preventing it from showing on the page.

[DrByte]

Sounds very much like the data in your database is in a format that doesn't match the character set used by your store or by your database-connection, thus the problems.

Remember, if you upgraded from v1.3.9 to v1.5.x and your 1.3.x database was NOT utf8 then you'll need to make sure your new site's language files are changed from utf8 to whatever character set they were using BEFORE the upgrade.

All explained here: http://www.zen-cart.com/content.php?...8859-1-to-utf8

[delia]

Well, thanks for that link again. I've got to start a spreadsheet of these links!

However, making those changes did not work. I am using the original configure files from 1.3.9 so I just added define('DB_CHARSET', 'latin1'); . It's the old english file as well and it does still have the line define('CHARSET', 'iso-8859-1'); and @setlocale(LC_TIME, 'en_US.ISO_8859-1');

When I ran into this before, I had started fresh with 1.5 and just upgraded the database so changing the 1.5 configure and english files worked for that one. I also remember the problems were more obvious and not so erratic as this.

[DrByte]

But you've determined that it's related to problems with the characters in the description, right?
Have you tried switching it back and forth to different character sets?
Are the myDebug errorlogs showing anything that PHp is complaining about?

[Limitless]

Delia,

I noticed this on a few products in my admin right after I added the 'additional product fields' mod and embedded a youtube video and/or attached a pdf.

It does not happen on all products (with or without the additional fields), and the data is visible within my Zen Cart Store Manager app (Windows Application) and visible on the catalog side, but all fields are blank on the admin side of things.

[delia]

yikes, there is a php error. It didn't stop the page so I didn't realize there was one.

[20-Sep-2012 08:10:41] PHP Warning: htmlspecialchars() [<a href='function.htmlspecialchars'>function.htmlspecialchars</a>]: Invalid multibyte sequence in argument in /home/ostermy/public_html/store/zenmin/includes/modules/product/collect_info.php on line 401

I did find your post about that and we are definitely then in the utf8 territory.

Limitless, the version of my product extra fields is not a 1.5 version. If you mean numinix, be sure you also have a 1.5 version. And find the correct forum thread to post this in.

[Limitless]

Ok, that's weird. I was 100% sure it said it was 1.5 compatible when I downloaded and installed it a few weeks ago.

Well, the plugin ~appears~ to work on 1.5, excluding random missing data in admin (not even sure if its related).

[delia]

I did the database conversion and that did kill off parts of the description. Looks like the client pasted in from word so we have curly apostrophes and the long dashes instead of hyphens. DrByte, what other character set would be a reasonable alternative? This is so not my area of expertise but I gotta learn about this stuff in more detail.

What if I go change just that one field? The choices are utf-8 bin, general and unicode. Your mod uses utf8_general_ci looks like. If I switch back and forth on that one field, can what disappears after the problem character reappear if I turn it back to the latin_general_ci?

The script has already changed the database collation to utf8 and that didn't change when I reimported his data. So I exported just the description field to be safe. Changed it to utf8_general_ci - description is still fully there on the website and lo and behold it also appeared in admin.

Yeah, I know a screwy way of doing this but it fixed his problem for the moment. Any dangers to this quick fix?

[DrByte]

If the data in your database has been fudged into being utf8 but your store is telling it to use iso88591 then I'm not surprised about the errors or any of the symptoms you mentioned.
That's probably the underlying root cause here: incorrectly changing stuff blindly.

If someone is going to use utf8, then it's best to use utf8 with everything. Or if they can't/won't/or don't-know-how to use utf8, then they should have just left it alone instead of changing it.

Summary: It sounds like your client has pushed you into having to convert everything. So, finish converting all the data to utf8 and update the language charset and db charset.

Make sure you test EVERYTHING to be sure that ALL the data in ALL the records in ALL the fields in ALL the tables is clean. Ya, that's a big job. But you don't wanna discover problems 6 months from now when you've got even more mixed-charset-data all over the place.

[delia]

I did do the conversion which then removed parts of those descriptions after the offending character on the website. So I reversed it. Clients tend to go ballistic when stuff disappears. Would have certainly preferred to do it correctly but he wasn't paying for me doing all that work on either. I told him to edit his descriptions, removing/changing the problem characters and when he was done, I'll do the utf8 full conversion.