EasyPopulate 4.0 Support Thread

[mc12345678]

Thanks, DrByte. :)@mc, I looked at the file and it seems changeable. Only, this line stumped me:$zco_notifier->notify('EP4_COLLATION_UTF8_ZC13X');Any chance you could take a look at the code and specify the changes needed?I also found UTF-8 references (in notation) in two other EP4 files. Can that be ignored?

That particular notifier that stumped you was added when notifiers were added to support expansion of the capability to support the bookx mod that was being developed by another zenner. It is only in "play" when being observed.Also, thank you DrByte for tieing the conversation together and not requiring reiteration. In my previous post, I HAD focussed on the database. After posting I thought about the files and other language that might have been also affected and at that time figured I would have to address it (but it was covered in that thread).Yes, can take a quick gander in a litle bit, but I think considering the situation that the change in the main file will correct the issue for this site.

[Feznizzle]

Thanks, MC!

Would these changes do the trick?

if ($ep_uses_mysqli) {
  $collation = mysqli_character_set_name($db->link); // should be either latin1 or utf8
} else {
  $collation = mysql_client_encoding(); // should be either latin1 or utf8
}
if ($collation == 'latin1') {
  if (function_exists('mb_internal_encoding')) {
    mb_internal_encoding("iso-8859-1");
  }
}

if (($collation == 'latin1') && ((substr($project, 0, 5) == "1.3.8") || (substr($project, 0, 5) == "1.3.9"))) {
  //mb_internal_encoding("iso-8859-1");
  $category_strlen_max = $category_strlen_max / 3;
  $categories_name_max_len = $categories_name_max_len / 3;
  $manufacturers_name_max_len = $manufacturers_name_max_len / 3;
  $products_model_max_len = $products_model_max_len / 3;
  $products_name_max_len = $products_name_max_len / 3;
  $products_url_max_len = $products_url_max_len / 3;
  $artists_name_max_len = $artists_name_max_len / 3;
  $record_company_name_max_len = $record_company_name_max_len / 3;
  $music_genre_name_max_len = $music_genre_name_max_len / 3;

  $zco_notifier->notify('EP4_COLLATION_UTF8_ZC13X');

}

Also, the reason I highlighted the reference to 1.3.9 is that the site I'm working on is a 1.3.9h. Should I change that as well?

Thanks again!

[mc12345678]

Thanks, MC!

Would these changes do the trick?

if ($ep_uses_mysqli) {
  $collation = mysqli_character_set_name($db->link); // should be either latin1 or utf8
} else {
  $collation = mysql_client_encoding(); // should be either latin1 or utf8
}
if ($collation == 'latin1') {
  if (function_exists('mb_internal_encoding')) {
    mb_internal_encoding("iso-8859-1");
  }
}

if (($collation == 'latin1') && ((substr($project, 0, 5) == "1.3.8") || (substr($project, 0, 5) == "1.3.9"))) {
  //mb_internal_encoding("iso-8859-1");
  $category_strlen_max = $category_strlen_max / 3;
  $categories_name_max_len = $categories_name_max_len / 3;
  $manufacturers_name_max_len = $manufacturers_name_max_len / 3;
  $products_model_max_len = $products_model_max_len / 3;
  $products_name_max_len = $products_name_max_len / 3;
  $products_url_max_len = $products_url_max_len / 3;
  $artists_name_max_len = $artists_name_max_len / 3;
  $record_company_name_max_len = $record_company_name_max_len / 3;
  $music_genre_name_max_len = $music_genre_name_max_len / 3;

  $zco_notifier->notify('EP4_COLLATION_UTF8_ZC13X');

}

Also, the reason I highlighted the reference to 1.3.9 is that the site I'm working on is a 1.3.9h. Should I change that as well?

Thanks again!

Couple of things. Let's start with what looks like a little bit of a history lesson on ZC (based on some independent review of old ZC code and system response to by a ZC 1.5.1 site) and why some of the above code is present. It would appear that back "in the day" (ie. before ZC 1.5.0), that the includes/configure.php and admin/includes/configure.php did not include a DB_CHARSET setting. Therefore part of the interface with the database was to respond/identify whatever characterset was otherwise associated or set in the database itself. A little bit about that though means that if the database responded that it was using UTF-8, then the maximum field length report for the associated fields would be reported related to UTF-8 sizing but expect that the software wouldn't be able to properly encode the characterset back into "multibyte" related utf8 (or in other words identify that there are more "letters" available than what the field would actually be able to take, basically reporting back the size of the field to be 3 times what it would accept). Now most of this was already written before I came along and looking back in the history these changes were incorporated ever more to maintain support for older systems while continuing to develop or provide for newer systems. The test for 1.3.9 looks at the first 5 characters of the $project variable.. Regardless of which sub-version of ZC 1.3.9, the first five characters are 1.3.9, so there is no change needed in that particular portion of the code.

Basically all that to say, that in the second part of the code, you wouldn't want to reduce the size of the available fields if the $collation was identified as latin1, therefore that section would remain unchanged.

A "bigger" problem though in using the more recent software is that it appears that there is a response indicating that the database may be responding back with utf8... And that is causing the "conversion" that is occurring of unusual characters in the csv file (when viewed in iso-8859-1 format)... If the problem is visible that soon in, then I have a few recommendations to "test" or check, but yes generally it looks like changing the code in this area is all that is necessary. (There may be an issue with the character used for the category name separator, but one step at a time).

So, here's what I suggest first:

if ($ep_uses_mysqli) {
  $collation = mysqli_character_set_name($db->link); // should be either latin1 or utf8
} else {
  $collation = mysql_client_encoding(); // should be either latin1 or utf8
}

echo 'collation: ' . $collation . "\n";

if ($collation == 'utf8') {
  if (function_exists('mb_internal_encoding')) {
    mb_internal_encoding("UTF-8");
  }
}

if ($collation == 'latin1') {
  if (function_exists('mb_internal_encoding')) {
    mb_internal_encoding("iso-8859-1");
  }
}

if (($collation == 'utf8') && ((substr($project, 0, 5) == "1.3.8") || (substr($project, 0, 5) == "1.3.9"))) {
  //mb_internal_encoding("UTF-8");
  $category_strlen_max = $category_strlen_max / 3;
  $categories_name_max_len = $categories_name_max_len / 3;
  $manufacturers_name_max_len = $manufacturers_name_max_len / 3;
  $products_model_max_len = $products_model_max_len / 3;
  $products_name_max_len = $products_name_max_len / 3;
  $products_url_max_len = $products_url_max_len / 3;
  $artists_name_max_len = $artists_name_max_len / 3;
  $record_company_name_max_len = $record_company_name_max_len / 3;
  $music_genre_name_max_len = $music_genre_name_max_len / 3;

  $zco_notifier->notify('EP4_COLLATION_UTF8_ZC13X');

}

The first "addition" is present to see what the actual returned value is, because in order to accomplish what is desired it may be necessary to not let EP4 think that it should be able to communicate in utf8, but instead push/force latin1 style formatting as a result of the observed outcome. There are other factors to review/confirm as well in the setup of the store and associated collation information. Basically EP4 expects the store to be setup correctly.

After making the first above change, when viewing the EP4 screen you should see text above the standard ZC admin screen indicating either utf8 or latin1. I am expecting that it says latin1 based on all of the discussion.

Can you also indicate what is displayed on the EP4 admin screen associated with the screen item: 'Internal Character Encoding:'?

Does it provide a result or does it say something like: ''mb_internal_encoding not available"?

The next thing to do after reporting all of that information would be to attempt to export, review the data.

When successful, export something "small" like a single category (disabled with disabled product?) that may have a single product with a really long product name, then without opening the file, attempt to upload that one product back and see if there are length/truncation result(s) reported (not expected). Then also look at the product itself and see if/what got modified about it (Should be nothing). Again export the product, compare the two files to see what if any differences now exist (again expect nothing).
Then attempt to open a copy of one of the files, turn around and save the file, again remembering the encoding.
Upload and then import the file (should be no changes on site).
Again export and compare that export with one of the previous (again no differences expected).

You're in an "opportune" position to be able to test/validate the operation because of the setup involved.

[Feznizzle]

Yes, it says latin1.

Internal Character Encoding says ISO-8859-1, same as before.

When I exported, imported, and then re-exported, the only difference between the export files is that the date/time changed to 0000-00-00 00:00:00. I don't think anything was truncated.

But when I opened the file to make some changes, I found garbage right away. I chose a category containing products with ™ symbols in the name. The symbols were replaced with: ô

Uggg.

[mc12345678]

Yes, it says latin1.

Internal Character Encoding says ISO-8859-1, same as before.

When I exported, imported, and then re-exported, the only difference between the export files is that the date/time changed to 0000-00-00 00:00:00. I don't think anything was truncated.

But when I opened the file to make some changes, I found garbage right away. I chose a category containing products with ™ symbols in the name. The symbols were replaced with: ô

Uggg.

So, there were two files exported. First, did both present the same "garbage" second, what encoding/viewer was used to view the file(s)?

Unusual that the date got set to 0000-00-00 instead of the database preferred 0001-01-01 style, but if the site itself could export and import the exported file without any character encoding issues, then that means that EP4 is doing its part just fine. The problem is in the next "tool" which is reading, expecting or converting the file to be utf8 which is then causing the observed "garbage" ie. looking at latin1 in utf8 can cause characters to look as junk.

[Feznizzle]

Yes, garbage in both files. I use Open Office. Can I switch CharSets in there?

[mc12345678]

Yes, garbage in both files. I use Open Office. Can I switch CharSets in there?

Of course.

When opening the file, you should be presented with an option to import text. The first box is the character set. Change that from unicode (utf-8) to Western Europe (ISO-8859-1) then follow the remainder of the settings to support import. I provide this information based on Apache Oppenoffice 4.1.3. I haven't updated or attempted to update recently to identify if/what differences might exist in the options to reach various features. That aspect is something that should be addressed through the tool of use's help/assistance.

[Feznizzle]

Doh! Can't believe I missed that. Well, the garbage is gone... but so is the trademark symbol.

[mc12345678]

Doh! Can't believe I missed that. Well, the garbage is gone... but so is the trademark symbol.

Did it remain between import/export of the original file and only disappear when loaded into open office?

I found this which suggests you may want to also adjust the font style in the process: https://wiki.openoffice.org/wiki/Doc...mark_symbol%3F

Otherwise I thought that there might be some differences that may help by using Libre?

[Feznizzle]

Same thing occurs in Libre.