EasyPopulate 4.0 Support Thread

eavinu:

I am having an issue with ZC 1.5.7c and EP 4.0.37.13.
I have tried to import categories from 1.3.9h in several languages and got the result "File uploaded Successfully: CategoryMeta-EP2021-08-24-121946.csv Issues with the CSV file delimiter(s)"
Lower in the EP page in the table it says "CSV Unknown Delimiter" and "Import Error".
So I tried to remove all the languages from the categories CSV and left only the main language which is EN and again tried to upload and I get the same result.
Have checked the file in OO and also in Notepad++ to make sure of the delimiter etc'.

Both ZC and EP are latest versions and on SG hosting.
Any ideas?
Is there a link in the upper right hand corner of the EP4 screen indicating to update/upgrade or just the remove/uninstall link? If update/upgrade is available select that first then review the condition of reporting about the CSV delimiter.

To help narrow down the issue(s), may I suggest the following test(s):
Create a file using one of the "export" options: Does the CSV Unknown Delimiter issue occur? If so, then it is a problem with the EP4 software.
The CSV file that has been generated, mentioned that using Open Office or similiar, is the file being stored (saved as and following additional action to ensure that the encoding is UTF-8) as a UTF-8 format file (not having a BOM)?
When viewing the CSV file, what are the first 4 lines of text? (These are used to evaluate the likelihood that the CSV delimiter has been properly applied/used.

mc12345678:

Is there a link in the upper right hand corner of the EP4 screen indicating to update/upgrade or just the remove/uninstall link? If update/upgrade is available select that first then review the condition of reporting about the CSV delimiter.

To help narrow down the issue(s), may I suggest the following test(s):
Create a file using one of the "export" options: Does the CSV Unknown Delimiter issue occur? If so, then it is a problem with the EP4 software.
The CSV file that has been generated, mentioned that using Open Office or similiar, is the file being stored (saved as and following additional action to ensure that the encoding is UTF-8) as a UTF-8 format file (not having a BOM)?
When viewing the CSV file, what are the first 4 lines of text? (These are used to evaluate the likelihood that the CSV delimiter has been properly applied/used.

I tried several versions of export and the same error occurs.
There is no update/upgrade, only uninstall.

First 4 rows look fine, the exported files are UTF-8 without BOM...

eavinu:

I tried several versions of export and the same error occurs.
There is no update/upgrade, only uninstall.

First 4 rows look fine, the exported files are UTF-8 without BOM...
To confirm, when say tried several versions of export and to be sure that we are each talking about the same thing (no assumptions) please identify what was done to perform the export(s). I know it sounds silly, but sometimes the difference/solution is in the communication.

So, what delimiter is identified in your configuration settings and what delimiter is being used? I have seen some systems where the export file includes a delimiter that was unexpected/unrequested. Also hate to say this, but the question was what do the first four rows include, as in please provide the first four rows for an independent review. While it may appear correct by one individual's review, extra eyes help find the issue.

Furthermore I'm not able to repeat the issue with 1.5.7 nor 1.5.8, though both are on the same server/host, one with php 7.4 the other 8.0.

mc12345678:

To confirm, when say tried several versions of export and to be sure that we are each talking about the same thing (no assumptions) please identify what was done to perform the export(s). I know it sounds silly, but sometimes the difference/solution is in the communication.

So, what delimiter is identified in your configuration settings and what delimiter is being used? I have seen some systems where the export file includes a delimiter that was unexpected/unrequested. Also hate to say this, but the question was what do the first four rows include, as in please provide the first four rows for an independent review. While it may appear correct by one individual's review, extra eyes help find the issue.

Furthermore I'm not able to repeat the issue with 1.5.7 nor 1.5.8, though both are on the same server/host, one with php 7.4 the other 8.0.

I failed to say that the DB is actually an import from 1.3.9h that is on Godaddy hosting (10 years old) and the installation is fresh 1.5.7c files and I did DB upgrade with zc_install of course.
I then saw an issue with the encoding for Hebrew, it was showing gibberish both in the DB and in the storefront while in the old website it showed gibberish in the DB but showed Hebrew in the storefront.
Then I started trying to resolve by only exporting and importing the categories_description table to see it it helps and still had gibberish instead of Hebrew.
I asked for assistance from Eran Ariel (eranariel) The Zenner that published the Hebrew UTF-8 ZC language pack and he suggested to try going with latest version of EP and then I installed EP4 on both installations.
After exporting from the old one and importing to the new one I realized this error.
I went over all the parameters and also inspected the exported categories file and everything looked okay.
Then I decided to check what happens if I export a file from the new website and try to import it back to the new website, I get the same error.
The error in the table shows after exporting and not only when importing.

To answer your questions, under "Category Export/Import Options" I chose Model/Category at first, later I also tried Categry with Meta Tags.
I also tried full export under Filterable Exports.
First 4 rows of category file:

v_products_model,v_categories_name_2,v_categories_name_3,v_categories_name_4,v_categories_name_5,v_categories_name_6,v_categories_name_7,v_categories_name_9,v_categories_name_10,v_categories_name_11,v_categories_name_12,v_categories_name_13,v_categories_name_1
"MG02","Round Herb Grinders^Metal grinders","Round Herb Grinders^Metal grinders","Round Herb Grinders^Grinders de metálico","Round Herb Grinders^Metal grinders","Round Herb Grinders^Metal grinders","Round Herb Grinders^Metal grinders","Round Herb Grinders^Metal grinders","Round Herb Grinders^Metal grinders","Round Herb Grinders^Metal grinders","Round Herb Grinders^Metal grinders","Round Herb Grinders^Metallmühlen","Round Herb Grinders^Metal grinders"
"MG08","Round Herb Grinders^Metal grinders","Round Herb Grinders^Metal grinders","Round Herb Grinders^Grinders de metálico","Round Herb Grinders^Metal grinders","Round Herb Grinders^Metal grinders","Round Herb Grinders^Metal grinders","Round Herb Grinders^Metal grinders","Round Herb Grinders^Metal grinders","Round Herb Grinders^Metal grinders","Round Herb Grinders^Metal grinders","Round Herb Grinders^Metallmühlen","Round Herb Grinders^Metal grinders"
"MG05","Round Herb Grinders^Metal grinders","Round Herb Grinders^Metal grinders","Round Herb Grinders^Grinders de metálico","Round Herb Grinders^Metal grinders","Round Herb Grinders^Metal grinders","Round Herb Grinders^Metal grinders","Round Herb Grinders^Metal grinders","Round Herb Grinders^Metal grinders","Round Herb Grinders^Metal grinders","Round Herb Grinders^Metal grinders","Round Herb Grinders^Metallmühlen","Round Herb Grinders^Metal grinders"

I just clicked now to export the file and I also have a new line added to the table below in the EP4 page that has this export file name and says CSV Unknown Delimiter and Import Error, I didn't try to upload and import this file...

That's the useful information that helps...

So a couple of things. One it would appear that probably need to convert the database to utf8/utf8mb4, though that may may/not correct the issue in total. Other things of "interest" would be the collation on the admin/catalog side as identified for the language being used at least on the admin side. E.g., is English being used for the admin or one of the other "foreign" languages? Another thing, is that as I recall (need to search a little more) I thought that with Hebrew, there was possibly some alternate effort needed. Not quite the same as Japanese or Russian, but similar... There is some guidance about what needed to be done differently within EP4 and/or one's server possibly 20 pages back? It was "recently" discussed and by recent in this case I mean within the last year...

It was discussed by an individual that has been busy updating/modifying code to support Japan so even looking up the Japanese language pack may find the individual and from there look at their posting history to see when/where posted to EP4 (or perhaps they are happily in receipt of this message and can pop in to provide some assistance though there is a "process" by which they may/may not see the message(s) right now.)

To me, what is curious though is that basically all languages use the "same" category names... I would have thought that there would have been more of a translation.

So, short summary, good to know that the database was "upgraded" using zc_install (hopefully did not run into issues that stopped updating of the configuration table which can be tested by attempting to update a configuration option of any setting, if there is a sql error then there is likely some additional "work" to be done/redone).
Next has to do with character encoding of the database and system. Then there is possibly some additional action to address languages; however, there is the discussion that creating a new CSV file has also exhibited a CSV delimiter issue... Again as an assumption, I believe the above provided data is from the CSV that was generated by EP4. The next thing to consider is if there is a "system" issue by generating a new CSV file directly (one line of data would be suggested in a single language) and attempting upload and import... Would suggest again posting the contents of that file here; however, this time please use the ```
[CODE][/CODE]

So I am happy to say that finally the Unknown CSV Delimiter issue was solved.
I don't know why I didn't try this before...
At first, since this was an upgraded website and had previous versions of EP (1.2....), there was a temp folder in the root and I left it there, changed in the EP4 configuration to false under Uploads directory Admin/Catalog.
Now I decided to move it to the Admin folder and changed the configuration to True and the Unknown CSV Delimiter indication turned to a "CSV ," indication.

So now I only need to deal with the other issue that I discovered today, going through checkout timesout... doesn't matter which shipping method or payment method is being chosen.
Hope to find the correct thread to get help with that LOL

Thanks a lot mc12345678 for your efforts.
Regards,
Eran

mc12345678:

To me, what is curious though is that basically all languages use the "same" category names... I would have thought that there would have been more of a translation.

So actually they have a lot of products and categories and many of them are not fully translated to all languages, this is actually why they decided to drop a few languages now so making my job a bit easier :)

eavinu:

So now I only need to deal with the other issue that I discovered today, going through checkout timesout... doesn't matter which shipping method or payment method is being chosen.
Hope to find the correct thread to get help with that LOL
Eran

So I can now report that I found the error for that one as well (checked Debug logs), the issue was sending emails so I cancelled the email sending options for now since there is no point in trying to configure that while still in SG staging area and not under the correct domain...

mc12345678:

"Easily" would be a matter of perspective; however, I can envision where the current "file read" code may be positioned to a function with some "trigger" being added to identify what the source is for the filetype and support properly moving through the file. While this operation is perhaps easier in newer PHP versions, it remains important to note that the process would work best by stepping into/through the file rather than loading it entirely into memory. The file reading occurs in a few of the modules as well as the main import file. There is also a field delimiter test that is performed against files that may need to be altered to exclude such xml files or provide an equivalent test if necessary.

Turns out on further checking i can actually make my own custom mapped feed at the suppliers website and in .csv format :). I've tried a few times. finally found how to add gtin as an option too and tried to upload again to get the following error messages "Missing primary key from file" and "File Import Completed with issues."

I've spent hours trying to solve this and even hired developers to install the module and get the feed working as i didn't fancy doing it myself. developers worked over the quoted time and between say 4 of us its getting frustrating lol. it's something simple being missed now i'm sure of it?

Anything that's commonly done that would cause such issues ie not including the "v_" beginning to product fields? i'm sure it was in this support thread i read your not supposed to include that part?

flappingfish:

Turns out on further checking i can actually make my own custom mapped feed at the suppliers website and in .csv format :). I've tried a few times. finally found how to add gtin as an option too and tried to upload again to get the following error messages "Missing primary key from file" and "File Import Completed with issues."

I've spent hours trying to solve this and even hired developers to install the module and get the feed working as i didn't fancy doing it myself. developers worked over the quoted time and between say 4 of us its getting frustrating lol. it's something simple being missed now i'm sure of it?

Anything that's commonly done that would cause such issues ie not including the "v_" beginning to product fields? i'm sure it was in this support thread i read your not supposed to include that part?
The message provided indicates that whatever is being used for your "unique record" is not present in the first line of the file... By default that field header is: v_products_model
By making changes in the configuration menu, it is possible to instead change to using: v_products_id
By way of code, there is an observer that can be used to select/use a completely different primary key.

Note that again, this "primary key" is what ties the data being imported to the record(s) in the database. Depending on what is used for the primary key, multiple database records may be affected. For example, if there was a word within the products description that was used as a "primary key", then every record that has that word would be impacted by importing a row of data with that word. This is why when using v_products_model if more than one product has that exact model designation that import of a record with that model designation will affect all such product.

As far as the reference to using/not using the prefix "v_", that is associated to identifying other product table fields that are to be used in the import/export. The place where the "v_" is omitted is within the configuration settings for the User Defined Products fields... When entering/adding such fields within the configuration menu, they should be added using whatever the field is actually called in the products table.

Perhaps the above helps? It may be of assistance to provide the configuration settings and other pertinent information to get to your solution.

mc12345678:

The message provided indicates that whatever is being used for your "unique record" is not present in the first line of the file... By default that field header is: v_products_model
By making changes in the configuration menu, it is possible to instead change to using: v_products_id
By way of code, there is an observer that can be used to select/use a completely different primary key.

Note that again, this "primary key" is what ties the data being imported to the record(s) in the database. Depending on what is used for the primary key, multiple database records may be affected. For example, if there was a word within the products description that was used as a "primary key", then every record that has that word would be impacted by importing a row of data with that word. This is why when using v_products_model if more than one product has that exact model designation that import of a record with that model designation will affect all such product.

As far as the reference to using/not using the prefix "v_", that is associated to identifying other product table fields that are to be used in the import/export. The place where the "v_" is omitted is within the configuration settings for the User Defined Products fields... When entering/adding such fields within the configuration menu, they should be added using whatever the field is actually called in the products table.

Perhaps the above helps? It may be of assistance to provide the configuration settings and other pertinent information to get to your solution.

adding the v_ prefix has got it working for me :D

Few slight issues though sadly if you can assist with any of these i'd be eternally grateful :) ..

1. The gtin field isn't being populated on my end even though its showing in the feed from supplier as "v_gtin". ive also tried it as "gtin" to no avail. after thought as typing should i try "v_products_gtin" it displays on my product listing with field name "gtin" and ive added it to the easy populate configuration (as shown in image below)

2.The supplier gives url's for the images but they also have not populated using "v_products_image"

3. I added the extra field support for the field "Condition" -as shown below, (ie new, used) however, i see it is showing on the page where you upload the feed file as the following...

User Defined Products Fields:
gtin: TRUE
Condition: FALSE

This would cause issues on google unless i set my merchant centre to assume items as new if not specified so number 3 i can live without solving for now

on checking again it appears it has accepted the link and attempted to display the picture on my website. clicking on the image title that is showing instead comes up as the following, think this is more specific to my site rather than issues porting the field or easy populate. ill ask my developer about this one or post a seperate thread if cant resolve so just points 1 and 3 that are technical help with the module whereas 2 is more site specific and might need its own thread if i'm struggling still

think i'm down to site specific issues or at least the feed from the suppliers, i don't want to clog up the support thread for the module so i have created a separate thread if anyone can spare a few moments to assist me or in the future if you have similar issues perhaps you will find your solution at the new thread i made https://www.zen-cart.com/showthread.php?228346-easy-populate-4-getting-a-feed-to-work-headache-please-help :)

Hi guys, im having some issues with my site.

So basically my website has just been setup on easy populate and i ran a feed, it put categories where i didnt want them ect and despite my best efforts i had to ask a developer who has since resolved this by editing easy pop to direct the items uploaded with the filename cointaining the word computing to be appended with Computing^ however a week or so later i'm seeing items selling from my dropshipper at cost price!

I can't see these items under admin and they also have "cPath=0" and "product_type=1" in the url (using ultimate seo urls)

The cost price issue was me picking product cost instead of product price at the dropshippers whilst setting up the feed, i've changed that now and when i re-ran the feed it only updated about half the items in the feed file.

I added this v product type of 1 field and i can't remember what i read that made me add this field but i think its responsible. do I remove the field from the .csv file? and once ive done that how do i delete these "ghost items" as i have affectionately called them?

Any help would be greatly appreciated as I think this is an issue i've investigated myself whilst my developers aren't at work over the weekend and it would be great to resolve it before anymore cost price items are sold by bargain hunters finding them via google shopping results :S

seems my mobile template is also not loading proper but i have no idea whats going on and been left in the predicament of pay more money for the fix, try myself or ask for help in at least working out how i go about removing these ghost items in the hope that fixes everything else sadly. developer doesnt want to seem to admit there are issues that need sorting under warranty and wont work unless i pay more :/

flappingfish:

Hi guys, im having some issues with my site.

So basically my website has just been setup on easy populate and i ran a feed, it put categories where i didnt want them ect and despite my best efforts i had to ask a developer who has since resolved this by editing easy pop to direct the items uploaded with the filename cointaining the word computing to be appended with Computing^ however a week or so later i'm seeing items selling from my dropshipper at cost price!

I can't see these items under admin and they also have "cPath=0" and "product_type=1" in the url (using ultimate seo urls)

The cost price issue was me picking product cost instead of product price at the dropshippers whilst setting up the feed, i've changed that now and when i re-ran the feed it only updated about half the items in the feed file.

I added this v product type of 1 field and i can't remember what i read that made me add this field but i think its responsible. do I remove the field from the .csv file? and once ive done that how do i delete these "ghost items" as i have affectionately called them?

Any help would be greatly appreciated as I think this is an issue i've investigated myself whilst my developers aren't at work over the weekend and it would be great to resolve it before anymore cost price items are sold by bargain hunters finding them via google shopping results :S

seems my mobile template is also not loading proper but i have no idea whats going on and been left in the predicament of pay more money for the fix, try myself or ask for help in at least working out how i go about removing these ghost items in the hope that fixes everything else sadly. developer doesnt want to seem to admit there are issues that need sorting under warranty and wont work unless i pay more :/

Ok, breakdown of things...

product_type = 1, that means that the product is a standard product, it is not a document product, it is not a music product, it is not a display only document type... Etc... Most people use the standard product and never use any of the other options, but they are there .

A default install would set that value to 1 if the field was not uploaded. EP4 has only attempted a different products_type when it believes that the information is describing a music product which then would assign a products_type=2.

So, the cPath=0 issue... That is because a product was incorporated into the database where the category was not properly captured or the product was given a master_categories_id of 0 or possibly some other mix up.

Thing is, and I believe I commented on this in another location, it seems that your database has been modified away from default settings or perhaps it has not been properly updated from some older version. It has been stated earlier that an import file had to include the fields products_quantity_order_units and products_quantity_order_max. A current installation defaults those to '1' and '0' respectively and therefore new product would not need to define that value when using EP4 because EP4 wouldn't interfere with applying the default value if the field was excluded. But again, that to me indicates that there has been something modified about the database in places that should not be modified which also then indicates the possibility that the issue described has come from some other abnormal operation.

At any rate, if you are able to determine the products_id of the product that have cPath=0, and either their products_model (likely visible when looking at the product) then further research can be done to identify what "simple" database changes may be necessary.

Price updating or otherwise requires matching the product in the file to the product in the database. As again previously discussed in this forum, a product is updated by matching the primary key information in the file to the database.... If only about "half" were updated, then those were the half that had a match for the products_model information and the other half did not have a match...

As far as deletion, again setting v_status to the value of 9 for the product in question will delete whatever product have the matching primary key data (products_model based on previous discussion).

I've looked through the export code for EP4 and it does not include a direct export of the product that are having this issue if the product has just enough information to display on the catalog side, but not on the admin. Such product still can be modified in the admin once they are "found". It just requires editing the admin browser path information to display the particular product.

Hmm.. Not entirely sure how helpful any of that is in fixing your issue though other than pointing out that the guesses made about the source of the issue are not involved with the problem(s). The lack of update as said is because the data in the file is not matching data in the database. That isn't an EP4 caused issue. Deletion of the "product" should be carefully considered: there are product and there are linked product and the issue may be with the linked version of the product or it may be with the product itself...

thanks mc, I seem to be getting a bit closer, spent the last few hours doing as you suggested to find the products before you typed this, Iv'e spent 5 days trying to figure it out though. eventually i managed to figure out if i went directly into my php admin i could filter the products table to mater category id of 0. deleted all of those then about 4 hours later i searched for category 0 or less then i found a further 25 items, deleted those and re ran the feed.
This time it stopped at 25 items like it had the last few times... finally decides to look at the .csv and i find item 26 has no category name!
I then remember asking if it was possible to have these "skipped due to no category" items diverted to a miscellaneous folder (which the developer did a code change for) A quick scan through and i deleted any other lines with no category and ran the file to find 1668 items succesfully added... i lost my cool with category links and decided to delete the whole category and re add it hence the whole lot going back in, only to find the feed file links products with it having items twice in different categories lol.

Maybe your right about the database or something else being old, i've removed the quantity max and minimum fields now but i see this on some items...
and the following when i add one of these items to basket and go to basket....

also getting some issues with the google product feeder crashing on processing. When i refreshed the page i see 2 files one ending lock.xml, i sent the usual one and its down about 200 items. log folder shows an error of running out of memory and showing the resulting file as having .xml issues in merchant centre but it's accepted the data for processing no problems? i dont think people can actually checkout with these items with the weird minimum 0 issue though?

One bit of good news is that the hidden items seemed to have been causing the issue's with the way the mobile template was displaying also so now they've gone at least my site looks appealing on mobile devices again :)

I just wanted to take a moment and thank everyone who has worked on EP4. I have used this in so many ways just in the last few days. We wanted to add some different categories to several hundred products. I was able to create all of those links in a matter of minutes. I wanted our sort order modified to mimic the SKU numbers (helps us with orders and inventory). Again, a few formulas in a spreadsheet along with copy & paste the values back to the CSV file and I was golden!

:cheers:

Chris

Even though i still have problems with my EP, i could not agree with you more. The feedback and help here are amazing.

g2ktcf:

I just wanted to take a moment and thank everyone who has worked on EP4. I have used this in so many ways just in the last few days. We wanted to add some different categories to several hundred products. I was able to create all of those links in a matter of minutes. I wanted our sort order modified to mimic the SKU numbers (helps us with orders and inventory). Again, a few formulas in a spreadsheet along with copy & paste the values back to the CSV file and I was golden!

:cheers:

Chris

kernheimer:

Even though i still have problems with my EP, i could not agree with you more. The feedback and help here are amazing.
@kernheimer, as you and I know, we have talked outside this forum. I would like to ask a few questions to find an answer.

In admin->Configuration->Easy Populate 4:

What are the settings for:

1. Import/Export Primary Key
2. Import Language Override

The problem described indicates a conflict.
To correct the issue I understand:
If 1. is products_id or blank_new, then 2. should NOT be products_model_only.
If 1. is products_model (historically the default), then 2. should NOT be products_id_only.

The above is for the problem described. The problem described has been, when importing a product file created by export, the system response is "No category provided for this new product".

I have more questions, but in this situation I want to slow down and address one issue at a time. Please continue to have patience.

I have just got the latest version from git hub and wanted to check if it is now a requirement that the folder in which you keep the uploaded files is under [YOURADMIN] directory.
The reason I ask is that I was having trouble with all the files showing CSV Unknow Delimiter. I tracked the code down to module easypopulate_4.php lines 438-444

$basepath = "";
  $realBase = realpath($basepath);
  $userpath = $basepath . $file;
  $realUserPath = realpath($userpath);
  if ($realUserPath === false || strpos($realUserPath, $realBase) !== 0) {
      return NULL; // return back to the function with a non-result?
  }

The issue is that
$file /MYSERVER/HOME/EASYPOPULATELOADDIR/EasyPopulateUploadDIR/Myfile.csv
$realBase /MYSERVER/HOME/EASYPOPULATELOADDIR/MYADMIN
$userpath /MYSERVER/HOME/EASYPOPULATELOADDIR/EasyPopulateUploadDIR/Myfile.csv
$realUserPath /MYSERVER/HOME/EASYPOPULATELOADDIR/EasyPopulateUploadDIR/Myfile.csv

does not pass the test unless EasyPopulateUploadDIR is under MYADMIN
thus $file /MYSERVER/HOME/EASYPOPULATELOADDIR/MYADMIN/EasyPopulateUploadDIR/Myfile.csv

I can leave the folder under MYADMIN just checking that this is now a requirement, OR if i have set something up wrongly.