EasyPopulate 4.0 Support Thread

The solution to the import error in place of the import button was the configuration option to store the eptemp folder outside of admin is the cause of the problem.
This leads to another question. Why is setting the eptemp folder outside the admin folder causing this import error?

wtfbbq:

The solution to the import error in place of the import button was the configuration option to store the eptemp folder outside of admin is the cause of the problem.
This leads to another question. Why is setting the eptemp folder outside the admin folder causing this import error?
Understanding that English is not the stronger language, I will try to answer what I may understand.

A problem was created in a code change where placing the EP4 upload/download folder outside of the admin directory would cause the files identified in that folder to not be processed for import by EP4. The file would be identified as having a CSV delimiter error. Although a solution was only recently publicly posted, the solution had been determined a while ago awaiting someone else to confirm that there was an issue and request resolution (see ). That solution appears to have worked.

I am not sure if the question above is now answered by the above text, or if a more technical explanation is requested about why the code works the way it does, so I am going to step out and attempt an explanation.

To attempt to limit the ability of database settings from pointing to folder locations that are not expected to represent "safe" spaces in the Zen Cart software, a method is incorporated in the software to report/determine the file's location that is executing the code. That location is then compared to the path of the file to be accessed. If the two sets of information align then the path is used, if the requested path is not in the file's path or downstream from it (in a sub-folder or some sub-sub-folder), then the path is not trusted. But, a folder that is placed outside of the admin folder would need to be at least one folder to the left of the current folder. Therefore to allow that path, additional effort/code is required to permit the parent folder to be allowed.

So why not just have the folder always point to the parent folder? Not everyone sets up their admin folder to only be a sub-folder off of the catalog, there has been discussion of removing the admin folder, and I consider the "safer" option for all CSV data to be imported and exported from within a secure area such as the admin folder. Therefore, I wrote the code to default running from the admin directory instead of from one directory above the current folder and allow any folder off of the catalog root to be permitted.

Otherwise, because I made a mistake. :)

mc12345678:

Understanding that English is not the stronger language, I will try to answer what I may understand.

A problem was created in a code change where placing the EP4 upload/download folder outside of the admin directory would cause the files identified in that folder to not be processed for import by EP4. The file would be identified as having a CSV delimiter error. Although a solution was only recently publicly posted, the solution had been determined a while ago awaiting someone else to confirm that there was an issue and request resolution (see https://www.zen-cart.com/showthread.php?190417-EasyPopulate-4-0-Support-Thread&p=1385080#post1385080). That solution appears to have worked.

I am not sure if the question above is now answered by the above text, or if a more technical explanation is requested about why the code works the way it does, so I am going to step out and attempt an explanation.

To attempt to limit the ability of database settings from pointing to folder locations that are not expected to represent "safe" spaces in the Zen Cart software, a method is incorporated in the software to report/determine the file's location that is executing the code. That location is then compared to the path of the file to be accessed. If the two sets of information align then the path is used, if the requested path is not in the file's path or downstream from it (in a sub-folder or some sub-sub-folder), then the path is not trusted. But, a folder that is placed outside of the admin folder would need to be at least one folder to the left of the current folder. Therefore to allow that path, additional effort/code is required to permit the parent folder to be allowed.

So why not just have the folder always point to the parent folder? Not everyone sets up their admin folder to only be a sub-folder off of the catalog, there has been discussion of removing the admin folder, and I consider the "safer" option for all CSV data to be imported and exported from within a secure area such as the admin folder. Therefore, I wrote the code to default running from the admin directory instead of from one directory above the current folder and allow any folder off of the catalog root to be permitted.

Otherwise, because I made a mistake. :)

The basic idea of the process is: if the database is modified by a bad actor, the worst that they can do/access is a file within the ZC catalog but not above it. In so accessing they would have to modify the file that is performing execution in order to affect something else with the idea that the user has placed their files in some sub-folder. Further, the admin directory itself is not stored in the database so that it can not be discovered by database review/inspection. While there is not a requirement right now for csv files to be in a sub-folder, I likely will incorporate such a requirement in a future release based on this evaluation.

mc12345678:

Almost have it. Forgot that the attribute exports don't use the products to categories table so having to generate the product list based on products in categories. I plan to create two export options for each basic and detailed to support of linked product and the second to exclude linked product.

Great thanks. Looking forward to the newer version with this feature that can selectively export a range of attributes (either by categories, manufacturers or date if possible).

waterbender:

Great thanks. Looking forward to the newer version with this feature that can selectively export a range of attributes (either by categories, manufacturers or date if possible).
Current plan is to support combination of categories and manufacturers through the existing dropdowns at the top of the EP4 admin screen. Not planning on addressing dates (newly added request) because of what I perceive as too many options to consider and dates to also address. Not to say that it couldn't be done. At the moment I see it as too much effort, likely too confusing for the user, with little in way of overall returns.

As to the category "issue" came across another wrinkle that am trying to ensure is addressed sufficiently. ZC 1.5.7 and older generated results in one format for a function, but upcoming ZC 1.5.8 will use a different response when working via the admin side. I want to "set it and forget it" by allowing a single body of code to function in both environments. So working through implementing the logic to address and for it to not add significant complexity.

mc12345678:

Current plan is to support combination of categories and manufacturers through the existing dropdowns at the top of the EP4 admin screen. Not planning on addressing dates (newly added request) because of what I perceive as too many options to consider and dates to also address. Not to say that it couldn't be done. At the moment I see it as too much effort, likely too confusing for the user, with little in way of overall returns.

As to the category "issue" came across another wrinkle that am trying to ensure is addressed sufficiently. ZC 1.5.7 and older generated results in one format for a function, but upcoming ZC 1.5.8 will use a different response when working via the admin side. I want to "set it and forget it" by allowing a single body of code to function in both environments. So working through implementing the logic to address and for it to not add significant complexity.

Further status update. After further consideration of the "configuration" of the associated function across at least those two versions of Zen Cart, I decided to draw the function into EP4 instead of using the Zen Cart provided "versions". The reasoning is this:
in earlier versions, an array is generated where the key to the array represents the "numbered" item in the list (5 product, the last key would be 5) and the value of the array represented the products_id. In the upcoming version, the key to the array represents the products_id and the value of the array represents the category path. So I thought I had a good system developed to be able to evaluate the array results and make a determination off of that as to which result was being provided. In general this would work when evaluating the differences of the below to arrays:
$myArrayOld[1] = 7
$myArrayOld[2] = 5
versus
$myArrayNew[7] = 6
$myArrayNew[5] = 3

Using the above, I could sort the array by key, then evaluate the number of array items and if the key of the last array item did not equal the number of items, then I was using the "new" system and could "withdraw" the products_id for processing. If the last key was equal to the number of array items, then I would use the values for products_id.

But... In the boundary condition where the products_id of the first few products (1 and 2 for example) were being returned, then:
$myArrayOld[1] = 1
$myArrayOld[2] = 2
versus
$myArrayNew[1] = 8
$myArrayNew[2] = 9

Would mistakenly in a new system attempt to process the array with consideration of the right side (value) being the products_id, although that value side (the result) is actually the category path of category 8 followed by category 9 instead of products_id 1 followed by 2.

A separate consideration was to attempt to identify where the associated function was "stored" with the old version being in the admin directory and the new version in the catalog. But, then modifications that one may have made to their site could increase the needed assistance (much like the issue generated by the current defaults for import/export of language related fields). So, I don't of course like that concept. Then there was the consideration of attempting to identify the number of parameters of the associated function, same issue of support and though I had a process developed to internally adjust for an earlier catalog side code discrepancy that is corrected for ZC 1.5.8, if one doesn't incorporate a specific change (missed it or didn't want to apply it for some reason), again additional support assistance would become necessary.

So... As a result, I have incorporated the ZC 1.5.8 version of the applicable function directly into the easypopulate_4_export.php file giving it a specific ep4 related name so that the code is otherwise written more simply. I considered adding it to the extra_functions file, but really I am looking to move away from that file and instead implement that code into an ep4 class instead so that the code would only load while using ep4 or at an on call basis.

I know, more than you wanted to know @waterbender, but I wanted to at least publish somewhere "public" why the changes I'm making were made the way they were. I would expect that as time wears on and the code requires significant change to support operation, that any consideration of using it on a version less than 1.5.8 will mean that could revert back to the Zen Cart core code, but as long as this code is supported on a ZC version before 1.5.8, it seems the best course of action for the code is to use the ep4 version of the function.

Hi, I apologize if this has been answered but I cannot locate the question in this thread. EP4 for 1.5.7 is working just great for me with ONE exception. When downloading the Basic Attributes, I am seeing only 1 line when I have many, many more lines of data. I attempted to split thinking the size of the file may have been an issue, but it's nowhere near that big. Is there a setting or known solution to basic attributes not displaying all lines of data? Thanks so much and once again, I apologize if this was addressed elsewhere. Thank you!

chuckrey:

Hi, I apologize if this has been answered but I cannot locate the question in this thread. EP4 for 1.5.7 is working just great for me with ONE exception. When downloading the Basic Attributes, I am seeing only 1 line when I have many, many more lines of data. I attempted to split thinking the size of the file may have been an issue, but it's nowhere near that big. Is there a setting or known solution to basic attributes not displaying all lines of data? Thanks so much and once again, I apologize if this was addressed elsewhere. Thank you!
So, I've only seen a "single" line of data when either there are no attributes assigned or there is a possibility that the method of defining the attributes to the product doesn't ensure a record in each of the tables: products_attributes (table a), products (table p), products_options (table o), and products_options_values (table v) and further that those tables are related by: a.products_id = p.products_id AND a.options_id = o.products_options_id AND a.options_values_id = v.products_options_values_id AND o.language_id in = v.language_id.
Those tables could possibly instead be referenced by left joins, but there may also be undesirable data found when doing that on the export...

Basically, there may be an issue with the data in the database, but I have done a comparison of the v4.0.37.13 software to what I am currently working on as an upgrade where my upgrade works (and a similar issue hadn't previously been identified) and haven't found anything that specifically limits the number of responses/records. The query that generates the list to be exported is:

SELECT
      a.products_attributes_id            as v_products_attributes_id,
      a.products_id                       as v_products_id,
      a.options_id                        as v_options_id,
      a.options_values_id                 as v_options_values_id,
      p.products_model            as v_products_model,
      o.products_options_id               as v_products_options_id,
      o.products_options_name             as v_products_options_name,
      o.products_options_type             as v_products_options_type,
      v.products_options_values_id        as v_products_options_values_id,
      v.products_options_values_name      as v_products_options_values_name,
      v.language_id                       as v_language_id
      FROM products_attributes as a, products as p, products_options as o, products_options_values as v
      WHERE
      a.products_id       = p.products_id AND
      a.options_id        = o.products_options_id AND
      a.options_values_id = v.products_options_values_id AND
      o.language_id       = v.language_id ORDER BY a.products_id, a.options_id, v.language_id, v.products_options_values_id

If that is only resulting in a single row and there are multiple product with attributes, then I would think there is something "wrong" (different than expected) with the data. The data may be handled differently and therefore this sql may need to be adjusted to accommodate ideally with the same fields being returned.

Thanks for the response, I am going to have a closer look at the data in the DB as well as compare to another DB that is including all data. I will update with my finding.
Thanks again!

chuckrey:

Thanks for the response, I am going to have a closer look at the data in the DB as well as compare to another DB that is including all data. I will update with my finding.
Thanks again!

If I may ask, are there multiple languages involved? What language(s)?

I do have dutch, english, french, german, russian and spanish folders located in the Includes/languages folder.
Dutch was installed w/ Printable Pricelist module
French German and Spanish came w/ the Template of "YourStore". However I have another site using it and Basic Attr works just fine.

I am not using any other language than English on the website.

chuckrey:

I do have dutch, english, french, german, russian and spanish folders located in the Includes/languages folder.
Dutch was installed w/ Printable Pricelist module
French German and Spanish came w/ the Template of "YourStore". However I have another site using it and Basic Attr works just fine.

I am not using any other language than English on the website.

So I'll note that the existence of language files for EP4 is not an issue, when/if Russian is used there may be an issue with the language content being populated properly such as importing the products_description related field, but it shouldn't cause the reported issue.

On another note, for the site having the issue, do the other exports also have an issue? E.g. the product export or price listing?

Another question, when looking at the EP4 admin screen, what index does the English language have? It's in the info on the upper right area.

The other exports do not have an issue, just the basic attributes.

Detail and full work great and export all without an issue.

Regarding the index question. Is this what you are looking for?
Installed Languages
1-en: English
Default Language: 1-English
Internal Character Encoding: UTF-8
DB Collation: utf8mb4

chuckrey:

The other exports do not have an issue, just the basic attributes.

Detail and full work great and export all without an issue.

Regarding the index question. Is this what you are looking for?
Installed Languages
1-en: English
Default Language: 1-English
Internal Character Encoding: UTF-8
DB Collation: utf8mb4

Yes. While haven't seen it in the wild, I've once caused the language_id for English to become like 3 or 4 and was the only language installed. Just wanted to confirm no unexpected issues likely from database language data. Thanks for that. Still suggesting until otherwise determined that is database related.

chuckrey:

The other exports do not have an issue, just the basic attributes.

Detail and full work great and export all without an issue.

Regarding the index question. Is this what you are looking for?
Installed Languages
1-en: English
Default Language: 1-English
Internal Character Encoding: UTF-8
DB Collation: utf8mb4

mc12345678:

Yes. While haven't seen it in the wild, I've once caused the language_id for English to become like 3 or 4 and was the only language installed. Just wanted to confirm no unexpected issues likely from database language data. Thanks for that. Still suggesting until otherwise determined that is database related.

Wait... detailed attribute export works as expected, but basic does not? Much of the query for both of those is the same. I mean they are each asking for different fields, but the from and where portions are nearly identical. A difference that stands out is that the detailed export adds a limitation of using the language assigned to the value of 1.

Fyi, in the past a recommendation has also been to remove and reinstall the software.

mc12345678:

Wait... detailed attribute export works as expected, but basic does not? Much of the query for both of those is the same. I mean they are each asking for different fields, but the from and where portions are nearly identical. A difference that stands out is that the detailed export adds a limitation of using the language assigned to the value of 1.

Fyi, in the past a recommendation has also been to remove and reinstall the software.

I am going to try that this evening and see if it resolves. Thanks again!!

Issue with images ... I've tried everything to figure this out. When using a CSV file with Easy Populate everything works fine but the images remain small when clicking on the "larger image" link. The pop-up will only display a small image. The only way to fix this is through the admin by clicking the edit button for the product, click preview, and then save. That fixes the issue for every item. But why does "Easy Populate" cause this problem with every CSV import? I've tried making the image names and extension lowercase (and lowercase as well in the CSV file) so they all match but it doesn't help. Any ideas?

mc12345678:

Wait... detailed attribute export works as expected, but basic does not? Much of the query for both of those is the same. I mean they are each asking for different fields, but the from and where portions are nearly identical. A difference that stands out is that the detailed export adds a limitation of using the language assigned to the value of 1.

Fyi, in the past a recommendation has also been to remove and reinstall the software.

So I just finished removing EP 4.0 and reinstalled the latest from Github. I did not have any luck, same result of not displaying the data from Basic Attributes.....

However.. I tested an older version I have on my hard drive. Specifically Easy Populate 4.0.30 - Beta 06-27-2015 and it worked where all data is now downloading without an issue. The version that was not allowing me to export Basic Attributes is Easy Populate 4.0.37.13 - 05-03-2021

chuckrey:

So I just finished removing EP 4.0 and reinstalled the latest from Github. I did not have any luck, same result of not displaying the data from Basic Attributes.....

However.. I tested an older version I have on my hard drive. Specifically Easy Populate 4.0.30 - Beta 06-27-2015 and it worked where all data is now downloading without an issue. The version that was not allowing me to export Basic Attributes is Easy Populate 4.0.37.13 - 05-03-2021

The question remains (for clarity) the detailed attributes export from the same store that is having an issue with the basic attributes export, works or only a single record? Could also identify php version and zen cart version, please? Done any phpmyadmin query testing of the database? I'll go back and try to review such an old version but without understanding why this one is not displaying all of the data for you but is for others makes it difficult to diagnose or claim specifically is a problem with the software, especially when the same software was reported as installed in a different store and reported as fully functional.

JNR345:

Issue with images ... I've tried everything to figure this out. When using a CSV file with Easy Populate everything works fine but the images remain small when clicking on the "larger image" link. The pop-up will only display a small image. The only way to fix this is through the admin by clicking the edit button for the product, click preview, and then save. That fixes the issue for every item. But why does "Easy Populate" cause this problem with every CSV import? I've tried making the image names and extension lowercase (and lowercase as well in the CSV file) so they all match but it doesn't help. Any ideas?

Only idea(s) that come to mind is that there is some other image handling software that does something when processing through the normal admin. Standard image processing reads the database file location for TY e base product image and then uses the "rules" for images to locate the large, additional, etc. Images for that product. I know that software such as image handler processes an image when it goes through the handler. I can't recall if the user doesn't see the results until the image is loaded a second time or if it's compression is observable on the first visit.

In other words, I have no explanation to why it happens on the site in question as there is nothing that is done that should cause or allow an issue such as that.