EasyPopulate 4.0 Support Thread

[reetp]

No, it is much simpler than that.

Your analogy is wrong - it is like opening a zip with a text editor, the editor allowing you to open the file, and edit it...... without throwing an error !

If a decision process runs and does not complete correctly for whatever reason then an error should be thrown advising the user. The assumption is the user is dumb (that's me !) and everything you can do to stop them making stupid errors should be encouraged - hence a million different error checks throughout Zencart.

In this instance the filename was incorrect which caused the program to carry on and report that everything was OK when clearly it wasn't.... (the data was right - just the filename being wrong)

That cost me a day of running around based on information reported by the process that was clearly wrong. It said it worked, when clearly it had not. That isn't right.

I'm not trying to be negative - I'd just like to see the code improved to the benefit of all as it is an indispensable tool for Zencart.

Here's a patch to check the filenames and log an error - it may not be perfect but it is about there - also miles quicker than running through all the code for a simple failure :-)

You can see my comment line - sometimes it checks for the filename, others it does a anything else check with <>

Either way, if the name is checked at the start then it gives a fast response to the problem. If you have new sections / filenames then just add the name to the array. Using stristr would give you more flexibility as per my note, but I have left as is for now.

It's also dead simple.....

--- easypopulate_4_import-orig.php 2015-06-18 11:18:05.000000000 +0200
+++ easypopulate_4_import.php 2015-06-18 14:00:21.000000000 +0200
@@ -8,6 +8,37 @@

$file = array('name' => $_GET['import']);
$display_output .= sprintf(EASYPOPULATE_4_DISPLAY_LOCAL_FILE_SPEC, $file['name']);
+
+
+
+/*
+ * These are the filenames that we look for
+ *
+strtolower(substr($file['name'],0,11)) == "featured-ep")
+strtolower(substr($file['name'],0,15)) == "attrib-basic-ep"
+strtolower(substr($file['name'],0,18)) == "attrib-detailed-ep")
+strtolower(substr($file['name'],0,15)) == "sba-detailed-ep")
+strtolower(substr($file['name'],0,12)) == "sba-stock-ep")
+strtolower(substr($file['name'],0,15)) <> "categorymeta-ep")
+strtolower(substr($file['name'],0,7)) <> "attrib-")
+strtolower(substr($file['name'],0,4)) <> "sba-")
+strtolower(substr($file['name'],0,14)) == "pricebreaks-ep")
+*/
+
+
+$fileNameArray = array ("featured-ep", "attrib-basic-ep", "attrib-detailed-ep", "sba-detailed-ep", "categorymeta-ep", "attrib-", "sba-", "pricebreaks-ep");
+$discoveredFile = false;
+
+foreach ($fileNameArray as $key) {
+
+ $length = strlen ($key);
+ // probably better to do stristr but all the lines then need changing further down
+ if (strtolower(substr($file['name'],0,$length)) == $key) {
+ $discoveredFile = true;
+ }
+}
+
+if ($discoveredFile == true) { // we'll proceed

$ep_update_count = 0; // product records updated
$ep_import_count = 0; // new products records imported
@@ -1637,5 +1668,17 @@
} else {
$messageStack->add("File Import Completed.", 'success');
}
+}
+else {
+ $display_output .= '<h3>Finished Processing Import File</h3>';
+ $display_output .= '<br /><h3 style="color:red">Filename is wrong</h3> ';
+ $display_output .= '<br />Please check that the filename <b>starts</b> with the correct term';
+ $display_output .= '<br />The following are the permissible names :';
+ foreach ($fileNameArray as $key) {
+ $display_output .= "<br />" . $key;
+ }
+ $messageStack->add("File Import Failed.", 'error');
+
+} // End file check loop
} // END FILE UPLOADS
-?>
\ No newline at end of file
+?>


And a small patch to easypopulate.php itself to clear a small warning that was annoying me :-)

--- easypopulate_4-orig.php 2015-06-17 19:03:42.000000000 +0200
+++ easypopulate_4.php 2015-06-18 14:47:28.000000000 +0200
@@ -28,9 +28,9 @@
$ep_music = (int)EASYPOPULATE_4_CONFIG_MUSIC_DATA; // 0-Disable, 1-Enable
$ep_uses_mysqli = (PROJECT_VERSION_MAJOR > '1' || PROJECT_VERSION_MINOR >= '5.3' ? true : false--- easypopulate_4-orig.php 2015-06-17 19:03:42.000000000 +0200
+++ easypopulate_4.php 2015-06-18 14:14:01.000000000 +0200
@@ -30,7 +30,7 @@

@set_time_limit($ep_execution); // executin limit in seconds. 300 = 5 minutes before timeout, 0 means no timelimit

-if (!$error) {
+if (!isset($error)) {
$upload_max_filesize=ini_get("upload_max_filesize");
if (preg_match("/([0-9]+)K/i",$upload_max_filesize,$tempregs)) $upload_max_filesize=$tempregs[1]*1024;
if (preg_match("/([0-9]+)M/i",$upload_max_filesize,$tempregs)) $upload_max_filesize=$tempregs[1]*1024*1024;);

-@set_time_limit($ep_execution); // executin limit in seconds. 300 = 5 minutes before timeout, 0 means no timelimit
+@set_time_limit($ep_execution); // execution limit in seconds. 300 = 5 minutes before timeout, 0 means no timelimit

-if (!$error) {
+if (!isset($error)) {
$upload_max_filesize=ini_get("upload_max_filesize");
if (preg_match("/([0-9]+)K/i",$upload_max_filesize,$tempregs)) $upload_max_filesize=$tempregs[1]*1024;
if (preg_match("/([0-9]+)M/i",$upload_max_filesize,$tempregs)) $upload_max_filesize=$tempregs[1]*1024*1024;


I'd have a go on github but not very experienced with it.... and I am not sure which repo to follow - is chaddro still working on this ?

B. Rgds
John

[ranger_lp]

Did anyone ever come up with a fix for this being displayed on the bottom of the page?


. www.evoluted.net Layout: Manny . www.tenka.co.uk REQUIREMENTS ============ This script requires PHP and GD2 if you wish to use the thumbnail functionality. INSTRUCTIONS ============ 1) Unzip all files 2) Edit this file, making sure everything is setup as required. 3) Upload to server 4) ?????? 5) Profit! CONFIGURATION ============= Edit the variables in this section to make the script work as you require. Start Directory - To list the files contained within the current directory enter '.', otherwise enter the path to the directory you wish to list. The path must be relative to the current directory. */ $startdir = DIR_FS_CATALOG . $tempdir; /* Show Thumbnails? - Set to true if you wish to use the scripts auto-thumbnail generation capabilities. This requires that GD2 is installed. */ $showthumbnails = true; /* Show Directories - Do you want to make subdirectories available? If not set this to false */ $showdirs = true; /* Force downloads - Do you want to force people to download the files rather than viewing them in their browser? */ $forcedownloads = false; /* Hide Files - If you wish to hide certain files or directories then enter their details here. The values entered are matched against the file/directory names. If any part of the name matches what is entered below then it is now shown. */ $hide = array( 'dlf', 'index.php', 'Thumbs', '.htaccess', '.htpasswd', 'fileList.php' ); /* Show index files - if an index file is found in a directory to you want to display that rather than the listing output from this script? */ $displayindex = false; /* Allow uploads? - If enabled users will be able to upload files to any viewable directory. You should really only enable this if the area this script is in is already password protected. */ $allowuploads = false; /* Overwrite files - If a user uploads a file with the same name as an existing file do you want the existing file to be overwritten? */ $overwrite = false; /* Index files - The follow array contains all the index files that will be used if $displayindex (above) is set to true. Feel free to add, delete or alter these */ $indexfiles = array ( 'index.html', 'index.htm', 'default.htm', 'default.html' ); /* File Icons - If you want to add your own special file icons use this section below. Each entry relates to the extension of the given file, in the form => . These files must be located within the dlf directory. */ $filetypes = array ( 'png' => 'jpg.gif', 'jpeg' => 'jpg.gif', 'bmp' => 'jpg.gif', 'jpg' => 'jpg.gif', 'gif' => 'gif.gif', 'zip' => 'archive.png', 'rar' => 'archive.png', 'exe' => 'exe.gif', 'setup' => 'setup.gif', 'txt' => 'text.png', 'htm' => 'html.gif', 'html' => 'html.gif', 'fla' => 'fla.gif', 'swf' => 'swf.gif', 'xls' => 'xls.gif', 'doc' => 'doc.gif', 'sig' => 'sig.gif', 'fh10' => 'fh10.gif', 'pdf' => 'pdf.gif', 'psd' => 'psd.gif', 'rm' => 'real.gif', 'mpg' => 'video.gif', 'mpeg' => 'video.gif', 'mov' => 'video2.gif', 'avi' => 'video.gif', 'eps' => 'eps.gif', 'gz' => 'archive.png', 'asc' => 'sig.gif', ); /* That's it! You are now ready to upload this script to the server. Only edit what is below this line if you are sure that you know what you are doing! */ error_reporting(0); if(!function_exists('imagecreatetruecolor')) $showthumbnails = false; $leadon = $startdir; if($leadon=='.') $leadon = ''; if((substr($leadon, -1, 1)!='/') && $leadon!='') $leadon = $leadon . '/'; $startdir = $leadon; if($_GET['dir']) { //check this is okay. if(substr($_GET['dir'], -1, 1)!='/') { $_GET['dir'] = $_GET['dir'] . '/'; } $dirok = true; $dirnames = split('/', $_GET['dir']); for($di=0; $di
File
Size
Last Modified
'.$files[$i].''; $thumb2 = ' onmouseover="o('.$i.', \''.urlencode($leadon . $files[$i]).'\');" onmouseout="f('.$i.');"'; } if($filetypes[$ext]) { $icon = $filetypes[$ext]; } $filename = $files[$i]; if(strlen($filename)>43) { $filename = substr($files[$i], 0, 40) . '...'; } $fileurl = $leadon . $files[$i]; if(!$forcedownloads) { $fileurl = $_SESSION['PHP_SELF'] . '?dir=' . urlencode($leadon) . '&download=' . urlencode($files[$i]); } $fileurl = "../".$tempdir.$files[$i]; ?>
0KB
Jan 01 1970 01:00:00 AM

Thanks in advance

Tony

[mc12345678]

Umm, unless can point to where this issue is addressed in this forum, nothing about that content has to do with this version of EP

[RFree190]

an error occurred while processing this directive

I'm trying to upload a new product list, with approx 6,800 products. Each time I do the upload, it crashes with the above error message displayed on a white screen. Also, the last product uploaded is different each time I run the upload, so I don't think there is an issue with uploaded data causing the crash. If I split the file into multiple uploads, (It splits this one into 2 uploads), the first one crashes, the second uploads fine.

I have looked for any error logs, but can't find one.

Any ideas??? Thanks!

[mc12345678]

an error occurred while processing this directive

I'm trying to upload a new product list, with approx 6,800 products. Each time I do the upload, it crashes with the above error message displayed on a white screen. Also, the last product uploaded is different each time I run the upload, so I don't think there is an issue with uploaded data causing the crash. If I split the file into multiple uploads, (It splits this one into 2 uploads), the first one crashes, the second uploads fine.

I have looked for any error logs, but can't find one.

Any ideas??? Thanks!

From my quick internet search, it would appear that the issue is more related to your host to some extent than to EP4 and that the error logs will be logged with your host or wherever they store logs related to the site's operation as compared to the ZC cache (ZC < 1.5.1)/logs (ZC >= 1.5.1) folder. The next thought is that the problematic file be split again (and again) until the culprit entry(ies) are identified. It may also be necessary to split the files to a more refined limit... What is the difference in number of entries between the two files? (Ie. what was the record split number as compared to the total number?) If say the cutoff were 5,000, then the second file would only contain about 1,800... But if the split were 3,400 then basically they are equal files in rows of content. The difference in row last seen before the error is probably a result of caching in one respect or another or because there is different system response and action for an update as compared to an insertion. About how long is it taking for the data to be processed? I don't recall which setting(s) affect it, but I have seen something about timeouts occurring when trying to process a larger file even if the code is still actively processing the file. Just some ideas. :) The other thing that I found was to verify your file and folder permissions, but not so sure I see that as the issue here... (644 for files, and 755 for folders)

[RFree190]

From my quick internet search, it would appear that the issue is more related to your host to some extent than to EP4 and that the error logs will be logged with your host or wherever they store logs related to the site's operation as compared to the ZC cache (ZC < 1.5.1)/logs (ZC >= 1.5.1) folder. The next thought is that the problematic file be split again (and again) until the culprit entry(ies) are identified. It may also be necessary to split the files to a more refined limit... What is the difference in number of entries between the two files? (Ie. what was the record split number as compared to the total number?) If say the cutoff were 5,000, then the second file would only contain about 1,800... But if the split were 3,400 then basically they are equal files in rows of content. The difference in row last seen before the error is probably a result of caching in one respect or another or because there is different system response and action for an update as compared to an insertion. About how long is it taking for the data to be processed? I don't recall which setting(s) affect it, but I have seen something about timeouts occurring when trying to process a larger file even if the code is still actively processing the file. Just some ideas. :) The other thing that I found was to verify your file and folder permissions, but not so sure I see that as the issue here... (644 for files, and 755 for folders)

Thanks for the quick reply.
I was doing some troubleshooting, using your idea of smaller "Splits" and think I tracked down the problem. I was using MS Access to compile a CSV of all my products. I have been doing this for years and have been using "chr(10) & chr(13)" to put line breaks and carriage returns into my product descriptions. For some reason, this broke my imports. Maybe it's the latest version of MS Access. I had been using Access 2007 for years, with success. This is my first time using Access 2013. I'm going to blame MS, as EP4 has worked for me in the past.

Thanks for your input. It helped me track down my issue.

[reetp]

...code

Slight mod - needed to allow the file 'full-ep'

--- easypopulate_4-orig.php 2015-06-17 19:03:42.000000000 +0200
+++ easypopulate_4.php 2015-06-18 14:47:28.000000000 +0200
@@ -28,9 +28,9 @@
$ep_music = (int)EASYPOPULATE_4_CONFIG_MUSIC_DATA; // 0-Disable, 1-Enable
$ep_uses_mysqli = (PROJECT_VERSION_MAJOR > '1' || PROJECT_VERSION_MINOR >= '5.3' ? true : false);

-@set_time_limit($ep_execution); // executin limit in seconds. 300 = 5 minutes before timeout, 0 means no timelimit
+@set_time_limit($ep_execution); // execution limit in seconds. 300 = 5 minutes before timeout, 0 means no timelimit

-if (!$error) {
+if (!isset($error)) {
$upload_max_filesize=ini_get("upload_max_filesize");
if (preg_match("/([0-9]+)K/i",$upload_max_filesize,$tempregs)) $upload_max_filesize=$tempregs[1]*1024;
if (preg_match("/([0-9]+)M/i",$upload_max_filesize,$tempregs)) $upload_max_filesize=$tempregs[1]*1024*1024;
[root@home myadmin]# diff -ruN easypopulate_4_import-orig.php easypopulate_4_import.php
--- easypopulate_4_import-orig.php 2015-06-18 11:18:05.000000000 +0200
+++ easypopulate_4_import.php 2015-06-26 15:19:07.000000000 +0200
@@ -8,6 +8,34 @@

$file = array('name' => $_GET['import']);
$display_output .= sprintf(EASYPOPULATE_4_DISPLAY_LOCAL_FILE_SPEC, $file['name']);
+
+/*
+ * These are the filenames that we look for
+ *
+strtolower(substr($file['name'],0,11)) == "featured-ep")
+strtolower(substr($file['name'],0,15)) == "attrib-basic-ep"
+strtolower(substr($file['name'],0,18)) == "attrib-detailed-ep")
+strtolower(substr($file['name'],0,15)) == "sba-detailed-ep")
+strtolower(substr($file['name'],0,12)) == "sba-stock-ep")
+strtolower(substr($file['name'],0,15)) <> "categorymeta-ep")
+strtolower(substr($file['name'],0,7)) <> "attrib-")
+strtolower(substr($file['name'],0,4)) <> "sba-")
+strtolower(substr($file['name'],0,14)) == "pricebreaks-ep")
+*/
+
+$fileNameArray = array ("full-ep", "featured-ep", "attrib-basic-ep", "attrib-detailed-ep", "sba-detailed-ep", "categorymeta-ep", "attrib-", "sba-", "pricebreaks-ep");
+$discoveredFile = false;
+
+foreach ($fileNameArray as $key) {
+
+ $length = strlen ($key);
+ // probably better to do stristr but all the lines then need changing further down
+ if (strtolower(substr($file['name'],0,$length)) == $key) {
+ $discoveredFile = true;
+ }
+}
+
+if ($discoveredFile == true) { // we'll proceed

$ep_update_count = 0; // product records updated
$ep_import_count = 0; // new products records imported
@@ -1637,5 +1665,17 @@
} else {
$messageStack->add("File Import Completed.", 'success');
}
+}
+else {
+ $display_output .= '<h3>Finished Processing Import File</h3>';
+ $display_output .= '<br /><h3 style="color:red">Filename is wrong</h3> ';
+ $display_output .= '<br />Please check that the filename <b>starts</b> with the correct term';
+ $display_output .= '<br />The following are the permissible names :';
+ foreach ($fileNameArray as $key) {
+ $display_output .= "<br />" . $key;
+ }
+ $messageStack->add("File Import Failed.", 'error');
+
+} // End file check loop
} // END FILE UPLOADS
-?>
\ No newline at end of file
+?>


And a small patch to easypopulate.php itself to clear a small warning that was annoying me :-)

--- easypopulate_4-orig.php 2015-06-17 19:03:42.000000000 +0200
+++ easypopulate_4.php 2015-06-18 14:47:28.000000000 +0200
@@ -28,9 +28,9 @@
$ep_music = (int)EASYPOPULATE_4_CONFIG_MUSIC_DATA; // 0-Disable, 1-Enable
$ep_uses_mysqli = (PROJECT_VERSION_MAJOR > '1' || PROJECT_VERSION_MINOR >= '5.3' ? true : false--- easypopulate_4-orig.php 2015-06-17 19:03:42.000000000 +0200
+++ easypopulate_4.php 2015-06-18 14:14:01.000000000 +0200
@@ -30,7 +30,7 @@

@set_time_limit($ep_execution); // executin limit in seconds. 300 = 5 minutes before timeout, 0 means no timelimit

-if (!$error) {
+if (!isset($error)) {
$upload_max_filesize=ini_get("upload_max_filesize");
if (preg_match("/([0-9]+)K/i",$upload_max_filesize,$tempregs)) $upload_max_filesize=$tempregs[1]*1024;
if (preg_match("/([0-9]+)M/i",$upload_max_filesize,$tempregs)) $upload_max_filesize=$tempregs[1]*1024*1024;);

-@set_time_limit($ep_execution); // executin limit in seconds. 300 = 5 minutes before timeout, 0 means no timelimit
+@set_time_limit($ep_execution); // execution limit in seconds. 300 = 5 minutes before timeout, 0 means no timelimit

-if (!$error) {
+if (!isset($error)) {
$upload_max_filesize=ini_get("upload_max_filesize");
if (preg_match("/([0-9]+)K/i",$upload_max_filesize,$tempregs)) $upload_max_filesize=$tempregs[1]*1024;
if (preg_match("/([0-9]+)M/i",$upload_max_filesize,$tempregs)) $upload_max_filesize=$tempregs[1]*1024*1024;

[mc12345678]

This entire week I've been trying to figure out how to provide comment to your code without running you away because these things don't get better without someone pointing out an issue. I have no control over your response and if it is to walk away or otherwise be upset, then so be it. Hopefully now you can look back at the suggestions of the last several posts, including the goal of being helpful to the newbie as well as not unnecessarily forcing existing users to make changes in how they operate and maybe see things a little differently.

In regards to the most recent post:
1) Wondered when you would figure that one out, and now 2) have added a restriction that didn't exist before and is truly unnecessary other than to suppport this error message. With that one single addition now all files must follow a specific naming convention not just the ones that require special/unique handling. I'll have my mod up in a bit, been trying to come up with decent terminology to display and maybe be acceptable to the eye. It doesn't restrict the ability to process files of any name, shows the naming convention expected to process that type of file, and provides the info on the first screen shot rather than after import "after the damage is done". The display of all file types on the first screen is switchable from all, to all in the designated directory to no help provided (ie, current view for those classic users :) ) with the default operation/install being all help. So those installing the update and wnting to not have this extra assistance will need to run a smidge of sql or store a constant somewhere to change the display of the min page away from the "full help".

[mc12345678]

Version 4.0.30 has been pushed to github: https://github.com/mc12345678/EasyPopulate-4.0/ If found to be acceptable will have it merged into the version presented at the first post of this thread.

Adds a new switch to the configuration menu. If this switch is not entered, set, or reset then the full new method of filename display will be provided which may be considered annoying by those that have become accustomed to using EP4 as it is and understand the need to name certain files a certain way as outlined in the instructions. (If so, I suggest installing the below SQL through the tools -> Install SQL patches option or by selecting the un-install/reinstall links on the EP4 page and the SQL will take care of itself. Though remember to note any special settings on the admin's configuration panel before uninstalling.)

Also, the generation of a warning has been corrected. Previous use of

if (!$error)

would result in execution if $error was not set, was = null, was = false, or any other non-true type result. Problem is that now in certain versions of PHP, if the variable hasn't been declared (not set), then a warning is thrown so, to maintain continued functionality and prevent those warnings, the statement has been changed to

if ((isset($error) && !$error) || !isset($error))

This way, if it is set then can evaluate that it is not an error, or if it is not set at all (or if equal to null) then in either regards it is considered as being "addressed" and the software can move on.

So that SQL if you want to be lazy and install the software but only incorporate the one additional change:

SELECT @config_id:=configuration_group_id FROM configuration_group where configuration_group_title= 'Easy Populate 4' and configuration_group_description = 'Configuration Options for Easy Populate 4';

INSERT INTO configuration (configuration_title, configuration_key, configuration_value, configuration_description, configuration_group_id, sort_order, last_modified, date_added, use_function, set_function) VALUES ('Show all EP4 Filetypes with Files', 'EP4_SHOW_ALL_FILETYPES', 'True', 'When looking at the EP4 Tools screen, should the filename prefix for all specific file types be displayed for all possible file types (True [default]), should only the method(s) that will be used to process the files present be displayed (False), or should there be no assistance be provided on filenaming on the main page (Hidden) like it was until this feature was added? (True, False, or Hidden)', @config_id , '25', NULL, now(), NULL, 'zen_cfg_select_option(array(\"True\", \"False\", \"Hidden\"),');

[kpshira]

I just had the module installed by my hosting company. I cannot download the Complete .csv file to edit (Attributes Not Included). I keep getting "WARNING: An Error occurred, please refresh the page and try again." even after I refresh the page.
I can download all the other files except this one (both the regular download and the temporary).

Thank you for any help or insight.