EasyPopulate 4.0 Support Thread

EasyPopulate 4.0 (EP4) Beta Support Thread. For Zencart 1.38/1.39/1.5

Download from at Github:
[strike]https://github.com/chaddro/EasyPopulate-4.0[/strike]

https://github.com/mc12345678/EasyPopulate-4.0

Look for the "ZIP" button to download. To install, upload all files to your admin directory maintaining directory structure.
Also upload the additional files for your version of Zencart - be sure to put thes in their correct folder.

Next login to your Admin, and go to Tools -> EasyPopulate 4.
At the top of the screen will be a couple warnings for your upload folder, and configuration keys missing... click the "Please install your configuration by clicking here" to install the configuration keys.

Now navigate to Configuration -> EasyPopulate 4
Be sure to set your "Uploads Directory" to a folder that exists. I recommend changing from from the default "temp/" to something obscure to frustrate hackers.
You may also need to check folder permissions. Default for most servers is 755, but you may need to set permissions to 777.

I recommend that you first enter a couple Categories and Products through the normal Zencart Admin. This will help you understand the exported file formats.

Things of Import:

1) EP4 works only with CSV data files, and for best cross platform compatibility please use OpenOffice to Export your CSV file.
You should select the Comma as your field delimiter, and the Double Quote as your text delimiter. You may not get proper results using Excel.

2) You MUST use products_models to distinguish your products for import.

3) Categories are handled differently from other versions of EP. You can now import mult-lingual categories (Like Products Names, etc.).
To achieve this, each v_categories_names_1, v_categories_names_2 correspond to a language installed in your system, and individual category names are separated by the Carat "^" symbol. For Example: Bar Supplies^Glass Washers^Brushes

Bar Supplies will be your Top level Category, with Glass Washers as a sub-directory, and Brushes as a sub-directory of Glass Washers.

Be careful when creating your category names. "Bar Supplies" is not equal to "BAR Supplies" ... this will result in two category entries.

As of 4.0.17, your lowest defined language (by language id) is needed to when creating categories. In the future I'll change this to the defined default language, but that
will take some additional work.

4) Work flow is somewhat different than other version of EP. With EP4 you first either upload your file, then click on Import. Optionally you can also Split an import file if it is too large. You can set the number of records to split on in the configuration settings. Default is 2000. If you have a powerful server, you can up this significantly. Testing on a VPS with an import file of 900,000 records, I broke into 50,000 segments. Import of each 50,000 records took about 250 seconds.

Also NO streaming upload or download. Sorry if you liked this feature. A lot of effort has been put into improving the code's performance and streaming the data was a real memory hog. To download your exported file, you will need to "right-click, save-as". You may be able to set your browser to automatically download CSV files. I'll come up with a better solution in due time.

5) File names act as a switch inside the script for Importing. Namely: PriceBreaks-EP, CategoryMeta-EP, and Attrib-EP ... thes import files must have names that start with these names.

6) Basic Attribute Import. Please read the notes in the Readme.txt file carefully for how to use this. I am still working on completing this feature. Currently, you can at Import create your products_options_name (your attribute name, ie. color) , products_options_type (checkbox, downdown, etc) and , products_options_values_name (red, green, blue).

Be sure to backup you data before importing your files. Remember, this is still "beta". I have made every attempt to make this a solid bug free product, but it does require more testing. I have added a lot of error trapping, but I'm sure I've missed things.

Finally, EasyPopulate 4 is an amalgamation of code and ideas from earlier versions of EP. All due recognition is given to these earlier works and those author's efforts.
Langer for bringing us 1.2.5.4 from OsCommerce, Phazie for his efforts to move to CSV, jbcholdings for his ideas in Attributr, and all the forum participation over the years from users ... Scrat, Elkbow, Schooboy... too many to mention!

I also want to thank all my Beta testers to date for given my version a try!

-chadderuski

@chadderuski, would you please package this up into a zip file (with a GPL License and instructions) and submit it to the Free Software Add Ons at the top left of this page? We try to encourage people to go to one single place to get mods, rather than have them scattered around.

swguy:

@chadderuski, would you please package this up into a zip file (with a GPL License and instructions) and submit it to the Free Software Add Ons at the top left of this page? We try to encourage people to go to one single place to get mods, rather than have them scattered around.

I had asked Dr. Byte if it was okay to keep on github while in "beta" and was told yes. I will add the GPL License (forgot about that, this is my first contrib). Once I finish rounding out the attributes and finish the docs, I'll make it an official submission.

Thank you! Developing, collaborating and testing off-site is great, but once it's really ready, this is the best place for it.

Right. Please package and upload new zips here (addons section) when you want people to use them or become aware of new features or milestones.

chadderuski,

Thanks for keeping this mod stable and viable, I think you've made a much needed and professional contribution to ZenCart!

I do need some help though. :)

I've got the mod installed and I've exported all of my product data to see what kind of formatting I need to use for the import process. My client currently has 5,000 products which were manually entered but has access to far more than that. They have been provided a massive amount of product data and I can manipulate that data in whichever way is needed for importing, but apparently I am missing something.

I tried a simple test by entering some mock data into what I [think] is a properly formatted CSV file but the product was not inserted into the database as expected.

I uploaded what I thought to be a properly formatted CSV file and then imported it. My results were as follows:

"Import Results

Filename: Test----NOW.csv"

What did I do wrong? Thanks!

Hi Keezee,

Did you use OpenOffice to export the CSV file?

Also, maintain the file naming convention: "Full-EP"+some text+.csv

You could also attached a sample of your data. Top dozen lines with headers.

-chadd

Hey chadd,

Wow, fast reply! Thanks.

I did try exporting with Open Office, but my file name was way off. I used a file name of ''Test----NOW.csv'

I changed that to "Full-EP2012TEST.csv"

Sadly, that did not work either. Can I use EP4 to simply add one product or will my upload always need to include all of my products?

My test file had only one line aside from headers as a quick test. Here is the top 12 lines of my original export:

Sample CSV file is attached, thanks again! Attachment #10012

I'll take a look at your file now.

No, I regularly use EP4 to export single sub-categories, modify and upload corrections/changes. Yes, you can just use 1 line of data. If fact, that is the very best first thing to do when you want to ensure your data is importing correctly.

Also, be sure you are doing a backup of your data first before running an import. There is an update MySQL database backup tool that work fine on most databases.

BRB

When using OpenOffice:

1) Select File -> Save As
2) Ensure File type: is Text CSV (.csv)
3) check the Edit Filter settings box
4) click save

In the Export of text Files box,

Select your Character set. Probably UTF-8 is you are on zen 1.5 now.
Field delimiter: Comma ,
Text delimiter: Double quote "
you can leave the save cell contents as show box checked.

Even so, your sample data imported to my server just fine.... so I do not know what you could be missing. It looks correct to me.
What version of zencart are you using?

Okay, tested your data on another server. Both 1.39h and 1.5 are pulling your data in okay.

What version of PHP are you running?

Thanks for all the help and tips.

I actually just tried to import the same file I sent to you and it didn't work in my end, odd! I'm using v1.3.9d

The only message I see after uploading and selecting 'import' is:

Import Results

Filename: Full-EP2012TEST.csv

I get no other information. I have idea where to start to fix this.

EDIT: Trying to confirm the version of PHP, but it should be up-to-date.
EDIT 2: ... PHP Version: PHP 4.x

Keezee:

EDIT 2: ... PHP Version: PHP 4.x

couple things:

1) I don't have a server running php 4... that's getting really out dated. It may be one of the functions I'm using will only work correctly in php 5.

2) Can you check your /cache folder to see if you have any error logs.

Your on to something...

[01-Jan-2012 22:11:09] PHP Fatal error: Call to undefined function: memory_get_usage() in /home/content/b/i/k/bikeral/html/admin/easypopulate_4_export.php on line 295

Ah, that function is only available in php 4.3.2 and above and php 5. This is part of the diagnostic code so you can see how much memory is being used by the script.
In some instances certain php setting need to be adjusted (php.ini) to allow more memory usage and/or longer execution time. We should be able to do a work around for you.

Hey chadderuski

Installed EP4 with no problems , Very easy :-)

I am currently looking at writing a little app with Autoit and Open office to add my products to a csv file for importing via EP4

Mostly my store is sells PDF's so I current have an Autoit script that adds products via a web browser and uploads the PDF's . It adds all the Product description and Attributes . Being downloads the attributes are important part of the process.


What I would like to do is add products thru EP4 . What Im wondering is it possible (once you finish the import side) to add product description and product attributes in the one file and then import. Or do you have another way of doing it ?

I'm guess the file needs the correct column headers and it will then import correctly . If this i the case I can code my little Autoit app in anticipation of your fully working script


Also doing a script to convert your output to a turbo lister import file


Thanks for your effort in working on EP4 :-)

Paul

Hasher:

Hey chadderuski
I am currently looking at writing a little app with Autoit and Open office to add my products to a csv file for importing via EP4

Mostly my store is sells PDF's so I current have an Autoit script that adds products via a web browser and uploads the PDF's . It adds all the Product description and Attributes . Being downloads the attributes are important part of the process.

What I would like to do is add products thru EP4 . What Im wondering is it possible (once you finish the import side) to add product description and product attributes in the one file and then import. Or do you have another way of doing it ?

I'm guess the file needs the correct column headers and it will then import correctly . If this i the case I can code my little Autoit app in anticipation of your fully working script

Paul

Hi Paul,

Currently, you will need to have two files. One to update products descriptions, and another to create and assign attributes. This process is much like what you would do in the Admin. Add a new product, then create your attributes, and assign them to the product.

Performing all three at once really complicates the code. Add to the mix proper multilingual support and its that add much more complex. So for right now it will have to remain a two-step process.

What kind of attributes are you creating and assigning to your downloadable products? You may have some detailed attribute information that I'm not addressing as yet. But it should be quite doable.

I will be working hard on the attribute importing code in the days/weeks to come.

-cj

hey cj

With downloading the 'Detailed Products Attributes' it seems to have most fields i use.

Im not sure how the zencart database works but in phpmyadmin I can see a table called 'products_attributes_download` which has 3 fields which I do use:

products_attributes_filename - name of media to d/l and sub folder eg /folder_sub/test.zip . This one seems important one your missing

products_attributes_maxdays and products_attributes_maxcount , These normal fill out to defaults in the attributes admin screen but may need to be added if doing thru the database directly


I think its fine doing a separate Product and attributes import file but Im wondering how your going to match Product description and attributes lines . Assign a products ID or match line for line.

Hope this helps :-)

Paul

Hasher:

hey cj

With downloading the 'Detailed Products Attributes' it seems to have most fields i use.

Im not sure how the zencart database works but in phpmyadmin I can see a table called 'products_attributes_download` which has 3 fields which I do use:

products_attributes_filename - name of media to d/l and sub folder eg /folder_sub/test.zip . This one seems important one your missing

products_attributes_maxdays and products_attributes_maxcount , These normal fill out to defaults in the attributes admin screen but may need to be added if doing thru the database directly


I think its fine doing a separate Product and attributes import file but Im wondering how your going to match Product description and attributes lines . Assign a products ID or match line for line.

Hope this helps :-)

Paul

Hi,

Please note that the attributes downloads links in EP are what I'm using to examine and understand the data's structure. There is no corresponding import of these fields - yet! Yes, there is ANOTHER table 'products_attributes_download' in addition to the products_attributes, products_options, products_options_names, products_options_type, and products_options_values_to_products_options.

These all work together to do your attributes! Now I'm sure the original designers of the attributes features understood them inside out, but I'm reverse engineering my understanding and their relationships so it's been... complex. Additionally, translating this into a functional way to creating/modify/update at import has been a real challenge. I'm on the right track, but it will take some dedicated time to complete this.

Langer had originally attempted to solve this by including every column for every combination of options names and values. If you had 30 attributes with 5 options each that'd be 150 columns. Not what I considered a practical solution.

My solution drastically simplifies the process. If you look at the sample Attrib-EP.csv file you'll see what I mean. Adding the other columns with be easier once I have the BASIC import <> export working correctly. I need to first make the BASIC export match my import file where the product, attribute type, option name, and option values export on line line.

I don't want to promise a solution date, but I will be working on this!

Thanks cf , understand what your trying to do now much better :-)