EasyPopulate 4.0 Support Thread

[Feznizzle]

I'm in the process of building a catalog with about 25,000 products. Even when the data is segmented into groups of 5,000 products, my spreadsheet (libreoffice) bogs down when all data is present (particularly the html for product description pages).

To make this manageable, I want to upload/insert in stages. First I'll CREATE products by inserting bare minimum info (model, ID, Category, Prod name, qty, status, order units, etc). After that, I'll come back and UPDATE those products by inserting additional data (image, price, description, etc).

The following questions are about working with product files (eg, Full-EP2017Sep26-165412.csv).

Product page creation questions:
1. When creating NEW product pages, can I leave the following columns blank (no data)?
v_products_description_1
v_products_price
v_products_weight
v_date_avail
v_date_added

If not, I'll insert small dummy info to replace during later updates.

2. When creating NEW product pages, can I *delete* columns I don't need at the moment (meta data, specials, etc) to reduce clutter? Or does EP4 freak out if columns it expects don't exist?

Updating product page questions:
3. When updating EXISTING product pages, can I *delete* columns I don't wish to change (category, product name) to reduce clutter? Obviously I'll keep the important ones (model, ID, etc).

4. If I have to keep all columns, can I leave the ones I don't wish to change blank?

[mc12345678]

Sorry for the delay in responding.

So... First thing and I know it doesn't directly help with the local effort of adding product to the spreadsheet, but EP4 offers the ability to split a file that is either exported to the server or uploaded to the server. So you could (later) do updates of existing data by exporting, then splitting then downloading so that when you open the file you aren't "troubled" by too many rows/too much data.

That said, if you haven't downloaded the github version which supports product export without including linked product, then when a set of data is split, there is the possibility, however remote, that a row of data for a product could be in one split file and another... the last one imported wins...

To your questions about uploading new product. The concept of the software is that when working with the main product information, only required fields need to be in the file. For new product that is a primary key and a category. For any field not included, generally speaking it will default to some form of emptiness. For dates, if the date_added field is not included, then it will default to now, whenever now is. If the field is included and it is blank or has a date less than 0001-01-01 00:00:00, then it will be set to that (which should be the default in the database, but that's a different story.) v_date_avail does not default to a now condition, but instead a "blank" condition.

As to product status, if it is not provided then it should default to either off (disabled) or the status default in your database. There is a consideration though as to if a product is uploaded that has 0 quantity and the store is set to deactivate product that have 0 quantity then the product will be deactivated regardless of the condition of the uploaded status. Sort of an EP4 working with ZC condition.

Otherwise, regarding your "plan" for updating. Again, when discussing the main product information (not yet incorporated in attributes and a couple other off-shoots), if you don't want to even possibly change a field, then the field column should and can be omitted. If the field is present, whatever is in that column will be attempted to be pushed to the database. If that means the column is blank, then the data will be removed, if it changed from "up" to "down" then "down" will be what remains...

Now, if I have incorrectly stated these things, such that the software does not operate as I have described, please identify as it more than likely is a mistake.

Now, attributes and category specific changes, I know attributes haven't been further modified to permit import with fields missing, but I think at the moment that category import is the same (operation hasn't been modified to support upload with missing fields).

Again, to your last question, there has not been a setting established to leave a field alone if the field is present but the row data is blank. I don't seem to recall anyone asking for that and I tend to look at it as extra work to blank a field to prevent an update as compared to possibly two different file imports or to leave the data as is. (it's possible in most spreadsheet programs to "lock" a field to prevent editing, so would recommend that route instead personally...)

[Feznizzle]

Sorry for the delay in responding.

So... First thing and I know it doesn't directly help with the local effort of adding product to the spreadsheet, but EP4 offers the ability to split a file that is either exported to the server or uploaded to the server. So you could (later) do updates of existing data by exporting, then splitting then downloading so that when you open the file you aren't "troubled" by too many rows/too much data.

That said, if you haven't downloaded the github version which supports product export without including linked product, then when a set of data is split, there is the possibility, however remote, that a row of data for a product could be in one split file and another... the last one imported wins...

To your questions about uploading new product. The concept of the software is that when working with the main product information, only required fields need to be in the file. For new product that is a primary key and a category. For any field not included, generally speaking it will default to some form of emptiness. For dates, if the date_added field is not included, then it will default to now, whenever now is. If the field is included and it is blank or has a date less than 0001-01-01 00:00:00, then it will be set to that (which should be the default in the database, but that's a different story.) v_date_avail does not default to a now condition, but instead a "blank" condition.

As to product status, if it is not provided then it should default to either off (disabled) or the status default in your database. There is a consideration though as to if a product is uploaded that has 0 quantity and the store is set to deactivate product that have 0 quantity then the product will be deactivated regardless of the condition of the uploaded status. Sort of an EP4 working with ZC condition.

Otherwise, regarding your "plan" for updating. Again, when discussing the main product information (not yet incorporated in attributes and a couple other off-shoots), if you don't want to even possibly change a field, then the field column should and can be omitted. If the field is present, whatever is in that column will be attempted to be pushed to the database. If that means the column is blank, then the data will be removed, if it changed from "up" to "down" then "down" will be what remains...

Now, if I have incorrectly stated these things, such that the software does not operate as I have described, please identify as it more than likely is a mistake.

Now, attributes and category specific changes, I know attributes haven't been further modified to permit import with fields missing, but I think at the moment that category import is the same (operation hasn't been modified to support upload with missing fields).

Again, to your last question, there has not been a setting established to leave a field alone if the field is present but the row data is blank. I don't seem to recall anyone asking for that and I tend to look at it as extra work to blank a field to prevent an update as compared to possibly two different file imports or to leave the data as is. (it's possible in most spreadsheet programs to "lock" a field to prevent editing, so would recommend that route instead personally...)

YAHOOO! That bolded out statement is faaaaantabulous!

To be super clear, you're saying that after a product has been added, future uploaded files can (and should) contain only the primary key (for me it's ID) and the column(s) containing data I wish to insert/change?

For instance, I could upload a csv with only two columns of data like so:
v_products_id and v_products_description_1

If so... awesome. This just got really easy. :)

Thank you so much for providing so much info!

[mc12345678]

YAHOOO! That bolded out statement is faaaaantabulous!

To be super clear, you're saying that after a product has been added, future uploaded files can (and should) contain only the primary key (for me it's ID) and the column(s) containing data I wish to insert/change?

For instance, I could upload a csv with only two columns of data like so:
v_products_id and v_products_description_1

If so... awesome. This just got really easy. :)

Thank you so much for providing so much info!

That's the expectation. If it doesn't deliver that way, please advise. There's a lot of possible columns and I don't think each individual scenario was attempted, but it was all based on a philosophy developed by the needs of those that asked and demonstrated how such feature(s) could be advantageous and make the work easier. :) So, if I lied to you, I'd like the opportunity to either fix the issue or explain why it is the way it is.

[Feznizzle]

I uploaded in two passes... worked like a charm, exactly as advertised!

FIRST UPLOAD (created 20k+ products)
I left all column headers from template in place. This file contained most of the real prod data (model, image, price, etc), except page html and meta title/description. Once uploaded, I let EP4 segment before I inserted.

SECOND UPLOAD (updated all the freshly created products)
This file contained only two columns of data, product ID (my control) and html. It worked perfectly, with only one minor problem.

Problem was that the csv was really big (54mb). When I used EP4 to upload, it failed for some reason so I ended up having to use FTP client for upload. Once in, I attempted to split the file using EP4. I have it set to split at 5000 records, so I expected 5 segments (same as first upload above). Ended up with like 100 segments! lol

No biggy. I manually split, uploaded and inserted. My smaller segments went thru without a problem.

Anyway, wanted to follow up and let you know it went smoothly. Today I'll be updating all meta data using the same limited column strategy. I don't anticipate any issues, but I'll report back if any occur.

Thanks a lot, MC. You da man!

[mc12345678]

Great information and glad that it worked so well (mostly) :). While I've added to this plugin, it certainly was built on a concept and with a design of others so I can't take all the credit. :)

Regarding the multi-file split that occurred. Did you happen to look at the contents of one or more of them to see what they held or be willing to try again and see?

I'm curious to find out what conditions were in place to cause that issue.

[Feznizzle]

About the split, I'll have to recreate for you.

But I just hit a massive roadblock. I've been at it for 12 hours, head throbbing, ready to smash my puter!

I just used EP4 to upload all meta titles, descriptions. Since I had flipped on all the meta statuses I wanted during my first product upload, all I was uploading was three columns of data (ID, Title, Description).

Nothing happened. I mean, meta title/description got inserted... but they didn't show up in page source. So I went to a product via my admin panel and tried to flip meta data on manually. I could see the meta stuff I wanted in Admin, but couldn't make them active.

I'll trouble shoot in the morning, when I can finally think again.

Is it possible I broke something during the initial upload by setting v_metatags_title_status to 1, but then leaving v_metatags_title_1 blank?

To test that possibility out, I'll revert back to before that first big upload. When I repeat the upload, next time it will have all the meta info with it.

But I think I may have actually broken something.... even before the EP4 insertion. The amin itself won't allow me to change meta settings.

Ug. Anyway, I sooooo done for the day. It's miller time. Better yet, it's Makers time!

[mc12345678]

So, Hmmm. Had a chance to reread what was provided above, look through the forum and read some code.

EP4 regarding metatags does still have one requirement which is to include the metatags_title (which you have done) in order to process in metatag information. The next thing for a ZC side, is that the applicable metatag information needs to have a status of 1 (true) in order for ZC to say, yes should display. In between those two is that EP4 needs to know whether it should try to process/export metatag information from the configuration menu.

Based on the provided process I'm thinking the individual status setting did not get set/establish for the records and so the data is present but not displayed?

[Feznizzle]

I think you're right.

I just used EP4 to upload, split, then import meta info again. This time I included all meta variable columns, set as I desired (0 for status I didn't want, 1 for status I wished to use).

Went smooth, no issues to report.

Thank you so much for all your help!

[cindygordon]

There are so many places to download, which is the place to download the most current version? Is it the one listed on the first post of this thread, that is dated April 2006 or is that just a beta?