Upgrading Strategy Advice? ... for specialized site

Hi,

I'm planning to deviate from the standard upgrade protocol, I'd appreciate it if anybody could poke holes in my plan.

MY PROPOSED STRATEGY (upgrading zc151 to zc154)
**PART 1 – Prepwork
**1. Create new subdomain (zc154.mysite), create new database, install fresh zc154.
2. Backup the live site db, import into fresh zc154, run the upgrade.
3. Install the responsive template.
4. Backup/import all the images from the live site.
5. Upgrade all mods, tweak until everything is running smooth.

Because time will have passed since Step2 above (upgrading Live site DB), I will need to manually capture any new customer data that may have occurred. So I will need to take the Live site db, upgrade it, dump the needed tables, and update the upgraded sites db. So...

PART 2 – Going live

1. Repeat Steps1&2 above on a new subdomain to get LIVE zc151 updated for zc154.
2. Track down and import all new traffic/customer data into the fully upgraded site db.
3. Port the upgraded site over to the Live domain.
4. Reinstall all modules (shipping, taxes, payment, etc) on Live.
5. Pray!

QUESTION

• Do you see any problems with this strategy?
• Can anybody tell me where I can find a list of tables that will change between db updating: address_book, orders, orders_status, admin_activity_log, and so on? If not, I can probably wing it. I may focus only on customers/orders because that is the most important bit and would limit the scope so I don't risk a screwup.
• Any thoughts? Suggestions to make this easier?

WHY DEVIATE FROM PROTOCOL?
This upgrade is about the toughest task I've ever faced (with web dev) and it's very much stretching my abilities.

I have three major problems. First, the site is heavily modded. Second, the site has tons of products, images, content (zipped, itweighs about 600mb). Third, the live version is active with new traffic, customers, orders, etc coming in every day.

One of the main goals I have is take the current template and upgrade it to a responsive version of itself. Since I'm trying to keep the same look for the primary version of the site (desktop), I need to work with all the categories/products/ezpages in place so I can match them up (style wise).

Because the site is so heavily modded (10+ addon mods, several of those have been heavily customized, plus there are few 100% custom mods), I have to upgrade the db numerous times as I upgrade the mods. I'm afraid that I'll forget critical database tweaks made subsequent to running various SQL updates. That could get super messy.

Plus, I'm very visual. I need to be able to see my progress and verify with my eyeballs that my changes are in place and working as they should. And the responsive version of the template is very different in some areas.

This just seems like the only way to go. For me.

What do you think?

My thought? Be prepared to take an extensive amount of time to get what you want.

Everything described can be accomplished with a single sql update file put together as you go along a single attempt of upgrading... you're far more likely to skip a critical step, object, piece of data following the above described deviation than to use the relatively tried and true method put together to provide an upgrade process applicable to the mass of people (thought I saw a number like tens of thousands?) that use ZC.

The above process as with the recommended process requires a moment of site outage in order to put things live while being up-to-date. So, that will be necessary regardless of the method. Further, from the above description the items of concern to "support" the newly upgraded site are things which you have total control over: product, category, etc. Description. These are things that can be updated via sql. The "hard" part may be to limit the live store product changes during the upgrade process, though those too could be captured in sql and then updated through the process...

Nothing described though seems like it necessitates the longer involved process of trying to correctly capture the differences in the two databases (at some point both database versions requiring an upgrade so that the information properly lines up).

Every online business tends to have some point in time when the access to the store is limited for a short duration. You can plan for this and notify your customers of an upcoming outage during which the final live upgrade would occur (file folder(s) moved, configure.php files modified as necessary or through the upgrade process, database upgraded with zc_install, sql patch file run for all the things that need to be captured as identified by doing the upgrade on the backup/sub-domain.)

My thoughts, asking for a lot of hurt... can be done, but the "incremental" modifications can be proven to yourself using the backup store and apply those in a single round rather than the outage that would exist to do each and every "upgrade" one at a time or try to individually recopy the changes made for the product(s) or other data to look as desired out the gate.

I must admit when I start to read the official instructions for upgrading my head starts to spin.

I have been upgrading to 155 for months but my process is.
Local install for development work, on php 7.

1. New Install of zc155a
2. Import real database
3. Upgrade database
4. Merge mods from old site but maintaining current template. Baby steps, and easy to see when something is out of place on the new site.
   Write comments in any changes to core files.
   In the process do a pile of spring cleaning and keep in mind the pending change of template.
   This is what takes me ages, as I snarl at mods I did years ago that are too ugly to leave as is and spend time fixing stuff that isn't broken.
5. Make notes of any database fiddling required...when I find something that needs changing I backup the database and create/test an sql query to do the job automatically and so gradually build up a set of commands to use in one go when the time comes for the real upgrade.
6. When all seems well and good, replicate the files on the server like in /shop_155.
   7)Import real database and upgrade again.
   8)Apply custom sql changes.
7. Test to death.
8. Take old shop offline.
9. Import db to new shop and upgrade do sql commands again.
10. rename old /shop to /shop_154 for example.
11. rename new /shop_155 to /shop and change configure.php to suit.
12. Switch on.

I estimate 5-10 minutes offline, if that.

This leaves the old working shop in place as a reference.

When all seems ok, think about implementing new template (locally etc...).

I do it slightly differently than torvista.

a) stage the whole thing on my own local server (new install, import db)
b) inspect all file differences, remaking all customizations into the new site, including upgrading of plugins
(Since I'm using a copy of the live db, I take notes of any new SQL changes I have to apply when upgrading plugins. Sometimes I apply those to the real live site before I actually go live, especially if I'm certain those changes won't negatively impact the live operation. This just helps minimize downtime when going live.)
c) test test test all along

When ready to go live:
d) copy the new /admin/ and /includes/ folders (and any other actual changed folders) to the live server as new_admin and new_includes
e) put the site down-for-maint
f) upload zc_install, and rename admin to old_admin, then new_admin to admin ... same with all other folders. This takes just a few seconds.
g) run zc_install to upgrade the db ... again just seconds to do
h) turn off down-for-maint

Done. Downtime is usually just a couple minutes. The critical part affecting shoppers is the swapping of the /includes/ folders from old to new, and the time spent with zc_install. Outside that shoppers aren't impacted, and no database data is lost or subjected to risk of damage/loss due to unnecessary export/import.

And I'd do template alterations as a completely separate step. That means I'd upgrade the present template by applying onto it any changes done to template_default between the versions. Then if a template change is needed it becomes a separate step. Again to minimize downtime.

While I would start with the template and torvista would add it later, regardless of that part (somewhat comes down to personal preference and that plugins are typically written to be functional with a base ZC install and therefore should continue to be functional with each other when properly incorporated to do so. But suggested part of all that, plan on creating a template directory. Then make the modifications desired/needed in it. This allows you to undo changes easily, to switch back to a default or even previously "functional" version of the template, etc... there is a plugin that is now available that supports backing up a modified template (expects all modifications to exist in the applicable template override directory).

This way, regardless of the final look (css, html tags) at least the operation(s) can be proven to function and when the template is actually applied, the "data" is there, and just have to work out the details.

I like the other way, such that as the new data is being added to the screen, other presentation modifications are made as necessary or the action location modified to get the desired result where wanted. Matter of taste for us and others. :)

Thanks for the input, MC.

At the bottom, you say:

the "incremental" modifications can be proven to yourself -- in a single round rather than -- doing each -- "upgrade" one at a time
Good catch, sorry! I missplaced Step 4 under PART 2, should appear as Step 6 under PART 1.

Also you point that manually importing data can easily lead to data loss. To address that possibility, I've come up with this alternative. As I'm working PART 1 I will keep a diary of all sql updates. By that I mean, if an addon upgrade (say Ceon URI) requires that I run an sql update, I'll note it and set the updater aside. Then I'll run it later on the current db before going live.

**Here is the revised plan:
****PART 1 – Prepwork
**1. Create new subdomain (zc154.mysite), create new database, install fresh zc154.
2. Backup the live site db, import into fresh zc154, run the upgrade.
3. Install the responsive template.
4. Backup/import all the images from the live site.
5. Upgrade all mods, noting any that require sql updates and setting the updaters aside.
6. Reinstall all standard modules (shipping, taxes, payment, etc).
7. Tweak until test site is running smooth.

**PART 2 – Going live
**1. Repeat Steps1&2 above on a new subdomain to get LIVE zc151 updated for zc154.
2. Rerun every sql update noted in Step 5 above.
3. Port the upgraded site files and db over to the Live domain.
4. Pray!

Is that better? Or do you still see the plan as problematic? Whatcha think, homey?

Btw, there will be no new prod/category/etc changes while i'm working. The only new data will be traffic related (new customers, orders, order updates, etc). But now that shouldn't matter, accept that if a new product is added the associated image will need to be re-uploaded (easy peasy).

I made my last comment before realizing drbyte and torvista had commented.

torvista:

I must admit when I start to read the official instructions for upgrading my head starts to spin.
AMEN!

Interesting that both of you say to upgrade the current template first. I'll have to think hard on that for a bit because my first instinct to go MC's route (start with template). Many (if not most) mods require template file alterations and overrides. Seems like a lot of extra work to have to do them all over again! But I can see how any problems with the upgrade would become obvious if you were looking at the site you already know so well.

Lot to process. Thanks so much for relaying your own methods.

Feznizzle:

Many (if not most) mods require template file alterations and overrides. Seems like a lot of extra work to have to do them all over again!
Well, the amount of template-specific changes between ZC versions is generally minimal. So identifying and applying those changes usually takes a fraction of the time in my experience.

You said the "instructions make your head spin". I'm not sure why. They're just saying the same thing I posted above, but in a long detailed step-by-step way so the person who doesn't grok the terminology or concepts has something to go on.
I did abbreviate it quite a bit in this post awhile back: http://www.zen-cart.com/entry.php?3-How-do-I-rebuild-my-site-on-the-new-version-instead-of-upgrading

One other thing. You've indicated that you're going to stage your upgrade in a subdomain of your site. Does your site employ a wild-card SSL certificate? If not, are you planning on purchasing an SSL certificate for that sub-domain?

FWIW, I usually stage an upgrade in a subdirectory off the target site (e.g. https://www.example.com/upgrade) so that I'm using the store's SSL certificate to validate the checkout-process and the SSL-side of the admin.

lat9, that's an awesome suggestion! Thank you!!!

But... now I have a really, really dumb question to ask. If new/upgraded store is in a subdirectory, will it obey .htaccess rules from the folder above?

Drbyte, your point about easily finding changes in the older template makes a lot of sense.

Regarding the head spin bit, it's all the extra info added to aid grokking for the inexperienced that makes it painful. I actually took the extended version and copy/pasted into a blank file, then when thru and deleted all the grok stuff and anything that didn't apply or I didn't need to manually arrive at a simplified overview of the process, similar to what you posted above. :)

DrByte:

You said the "instructions make your head spin". I'm not sure why. They're just saying the same thing I posted above, but in a long detailed step-by-step way so the person who doesn't grok the terminology or concepts has something to go on.
I can remember my first attempt at upgrading. I had "temporarily" installed version 1.3.9.h I think it was, it might have been g using an autoinstaller from the host. I didn't know a thing about how to establish a database, how I was going to upload the files, etc. But there was a convenient button that would do it all. I then realized that zen cart as a larger group existed beyond the little button that was there.

Found that there was in fact a newer version, started reading that same help file/location which was different then other direction that was found, but realized that in the first few sentences it was outlining a "philosophy" of what was needed. The details had to be filled in on how to do it.

I got confused on old database, new database, push this here, that there. I ended up drawing a sort of road map. Then when I saw someone else either describe or try to describe in such a "simple" matter, it really fell into place.

In that "research" time I carried a printout of those instructions with me for days and would reference back to them when my mind got to where it seemed like it understood but wasn't sure. So I too would say that it has been a perplexing "set of instructions" but at the same time spot on. :)

I think in the end I relate it to the game of towers of Hanoi in the incremental strategy of targeting a goal, sometimes repeating a previous action (for good/required reason), and each action is performed to prepare for the next

My thing about doing template later, learned from a lesson of eventually trying out different templates after the "other stuff" was mostly how desired was that I came along one (still offered on this site) that pretty much hijacked the catalog side and also various settings on the admin side... after installation, I wasn't able to swap to a different template that I had already installed to check out differences and get a feel for what was to be in place. I was able to back track the modifications and restore to the previous functional test site, but it was truly a pain. Also I have seen some templates that will snip a section out of one portion of the template and put it elsewhere, so any related code had to also move in order to keep the results similar to the desired effect. Further some things when presented as originally designed (placed in the template) don't end up looking right when applied to whatever other template that was desired, so... made a lot of sense (to me) to pick the template first, then where something modifies the look of the template to incorporate it where needed/desired.

But it is like solving for say the length of a line from two corners of a parallelogram. Go one way first and then the other, whatever is done get to the end. :)

Feznizzle:

lat9, that's an awesome suggestion! Thank you!!!

But... now I have a really, really dumb question to ask. If new/upgraded store is in a subdirectory, will it obey .htaccess rules from the folder above?

Drbyte, your point about easily finding changes in the older template makes a lot of sense.

Regarding the head spin bit, it's all the extra info added to aid grokking for the inexperienced that makes it painful. I actually took the extended version and copy/pasted into a blank file, then when thru and deleted all the grok stuff and anything that didn't apply or I didn't need to manually arrive at a simplified overview of the process, similar to what you posted above. :)
Regarding htaccess, yes it will, exactly as written which could work for or against you when considering your original rules may have been written to target one aspect specifically but considered to only apply to one site, where here you would have "two" sites in different Web facing directories.

Feznizzle:

Regarding the head spin bit, it's all the extra info added to aid grokking for the inexperienced that makes it painful.
I did abbreviate it quite a bit awhile back here: http://www.zen-cart.com/entry.php?3-How-do-I-rebuild-my-site-on-the-new-version-instead-of-upgrading

Feznizzle:

Thanks for the input, MC.

At the bottom, you say:

Good catch, sorry! I missplaced Step 4 under PART 2, should appear as Step 6 under PART 1.

Also you point that manually importing data can easily lead to data loss. To address that possibility, I've come up with this alternative. As I'm working PART 1 I will keep a diary of all sql updates. By that I mean, if an addon upgrade (say Ceon URI) requires that I run an sql update, I'll note it and set the updater aside. Then I'll run it later on the current db before going live.

**Here is the revised plan:
****PART 1 – Prepwork
**1. Create new subdomain (zc154.mysite), create new database, install fresh zc154.
2. Backup the live site db, import into fresh zc154, run the upgrade.
3. Install the responsive template.
4. Backup/import all the images from the live site.
5. Upgrade all mods, noting any that require sql updates and setting the updaters aside.
6. Reinstall all standard modules (shipping, taxes, payment, etc).
7. Tweak until test site is running smooth.

**PART 2 – Going live
**1. Repeat Steps1&2 above on a new subdomain to get LIVE zc151 updated for zc154.
2. Rerun every sql update noted in Step 5 above.
3. Port the upgraded site files and db over to the Live domain.
4. Pray!

Is that better? Or do you still see the plan as problematic? Whatcha think, homey?

Btw, there will be no new prod/category/etc changes while i'm working. The only new data will be traffic related (new customers, orders, order updates, etc). But now that shouldn't matter, accept that if a new product is added the associated image will need to be re-uploaded (easy peasy).
Far less problematic. In the details, but "section 2" doesn't go back over the reinstallation of all of the "modules" (shipping, payments, etc.) and of course with regards to ALL the other sql, the "running" file that torvista suggested can address that. DrByte's suggestion of parallel upgrading both the live and new store for some cases, this can work even with some of the ZC core code as well, though it tends to be best suggested to get the rest of the files and not just patch sections of the code to bring them up to the latest. That's somewhat of a time relative thing. Today, tomorrow, okay, next year, not so good. :P

Kind of missed a part 1.5. Now that part 1 was successful after a bit of work. Is everything in place/available to repeat it and get the same result? (helps shorten the prayers... :P)

Images, want to be sure that only porting in images that you are familiar with. Ie. Truly be sure that 1) the files in the folder are images, 2) that they are the ones you expect/know about (Ie. Follow whatever naming convention, there's a tool that will help identify extra pictures that are not mapped and vice versa, etc...) not just blindly copy the directory. So something like backup/import all of the applicable, necessary and/or expected images from the live site to the backup site.
Although really, unless the specific images are needed for some form of formatting or other "arrangement" (ie. Template images generally) I don't worry about copying the images over, but do work through some form of weeding/review process on the live site mostly because of the modified part 2s that were described by torvista and DrByte so I kind of don't feel like reproducing their efforts.

As said, a lot of different ways to get to the finish line. In the end the down time really should be negligible. I tend to keep the site off line a little longer to manually run through a purchase cycle, but there are also possible "automated" things that could be done to at least verify the operation. (pre-populate a cart, allowing to go straight to and through checkout, automate a login, etc...) remember the whole thing is open source. Can set up to do/have anything. Just be sure to remove such tools before the site is put back online (though that too could be automated...)

DrByte:

(Since I'm using a copy of the live db, I take notes of any new SQL changes I have to apply when upgrading plugins. Sometimes I apply those to the real live site before I actually go live, especially if I'm certain those changes won't negatively impact the live operation. This just helps minimize downtime when going live.)

i am also a fan on modifying the live DB for mods not related to the core. in my experience, most of the mods add fields or change lengths and will work fine on the older release. one could also keep a separate file of all the sql commands as you make them and then apply all of them when doing the upgrade.

i have also found using a source code repository and a version control system to be invaluable. its a bunch more work in the beginning (if you have never used them before) but i have found it invaluable as the project gets bigger and bigger.

good luck!

I recently did an upgrade of a heavily modded zc151 site to zc155a for a client

Feznizzle:

.....
MY PROPOSED STRATEGY (upgrading zc151 to zc154)

Why aren't you upgrading straight to zc155a? Is there any particular reason? Saves you another headache later.

Feznizzle:

.....
I have three major problems. First, the site is heavily modded. Second, the site has tons of products, images, content (zipped, itweighs about 600mb). Third, the live version is active with new traffic, customers, orders, etc coming in every day.

One of the main goals I have is take the current template and upgrade it to a responsive version of itself. Since I'm trying to keep the same look for the primary version of the site (desktop), I need to work with all the categories/products/ezpages in place so I can match them up (style wise).

Because the site is so heavily modded (10+ addon mods, several of those have been heavily customized, plus there are few 100% custom mods), I have to upgrade the db numerous times as I upgrade the mods. I'm afraid that I'll forget critical database tweaks made subsequent to running various SQL updates. That could get super messy.

Plus, I'm very visual. I need to be able to see my progress and verify with my eyeballs that my changes are in place and working as they should. And the responsive version of the template is very different in some areas....

I do my upgrades on an identical clone of the live site which I have on my local development server (desktop if you want). The only coding difference between the live shop and the local clone are the 2 configure.php files, and of course I don't get orders on the local dev server.

It really makes no difference at all if a live store is super busy or a slowmo, if I have 1000's of products or only a few, 1000's of images or only some, many mods or none. Whilst I am going about my business of upgrading locally, the live store is just business as usual.

It also helps if you have a list of addons/changes/mods, you name it, whatever changes you made in the past.

When I get a heavily modified site like yours to upgrade, I first look at the installed plugins. Are they compatible with the new ZC version I am upgrading to? Are there up-to-date versions of the plugins?

If so, then all is good and I can move on and get my trusted Beyond Compare to work and make the core file changes between ZC versions, then apply the latest versions of the plugins, uninstall shipping and payment modules and finally run a local zc_install which updates my 2 configure.php files and the DB (all locally). Now re-install shipping and payment modules. Then, the acid test, and more testing, testing etc etc (need to stop here).

If not (plugins are not compatible with the latest ZC version), then I need to uninstall them first before I can proceed as in the previous para.

Important to note at this point which files in the new ZC version are obsolete and have to be deleted (locally and live).

In fact, make notes about everything which may be useful when doing the live upgrade later

You can do all the stuff above at your leisure when you have time at your own pace. No harm done....

Now, let's assume the local testing comes up trumps all around, the serious part begins:

1. pick a quiet time on the live server/site
2. do another full backup (files and DB), zip it up and stow it away safely, just in case.
3. live site down-to-maint
4. uninstall paymet and shipping modules on the live site (take not of all settings before your do)
5. upload all files from local installation to live server (minus the 2 configure.php files, best to set them to 400 on the live site)
6. delete obsolete files on live server as per list above (your notes)
7. run zc_install, you will be prompted by ZC, it complains about anything which you need to do before proceeding
8. all done? good, delete zc_install folder
9. log into admin
10. re-install shipping and payment modules, apply your settings

Now you can still do some live testing if you want or take the site out of maintenance.

That's it.

I had not a single hiccup on that heavily modded zc151 site which I recently upgraded (for a client) to zc155.

Just repeating: when upgrading a site like yours, it is imperative to make notes on a progressive basis so you can apply them later when you do the live upgrade

One small note on plugins for zc155: most of them have auto-installers these days or no sql patch needs to be run manually.

Cheers/Frank

Again, huge thanks to all for the great advice. I feel a lot better now, seeing that everybody has developed their own way of doing things. I was deeply concerned about carving my own path.

Frank, you asked why not go all the way up to 155a. That was the original goal, tried very hard to pull it off. But my main directive was to upgrade to a responsive template and there are none for 155 yet. I attempted to take a responsive template for 154 and upgrade for 155 but failed.

As unhappy as I was about that, deciding not to go all the way up was probably for the best. Many other addons are also not 155 ready.

Feznizzle:

Again, huge thanks to all for the great advice. I feel a lot better now, seeing that everybody has developed their own way of doing things. I was deeply concerned about carving my own path.

Frank, you asked why not go all the way up to 155a. That was the original goal, tried very hard to pull it off. But my main directive was to upgrade to a responsive template and there are none for 155 yet. I attempted to take a responsive template for 154 and upgrade for 155 but failed.

As unhappy as I was about that, deciding not to go all the way up was probably for the best. Many other addons are also not 155 ready.

Okay, so there's the default one included in ZC 1.5.5 and there are some modifications that can be made to at least Anne's templates to get them working on ZC 1.5.5. Instructions are relatively outlined in the following thread: https://www.zen-cart.com/showthread.php?219801-Making-v154-DIY-Picaflor-Azul-templates-work-with-v1-5-5
Yes requires a little work, but...

mc12345678:

Yes requires a little work, but...

Indeed, however it appears beyond me. And thanks for the link, though I've been there done that. I gave it a go, ended up with gobs of glitches. Then I paid a skilled zenner to give it a go, ended up with just as many glitches. Had the template worked out, I still would have faced headaches trying to upgrade necessary mods on my own.

At this point, it's now time to just git 'er done! :)