Duplicate Orders

[lhungil]

Just a short list of quick thoughts / speculation. Probably things you've already contemplated / checked. If you cut over the DNS does the lesser VPS still keep up (or does it start to lag)?

Similiar load on both sites (connections, end users, etc)? Same server farm (and network) for both VPS? Same underlying OS / configuration / optimizations? Same PHP <==> HTTP server setup? Any noticeable pattern looking at page parse / SQL times between the two? Is one noticeably slower when writing to the DB via Zen Cart (possibly test using EZ Populate or similiar)? Any "security" software on either? Any proxy / cache / CDN setups? Any "limiting" or QOS? Any output filters (PHP or HTTPd)? Similiar physical hardware for both hosts (CPU, mem, hdd - speed, latency, bandwidth, etc)?

[RodG]

If you cut over the DNS does the lesser VPS still keep up (or does it start to lag)?

Sorry. I don't quite follow. Both sites are using the same DNS. The original site is "www" and the copied site is call "www2"

Not only was this the easiest approach, but I figured this would take DNS out of the equation.

Similiar load on both sites (connections, end users, etc)?

No. The copy site only has me accessing it, so the live site is certainly going to have a heavier load, but certainly not enough to cause a performance issue - And only with some functions??

Same server farm (and network) for both VPS?

No. totally different - intentionally - just in case it is/was caused by some inexplicable networking issue within the server farm.

I'm not sure if I've proven this to be true or not though - It is one of the few things different between the sites.

Same underlying OS / configuration / optimizations? Same PHP <==> HTTP server setup?

As near as possible. All versions are the same on both servers, but I've needed to tweak the mysql settings on the copy server because of the smaller memory. On the live server I have 3Gb 'reserved' just for mySQL, on the copy server it is using the default memory allocations (a lot less).

Hmmm, perhaps I should reset the mysql settings on the live server back to the defaults. I can't imagine this will improve things. Those settings haven't been change since I 1st set this server up about 18 months ago.

[/QUOTE]
Any noticeable pattern looking at page parse / SQL times between the two?
[/QUOTE]

For general page loads, both are on par with each other (the live server being actually a little bit faster - which is pretty much expected considering it has double the grunt).

Is one noticeably slower when writing to the DB via Zen Cart (possibly test using EZ Populate or similiar)?

I don't use EZ Populate or similiar - but again, almost everything about the site(s) are fine - adding/removing from cart, navigating pages, logging on/off, adding/editing products, and so on - The problem only shows with the 'confirm order' (customer purchase) and the 'update status' (admin).

Oh, another finding - I initally said one of the sites had the update problem and the other had the confirm order problem - This turned out to be incorrect - both sites exhibit the same two problems.

One of them (mine) is still using Zen 1.3.9 (I use it mostly for ozpost compatibility testing), the other is using Zen V1.5.4

Any "security" software on either?

Yes, but all have been disabled at various stages of debugging. The problem continued.

Any proxy / cache / CDN setups?

There is obviously caching going on somewhere along the line (local browser, ISP transparent proxy, and obviously? the ImageHandler cache) - but these will be consistant with either site. The servers aren't proxied (that I'm aware of - I'd be piffed if they were). We do use a CDN for both sites and servers, but disabling this didn't change the problem one iota.

Any "limiting" or QOS?

Nothing I've implemented. Who really knows what the hosts have going on behind the VPS.

Any output filters (PHP or HTTPd)?

mod_security (default settings) on both servers - No difference when I've disabled it.

Similiar physical hardware

No way of knowing - they are both VPS's rather than bare metal servers.

for both hosts (CPU, mem, hdd - speed, latency, bandwidth, etc)?

2core vs 4core (4core has the problem)
2Gb RAM vs 5Gb RAM (%Gb has the problem)
Virtualised HDD vs SDD. The HDD is the one with the problems, but there is no disk swapping going on (in fact there is enough memory in reserve on this VPS that I didn't even bother creating a swap file :-)

So, to sum up, yes there *are* differences between the specs of the live server and the copy/test server - but it is the server with the better specs that is giving the problems, and that wasn't what I was expecting when I copied the site to a less powerful server - I was actually expecting the symptoms/problems to escalate (partly why I chose a *less* powerful server.

The fact that this less powerful server isn't exhibiting any issues at all has actually thrown a spanner in the works for me - because the intent was after 'proving' the problem still exists on the new server I'd be able to add a few 'traps' into various parts of the code to follow exactly where it is when it hangs. This is a bit pointless now because the new server doesn't have any issues at all.

So, unless I can get a better grasp on what could possible cause this, my only options seem to be some 'heavy duty' debugging on the live site - OR, find another 'permanent' home for the site (I can't/won't leave it on the current test site because it isn't a 'conventional' server/host). It is a 'Droplet' - SAAS (pay for what you use) and being relatively new technology (at least to me) I've no idea what to expect in regards to long term reliability and costs. I must say though, it is an ideal system for testing various servers and configurations.

If you (or any other developer) has never heard of 'Droplets', a little more info can be found here https://www.digitalocean.com/

-----------------

Thanks for your thoughts. You have prompted me to try one more thing on the live server (set the mySql configs back to the 'small server' default. I really can't see this *fixing* the problem, but it is one of the few things I've not tried yet.

Cheers
RodG

[lhungil]

I guess if it was me (and the hosting provider could not help or was expensive for labor on an unmanaged VPS), I'd look into a new home.

By cutting over I meant temporarily switching which server the DNS for the live site pointed to. But if the test site is SAAS, not sure what cost might be incured just to test... Not sure the hassle (especially if an order foes through) would be worth the effort.

And yeah, many of my questions could only be answered by the hosting provider... I mentioned the underlying hardware and load as a potential because I would consider the underlying (shared) physical hardware and mappings to virtual resources (shown to VPS) to be one of the biggest potential factors after configuration / network contention.

[RodG]

I guess if it was me (and the hosting provider could not help or was expensive for labor on an unmanaged VPS), I'd look into a new home.

TBH, I've not involved the hosting provider in this at all - The nature of the issue (some ZenCart functions only - all else OK) means that even if I did contact them and somehow convinced them the problem is real, they'd simply tell me to find a ZenCart expert.

The only way I've been able to prove to *myself* that it *isn't* a zencart issue (per se) is by replicating the system on the SAAS server.

I've also had many years of experience behind me dealing with web hosting providers - and as much as I hate to say this, they are pretty much all manned, staffed and run by people without a clue. I don't even waste my time with them these days. If/when I have a problem I've always (until now) been able to diagnose and fix it myself, OR, provide the host with enough documentation and evidence of my findings that they can't possibly deny that it a problem that *they* need to fix (and even then it takes a lot of convincing).

Some hosts are better than others of course - but the bottom line is, with this particular problem, for all intents and purposes it *looks* like a ZenCart issue - I don't know what I'd need to do to convince them its not - Just saying/demonstrating that the same setup works on another server just won't cut it.

This is why I'm seeking help, input and suggestions here. I honestly believe that many of the folk around here have a lot more knowledge, skills and understanding of 'how things work' than the 'support' staff of any webhost you'd care to name.

I will also add, that I really don't expect anyone here to actually provide an answer/explanation - I know I'm often quite 'up myself' and very arrogant and opinionated, but beyond that there really haven't been too many computer related 'mysteries' over the years that I haven't been able to resolve and rectify. This one isn't much of an exception - I can resolve/fix the problem by moving to another host, BUT, it is the "bizarre" symptoms that has me puzzled, and I can't think of a single thing that could explain it.

Why is the delay only with the processes after pressing the 'confirm order' or 'update order' buttons? What is the common denominator between these two functions. What could possible cause a server response to take between 1-3minutes before it returns from the call? I've checked both the php and html response timeout settings - which I've also tried adjusting from 15 seconds up to 3mins (our default is 60 seconds). It doesn't make any difference whatsoever.

And yeah, many of my questions could only be answered by the hosting provider..
I mentioned the underlying hardware and load as a potential because I would consider the underlying (shared) physical hardware and mappings to virtual resources (shown to VPS) to be one of the biggest potential factors after configuration / network contention.

Do any hosts specify the underlying hardware of their VPS's? Or how many VPS's are actually hosted on any given hardware? Not that I know of.

Even so, let's assume that it is something related to the underlying hardware (the fact that all is ok on another VPS neither proves or eliminates this possibility) the question still remains as to what it would be with this hardware that could possible cause a significant delay with certain http requests. My logic would suggest that this is far more likely to be a software problem rather than a hardware problem - but even this doesn't help - Basically, the symptoms I'm experiencing *shouldn't* happen.

Just in case it is of any help, here is a link to the live site where I've got a 1cent test product setup. Feel free to create an account and proceed with the 'purchase' (select any payment method except PayPal - unless you really want to spend the 1cent :-)

http://www.plussizedclothing.com.au/...2857&cPath=138

The copy of this store can be found at 'www2' (IOW, same link as above, but add a '2' after the 'www').

The 'copy' site should return with the 'checkout_success' page within a few seconds of pressing the 'confirm' button. The 'live' site will/should sit there for almost eternity before it returns. It always returns if you with long enough.

Oh, And if you press the 'Refresh' button nothing appears to happen - meanwhile, each time it gets pressed a new/duplicate order is added to the store??????

When the checkout success page does appear (after a few minutes) it gives an order number, and that number is a last one entered into the DB ---
IOW, if the last order number was '1000', and a customer presses the 'Refresh' button 3 times, there will be order numbers of #1001, #1002, #1003 with the final order #1004 being displayed on the checkout success page. If the customer waits unto the page refreshes itself, it will only record a single order with the correct #1001.

Note: If your observations are the same as mine, the 'live' site is just a little bit more zippier than the copy site when browsing products, etc - Yet this is the one that goes into limbo on order confirm (and admin updates) - To *me* that pretty much eliminates serverload as being a possible factor.

It's a mystery.

I don't like unsolved mysteries. :-(

Cheers
RodG

[lhungil]

As I'm halfway around the world, both sites are a bit slow for me. Just loading the test product page is taking me 6-12 seconds (with over three before I receive the first byte). The "live" server appears to be around 200 - 500 ms faster on the test product page load.

I did notice the page load after submitting the creation of a new account appeared slower on both (compared to loading the product pages). On the "live" server significantly slower (more than 30 seconds).

The site is not letting me select a shipping method to continue checkout (probably because of the bogus address I used on the test accounts). Does not display any error just keeps reloading the checkout page.

Have you checked / repaired the DB tables (doubt this would be the cause, but....)? Have you tried creating a test to examine SQL write performance between the two servers?

[carlwhat]

my guess is a problem with sending of emails. process works something like:

- order gets created
- email gets sent
- cart gets cleared

the extended wait is ZC trying to send an email. i bet if you placed an order with email notifications turned off the response time would be about a second... again just a guess...

see:

https://www.zen-cart.com/showthread....g-order-emails

[lhungil]

Hmmm, good thought carlwhat!

A delay with communication to the email server could make sense (and would be triggered in checkout, new account creation, and status change w/ notification). Can probably test in the Zen Cart admin to see if sending an email shows a similiar delay.

I've read reports of PHP mail() sometimes being slow if DNS resolution takes an extended time. And SMTP could also be delayed by QOS / rate limiting.

[RodG]

my guess is a problem with sending of emails.

Spot on. In fact I'd just logged in here to report that I'd identified this as the problem.

I've not fixed it yet (not looked), I thought I'd take a little bit of 'time out' to celebrate.

Yeah, some may say it's a little premature - but having identified the process causing the delay is as good as a solution AFAIK. :-)

i bet if you placed an order with email notifications turned off the response time would be about a second... again just a guess...

I've not done this particular test/experiment yet, but I was able to confirm/verify this with the order update function, so yeah, basically the same test/proof.

What *eventually* 'clicked' for me was that the problem only occurred (or seemed to only occur) with the order confirmation and the order update functions. For over a week I've been puzzling over what the common denominator was with these two functions, and it just 'hit' me while I was waiting for another update to return, and from then it was easy to 'prove'. :-)

My next question @carlwhat is, Where the heck were you with this suggestion a few days ago? <g>

Seriously though, although an hour or two 'late' I really do appreciate the response/suggestion. I 'knew' that if anyone would come up with a viable explanation it would be another Zenner :-)


Cheers
RodG

[RodG]

As I'm halfway around the world, both sites are a bit slow for me.

This is comment that shouldn't be made without a little more research :-)

FWI though, one of the servers is hosted in a Chicago data center, the other is in New York.

Distance doesn't equate to speed (or lack of) though - This is dependent of the path that the data packets travel and the 'weight' of the paths. A fast path from Melbourne to NewYork then back to a server in Sydney can often be a lot faster than a slower path from Melbourne direct to Sydney.

Besides that, although I've told you where the servers are physically located, you might be interested to see where the pages *you* receive are originating from. You can do this with the 'traceroute' (*nix) or 'tracert' (windows) commands.
You'll find that the pages are being served from a host in Melbourne (ot, if you live in Sydney, then it'll be from a sydney based server).
Folk in other countries will find the site is apparently hosted in their own country. Typically, either of these sites should ever be more than 8-10 'hops' away from the client.
I'm getting typical response times of 40-50ms (which is many times better than I get from me sites that are 'hosted in Australia'

Anyway, my point being is that no one should ever consider physical distance to be a determine factor as to how 'fast' a site will respond.

I feel that this is important for Australians to know, because most Australian Webhosts depend in this fallacy in order to survive, and most will charge a 'Premium' to host with them. Now, I'm all for supporting Australian businesses, but not when the try to gouge every last cent out of you/us.

If I were to host this VPS in Australia I'd be looking at least $300-$500pm (about 3 times what we are currently paying). It ain't going to happen. ;-)

OTOH, if *you* know of an Ozzy hosting provider that can supply a VPS with 4cores and 4-5Gb RAM, with ~200gb p/m data transfer for ~$100pm please PM the details.

Anyways, thanks again for your input and suggestions.. You didn't get the money shot, but I appreciate your efforts nonetheless, and yes, you did give me a few things to think about and consider :)

Thanks again.
Cheers
RodG

[carlwhat]

My next question @carlwhat is, Where the heck were you with this suggestion a few days ago? <g>

- working on my PCI compliance. ;)
- still mourning the loss of David Bowie.
- wondering how people at themeforest could charge for 1.5.4 responsive design templates that work like crap with bugs and outdated code from older releases.
- debugging (yet again) the crap (IMHO) that is MY_ADMIN/includes/functions/extra_functions/edit_orders_functions.php.

shall i go on?

seriously, as i stated in the link that i provided, i think it's #1 a design flaw that despite the settings one has, ZC will create the order without clearing the cart. the idea that sending an email order confirmation in the middle of that process which could derail the process is inherently bad... again IMHO. and #2 using the built-in ZC logging function would have saved us both many hours of time and frustration as it would have pointed us both in the right direction as to the cause of the problem.

glad you were able to identify the problem. indeed, identifying the problem is much more difficult and worthy of celebration that implementing a solution.