Duplicate Orders

[Friedrich72]

Love Zen Cart. Everything works great except this, kind of big, problem.
We could really use some help determining the cause of random, duplicate orders.
I have installed a few Zen Cart stores and this is the only one that has the problem.
This store is different in that it receives a lot of traffic compared to the others. And there are some different modules installed.

ZC: 1.5.1
Upgraded: from 1.3.9h to 1.5.1 on April 9 in hopes that it would solve the duplicate orders problem.
PHP: 5.2.17
Host: 1and1
Mods:

• ceon adv shipper 5.x
• ceon manual card
• image handler 4
• column layout grid
• edit orders 4.0.4
• Zen Lightbox 1.6.3
• Minimum Order

Performed Manual Install via Filezilla
We don't use paypal or any other type of processing than listed above.

I have read through this thread http://www.zen-cart.com/showthread.p...-Confirmation/ and it sounds like the same problem except our problem is also with 1.5.1

This is what happens:
Maybe 5 times a month roughly an order is duplicated at the time of purchase or a few seconds after, or a few minutes after.
The customer is only charged once.
The customer receives a notification on the second order.
Same order number on duplicate.

Suspicion:
possibly host processing lag is causing it
1and1 seemed to have slow processing time on their secure server. I contacted them. They confirmed that they were dealing with some issues. I asked them if the duplicates could be attributed to it but they said no.

Can someone tell me the best way to analyze what is happening?
I need something like the error log that is built into ZC.
However there are no errors in that log resulting from the duplicates.

Is anyone else from the thread link above still having issues?

[DrByte]

This is what happens:
Maybe 5 times a month roughly an order is duplicated at the time of purchase or a few seconds after, or a few minutes after.
The customer is only charged once.
The customer receives a notification on the second order.
Same order number on duplicate.

The only payment module you've mentioned is CEON Manual Order ... which merely emails the encrypted card number, immediately, no time delay.
So, since an order is saved/generated immediately after the payment module completes, then your problem is either:
- customer is submitting payment multiple times
- or you're using another payment module or some other addon that's triggering something else to happen in addition to normal operation
The way the core of ZC works it's impossible for a "duplicate" to be generated several minutes later unless something outside normal core code, or an external service, is causing it.

But without seeing timestamps and IP addresses and module info etc of actual orders, we can't do anything other than hypothesize. You've got all the facts.

[Friedrich72]

Thank you for responding, Dr. Byte. Direction like this, from you, is so appreciated!
I have orders from April for the last five duplicates.
What else do we need to pin point the problem? I am thinking of doing a test where we would disable the edit orders module except for certain times when we need it. Do you think that would be helpful?
All but the second one in the list used credit.

99.66.1.26
Tue Apr 30 2013 12:37:01
Tue Apr 30 2013 12:37:06

70.45.115.131 - this one used check
Sat Apr 27 2013 9:00:34
Sat Apr 27 2013 9:00:26

96.248.6.178
Fri Apr 26 2013 14:38:01
Fri Apr 26 2013 14:37:52

108.227.141.135
Tue Apr 23 2013 17:57:36
Tue Apr 23 2013 17:57:26

50.82.241.218
Mon Apr 22 2013 21:46:30
Mon Apr 22 2013 21:46:15

[DivaVocals]

Thank you for responding, Dr. Byte. Direction like this, from you, is so appreciated!
I have orders from April for the last five duplicates.
What else do we need to pin point the problem? I am thinking of doing a test where we would disable the edit orders module except for certain times when we need it. Do you think that would be helpful?
All but the second one in the list used credit.

99.66.1.26
Tue Apr 30 2013 12:37:01
Tue Apr 30 2013 12:37:06

70.45.115.131 - this one used check
Sat Apr 27 2013 9:00:34
Sat Apr 27 2013 9:00:26

96.248.6.178
Fri Apr 26 2013 14:38:01
Fri Apr 26 2013 14:37:52

108.227.141.135
Tue Apr 23 2013 17:57:36
Tue Apr 23 2013 17:57:26

50.82.241.218
Mon Apr 22 2013 21:46:30
Mon Apr 22 2013 21:46:15

Edit Orders doesn't affect checkout IN ANY WAY. It is a post order editing module so disabling it won't solve your issue..

[RodG]

Hi DrB (and others)

I know this is a rather old thread that doesn't seem to be resolved, but I've got both a similar and identical problem that frankly has be somewhat stumped.

The 'upside' is that both problems are easy for me to replicate and are very consistant. Alas, this doesn't seem to be helping me.

Two stores on the same server:

Store#1 works perfectly in every way *except* when I hit the button to update an orders status (doesn't matter what order). It just hangs - For almost two minutes before the the screen refreshes with the update succeful message. If I re-press the button before it returns I'm greeted with a message along the lines of 'nothing to update'.

Store#2 has a different problem - No problem with the order updates, but there is a huge delay btween the time the 'confirm order' button is pressed and the 'checkout success' page is shown - Typically about 1 minute - but after patiently waiting, it has taken up to 3 minutes. The actual order gets processed and updated into the database almost instantly - it is only the 'checkout success page that is slow. Performing a refresh before the success page is shown results in a duplicate order - Most customers aren't waiting this 1-3mins (I can't blame them) so almost every order is being duplicated.

I'm NOT having these issues with the same sites hosted on another server. The current server is far from being overloaded in any way - it a VPS with 5gb Ram and 4 cores. The other VPS server that I've set up to test/compare with only has 2gb RAM and 2 cores - IOW, about half of the resources than the server with the problems.

Oh, it makes no difference what payment or shipping methods are chosen. The site usually uses the FEC module (has done for years), but the problem persists no matter if I use FEC in single or multiple page checkout mode, and it even persists using the native zencart checkout - In short - Regardless of the chackout method/options - as soon as that 'confirm' button is hit, the order is processed, but the 'checkout success' page doesn't appear for thew 1-3 minutes.

The way the core of ZC works it's impossible for a "duplicate" to be generated several minutes later unless something outside normal core code, or an external service, is causing it.

That's what *I* thought.

But without seeing timestamps and IP addresses and module info etc of actual orders, we can't do anything other than hypothesize. You've got all the facts.

I've got these (from both servers). Not much help really though - other than to confirm the delay.

------------- Server with problem --------------------------------
[24/Jan/2016:13:28:31 +1100] "POST /index.php?main_page=checkout_process HTTP/1.1" 302 20
"http://www.xxxxxxx.com/index.php?main_page=fec_confirmation&fecaction=process" "Mozilla/5.0 (X11; Linux x86_64)


[24/Jan/2016:13:30:03 +1100] "GET /index.php?main_page=checkout_success HTTP/1.1" 200 19763
"http://www.xxxxxx.com/index.php?main_page=fec_confirmation&fecaction=process" "Mozilla/5.0 (X11; Linux x86_64)



Note: This example shows ~90 seconds elapsed between the 'checkout_process' and the 'checkout_success'.


----- Testing server - No problem. --------------------------------
[23/Jan/2016:21:41:05 -0500] "POST /index.php?main_page=checkout_process HTTP/1.1" 302 20
"http://www2.xxxxxx.com.au/index.php?main_page=fec_confirmation&fecaction=process" "Mozilla/5.0 (X11; Linux x86_64)


[23/Jan/2016:21:41:06 -0500] "GET /index.php?main_page=checkout_success HTTP/1.1" 200 19704
"http://www2.xxxxxx.com/index.php?main_page=fec_confirmation&fecaction=process" "Mozilla/5.0 (X11; Linux x86_64)

Note: A mere 1second had elapsed between the 'checkout_process' and the 'checkout_success'.

I'm confident that this isn't technically a ZenCart issue (same code both sites), and if it were a server overload issue (my initial thoughts) then I'd have expected the problem to worsen on the lower powered VPS (besides, logs and graphs indicate that the troublesome server is barely using 5% of its CPU with lots of unutilised RAM (nothing swapping to disk).

There doesn't appear to be any other slowdowns in any other aspect of any of these sites - The problem(s) appears to be pretty specific and consistant to these functions.

All in all, this problem really isn't making any sense to me at all. I can see/observe the problems, but I can't explain how or why they are occurring. (Or why the two sites on the same server are giving different symptoms).

I realise that now I've come this far of setting up another VPS for testing/comparison I could quite easily update the DNS and make this the 'live' site (and solve the problem(s) - And I suspect it may even come to that anyway, BUT, computers aren't 'magical black boxes' - There *has* to be a logical explanation.

Any ideas? Anyone? (Even speculation could be useful).

I don't even know the next step to fault finding this one. Suggestions?

Oh, I should add this problem just started 'out of the blue' (Mid December).

Thanks for any input.

Cheers
RodG

[lhungil]

Just a short list of quick thoughts / speculation. Probably things you've already contemplated / checked. If you cut over the DNS does the lesser VPS still keep up (or does it start to lag)?

Similiar load on both sites (connections, end users, etc)? Same server farm (and network) for both VPS? Same underlying OS / configuration / optimizations? Same PHP <==> HTTP server setup? Any noticeable pattern looking at page parse / SQL times between the two? Is one noticeably slower when writing to the DB via Zen Cart (possibly test using EZ Populate or similiar)? Any "security" software on either? Any proxy / cache / CDN setups? Any "limiting" or QOS? Any output filters (PHP or HTTPd)? Similiar physical hardware for both hosts (CPU, mem, hdd - speed, latency, bandwidth, etc)?

[RodG]

If you cut over the DNS does the lesser VPS still keep up (or does it start to lag)?

Sorry. I don't quite follow. Both sites are using the same DNS. The original site is "www" and the copied site is call "www2"

Not only was this the easiest approach, but I figured this would take DNS out of the equation.

Similiar load on both sites (connections, end users, etc)?

No. The copy site only has me accessing it, so the live site is certainly going to have a heavier load, but certainly not enough to cause a performance issue - And only with some functions??

Same server farm (and network) for both VPS?

No. totally different - intentionally - just in case it is/was caused by some inexplicable networking issue within the server farm.

I'm not sure if I've proven this to be true or not though - It is one of the few things different between the sites.

Same underlying OS / configuration / optimizations? Same PHP <==> HTTP server setup?

As near as possible. All versions are the same on both servers, but I've needed to tweak the mysql settings on the copy server because of the smaller memory. On the live server I have 3Gb 'reserved' just for mySQL, on the copy server it is using the default memory allocations (a lot less).

Hmmm, perhaps I should reset the mysql settings on the live server back to the defaults. I can't imagine this will improve things. Those settings haven't been change since I 1st set this server up about 18 months ago.

[/QUOTE]
Any noticeable pattern looking at page parse / SQL times between the two?
[/QUOTE]

For general page loads, both are on par with each other (the live server being actually a little bit faster - which is pretty much expected considering it has double the grunt).

Is one noticeably slower when writing to the DB via Zen Cart (possibly test using EZ Populate or similiar)?

I don't use EZ Populate or similiar - but again, almost everything about the site(s) are fine - adding/removing from cart, navigating pages, logging on/off, adding/editing products, and so on - The problem only shows with the 'confirm order' (customer purchase) and the 'update status' (admin).

Oh, another finding - I initally said one of the sites had the update problem and the other had the confirm order problem - This turned out to be incorrect - both sites exhibit the same two problems.

One of them (mine) is still using Zen 1.3.9 (I use it mostly for ozpost compatibility testing), the other is using Zen V1.5.4

Any "security" software on either?

Yes, but all have been disabled at various stages of debugging. The problem continued.

Any proxy / cache / CDN setups?

There is obviously caching going on somewhere along the line (local browser, ISP transparent proxy, and obviously? the ImageHandler cache) - but these will be consistant with either site. The servers aren't proxied (that I'm aware of - I'd be piffed if they were). We do use a CDN for both sites and servers, but disabling this didn't change the problem one iota.

Any "limiting" or QOS?

Nothing I've implemented. Who really knows what the hosts have going on behind the VPS.

Any output filters (PHP or HTTPd)?

mod_security (default settings) on both servers - No difference when I've disabled it.

Similiar physical hardware

No way of knowing - they are both VPS's rather than bare metal servers.

for both hosts (CPU, mem, hdd - speed, latency, bandwidth, etc)?

2core vs 4core (4core has the problem)
2Gb RAM vs 5Gb RAM (%Gb has the problem)
Virtualised HDD vs SDD. The HDD is the one with the problems, but there is no disk swapping going on (in fact there is enough memory in reserve on this VPS that I didn't even bother creating a swap file :-)

So, to sum up, yes there *are* differences between the specs of the live server and the copy/test server - but it is the server with the better specs that is giving the problems, and that wasn't what I was expecting when I copied the site to a less powerful server - I was actually expecting the symptoms/problems to escalate (partly why I chose a *less* powerful server.

The fact that this less powerful server isn't exhibiting any issues at all has actually thrown a spanner in the works for me - because the intent was after 'proving' the problem still exists on the new server I'd be able to add a few 'traps' into various parts of the code to follow exactly where it is when it hangs. This is a bit pointless now because the new server doesn't have any issues at all.

So, unless I can get a better grasp on what could possible cause this, my only options seem to be some 'heavy duty' debugging on the live site - OR, find another 'permanent' home for the site (I can't/won't leave it on the current test site because it isn't a 'conventional' server/host). It is a 'Droplet' - SAAS (pay for what you use) and being relatively new technology (at least to me) I've no idea what to expect in regards to long term reliability and costs. I must say though, it is an ideal system for testing various servers and configurations.

If you (or any other developer) has never heard of 'Droplets', a little more info can be found here https://www.digitalocean.com/

-----------------

Thanks for your thoughts. You have prompted me to try one more thing on the live server (set the mySql configs back to the 'small server' default. I really can't see this *fixing* the problem, but it is one of the few things I've not tried yet.

Cheers
RodG

[lhungil]

I guess if it was me (and the hosting provider could not help or was expensive for labor on an unmanaged VPS), I'd look into a new home.

By cutting over I meant temporarily switching which server the DNS for the live site pointed to. But if the test site is SAAS, not sure what cost might be incured just to test... Not sure the hassle (especially if an order foes through) would be worth the effort.

And yeah, many of my questions could only be answered by the hosting provider... I mentioned the underlying hardware and load as a potential because I would consider the underlying (shared) physical hardware and mappings to virtual resources (shown to VPS) to be one of the biggest potential factors after configuration / network contention.

[carlwhat]

my guess is a problem with sending of emails. process works something like:

- order gets created
- email gets sent
- cart gets cleared

the extended wait is ZC trying to send an email. i bet if you placed an order with email notifications turned off the response time would be about a second... again just a guess...

see:

https://www.zen-cart.com/showthread....g-order-emails

[lhungil]

Hmmm, good thought carlwhat!

A delay with communication to the email server could make sense (and would be triggered in checkout, new account creation, and status change w/ notification). Can probably test in the Zen Cart admin to see if sending an email shows a similiar delay.

I've read reports of PHP mail() sometimes being slow if DNS resolution takes an extended time. And SMTP could also be delayed by QOS / rate limiting.