Sudden duplicate authorize.net transactions? (Single order, double CC charges)

We have a relatively busy site that has started creating duplicate transactions/invoices in Authorize.net (for a single order, obviously).

Sometimes the zen cart orders aren't even generated (but an order ID's are), the customer's card is charged, and the Order ID isn't assigned within the zen cart site -- so the next customer takes the order ID. I"m mentioning it because it was even odder than a duplicate transaction.

I've only seen that behavior once. Most of the time it is just two transactions for one order and the order is saved correctly in the system (but the customer gets double-billed).

This started right around the beginning of the month, last week. I was hoping it was an authorize.net glitch, so I waited through the weekend to see if it would stop on it's own. Unfortunately we had four more incidents, so now I have to dig deeper.

We haven't made any coding or options changes to the site. About a month ago we added new items to the catalog, but that's it.

I'm aware that Authorize.net is in the process of upgrading their transaction gateway for high availability, using Akamai SureRoute, but I haven't changed to their new transaction URLs and their deadline to make the old URLs forward won't be until 6-30-2016.

FAQ here: http://www.authorize.net/support/akamaifaqs/

At first I was thinking this was the most likely culprit until I looked up their cut-over date. Now I'm not so sure.

Has anyone else noticed this kind of behavior? (Eg, duplicate transactions with authorize.net or transactions happening without an order being saved in the db?)

Trying to gather more clues to work with. Thanks for any input.

Patrick.

The "cutover date" is irrelevant: On the cutover date, their old URL will automatically point to the new one, so you won't need to make any code changes. Basically it won't matter. The only reason anyone would want to use the new URL before they switch the old one over is if their site is regularly encountering failed transactions due to very high volume specifically at times when Authorize.net is trying to do system maintenance.
That said, since you said your site is "relatively busy", I'd recommend you "try it out" by making the small change: "secure" becomes "secure2" in the URL.

But, giving any other feedback/advice is very difficult since you've not mentioned which Authorize.net module you're using.
And you haven't mentioned a list of what plugins you've installed.
Also helpful would be your PHP version and your hosting company.
I'd also suggest turning on Debug Log To File in your Authorize.net module so you can audit what's happening with each transaction by looking at the logs it generates in your /logs/ folder on your server.

The cutover date is entirely relevant to point away from the Akamai SureRoute as being the potential cause (even though not entirely conclusive due to the potential of back-end work on their systems). Not sure why you had to hammer me about something I already concluded, when I thought I clearly was showing the reasoning we agree with. Crunched for time? Bad day, maybe?

As for your "requests" for more information

Authorize.net Aim module
PHP version 5.5.31
Linux VPS inmotion hosting
No additional modules
Debug mode was set to Log and Email, now changed to Log File

Thanks,
Patrick.

Patrick Vincent:

The cutover date is entirely relevant to point away from the Akamai SureRoute as being the potential cause (even though not entirely conclusive due to the potential of back-end work on their systems). Not sure why you had to hammer me about something I already concluded, when I thought I clearly was showing the reasoning we agree with. Crunched for time? Bad day, maybe?
Eeks. Sorry, I didn't intend that to sound like hammering at all. I was just stating my opinion, which reflects the points I'd made in a few prior posts where people asked whether they needed to act on the akamai thing. Apologies if the tone was offensive. It wasn't meant to be.

Given that you've been doing some logging ... to email or to file, what are the logs showing when these duplicate occurred/occur?

Strictly speaking, a duplicate transaction suggests that the customer clicked the submit button multiple times, despite the (default Zen Cart) javascript that disables the button on first click.
Secondarily, if an AIM error was reported (even though card was charged, which would be very odd) and they submitted again, that would be another case. But of course then one would need to investigate the AIM error. Hence the logging. (files would be AIM_xxxx.log)
Similar for any PHP errors. (files would be myDebug-xxxxxx.log)

hope that helps you dig further

That was superb. Thank you. I'll find out what I can.

There are no mydebug files in the timeframe of the recent duplicate transactions. Most recent mydebug file is from 5-27.

One duplicate that I'm sure of (because I remember the last name) shows two AIM log files, timestamped a little over five minutes from each other. This points in the direction of the duplicate submit button pressing, as you brought up.

I remembered his name because there were two auth.net transactions, but the order ID that was generated with those transactions was never applied to an order, and a later customer took it. That customer complained that the site was 'going down' each time he tried to check out.

I'll get the list of duplicate transactions from our customer service rep today and look at the rest of the logs. Perhaps all of these users are having a similar problem and pressing submit twice because they are getting some sort of error.

It stopped as mysteriously as it began. Four days without an incident.

I did gather the associated transaction printouts and was getting ready to matrix those things together with processing times and digging into whatever logging and info I could get on my host's time-corresponded issues -- but the problem just stopped.

I'm going to guess it was network or server congestion, causing the returned zen cart page to timeout or error.

The only proactive action I can think of that might make an impact in the future is to increase the script timeout settings... maybe.

I might also instill with our cust service rep to gather more information about the customers if it starts again (browser, displayed error, etc..). Perhaps I'll make out a small template for her to email customers. A short questionnaire.

Is there anything else you can think of?

(Oh, and PS, I think I can relate your position on the Authorize/Akamai stuff. I've been trying to explain to people since the late 90's that cookies aren't viruses, but are pretty useful when you don't have a login over a stateless protocol. To this day people still ask about them and I get a little tense - I suspect from repeating myself for nearly two decades.)

Be well. And thanks so much for the attention. If I ever do find a definitive cause, I'll be sure to come back and fill in what blanks I can.

wild guess... your hosting company (or you) changed the email setting.

i use authorize.net, however i have modified the timing of the transaction submission due to clients needs. so i would not get into this situation. however, when the sending of an order email fails, it screws up the whole order process. see:

https://www.zen-cart.com/showthread.php?216010-Duplicate-orders-in-1-5-3&p=1311441#post1311441

frankly, i'm not convinced that this is the problem, but you were asking for anything else...

it does seem like a network problem. especially when you say the order ID gets assigned to the next customer's orders. i have had experiences similar when there are problems with queuing on other systems...

these are always the trickiest to figure out.

good luck!

I did ask for anything, yes, good input, and thank you I'll add it to the list of things to look out for. All troubleshooting ideas should be welcome with people (just be careful doing it in stackoverflow -- generally speaking, the community is far too officious and competitive for things like good ideas).

Update: the problems started again and I found I was able to get errors in the contact form by repeatedly submitting it (thanks for the pointer on email being a potential culprit).

From poking around I found that the setting for Email Friendly-Errors was set to false. I set it to true and no longer saw errors in the contact form after a nauseating number of re-submissions. (Note: when the contact form was still showing the error, emails were still being sent/delivered).

Error displaying on-submit of contact form:

Email Error: SMTP Error: Data not accepted.
SMTP server error:

I'm not sure if this was 100% of the problem, but based on the good change in behavior of the contact form after I set the option to true (no more errors), I'm hoping it clears up most of the ordering issues. Why it didn't cause more of a problem with that setting set to false for so long baffles me.

I'm concluding that something must have changed on the hosting/vps side even though the host support staff were less-than-willing to investigate that part. Perhaps a cpanel/whm/exim(email) update that altered the response code? Maybe? I can't be certain, especially considering how the problem seems to start/stop/start again.

Thanks again for the attention. If the mystery unravels further, I'll update the thread so others can benefit.

This change in v155 causes the email errors to be logged instead of dumped to the screen:
https://github.com/zencart/zencart/commit/1b20771c8f590c982245190414f65f681370b9a0

Might be useful for keeping tabs on frequency of such issues.

I will definitely do that. Thank you.

Also, to append more info, I got word from another of the tech support reps at IMH who dug a bit deeper than the others. He said he found in the logs that a foreign mail server had rejected our SSL encryption level when trying to send a copy of the order to the customer. (Due to PCI compliance we disabled some of the lower encryption levels recently).

He also explained that after the initial ssl transmission failed, the server then creates a new unencrypted connection to send mail (both issues visible in the log snippet he sent).

That makes sense with the behavior in the contact form. The SMTP error was displayed, but an email was still sent. It doesn't explain 100% of the situation, but if the script were failing due the line highlighted below, it at least explains part of it.

Log snippet from their support staff:
(my domains/my IP/my server/customer email address modified as a precaution)

2016-06-14 08:54:19 [20909] 1bCnqv-0005RF-Oe H=myserver.inmotionhosting.com (zencarthostedsite.com) [MY.SER.VER.IP]:53339 I=[MY.SER.VER.IP]:25 Warning: "SpamAssassin as cpaneleximscanner detected OUTGOING smtp message as NOT spam (-1.0)"
2016-06-14 08:54:19 [20909] 1bCnqv-0005RF-Oe <= [email protected] H=myserver.inmotionhosting.com (zencarthostedsite.com) [MY.SER.VER.IP]:53339 I=[MY.SER.VER.IP]:25 P=esmtpa A=dovecot_login:mycpanelusername S=31793 M8S=0 id=[email protected] T="Order Confirmation No: 6153" from [email protected] for [email protected]
2016-06-14 08:54:19 [20919] cwd=/var/spool/exim 3 args: /usr/sbin/exim -Mc 1bCnqv-0005RF-Oe
2016-06-14 08:54:19 [20919] 1bCnqv-0005RF-Oe SMTP connection outbound 1465908859 1bCnqv-0005RF-Oe privatedomain.com [email protected]
2016-06-14 08:54:19 [20922] 1bCnqv-0005RF-Oe H=smtpin.ptd.net [207.44.97.35] TLS error on connection (SSL_connect): error:14077102:SSL routines:SSL23_GET_SERVER_HELLO:unsupported protocol
2016-06-14 08:54:19 [20922] 1bCnqv-0005RF-Oe TLS session failure: delivering unencrypted to smtpin.ptd.net [207.44.97.35] (not in hosts_require_tls)
2016-06-14 08:54:20 [20919] 1bCnqv-0005RF-Oe => [email protected] F=[email protected] P=[email protected] R=dkim_lookuphost T=dkim_remote_smtp S=32871 H=smtpin.ptd.net [207.44.97.35]:25 I=[MY.SER.VER.IP]:53182 C="250 2.0.0 Ok: queued as ED5CA6A3B0E" QT=23s DT=1s
2016-06-14 08:54:20 [20919] 1bCnqv-0005RF-Oe Completed QT=23s

Ya, definitely useful to know about.

You could consider outsourcing your email delivery to be handled by another service, and let them decide how they want to handle ensuring your emails to such destinations are completed successfully.

I brought up the idea of using mandrill with the senior management earlier this morning. I really agree, that might be the best route.

Mailgun might be a more affordable starting point, until you ascertain volume.

Understood. I have to offer some pricing plans to the powers that be, so that's good to know. Thank you.