Paypal IPN Failing suddenly

Other than following DrBytes suggestions I'm almost out of ideas too (although I think that for LP to "completely disabled the firewall for a week of testing" seems a bit sus. Few hosts would do that for a single user complaint, so if it is something they did do then clearly you are not the only person with this issue, OR, they are simply feeding you bulldust).

Anyway, I'm not sure if it will provide any clues or not, but if you have command line access to your server, please enter the following commands and report your results:

nslookup api.paypal.com
traceroute api.paypal.com
ping api.paypal.com

Cheers
Rod

They only did the firewall disable after over a week of other testing support, and only did it after phone verification from me. It was their "last ditch effort" to support this problem. I dont think I have command line access, but I will ask lunarpages to run the commands and send me the results.

BTW I have a dedicated server, so I dont believe that means they would be disabling the firewall for everyone.

chriscana:

They only did the firewall disable after over a week of other testing support, and only did it after phone verification from me. It was their "last ditch effort" to support this problem.

Nonetheless, this is extremely accommodating for them, for just one user. I still smell something 'fishy'.

chriscana:

BTW I have a dedicated server, so I dont believe that means they would be disabling the firewall for everyone

You may be right, but this depends on whether your 'dedicated server' is a Virtual Machine (VPN/VPS) or whether your 'dedicated server' is 'bare metal'.
If a VM then the same firewall will be used for all hosts on that particular VM (not just your assigned portion of it).

Either way, my question now is, since you DO have a dedicated server, why is it being firewalled in the first place? Was this something you requested? (I suspect not). If they are firewalling your server of their own accord I'd be asking them why? How do they know which ports you want blocked (and from where)?

I see that you also have your 'own' name servers. One of the tests I asked was to ensure these are working correctly, and since LP may or may not know if you are running your own nameservers or not they wouldn't know if port 53 (DNS) needs to be blocked for you or not. If they have got this blocked then your store wouldn't be able to locate the PayPal servers so no requests will be sent. This doesn't exactly the symptoms being reported, but the point I'm trying to make is that you really don't need your store to be firewalled at all, and one of the first rules in isolating network related problems is to disable all firewall rules until the problem is resolved, because all they are doing is adding an additional source of problems.
I often used to neglect this 'rule' in my early days by following the traditional advice of 'make one change at a time, and if it doesn't fix the problem revert back to how things were before trying something else'). I got 'wise' after spending many hours looking for a problem only to find I had a firewall problem PLUS another problem (routing) that never showed whlle the firewall was enabled. Needless to say, I was just going around and around in circles.

chriscana:

but I will ask lunarpages to run the commands and send me the results

With a dedicated server you'll almost certainly have SSH access (which is all you need to run the commands). If you aren't familiar with SSH you may be lucky in that you can gain access to the command line by navigating to your cPanel (assuming this is what you use), clicking the SSH/Shell access icon and then selecting the 'connect using SSHterm' option (if available). This will launch a java applet where you can enter the commands.

Cheers
Rod

No SSHterm in CPanel. Lunarpages ran the commends for me:

root@server [~]# nslookup api.paypal.com
Server: 4.2.2.2
Address: 4.2.2.2#53

Non-authoritative answer:
Name: api.paypal.com
Address: 173.0.88.66
Name: api.paypal.com
Address: 173.0.88.98
Name: api.paypal.com
Address: 173.0.84.66
Name: api.paypal.com
Address: 173.0.84.98

root@server []# traceroute api.paypal.com
traceroute to api.paypal.com (173.0.84.98), 30 hops max, 40 byte packets
1 sd2.lunarpages.com (74.50.27.3) 1.156 ms 1.149 ms 1.137 ms
2 * * *
3 144.232.24.225 (144.232.24.225) 4.041 ms 144.232.25.105 (144.232.25.105) 3.996 ms *
4 * * *
5 * * *
6 * * *
7 * * *
8 sl-payp2-413771-0.sprintlink.net (144.232.191.210) 13.625 ms 13.756 ms 13.699 ms
9 * * *
10 * * *
11 * * *
12 * * *
13 * * *
14 * * *
15 * * *
16 * * *
17 * * *
18 * * *
19 * * *
20 * * *
21 * * *
22 * * *
23 * * *
24 * * *
25 * * *
26 * * *
27 * * *
28 * * *
29 * * *
30 * * *
root@server []#

root@server [~]# ping -c 3 api.paypal.com
PING api.paypal.com (173.0.84.98) 56(84) bytes of data.

--- api.paypal.com ping statistics ---
3 packets transmitted, 0 received, 100% packet loss, time 1999ms

chriscana:

No SSHterm in CPanel. Lunarpages ran the commends for me:

The tests/results are worthless.

They ran the tests from sd2.lunarpages.com (74.50.27.3)
Your store is located di7s00044.lunarbreeze.com. (64.50.172.35)

They should know better.

I now wonder if the firewall that they disabled for you was the correct one for your server? If they disabled the one for sd2.lunarpages.com (74.50.27.3) it doesn't come as any surprise that it had no effect on your server.

Are you sure your cPanel doesn't have the SSH/Shell option? (It's generally located under the section labelled 'security').

Cheers
Rod

According to them they did run it from the correct server/IP. See their response below, and also the output of iptables -L -n for the firewall that is attached to the message (too big for post).

Please note that the firewall was indeed re-enabled on your own server, and all the commands provided above were run on your own server, not sd2.lunarpages.com (74.50.27.3) which is one of our internal servers. If you believe that these commands were ran on this server due to the first node in the traceroute command showing this host, please note that the traceroute would normally not show the local interface, but rather the first host it contacts (usually its gateway). As such, I can confirm that all of these commands were in fact ran on your server. In order to prove this, below you can find the output of the ssh connection to your server, along with the last 20 lines of history (the commands which were executed on your server through ssh):

ssh [email protected]
Warning: Permanently added '64.50.172.174' (RSA) to the list of known hosts.
reverse mapping checking getaddrinfo for di7s00044.lunarbreeze.com failed - POSSIBLE BREAK-IN ATTEMPT!
[email protected]'s password:
Last login: Sun Aug 12 08:36:46 2012 from 64.50.162.9
root@server []# history | tail -20
982 /etc/init.d/httpd stop
983 netstat -nt | grep -v Proto | grep -v Active | grep ":80 " | awk '{print$5}' | awk -F ":" '{R=NF-1; print$R}' | sort | uniq -c | sort -n
984 ps aux | grep http
985 /etc/init.d/httpd start
986 top
987 apf --help
988 apf -f
989 ping api.paypal.com
990 apf -s
991 exit
992 nslookup api.paypal.com
993 cat /etc/hosts
994 traceroute api.paypal.com
995 ping api.paypal.com
996 ping -t c3 api.paypal.com
997 ping -c3 api.paypal.com
998 ping -h
999 ping -c 3 api.paypal.com
1000 exit
1001 history | tail -20
root@server []#

Furthermore, the firewall is up and running, and you can confirm this by simply running the "iptables -L -n" command which will show you all the rules currently set in place. (See attached)

Attachment #10971

I'm less inclined to suspect networking issues. Instead I'm now leaning toward it being damaged code.
Investigating all the ways in which the site's code differs from original Zen Cart code is key. http://www.zen-cart.com/wiki/index.php/Troubleshoot_-_Diagnosing_Obscure_Issues

Happened with my 1.3.8 install, also with 1.5 install. Did a complete fresh install and new database for idoseraudio.com, only changing template and populating with my own products - no changed code. Seems issue happens no matter installation or code base.

Okay. Investigation would require testing code execution at numerous stages in all the various scripts involved in processing the incoming request, including but not limited to most of the files whose filename contains the string 'paypal' in some form.

But the default core code already does debug logging with PayPal transactions, if the switch is enabled in your ZC paypal module settings. If that's not creating any ipn_xxxxxx.log files when IPNs are submitted, then either the folder isn't writable or the execution is halting before any of the core code runs. Maybe your server config is blocking POST requests to ipn_main_handler.php for some bizarre unknown reason.

I've gone through dozens of emails and support requests between paypal and lunarpages - not to mention the time you all have put into it (thanks kindly). I don't know if a solution can be reached at this point. I'm tempted to just disable IPNs on the paypal side and call it a day.

You said the only thing IPN does is echeck and refunds? Why would I get IPN failure posts daily if my shop hasn't had any refunds or echecks? If thats all I have to worry about, I can handle that without needed the ipn communication. I don't know why it would post multiple daily IPN requests if none of those are happening, though. Just want to be sure.

chriscana:

You said the only thing IPN does is echeck and refunds?
No. That's not exactly what I said.

DrByte:

When using Express Checkout and Payments Pro, the only value that IPNs provide is to automatically update orders in your store when a change takes place with one of those transactions, such as a refund or void or echeck clearing. If you don't need any of that, then simply turn off the IPN notification option in your Paypal account and leave it.
An IPN is sent EVERY time ANY transaction is added or updated in your account.
The only thing Express Checkout benefits from is when any of those updates are related to echecks or refunds.
But IPNs are still sent for everything. Your store will receive and acknowledge them all, but then just ignores what doesn't apply to existing EC transactions.

(Same for Payments Pro too, by the way, but that's beyond your present post.)

And, since yours have been failing for a long time, it's certainly not an urgent thing you need to fix.

In response to IPN log file writing, debug was set to off. I have enabled it for log writing. Will check back one those have written.

There should be some logs now ;)

So far there are logs with titles beginning with paypalwpp_Paypal_Action and Paypal_CURL but none with IPN listed yet. Checking in /includes/modules/payment/paypal/logs and logs dir is 777.

Do you get one if you visit https://idosersoftware.com/ipn_main_handler.php via your browser?

In the browser I get: Nothing to process. Please return to home page.
Yes. I am now getting IPN logs. See attached

Attachment #10972

Right. So it works when not submitting data. But when anything tries to submit (POST), it just aborts. Not what it's designed to do.

chriscana:

According to them they did run it from the correct server/IP. See their response below, and also the output of iptables -L -n for the firewall that is attached to the message (too big for post).

Apologies. My mistake, they did indeed run it from your server.

I see nothing amiss with any of the tests made, and although the firewall rules are somewhat convoluted (I've seen a lot worse), they don't **appear **to be the source of the problem.

DrByte:

I'm less inclined to suspect networking issues. Instead I'm now leaning toward it being damaged code.

Upon further investigation/consideration I'm also now less inclined to suspect networking/firewall issues. However, I'm not so much leaning towards 'damaged code', but more towards a server/configuration problem (Perhaps mod_proxy?).

I'm not in any way saying it is not 'damaged code' because there is still a bit of doubt as to whether you really do have a new 'clean install', or whether you did a partial/full upgrade... Those mixed character sets (collation error) previously mentioned do nothing to clear these muddy waters.

I'll get back to you if I can think of anything else, or have other suggestions for things to test, but in the meantime I can only suggest you follow DrBytes advice.

Oh, just out of curiousity, are you experiencing any other problems? (even seemingly unrelated ones). For example, do you have any issues updating or adding products? or is the IPN your only problem?

Cheers
Rod.