Paypal IPN Failing suddenly

[RodG]

Other than following DrBytes suggestions I'm almost out of ideas too (although I think that for LP to "completely disabled the firewall for a week of testing" seems a bit sus. Few hosts would do that for a single user complaint, so if it is something they did do then clearly you are not the only person with this issue, OR, they are simply feeding you bulldust).

Anyway, I'm not sure if it will provide any clues or not, but if you have command line access to your server, please enter the following commands and report your results:

nslookup api.paypal.com
traceroute api.paypal.com
ping api.paypal.com

Cheers
Rod

[chriscana]

They only did the firewall disable after over a week of other testing support, and only did it after phone verification from me. It was their "last ditch effort" to support this problem. I dont think I have command line access, but I will ask lunarpages to run the commands and send me the results.

[chriscana]

BTW I have a dedicated server, so I dont believe that means they would be disabling the firewall for everyone.

[RodG]

They only did the firewall disable after over a week of other testing support, and only did it after phone verification from me. It was their "last ditch effort" to support this problem.

Nonetheless, this is extremely accommodating for them, for just one user. I still smell something 'fishy'.

BTW I have a dedicated server, so I dont believe that means they would be disabling the firewall for everyone

You may be right, but this depends on whether your 'dedicated server' is a Virtual Machine (VPN/VPS) or whether your 'dedicated server' is 'bare metal'.
If a VM then the same firewall will be used for all hosts on that particular VM (not just your assigned portion of it).

Either way, my question now is, since you DO have a dedicated server, why is it being firewalled in the first place? Was this something *you* requested? (I suspect not). If *they* are firewalling your server of their own accord I'd be asking them why? How do they know which ports you want blocked (and from where)?

I see that you also have your 'own' name servers. One of the tests I asked was to ensure these are working correctly, and since LP may or may not know if you are running your own nameservers or not they wouldn't know if port 53 (DNS) needs to be blocked for you or not. If they have got this blocked then your store wouldn't be able to locate the PayPal servers so no requests will be sent. This doesn't exactly the symptoms being reported, but the point I'm trying to make is that you really don't *need* your store to be firewalled at all, and one of the first rules in isolating network related problems is to disable all firewall rules until the problem is resolved, because all they are doing is adding an additional source of problems.
I often used to neglect this 'rule' in my early days by following the traditional advice of 'make one change at a time, and if it doesn't fix the problem revert back to how things were before trying something else'). I got 'wise' after spending many hours looking for *a* problem only to find I had a firewall problem PLUS another problem (routing) that never showed whlle the firewall was enabled. Needless to say, I was just going around and around in circles.

but I will ask lunarpages to run the commands and send me the results

With a dedicated server you'll almost certainly have SSH access (which is all you need to run the commands). If you aren't familiar with SSH you may be lucky in that you can gain access to the command line by navigating to your cPanel (assuming this is what you use), clicking the SSH/Shell access icon and then selecting the 'connect using SSHterm' option (if available). This will launch a java applet where you can enter the commands.

Cheers
Rod

[chriscana]

No SSHterm in CPanel. Lunarpages ran the commends for me:

root@server [~]# nslookup api.paypal.com
Server: 4.2.2.2
Address: 4.2.2.2#53

Non-authoritative answer:
Name: api.paypal.com
Address: 173.0.88.66
Name: api.paypal.com
Address: 173.0.88.98
Name: api.paypal.com
Address: 173.0.84.66
Name: api.paypal.com
Address: 173.0.84.98

root@server [~]# traceroute api.paypal.com
traceroute to api.paypal.com (173.0.84.98), 30 hops max, 40 byte packets
1 sd2.lunarpages.com (74.50.27.3) 1.156 ms 1.149 ms 1.137 ms
2 * * *
3 144.232.24.225 (144.232.24.225) 4.041 ms 144.232.25.105 (144.232.25.105) 3.996 ms *
4 * * *
5 * * *
6 * * *
7 * * *
8 sl-payp2-413771-0.sprintlink.net (144.232.191.210) 13.625 ms 13.756 ms 13.699 ms
9 * * *
10 * * *
11 * * *
12 * * *
13 * * *
14 * * *
15 * * *
16 * * *
17 * * *
18 * * *
19 * * *
20 * * *
21 * * *
22 * * *
23 * * *
24 * * *
25 * * *
26 * * *
27 * * *
28 * * *
29 * * *
30 * * *
root@server [~]#

root@server [~]# ping -c 3 api.paypal.com
PING api.paypal.com (173.0.84.98) 56(84) bytes of data.

--- api.paypal.com ping statistics ---
3 packets transmitted, 0 received, 100% packet loss, time 1999ms

[RodG]

No SSHterm in CPanel. Lunarpages ran the commends for me:

The tests/results are worthless.

They ran the tests from sd2.lunarpages.com (74.50.27.3)
Your store is located di7s00044.lunarbreeze.com. (64.50.172.35)

They should know better.

I now wonder if the firewall that they disabled for you was the correct one for your server? If they disabled the one for sd2.lunarpages.com (74.50.27.3) it doesn't come as any surprise that it had no effect on your server.

Are you *sure* your cPanel doesn't have the SSH/Shell option? (It's generally located under the section labelled 'security').

Cheers
Rod

[chriscana]

According to them they did run it from the correct server/IP. See their response below, and also the output of iptables -L -n for the firewall that is attached to the message (too big for post).

Please note that the firewall was indeed re-enabled on your own server, and all the commands provided above were run on your own server, not sd2.lunarpages.com (74.50.27.3) which is one of our internal servers. If you believe that these commands were ran on this server due to the first node in the traceroute command showing this host, please note that the traceroute would normally not show the local interface, but rather the first host it contacts (usually its gateway). As such, I can confirm that all of these commands were in fact ran on your server. In order to prove this, below you can find the output of the ssh connection to your server, along with the last 20 lines of history (the commands which were executed on your server through ssh):

ssh [email protected]
Warning: Permanently added '64.50.172.174' (RSA) to the list of known hosts.
reverse mapping checking getaddrinfo for di7s00044.lunarbreeze.com failed - POSSIBLE BREAK-IN ATTEMPT!
[email protected]'s password:
Last login: Sun Aug 12 08:36:46 2012 from 64.50.162.9
root@server [~]# history | tail -20
982 /etc/init.d/httpd stop
983 netstat -nt | grep -v Proto | grep -v Active | grep ":80 " | awk '{print$5}' | awk -F ":" '{R=NF-1; print$R}' | sort | uniq -c | sort -n
984 ps aux | grep http
985 /etc/init.d/httpd start
986 top
987 apf --help
988 apf -f
989 ping api.paypal.com
990 apf -s
991 exit
992 nslookup api.paypal.com
993 cat /etc/hosts
994 traceroute api.paypal.com
995 ping api.paypal.com
996 ping -t c3 api.paypal.com
997 ping -c3 api.paypal.com
998 ping -h
999 ping -c 3 api.paypal.com
1000 exit
1001 history | tail -20
root@server [~]#

Furthermore, the firewall is up and running, and you can confirm this by simply running the "iptables -L -n" command which will show you all the rules currently set in place. (See attached)

iptables.zip

[DrByte]

I'm less inclined to suspect networking issues. Instead I'm now leaning toward it being damaged code.
Investigating all the ways in which the site's code differs from original Zen Cart code is key. http://www.zen-cart.com/wiki/index.p...Obscure_Issues

[chriscana]

Happened with my 1.3.8 install, also with 1.5 install. Did a complete fresh install and new database for idoseraudio.com, only changing template and populating with my own products - no changed code. Seems issue happens no matter installation or code base.

[DrByte]

Okay. Investigation would require testing code execution at numerous stages in all the various scripts involved in processing the incoming request, including but not limited to most of the files whose filename contains the string 'paypal' in some form.

But the default core code already does debug logging with PayPal transactions, if the switch is enabled in your ZC paypal module settings. If that's not creating any ipn_xxxxxx.log files when IPNs are submitted, then either the folder isn't writable or the execution is halting before any of the core code runs. Maybe your server config is blocking POST requests to ipn_main_handler.php for some bizarre unknown reason.