Paypal IPN Failing suddenly

I waited until friday but still got IPN failure warning emails from paypal every day. Eventually, on Sat, they disabled my IPN on paypals side after too many failures. I re-enabled it sat, and then sunday still getting the failures.

These are the previous two incidents that have been opened.

Incident: 120628-000140
Incident: 120722-000025

Here is the new one, so 3 total now with no resolution:

incident: 120805-000028

and of course, new incident gets the generic paypal response that all others have received with no information whatsoever.

Remember, the ticket is probably set to "Closed" by default, and if you don't reply within a couple days it'll then be closed without the ability to re-open it.
But, if you do reply, it'll stay open until a tech replies again.

Post back asking for the SPECIFIC HTTP response that their servers are getting. And remind them that a generic "not a 200 response" isn't a helpful answer. Tell them you want it escalated to a senior support person.

Replied back to the incident several times - basically begged them for more information. All I got was: "I have escalated this to the engineers and they confirmed that we are NOT getting a header response from your server."

If that's exactly what they said, then they have no idea what they're talking about.
They are getting a header response. The question is WHAT header response they're getting.
It's already established that they're looking for a "200" response and not getting it.
We need to figure out WHAT response they ARE getting so we can research that with the hosting company.

That was their exact response. Replied again today with what you said, and got another one line response just requesting I send them the script so they can test it.

Tell them they can replicate the problem using their IPN Simulator tool, pointing it to the same URL that the "failing" IPNs are correctly pointing to.
They have all they need to do that.

When asking them to check through the simulator the response is: I would like to test the script you are using so we can eliminate any copy error. Can you please provide the script you are using.

I don't understand their question. Sounds like they're completely avoiding the issue instead of looking at what actual response their servers are getting.

But, to satisfy them, point them to: https://www.zen-cart.com/getit

I don't know whether they've updated the ticket for you or not, but apparently your IPN setting got disabled again. Please turn it on again.
And post back here with what URL you've set your PayPal account to use when sending IPNs (the URL on that same screen where you check the box to turn the IPN feature on).

They're complaining that the URL you mentioned in your support ticket isn't the same as what your store is sending with the transactions.
But I know Zen Cart sends the correct URL in its transactions, so it's not the submissions that are wrong. I've advised as much, but I'd like to know what IPN Notification URL is set in your PayPal account.

https://idosersoftware.com/ipn_main_handler.php

It is re-enabled again. Ive tried other combinations in the past per your suggestion with and without https

This was their last response, but no idea what they are asking to do or if I should do it:

Here is below a code for you to test the IPN url and you will see for yourself there is no headers info returned.
Simply add the following code also in attachment and pass the url https://idosersoftware.com/ipn_main_handler.php and you will see the array is empty.

Thanks
Messan

[:][:][:][:][:][:][:][:][:][:][:][:][:][:][:][:][:][:][:][:][:][:][:][:][:][:][:][:][:][:][:][:][:][:]

<form method="post" action="headers.php"> Enter Your URL:<input type="text" name="endpoint" value="" size="50"/><br> <input type="submit" name="submit" value="Get Headers" /> </form> <?php $url = $_POST['endpoint']; // set the URL $user_agent = "Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0)"; $Action = "header_test"; $version = "1.0"; $POSTrequest = <<< End_Of_Quote Action=$Action&Version=$version End_Of_Quote; $ch = curl_init(); curl_setopt($ch, CURLOPT_URL, $url); //curl_setopt($ch, CURLOPT_USERAGENT, $user_agent); //curl_setopt($ch, CURLOPT_HTTPHEADER, $headers); curl_setopt($ch, CURLOPT_HEADER, 1); // tells curl to include headers in response curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1); // return into a variable curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 0); curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, 0); curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, 2); curl_setopt($ch, CURLOPT_POSTFIELDS, $POSTrequest); curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1); curl_setopt($ch, CURLOPT_TIMEOUT, 300); $mydata = curl_exec ($ch); echo '<br><br>'; echo "The Response: ".$mydata; echo '<br><br>'; print_r(explode(': ', $mydata, 10)); echo '<br><a href="xxxxxxxxxxxxxxxxxxx">Back</a><br>'; ?>

Okay, his script is returning literally nothing when pointing it to your IPN URL. First I've ever seen that. I can't even replicate that in any tests here after intentionally breaking all kinds of things.
Extremely bizarre.

Are you doing any URL-rewriting in your store? If so, can you turn it off?
What's in your /.htaccess file, if you have one? (ie: in the same folder as nddbc.html)
What addons is your store using?
What errors are you seeing in the myDebug-xxxxxxxx.log files in your /cache/ folder?

To rule out several things including hosting issues and maybe upgrade problems, can you set up a quick fresh install of a NEW copy of Zen Cart, using demo data, in another folder (and database), in your hosting account? Configure its PayPal EC module the same way you've configured your live site's Paypal EC module.
(You DO NOT need to change anything in your live Paypal account. The IPN URL there can be left alone.)
Then try an EC transaction on a low-priced item. (You could change an item's price to ten-cents for this test.)
What happens with the IPN in that case? What is the URL to that new test site's ipn_main_handler.php?

This is a brand new install of 1.5. Did it to support this same issue happening in 1.3.8. Fresh install, not upgraded.
There is no url-rewriting.
There is no .htaccess file in the root with nddbc
Only addon store has is a very simple sales report.

Here is an example debug from the cache folder:

[08-Aug-2012 15:24:10] PHP Fatal error: 1267:Illegal mix of collations (latin1_swedish_ci,IMPLICIT) and (utf8_general_ci,COERCIBLE) for operation 'like' :: select count(distinct p.products_id) as total FROM (products p LEFT JOIN manufacturers m USING(manufacturers_id), products_description pd, categories c, products_to_categories p2c ) LEFT JOIN meta_tags_products_description mtpd ON mtpd.products_id= p2c.products_id AND mtpd.language_id = 1 WHERE (p.products_status = 1 AND p.products_id = pd.products_id AND pd.language_id = 1 AND p.products_id = p2c.products_id AND p2c.categories_id = c.categories_id AND ((pd.products_name LIKE '%concentração%' OR p.products_model LIKE '%concentração%' OR m.manufacturers_name LIKE '%concentração%' OR (mtpd.metatags_keywords LIKE '%concentração%' AND mtpd.metatags_keywords !='') OR (mtpd.metatags_description LIKE '%concentração%' AND mtpd.metatags_description !='') OR pd.products_description LIKE '%concentração%') )) in /home/idoser1/public_html/idosersoftware.com/includes/classes/db/mysql/query_factory.php on line 101

You said "fresh install, not upgraded". I assume you're talking about PHP files. What about your store data? It looks like you imported the old data and upgraded it, right?
The collations problem in the error message strongly suggests that this is an upgrade, not a new install.

Would you be willing to do the new test install using demo data, as I mentioned earlier?

Kind of already did this with our other store: https://www.idoseraudio.com/ipn_main_handler.php

This was a brand new install of 1.5, brand new database, and manually entered products. IPN fails emails from paypal also list these IPNs as failing.

It has the same symptoms.

But, since I have no way to know what exactly is different between the code on your sites and the code in the official Zen Cart distribution zip, and can't replicate your symptoms using the code from the official Zen Cart distribution zip, there's not much more I can do.

When using Express Checkout and Payments Pro, the only value that IPNs provide is to automatically update orders in your store when a change takes place with one of those transactions, such as a refund or void or echeck clearing. If you don't need any of that, then simply turn off the IPN notification option in your Paypal account and leave it.

I concede that something is definitely wrong on your site, but after many hours working with official code and creating numerous scenarios, I still can't determine what it is. Your symptoms are most bizarre.
This is where I'd start: http://www.zen-cart.com/wiki/index.php/Troubleshoot_-_Diagnosing_Obscure_Issues

chriscana:

https://www.idosersoftware.com/extras/ipncheck.php
Comes back OK.

Indeed it does. Ergo, logic dictates that PayPal should also be getting some kind of response. The fact that they are getting nothing back only leads me to one possible conclusion. You have a network related issue.

chriscana:

Lunarpages added all paypal IPs to firewall, etc. Still failing.

And I suspect that this is where the issue lies.

The thing is, by default no server/host will ever need to 'add PayPal IP's to their firewall'. If anything they will need to exclude these IP's from being firewalled, but the only time this will occur is if the server administrators are so anally paranoid that they've blocked HTTP/HTTPS requests in the first place (in which case I ask them why the hell are they running publicly accessible servers if the 1st thing they do is block any site that isn't specifically allowed)?

Sorry, I don't expect you to know why they did this (if indeed they have) but few people actually know how to correctly configure firewalls (and I'm not talking about the home based router/firewall appliances here, although even these leave most peoples heads spinning to the extent that most will stay with the default configurations anyway). The firewalls used by ISP's tend to be dedicated Cisco (or similar devices), which takes some highly developed technical skills to configure correctly, and mistakes are easy to make.
I'm not saying that the folks at Lunar pages don't have these skills, but the fact that they have 'added all paypal IPs to firewall, etc' really makes me wonder.

Now, before anyone jumps up and down proclaiming that if it is a firewall issue then surely all stores using the same host will experience the same problem, I would like to remind you that since the site has its own SSL certificate it will also have its own IP address, which unless correctly masked will be subject to its own 'unique' firewall rules.

Having said that, it could also be a routing issue rather than a firewall issue, but the same constraints apply. Firewalls and routers share many common traits.

As much as I try, I can't think of anything else that would cause the symptoms that have been discussed in this thread. I've no problem if someone proves me wrong though, but as stated, this is about the only cause that seems to fit all the symptoms.

Cheers
Rod

Went through over a week of support with lunarpages also and was told they completely disabled the firewall for a week of testing and was still getting IPN fails.

Also, issue was happening before we went ssl. Same issue with 1.3.8 which is why we did a clean install of 1.5, and still getting IPN issues.