Default 'extra info' email content triggering gmail spam detection

ZC 1.5.5f

Not sure if this is really a Bug report (it only affects people using gmail as their email host, I think), but I think it may warrant a change to the default Extra Info content in admin emails, which includes: New order email, Contact Us email, and a few others. If this is a real problem affecting others, I would suggest changing the default ZC email_collect_extra_info() function so that when EXTRA_INFO is included in emails, it does not contain any IP address or hostnames. If not, please move this to non-Bug forum, because I think it's still good info for some people.

Long story short.. the ip address and hostname included in the 'Office use only' section of emails generated by Zen Cart, when sent via a gmail account, tends to cause gmail to block the sending account because they look like spam, and your Zen Cart shop will be unable to send any emails. Not resolving this problem leads to the account being unrecoverably blocked for 24 hours at a time (due to google policy).

Just a heads up to anyone else struggling with this, we have had a few weeks of grief using a G Suite (gmail) account as our SMTP Email configuration in Zen Cart. Our account would be mysteriously blocked, and we would start to see this error message on the ZC web interface after trying to send an email:

"... has exceeded the Gmail sending limit by sending too many messages that were rejected as spam"

Looking into our g suite admin console for the User we're sending as (our EMAIL_SMTPAUTH_MAILBOX) we saw a red banner at the top of the screen with the same message and a 'RESTORE GMAIL' button that can only be used 5 or 6 times a year.

We did get a small number of emails sent back to us with the subject "Delivery Status Notification (Failure)" with the failed message attached.

We realised that there was a pattern to which messages failed, they were almost all in response to the Contact Us form on Zen Cart, or our replies to those messages. Both of these include the 'Extra Info' content. Once I removed that content, the problem went away.

I realise this only affects admin emails .. normal emails to customers don't include the extra info, so wouldn't trigger the spam detection. However it only took a few hours of our business operating normally (10 Contact Us messages a day, perhaps) to trigger the spam detection, then our gmail account was blocked, then all emails stopped being sent .. new order notifications, order update notifications, and a bunch of other order management stuff we've added to ZC.

Fix: Remove the ip address and hostname content from the emails.

Our workaround: We don't use that information anyway, so I have edited email_template_contact_us.html to completely remove the EXTRA_INFO block.

This was extremely painful to our business, with many support staff unable to contact customers for a couple of days, so I really hope this info could help someone else facing this problem :)

While we all have our gmail accounts, I would not have entertained the thought of using it to send from a site. If I see an email come in from gmail, hotmail, yahoo, etc. my first thought is "fly by night".

Had a battle once with an owner of an international corporation with offices in 8 countries. His business card had james########################. He insisted it made it easier to check his mail no matter where he was. In almost the same breath, he complained of the unsolicited emails he got.

Most hosts provide mail servers and mail. Why not look at little more professional with support@your_store.com versus jimbob123######################? Even if you were able to obtain microsoft######################, I still don't think any recipient wouldn't think about hitting the s**p**a**m button.

And, some of the wording you are removing is required.

With the proper settings, your emails should not be having problems getting through. AOL, sbcglobal, and AT&T are notorious about blocking emails and never notifying anyone.:( Still, they will pass almost anything from a non-RBL listed server when properly configured.

Work with your host on getting emails properly set up and professional would be my advice.

Looks like even the forum is not too thrilled with G M A I L

Quote:

---

Originally Posted by dbltoe

While we all have our gmail accounts, I would not have entertained the thought of using it to send from a site. If I see an email come in from gmail, hotmail, yahoo, etc. my first thought is "fly by night".

---

Hi :smile: It sounds like you think the email address used ends with gmail dot com, but it does not. We have a G Suite business account and use our own company's domain name in all email addresses. To the average user it looks like a normal business, only if you examine the full headers will you see it was routed through Google's mail servers. Under the hood it's all still hosted on gmail and accessible via their normal webmail interface, though we use desktop mail clients via IMAP.

Quote:

---

Originally Posted by dbltoe

And, some of the wording you are removing is required.

---

I'm curious, why is it required? Do you mean it's simply useful? I'm considering putting back in some of the simple information like the customer name and logged in email address, since that could be useful, especially in the case of typos when they type their details on the form and they are actually logged in.

Quote:

---

Originally Posted by dbltoe

With the proper settings, your emails should not be having problems getting through. AOL, sbcglobal, and AT&T are notorious about blocking emails and never notifying anyone.:( Still, they will pass almost anything from a non-RBL listed server when properly configured.

---

I would have agreed with you. As I said, we got specific Delivery Status Notification emails with attached messages that had been blocked, and they were pretty much all Contact Us emails. I spent quite some time on call with Google and the first thing we did was ensure our DNS records have DKIM and SPF properly set up - they were not, I was unaware that when using a third party mail provider like google, if your emails come from 'yourcompany dot com' then your DNS records at the nameserver must still have DKIM and SPF (and DMARC, probably) records properly configured. Even with these set up, our account still got regularly blocked and we had to spend our rapidly dwindling "Restore Gmail" attempts to unblock it. This was pretty surprising to me, I'd have thought that a properly validated email would bypass other spam detection measures but apparently not.

They also pointed out that a section like we see in the Extra Info section e.g. "Host Address: cpc108457-cowc8-2-0-cust438.14-2.cable.virginm.net" may trigger the spam detection (they were deliberately vague about it, of course with call center tech support staff sometimes they either cannot tell you full details because of company policy, or they want to cut the call short without getting into too many details, or they simply don't know more details but aren't allowed to tell you).

By a little trial and error it seems pretty clear from my testing that removing this information makes the spam detection issues stop happening.

While this may not be considered a Bug by many standards, I posted to see if any other users have been suffering similar problems (and may not be aware of the root cause but googling may lead them here, I certainly wish I'd seen this post a week ago, would have saved an awful lot of trouble in our business :smile:). It would be very interesting to hear from anyone who has had experience of spam detection due to the Extra Info and I encourage them to reply here so we can pool knowledge.

ZC is an e-commerce shopping cart. it is not a mail server.

ZC sends e-mail. there are a multitude of ways that ZC can send email.

according to your initial post, you are using g-mail to send your mail to its final destination. i am a fan of gmail; i use it and i have clients that use their g-suite product.

that said, i have no one that uses their SMTP servers to send mail from their website. i am not saying it is not a good idea, i just have no one that does it.

excuse me if i'm splaining; client sends mail to SMTP server => SMTP server sends mail to receiving mail server => recipient retrieves email from said server. this is now email works.

in your situation, the client is your web site; and your SMTP server is rejecting your mail. this happens due to spam...

DKIM is a method of signing email from your SMTP server so that the receiving email server knows the SMTP server is authorized to send email on your behalf. how setting up DKIM would address this problem, you got me... perhaps you can elaborate?

i am not sure about your host, or if you have a slice, but i would think you might be better off sending email utilizing a mail server on your host (where your website resides) and configuring an SPF record and DKIM for your host and bypassing gmails SMTP server.

i am not disputing that what you are doing is working; but to me there is no guarantee that it will continue to work in the future. the idea that you can not convince your SMTP server that you are authorized to send email, and the only way to do this is to remove IP address information is silly to me... SPF, DKIM and DMARC are the gold standard for ensuring mail delivery from SMTP server to the receiving email server. but you are saying the problem is happening before that. which strikes me as messed up and worthy of determining a better SMTP server.

if i am wrong, i would gladly like to be enlightened on it. cuz email is hard!

Hi carlwhat :smile: Thanks for your points and I know this situation is complicated, more than the average observer on this forum may understand from their experience. You do sound clued up, though.

Quote:

---

Originally Posted by carlwhat

in your situation, the client is your web site; and your SMTP server is rejecting your mail. this happens due to spam...

---

My point here is that, apparently fairly recently, the rules of gmail's spam classification seem to have changed (we have seen a change in behaviour). I base this on the fact that we've used google business mail as our primary address for quite a while and had no problem, and within the last two weeks or so there has been a pretty consistent classification of some emails sent by ZC as spam (which have led to a very serious consequence of our main email account on gmail being blocked from sending emails, which makes our order management system via email non functional, as I detailed above). As I said before, these cases seem to be primarily cases where the EXTRA_INFO block is included in an email. None of the normal order update etc emails to customers are classified as spam.

Quote:

---

Originally Posted by carlwhat

DKIM is a method of signing email from your SMTP server so that the receiving email server knows the SMTP server is authorized to send email on your behalf. how setting up DKIM would address this problem, you got me... perhaps you can elaborate?

---

When we suspected google were detecting our emails as "spammy" we looked at ways this might be the case. One is that DNS records like DKIM and SPF are not set up correctly. So, setting these up correctly would remove them as potential reasons google would mark our emails as spammy. Yet, after doing this, the problem persisted.

Quote:

---

Originally Posted by carlwhat

i am not sure about your host, or if you have a slice, but i would think you might be better off sending email utilizing a mail server on your host (where your website resides) and configuring an SPF record and DKIM for your host and bypassing gmails SMTP server.

---

The situation we are discussing is related to google's spam detection of emails sent via their SMTP service. The DNS records of our host (the domain in the 'From' header of the email) should contain records that can validate the sender (DKIM/SPF/etc). Our physical host is not really part of that equation.

We did used to run our own MTA (exim) and had some hard lessons learned as we were marked as spammy and put on RBLs (our host was insecure in various ways I won't detail here), that is one reason we moved to G Suite (gmail business account) to handle our emails, many months ago. So we've tried what you suggest ("utilizing a mail server on your host") but I don't see how, if we are using gmail as a mail host now, that would improve the situation. It would be a change, sure, but the improvement is not qualified, and it would be a regression (not that you knew that :smile:).

Quote:

---

Originally Posted by carlwhat

i am not disputing that what you are doing is working; but to me there is no guarantee that it will continue to work in the future. the idea that you can not convince your SMTP server that you are authorized to send email, and the only way to do this is to remove IP address information is silly to me... SPF, DKIM and DMARC are the gold standard for ensuring mail delivery from SMTP server to the receiving email server. but you are saying the problem is happening before that. which strikes me as messed up and worthy of determining a better SMTP server.

---

The problem isn't that we can't convince our SMTP server that we are authorized. That has been achieved by several steps (basic auth, DKIm, SPF etc) .. the problem occurs because of physical content in the emails that are sent. This is outside of host, authentication, authorisation, and as far as I can tell, reputation (as far as that goes in the email/spam world).

The problem doesn't happen "before that" (by which I think you mean the point of sending the email, can our sending action be considered valid). It happens exactly after the email is sent, and we get back a Delivery Status Notification email, and after a few occurances of that, the sending account is blocked on gmail for "sending spam". My investigation seems to (over the past 3 days now) have proved that removing certain content (so far I've narrowed it down to the ip address and host address, see email_collect_extra_info in functions_email.php) from the emails stops this spam classification.

Your point of a 'better SMTP server' is a little weird in that we're talking about google/gmail here, I think we can generally agree it's world class. The emails in question tend to be from and to gmail, i.e. when a customer submits the Contact Us form, the only email sent that includes the Office Use Only section is the one sent from and to the ZC host's own email address, i.e. this email goes from 'us at ourcompany dot com' to the same address, and these are being blocked, a Delivery Status Notification email is sent, and after about 10 attempts of this our sending gmail account is blocked on gmail. I think they're generally doing a great job, and I'm just trying to help us and everyone else work with them.

Quote:

---

Originally Posted by carlwhat

cuz email is hard!

---

It is a massive pain in the balls to admin, for sure :smile:

Just wanted to add, after re-reading my own words, some people might legitimately think that our Contact Us form is being abused by spammers, and that content is then being classified as spam by google. This actually happened to us about a year ago, so we installed the recaptcha add-on. Since then we've had no problem with the Contact Us form being abused (great add-on :smile:) All this is water under the bridge and, in my considered opinion, nothing to do with this thread.

Recently had issues with deliverability to gmail addresses and getting DMARC setup correctly was thing that fixed it.

listen, if you got your email working, great. but i submit to you and others here that ip address and host address information is valid information in an email, and one can send ham emails with that information. i do it all the time.

Quote:

---

Originally Posted by neekfenwick

None of the normal order update etc emails to customers are classified as spam.

---

how would you know? did you contact everyone of them?

Quote:

---

Originally Posted by neekfenwick

We did used to run our own MTA (exim) and had some hard lessons learned as we were marked as spammy and put on RBLs (our host was insecure in various ways I won't detail here), that is one reason we moved to G Suite (gmail business account) to handle our emails, many months ago. So we've tried what you suggest ("utilizing a mail server on your host") but I don't see how, if we are using gmail as a mail host now, that would improve the situation. It would be a change, sure, but the improvement is not qualified, and it would be a regression (not that you knew that :smile:).

---

the fact that you could not get exim configured properly and ended up on RBL lists is on you and your host. and if you are using a shared host, well it could be even harder based on who your host is. running exim4 on a debian host and getting that configured correctly, is to me, far preferable to using gmail for sending your email from your website.

in your previous setup, you got put on some RBLs, because you and your host could not get exim setup properly. now gmail is telling you, you can't use our SMTP server to send spammy email. looks like the same problem; you have just moved it to a different point in the email chain.

email is hard. and i'm not disputing that what you are doing is working for you. but i think the setup is less than ideal, as again gmail is telling you your email to yourself is spammy. and if you want IP information in that extra info email, who the heck is gmail to tell you you can't have it? especially after you are paying them?

so, again, i submit to you that your email could be setup to receive that information.

I agree with @carlwhat.

With the changes Google made, it's getting harder to use anything other then there apps and browser to access gmail accounts. they tell me my mail software is unsecured and I have to give my left arm and right two fingers to continue to use it.. (Thunderbird)

Google trying to take over the world again...

With that... remember email from your shop is originating from your shop!! Just read the header, so it has no way for you to tack back to the order individual, contact-er, so on.. the tracer ends at your shop/host.. adding IP, host address to the admin side email lets you have some way to trace back if the sender is legit or not. Lest you have it for your records...

I left one host for not updating mail server which was constantly getting hijacked.. the host I'm with now keeps there server up to date and I've not had any issues with it.

Goodbye zen cart forum.

Must've been what Scott said, right guys?:dontgetit

I have that effect on people.

Did I over do it with the pinky and the brain comment about Gooo...

It should be a point of pride for all contributors that the Zen Cart forum is, for the most part, a friendly and welcoming place.
(Especially compared to other forums, which will not be named, although they rhyme with Shmesta-Shop.)

Whatever the case for departure, I would say that if the ip-address and/or hostname were an issue, that they could still be included in the email perhaps with some massaging.

For example instead of the ip address being four groups of information separated by three periods that the period have a space before and aftewr it and any link coded into the numbers just be removed or similar. Or perhaps (haven't looked at the detail in a while) some additional htmlencoding be applied...

Unfortunately there could still be some other reason that Gmail (even when using the g-suite portion) is flagging the email. We haven't even really talked about the settings that have been applied... smtp? Smtpauth? Whatever the new Gmail server option is? All/each of these provides different information to the servers along the way. Last time I looked at using the g suite for a domain, there seemed like a bunch of things needed to be done to have the email come/go through their servers related to dns records. Sounds like those may have all be addressed, but never hurts to ask or look again from the perspective of trust by the server(s).

Have you tried setting up an email address that would be used for sending such messages but perhaps not used as the recipient as well? Wondering if part of the issue is the receiver is the sender type thing...

Overall, doesn't surprise me that messages with more detail that a basic shopper gets when making a purchase get "treated" differently.

One common tactic used by spammers (er, those who have hijacked a legit mail server) is to send emails "from oneself, to oneself". I can see where spam-trappers could flag that up if other heuristics were a match. Maybe in this case the "other" patterns are the presence of these ip/host details.

Thus, another approach you could try is (as hinted at in one of the comments above) to set up a separate mailbox for contact-us messages, so the send-from and send-to aren't the same anymore, in the case of contact-us anyway. Helpful to try as a test to rule out whether that's a notable contributor to the issue you're having. You might then also send all your admin-copy-of-order-emails to something different than the send-from as well.

As to ripping out the ip/host details from the extra_info section, sure you can do that. As discussed maybe removing the whole section is overkill.

Quote:

---

Originally Posted by barco57

Recently had issues with deliverability to gmail addresses and getting DMARC setup correctly was thing that fixed it.

---

Thanks for the tip. You may have a point as we don't have DMARC set up. I'd hoped that DKIM and SPF would do it, and to be honest I found the DKIM documentation pretty hard to follow. I'll have another bash at it, and try re-introducing the Extra Info content to see if the detection is still triggered.

Sorry, your reply rubbed me up the wrong way the other day and I responded over the top, apologies for that.

Quote:

---

Originally Posted by carlwhat

listen, if you got your email working, great. but i submit to you and others here that ip address and host address information is valid information in an email, and one can send ham emails with that information. i do it all the time.

---

Yes, I do think ip/host is valid content, and we've used the default ZC behaviour for 10 years, moving to gmail about a year or more ago, with no problem. Suddenly in the past couple of weeks we're getting that same kind of content flagged as spam. So either Gmail have changed their spam detection algorithm, or the hostnames being included have changed to include content that trigger the old detection algorithm, or something else is going on. Google aren't willing to disclose any info about their algorithm, at least to me :smile:

Quote:

---

Originally Posted by carlwhat

how would you know? did you contact everyone of them?

---

I said the emails are not 'classified as spam' because, for the ones that we believe are (the Contact Us ones), we immediately get an email from Google with the subject 'Delivery Status Notification' saying it was blocked, and we don't see that behavious with any of the (far more voluminous) normal order emails. Certainly, some emails to customers may be being marked as spam, but the real indicator for me was that after a series of the reports that we do get, our gmail account is blocked (and we have to go and hit the 'Restore gmail' button), then, after making the change to the Contact Us email content, we haven't had to do that once, despite still sending hundreds of normal order update (and all the other types) of email from ZC each day.

Quote:

---

Originally Posted by carlwhat

the fact that you could not get exim configured properly and ended up on RBL lists is on you and your host. and if you are using a shared host, well it could be even harder based on who your host is. running exim4 on a debian host and getting that configured correctly, is to me, far preferable to using gmail for sending your email from your website.

---

All fair points, and I didn't want to get into a discussion about it (it's a whole other subject) I just wanted to be clear that we have tried that approach, i.e. I wasn't just ignoring the suggestion to try doing that.

Quote:

---

Originally Posted by carlwhat

in your previous setup, you got put on some RBLs, because you and your host could not get exim setup properly. now gmail is telling you, you can't use our SMTP server to send spammy email. looks like the same problem; you have just moved it to a different point in the email chain.

---

exim config certainly is complicated and we had a series of issues that were years ago now and I can't remember precise details. As I recall we got put on RBLs because our host was compromised and used as a relay to send actual spam email by spammers. Once our domain name was blacklisted it was quite painful to work out of that situation. It wasn't the same situation as this one of ZC content being considered spammy. I could rant on about that and past mistakes but it's not relevant here. We moved to gmail partly because we thought it would remove all possibility of being considered spammy, so it's a kind of dark humour that we've run into this position :smile:

Quote:

---

Originally Posted by carlwhat

email is hard. and i'm not disputing that what you are doing is working for you. but i think the setup is less than ideal, as again gmail is telling you your email to yourself is spammy. and if you want IP information in that extra info email, who the heck is gmail to tell you you can't have it? especially after you are paying them?

---

Yes, if I could dictate to gmail what is spam and what isn't, life would be great, but even if I pay for a business account we have no actual control over their spam filters. One suggestion to fix the situation is certainly to change email provider (or self-host as you suggest).

If your overall suggestion is to 'not use gmail' then that's fine, I was posting to try to help those who _are_ using gmail and have suddenly found their accounts being blocked in this way. The more I read here, it sounds like the issue on gmail is not considered something that would warrant a core change to ZC (i.e. a Bug), though I haven't yet tried the final step of setting up DMARC in our DNS records, I'll have a bash at that and try to report back.

Quote:

---

Originally Posted by DrByte

One common tactic used by spammers (er, those who have hijacked a legit mail server) is to send emails "from oneself, to oneself". I can see where spam-trappers could flag that up if other heuristics were a match. Maybe in this case the "other" patterns are the presence of these ip/host details.

---

Yes I agree, one of my first thoughts was the pattern that the Contact Us email actually sends an email 'to' us, 'from' us but with a 'Reply-To' header of the customer's email address (for easy Reply action), which I thought could be considered a kind of hi-jack attempt. But this idea was fairly quickly quashed by the Extra Info thing.

Quote:

---

Originally Posted by DrByte

As to ripping out the ip/host details from the extra_info section, sure you can do that. As discussed maybe removing the whole section is overkill.

---

Fair point, I've now re-instated almost everything except the ip/host details, and emails are still coming through fine. (I also took out the disclaimer and footer content as that didn't seem useful for our staff, though now I think about it perhaps it should stay in because when staff Reply to the email, that content is no longer included in the content they receive...)

Of course we're all busy, this is all around a very busy normal day job, I do appreciate the replies.

Quote:

---

Originally Posted by neekfenwick

(I also took out the disclaimer and footer content as that didn't seem useful for our staff, though now I think about it perhaps it should stay in because when staff Reply to the email, that content is no longer included in the content they receive...)

---

We've opted to remove those disclaimers in the default install of Zen Cart starting with v1.5.7, as they can sometimes trigger false-positives for spam flags.

Quote:

---

Originally Posted by DrByte

We've opted to remove those disclaimers in the default install of Zen Cart starting with v1.5.7, as they can sometimes trigger false-positives for spam flags.

---

That's interesting. I put the disclaimer and footer back in, so the only thing missing was the IP/Hostname content, and things seemed fine. Then I set up a DMARC TXT record in our DNS (https://mxtoolbox.com/DMARCRecordGenerator.aspx seems to generate the record well) and put the IP/Hostname back in, things seemed fine. Then I took the DMARC record _out_, and things still seemed fine! Not sure if there was some latency/caching issues in various parts of the system that made my changes not take immediate effect (DNS records had TTL=300 but still, there are plenty of other places latency can have effect, such as some hidden scoring mechanism inside google's spam detection).

This is a busy production server that I can't really mess much with, but I'm left wondering if google had a temporary tightening of their spam detected which they have now relaxed (perhaps due to lots of complaints, I've no idea).

So perhaps this whole problem has just "gone away" but you never know, it may hit someone else at some time so I hope the thread proves useful to someone, somewhere :smile:

I'm glad you found a combination of things that's working well now.

As with all things "google", keep an eye out for sudden unannounced unexplained un-repented-for changes. :)

LOL it's normally us who have to repent to and propitiate Google. :)