Default 'extra info' email content triggering gmail spam detection

[neekfenwick]

ZC 1.5.5f

Not sure if this is really a Bug report (it only affects people using gmail as their email host, I think), but I think it may warrant a change to the default Extra Info content in admin emails, which includes: New order email, Contact Us email, and a few others. If this is a real problem affecting others, I would suggest changing the default ZC email_collect_extra_info() function so that when EXTRA_INFO is included in emails, it does not contain any IP address or hostnames. If not, please move this to non-Bug forum, because I think it's still good info for some people.

Long story short.. the ip address and hostname included in the 'Office use only' section of emails generated by Zen Cart, when sent via a gmail account, tends to cause gmail to block the sending account because they look like spam, and your Zen Cart shop will be unable to send any emails. Not resolving this problem leads to the account being unrecoverably blocked for 24 hours at a time (due to google policy).

Just a heads up to anyone else struggling with this, we have had a few weeks of grief using a G Suite (gmail) account as our SMTP Email configuration in Zen Cart. Our account would be mysteriously blocked, and we would start to see this error message on the ZC web interface after trying to send an email:

"... has exceeded the Gmail sending limit by sending too many messages that were rejected as spam"

Looking into our g suite admin console for the User we're sending as (our EMAIL_SMTPAUTH_MAILBOX) we saw a red banner at the top of the screen with the same message and a 'RESTORE GMAIL' button that can only be used 5 or 6 times a year.

We did get a small number of emails sent back to us with the subject "Delivery Status Notification (Failure)" with the failed message attached.

We realised that there was a pattern to which messages failed, they were almost all in response to the Contact Us form on Zen Cart, or our replies to those messages. Both of these include the 'Extra Info' content. Once I removed that content, the problem went away.

I realise this only affects admin emails .. normal emails to customers don't include the extra info, so wouldn't trigger the spam detection. However it only took a few hours of our business operating normally (10 Contact Us messages a day, perhaps) to trigger the spam detection, then our gmail account was blocked, then all emails stopped being sent .. new order notifications, order update notifications, and a bunch of other order management stuff we've added to ZC.

Fix: Remove the ip address and hostname content from the emails.

Our workaround: We don't use that information anyway, so I have edited email_template_contact_us.html to completely remove the EXTRA_INFO block.

This was extremely painful to our business, with many support staff unable to contact customers for a couple of days, so I really hope this info could help someone else facing this problem :)

[dbltoe]

While we all have our gmail accounts, I would not have entertained the thought of using it to send from a site. If I see an email come in from gmail, hotmail, yahoo, etc. my first thought is "fly by night".

Had a battle once with an owner of an international corporation with offices in 8 countries. His business card had james########################. He insisted it made it easier to check his mail no matter where he was. In almost the same breath, he complained of the unsolicited emails he got.

Most hosts provide mail servers and mail. Why not look at little more professional with support@your_store.com versus jimbob123######################? Even if you were able to obtain microsoft######################, I still don't think any recipient wouldn't think about hitting the s**p**a**m button.

And, some of the wording you are removing is required.

With the proper settings, your emails should not be having problems getting through. AOL, sbcglobal, and AT&T are notorious about blocking emails and never notifying anyone. Still, they will pass almost anything from a non-RBL listed server when properly configured.

Work with your host on getting emails properly set up and professional would be my advice.

Looks like even the forum is not too thrilled with G M A I L

[neekfenwick]

While we all have our gmail accounts, I would not have entertained the thought of using it to send from a site. If I see an email come in from gmail, hotmail, yahoo, etc. my first thought is "fly by night".

Hi It sounds like you think the email address used ends with gmail dot com, but it does not. We have a G Suite business account and use our own company's domain name in all email addresses. To the average user it looks like a normal business, only if you examine the full headers will you see it was routed through Google's mail servers. Under the hood it's all still hosted on gmail and accessible via their normal webmail interface, though we use desktop mail clients via IMAP.

And, some of the wording you are removing is required.

I'm curious, why is it required? Do you mean it's simply useful? I'm considering putting back in some of the simple information like the customer name and logged in email address, since that could be useful, especially in the case of typos when they type their details on the form and they are actually logged in.

With the proper settings, your emails should not be having problems getting through. AOL, sbcglobal, and AT&T are notorious about blocking emails and never notifying anyone. Still, they will pass almost anything from a non-RBL listed server when properly configured.

I would have agreed with you. As I said, we got specific Delivery Status Notification emails with attached messages that had been blocked, and they were pretty much all Contact Us emails. I spent quite some time on call with Google and the first thing we did was ensure our DNS records have DKIM and SPF properly set up - they were not, I was unaware that when using a third party mail provider like google, if your emails come from 'yourcompany dot com' then your DNS records at the nameserver must still have DKIM and SPF (and DMARC, probably) records properly configured. Even with these set up, our account still got regularly blocked and we had to spend our rapidly dwindling "Restore Gmail" attempts to unblock it. This was pretty surprising to me, I'd have thought that a properly validated email would bypass other spam detection measures but apparently not.

They also pointed out that a section like we see in the Extra Info section e.g. "Host Address: cpc108457-cowc8-2-0-cust438.14-2.cable.virginm.net" may trigger the spam detection (they were deliberately vague about it, of course with call center tech support staff sometimes they either cannot tell you full details because of company policy, or they want to cut the call short without getting into too many details, or they simply don't know more details but aren't allowed to tell you).

By a little trial and error it seems pretty clear from my testing that removing this information makes the spam detection issues stop happening.

While this may not be considered a Bug by many standards, I posted to see if any other users have been suffering similar problems (and may not be aware of the root cause but googling may lead them here, I certainly wish I'd seen this post a week ago, would have saved an awful lot of trouble in our business ). It would be very interesting to hear from anyone who has had experience of spam detection due to the Extra Info and I encourage them to reply here so we can pool knowledge.

[carlwhat]

ZC is an e-commerce shopping cart. it is not a mail server.

ZC sends e-mail. there are a multitude of ways that ZC can send email.

according to your initial post, you are using g-mail to send your mail to its final destination. i am a fan of gmail; i use it and i have clients that use their g-suite product.

that said, i have no one that uses their SMTP servers to send mail from their website. i am not saying it is not a good idea, i just have no one that does it.

excuse me if i'm splaining; client sends mail to SMTP server => SMTP server sends mail to receiving mail server => recipient retrieves email from said server. this is now email works.

in your situation, the client is your web site; and your SMTP server is rejecting your mail. this happens due to spam...

DKIM is a method of signing email from your SMTP server so that the receiving email server knows the SMTP server is authorized to send email on your behalf. how setting up DKIM would address this problem, you got me... perhaps you can elaborate?

i am not sure about your host, or if you have a slice, but i would think you might be better off sending email utilizing a mail server on your host (where your website resides) and configuring an SPF record and DKIM for your host and bypassing gmails SMTP server.

i am not disputing that what you are doing is working; but to me there is no guarantee that it will continue to work in the future. the idea that you can not convince your SMTP server that you are authorized to send email, and the only way to do this is to remove IP address information is silly to me... SPF, DKIM and DMARC are the gold standard for ensuring mail delivery from SMTP server to the receiving email server. but you are saying the problem is happening before that. which strikes me as messed up and worthy of determining a better SMTP server.

if i am wrong, i would gladly like to be enlightened on it. cuz email is hard!

[neekfenwick]

Hi carlwhat Thanks for your points and I know this situation is complicated, more than the average observer on this forum may understand from their experience. You do sound clued up, though.

in your situation, the client is your web site; and your SMTP server is rejecting your mail. this happens due to spam...

My point here is that, apparently fairly recently, the rules of gmail's spam classification seem to have changed (we have seen a change in behaviour). I base this on the fact that we've used google business mail as our primary address for quite a while and had no problem, and within the last two weeks or so there has been a pretty consistent classification of some emails sent by ZC as spam (which have led to a very serious consequence of our main email account on gmail being blocked from sending emails, which makes our order management system via email non functional, as I detailed above). As I said before, these cases seem to be primarily cases where the EXTRA_INFO block is included in an email. None of the normal order update etc emails to customers are classified as spam.

DKIM is a method of signing email from your SMTP server so that the receiving email server knows the SMTP server is authorized to send email on your behalf. how setting up DKIM would address this problem, you got me... perhaps you can elaborate?

When we suspected google were detecting our emails as "spammy" we looked at ways this might be the case. One is that DNS records like DKIM and SPF are not set up correctly. So, setting these up correctly would remove them as potential reasons google would mark our emails as spammy. Yet, after doing this, the problem persisted.

i am not sure about your host, or if you have a slice, but i would think you might be better off sending email utilizing a mail server on your host (where your website resides) and configuring an SPF record and DKIM for your host and bypassing gmails SMTP server.

The situation we are discussing is related to google's spam detection of emails sent via their SMTP service. The DNS records of our host (the domain in the 'From' header of the email) should contain records that can validate the sender (DKIM/SPF/etc). Our physical host is not really part of that equation.

We did used to run our own MTA (exim) and had some hard lessons learned as we were marked as spammy and put on RBLs (our host was insecure in various ways I won't detail here), that is one reason we moved to G Suite (gmail business account) to handle our emails, many months ago. So we've tried what you suggest ("utilizing a mail server on your host") but I don't see how, if we are using gmail as a mail host now, that would improve the situation. It would be a change, sure, but the improvement is not qualified, and it would be a regression (not that you knew that ).

i am not disputing that what you are doing is working; but to me there is no guarantee that it will continue to work in the future. the idea that you can not convince your SMTP server that you are authorized to send email, and the only way to do this is to remove IP address information is silly to me... SPF, DKIM and DMARC are the gold standard for ensuring mail delivery from SMTP server to the receiving email server. but you are saying the problem is happening before that. which strikes me as messed up and worthy of determining a better SMTP server.

The problem isn't that we can't convince our SMTP server that we are authorized. That has been achieved by several steps (basic auth, DKIm, SPF etc) .. the problem occurs because of physical content in the emails that are sent. This is outside of host, authentication, authorisation, and as far as I can tell, reputation (as far as that goes in the email/spam world).

The problem doesn't happen "before that" (by which I think you mean the point of sending the email, can our sending action be considered valid). It happens exactly after the email is sent, and we get back a Delivery Status Notification email, and after a few occurances of that, the sending account is blocked on gmail for "sending spam". My investigation seems to (over the past 3 days now) have proved that removing certain content (so far I've narrowed it down to the ip address and host address, see email_collect_extra_info in functions_email.php) from the emails stops this spam classification.

Your point of a 'better SMTP server' is a little weird in that we're talking about google/gmail here, I think we can generally agree it's world class. The emails in question tend to be from and to gmail, i.e. when a customer submits the Contact Us form, the only email sent that includes the Office Use Only section is the one sent from and to the ZC host's own email address, i.e. this email goes from 'us at ourcompany dot com' to the same address, and these are being blocked, a Delivery Status Notification email is sent, and after about 10 attempts of this our sending gmail account is blocked on gmail. I think they're generally doing a great job, and I'm just trying to help us and everyone else work with them.

cuz email is hard!

It is a massive pain in the balls to admin, for sure

[neekfenwick]

Just wanted to add, after re-reading my own words, some people might legitimately think that our Contact Us form is being abused by spammers, and that content is then being classified as spam by google. This actually happened to us about a year ago, so we installed the recaptcha add-on. Since then we've had no problem with the Contact Us form being abused (great add-on ) All this is water under the bridge and, in my considered opinion, nothing to do with this thread.

[barco57]

Recently had issues with deliverability to gmail addresses and getting DMARC setup correctly was thing that fixed it.

[carlwhat]

listen, if you got your email working, great. but i submit to you and others here that ip address and host address information is valid information in an email, and one can send ham emails with that information. i do it all the time.

None of the normal order update etc emails to customers are classified as spam.

how would you know? did you contact everyone of them?

We did used to run our own MTA (exim) and had some hard lessons learned as we were marked as spammy and put on RBLs (our host was insecure in various ways I won't detail here), that is one reason we moved to G Suite (gmail business account) to handle our emails, many months ago. So we've tried what you suggest ("utilizing a mail server on your host") but I don't see how, if we are using gmail as a mail host now, that would improve the situation. It would be a change, sure, but the improvement is not qualified, and it would be a regression (not that you knew that ).

the fact that you could not get exim configured properly and ended up on RBL lists is on you and your host. and if you are using a shared host, well it could be even harder based on who your host is. running exim4 on a debian host and getting that configured correctly, is to me, far preferable to using gmail for sending your email from your website.

in your previous setup, you got put on some RBLs, because you and your host could not get exim setup properly. now gmail is telling you, you can't use our SMTP server to send spammy email. looks like the same problem; you have just moved it to a different point in the email chain.

email is hard. and i'm not disputing that what you are doing is working for you. but i think the setup is less than ideal, as again gmail is telling you your email to yourself is spammy. and if you want IP information in that extra info email, who the heck is gmail to tell you you can't have it? especially after you are paying them?

so, again, i submit to you that your email could be setup to receive that information.

[swguy]

I agree with @carlwhat.

[davewest]

With the changes Google made, it's getting harder to use anything other then there apps and browser to access gmail accounts. they tell me my mail software is unsecured and I have to give my left arm and right two fingers to continue to use it.. (Thunderbird)

Google trying to take over the world again...

With that... remember email from your shop is originating from your shop!! Just read the header, so it has no way for you to tack back to the order individual, contact-er, so on.. the tracer ends at your shop/host.. adding IP, host address to the admin side email lets you have some way to trace back if the sender is legit or not. Lest you have it for your records...

I left one host for not updating mail server which was constantly getting hijacked.. the host I'm with now keeps there server up to date and I've not had any issues with it.