Difference between revisions of "Troubleshoot - Understanding Causes Of Lost Emails"

From Zen Cart(tm) Wiki
Jump to: navigation, search
m
Line 6: Line 6:
 
If you're having difficulties with email and your online store, this article will hopefully assist in resolving those problems.
 
If you're having difficulties with email and your online store, this article will hopefully assist in resolving those problems.
  
==Executive Summary==
+
==TL;DR Summary==
 
#Make sure your domain is properly configured. We will cover the various settings and entities you need to have configured for your site domain to be able to send email.
 
#Make sure your domain is properly configured. We will cover the various settings and entities you need to have configured for your site domain to be able to send email.
 
#Make sure your email addresses are properly configured. We'll discuss the required and recommended email addresses for your store, and some do-s and don'ts.
 
#Make sure your email addresses are properly configured. We'll discuss the required and recommended email addresses for your store, and some do-s and don'ts.

Revision as of 19:31, 30 December 2016

In Depth Guide to Email Troubleshooting Issues

Copyright (c) 2008 by Chuck Redman, for Zen Cart®, All Rights Reserved

Distributed under Creative Commons License

If you're having difficulties with email and your online store, this article will hopefully assist in resolving those problems.

TL;DR Summary

  1. Make sure your domain is properly configured. We will cover the various settings and entities you need to have configured for your site domain to be able to send email.
  2. Make sure your email addresses are properly configured. We'll discuss the required and recommended email addresses for your store, and some do-s and don'ts.
  3. Set up your Zen Cart mail system. We'll cover the different email transports and the implications of their use.
  4. Email, black holes and singularities. I offer some sage advice on cybernetic entomology.


Email And Your Online Store

You've battled through the menus, the templates, the overrides, the tears and the joy, and your site is now ready. No it isn't.

One of the most important aspects of your online creation, and one of the most overlooked, is the email system. Your entire venture depends on your site being able to communicate with both you and your clients.

And communication doesn't mean just the visual appearance of the site and the product selection and layout. It means the much more mundane world of order confirmations and notifications, newsletters, password resets and delivery confirmations.

You will find that contrary to your experience, sending email is not simply clicking the send button and submitting an email to a smiling group of swift-footed Mercury's eager and willing to speed it to its destination. The mail servers of the world more resemble a glowering bunch of Trolls or Vogon beaurocrats meticulously eager to find any excuse to consign your email to the pending file or the shredder.

Fortunately the cure is usually relatively simple. The Zen Cart developers go to great lengths to ensure that any emails generated by the system are as standards-compliant as possible, so that they are unlikely to attract the attention of Spam filters or 'purist' mail server set-ups. You should certainly do the same. Unfortunately there are some instances where the Zen Cart mail set-up and your hosting company mail servers have no compatible mail transports. While that may be considered by some to be a good enough reason not to use their services, this is an extreme case and it is usually possible to find a workable compromise.

QUICK TIPS

For those looking for the quick-action tips, note the following: To minimize the likelihood of your emails getting rejected during a delivery attempt, we recommend:

  • use a "real", dedicated, paid-for, domain email-address as your store's primary send-from address. Using a "free" email account as your send-from address will often cause your store emails to be dumped into a black hole.
  • set "emails must send from known domain" to "true" (so that the email doesn't look like it's coming from someplace else)
  • using the "PHP" transport method is common, and works for many hosts; however, SMTPAUTH tends to be less likely to land your sent messages into the bit-bucket, since to send the messages your account has to be validated--making things appear to be more trustworthy.
  • if you are finding that your emails aren't getting sent properly, whether due to specific error messages, or just apparently not being delivered, check out Section 4 near the end of this article for some suggested troubleshooting tips.
  • make sure that your domain email setup is correct and complete, and that you can send and receive email from it using common email clients.

The rest of this article gets fairly technical, and will be most suitable to read with a mug of your favourite warm drink in-hand.


The following is a non exhaustive checklist of things that are usually necessary in order for you to have a smoothly operating email system on your website; or if you prefer, 'To enhance and enrich your online ecommerce experience'.

The items discussed and the explanations should not be regarded as definitive or legal or complete. They are painted with a very broad brush, and are there to help you better understand what is involved, and not as academic or legal reference texts. Correspondence on errors, omissions or additions is welcome; that on nitpicking nuances and interpretations will be ignored. (Insert 15 page legal disclaimer here).


There are several areas to consider when attempting to understand how email works. If your online store is having troubles getting emails to your customers, it's wise to remember that there are numerous factors involved: domain names and configuration, send-FROM settings and address, send-TO addresses, hosting account configurations, formats of emails, reliability of the servers being used at various stages, reputation for spam origins, and lots more. Read along to get a broader understanding of the many factors involved and the many things that can affect *whether* and *how* any given message gets handled and delivered.


1. Your domain.

One of the first steps in setting up an online site is deciding on a name and website URL or address for the site. Once this is done, either through a domain registrar or a hosting company, money changes hands and little further thought is given to the matter, except at renewal time. It's important to have some understanding of exactly what this means. It means you have a method of being uniquely identified on the internet by the sequence of characters you have chosen as your domain name. Your domain name does not include www, it is the rest of the identifier - mydomain.com, mysuperdupernewstore.com etc. The www bit is a subdomain, by convention usually reserved for websites, but you are free to create an infinite number of subdomains - myfirststore.mydomain.com, mysecondstore.mydomain.com, and indeed www.myfirststore.mydomain.com www.mygranniesphotogalleryofhertriptoelko.mydomain.com, on and on till boredom or exhaustion set in. Note that all of these are added to the left of the domain name. Adding bits to the right is a no-no, or cost money, except in specific circumstances, such as when they refer to folders on a website or ftp site - www.mydomain.com/shop. www.mydomain.com/gallery. Again, repeat ad nauseum.

In order to make all of this work, each and every one of the domains and subdomains has an entry in a table (somewhat akin to a telephone book) which cross references your domain or subdomain name to an IP address where that domain may be found. These tables are maintained by 'name servers' and are the backbone of the DNS or Domain Name System. Theses entries take the form:

 @  IN   A  10.10.10.10
 mydomain.com. IN   A  10.10.10.10
 www IN CNAME mydomain.com
 granniesphotos.mydomain.com IN A 10.10.10.10

... which means 'You can find mydomain.com at IP address 10.10.10.10, and when I say www.mydomain.com I really mean mydomain.com, it's an alias, again in an infinite number of combinations.

Depending on your domain registrar or hosting company you may have direct access to updating your DNS settings, or you may have to request any changes you want made.


You may be asking, What does this have to do with your website your shop and email? Good question. Read on ...

MX RECORDS

All of those domains and subdomains can have email addresses. And in exactly the same way, they need DNS entries for the mail server(s) involved in handling mail for the domain. The plural is there because domains often have multiple mail servers defined, with an associated 'priority' , where the ultimate destination, the mail server that actually handles the mail delivery to the user, is the one with the lowest 'priority' number and thus highest priority. These entries take the form of an MX or mail record:

 mydomain.com. IN MX 10 fred.mydomain.com.

In this example, we see that mail for mydomain.com is handled by a mail server fred.mydomain.com I have intentionally not used mail.mydomain.com there, although that is the usual convention. So usual that some mail servers will automatically alias mail.mydomain.com to mydomain.com. Do not rely on this, it is NOT a standard, merely a convention. It is not necessary that the MX record point at a mail server in the same domain, it can be any valid address. i.e. the MX record for mydomain.com can point to the mail server at myhostingcompany.com or mydomainregistrar.com.

Points to notice: those periods, dots, full stops '.' (whatever you wish to call them) at the end of the domain name are important. It means 'this is it in the domain name definition, do NOT append mydomain.com' i.e. without it, mydomain.com becomes mydomain.com.mydomain.com

MX records may NOT point to IP addresses or CNAME entries. This implies that whatever is listed in the target part of an MX record, must ALSO have a valid DNS entry, pointing at its IP address.

[ALERT] To repeat: any mail server pointed to by an MX record must in turn have a DNS entry for itself, pointing to its IP address. Exceptions to this are known as 'phantom' mail servers, and are frowned upon.

Another common pitfall occurs when you register your domain with a domain registrar, and then choose a hosting company and sign up for their services. They are separate companies and know nothing of each other. Many registrars obligingly set up the DNS to point to their mail servers, and wait for you to define forwarding rules and email addresses. Similarly your hosting service sets up your hosting with MX records pointing to their mail servers in their DNS. You end up with a number of mail servers, all with the same priority of 10 or so, all happily accepting mail for your domain, and it goes nowhere. Or sometimes you can get the mail, and sometimes not. It is ESSENTIAL that you ensure that any mail servers defined in MX records have a consistent hierarchy that ensures that mail will be delivered to a single destination server, from which you can access it. It is also a common tactic for spammers to submit mail to a low priority mail server, in the hope that this will bypass Spam checks on the high priority server. i.e. it may be possible to have too many mail servers for your own good. Delete any that serve no purpose.


SENDING-MAIL SERVERS

Note that all of the above refers mainly to RECEIVING mail servers. It defines in the first instance how mail for your domain gets to you. But consider SENDING mail servers. Any machine with mail server software anywhere can connect to another mail server and say 'I have some mail here for joe@hisdomain.com, for which you are a mail server, it's from fred@dodgymail.com, here it comes'. You do it every day when you hit the send button on your mail client. Or when you're sitting in (Insert any well known yuppie coffee shop chain of choice) with your portable phone or laptop, using their wireless access point. A spammer's dream.

To try and impose some semblance of order on this, mail servers will do a variety of things with incoming mail connections. They will check for MX records, for reverse DNS entries, for blacklisted domains and IP addresses, and for DNS verification records like SPF records.

Reverse DNS is simply a 'backwards DNS pointer entry', with 'in-addr.arpa.' appended. If your domain IP address is 1.2.3.4 the reverse DNS entry would be:

 4.3.2.1.in-addr.arpa. IN PTR  mydomain.com.

Again, it must be an A record not an IP or CNAME alias.

This allows receiving mail servers to query by IP address to ensure that the sending mail server is who or what it claims to be. MX records can also be checked to see if it belongs to that domain.


BLACKLISTING

Blacklisting is complex, insofar as there is no single definitive 'blacklist'. There are LOTS of them, some good, some bad, some insufferably pompous, righteous and infallible. Check. If you find your domain on a blacklist, all you can do is attempt to get it removed. As a general rule ALL dynamic IP addresses i.e. those from home broadband connections and similar are almost automatically blacklisted. If you want to run your site from a computer on your desk, you are going to have to get a static IP address.

SPF RECORDS

An SPF record is yet another DNS record, this time a TXT record that allows you to specify which machines are ALLOWED to send mail for your domain. The set-up has many possibilities, and it is easiest to visit http://www.openspf.org/ where they have a wizard to assist. It is easy to succumb to the temptation to make the SPF definition broader than it should be. Resist. Another point that is easy to overlook with SPF records is that the mail server that sends out the email for your site should be included in your SPF record. I'll repeat that. The mail server that sends out the email for your site MUST be included in your SPF record. If it is not, why are you doing this?


Any, all or none of the above (DNS, MX, SPF, etc) may be set up for you by your domain registrar or hosting service. What is much more important is that you check, and fix anything that is not correct.


All of the above can be checked by visiting http://www.dnsstuff.com and doing a dnsReport on your domain. Check the mail section of the report, and fix any reported warning, error or problem that is within your power.

Checking DNS entries is easy using nslookup in Windows, or dig on Unix/Linux machines. Understanding the command line options and the output is unfortunately not quite so simple. Here are some starting points:

 nslookup -d mydomain.com
   or
 nslookup -t MX mydomain.com
   or
 dig MX mydomain.com


2. Email Addresses

When you purchased your domain, you also, as a side effect obtained the ability to create an unlimited number of email addresses for that domain. If you feel that the world needs myfavouriteauntsally@myshinynewstore.com, go ahead. When Aunt Sally gets an internet connection, I'm sure she'll be very happy.

What it did not necessarily get you is a site where that mail is hosted. In most cases your hosting service or registrar will offer this, but there is no standard package. Some offer only forwarding or similar, or charge extra per email address.

There is, again, no standard list of what email addresses are necessary for a site. From a standards point of view, one is absolutely essential, and that is postmaster@mydomain.com. bounce@mydomain.com is also highly desirable. This is not in jest, the standards stipulate that all mail domains MUST have a postmaster mailbox, and some 'purist' mail server set-ups will check, and refuse mail from the domain if it is not there.

Typically, websites will have something like sales@ , info@ , support@ , admin@ , webmaster@ , and anything else you care to define. These can be aliases of a single mailbox, but it is better to explicitly define them than to rely on a 'catchall' define to accept any mail to the domain. i.e. if you use an email address on the website, then define and create it, even if it is just as an alias for admin@ or whatever, it is desirable for it to exist as an explicit definition, than as a vague anything@.

FORWARDING?

The subject of forwarding is contentious, as it is very tempting to define a single mailbox, with all other addresses as aliases of that mailbox, and set it to forward to your 'home' or usual email address. If that usual address is in the same domain or on the same mail server, I would give a cautious yes. Any other mail server or domain, I say no. it is all too easy for an intermediate or the destination mail server to decide it is Spam or junk or insignificant and bin it or delete it. Ideally, use a separate mailbox and set your mail client to explicitly fetch the mail from the server as an additional account in your email client. Do not forward.

Note that hosting companies and registrars may set up all or none of the above for you, or give you a control panel where you can do much of it yourself.

FREE EMAIL ADDRESSES

Similarly, much has been said about using 'free' email addresses on websites. I believe there is a simple test. Ask yourself whether you would buy from a website whose contact address was dodgyventures@freewebmail.com, hosted who knows where? (Apologies to freewebmail.com if they exist. I'm sure they are a highly reputable organisation. The comment is illustrative, not factual). Be wary.

There is also the issue of their Terms and Conditions in which you agree to allow content scanning of your mail to target ads at you. I'm not sure how well that squares with the various customer privacy enacted in various parts of the world, when it comes to your customers personal and payment details, order details, purchase history etc. even if it is just automated scanning.

You should, however, use such services for testing when you have everything else set up and working, as it is not wise to tell potential and actual customers which email addresses they may and may not use. Be aware however that these large entities make it as difficult as possible to actually send any email to their addresses. If any of the settings/setup mentioned above (part 1 of this article, ie: DNS, MX, SPF, valid accounts, etc) are not in place, you will find your site mail disappearing (dumped in a black hole and never delivered because it fails various rules/tests) or ending up in the Spam bin. They are all fairly ruthless about that, and curing it can be prolonged and more educational than you might choose.

Whenever I suggest this, I am told 'But XXXXX (insert as applicable) are an adorable, big fwuffy bunny huggy lovable company. They would never treat their customers like this!!! How can you be so mean?' I reply ' Yes they can, but in fact, they do treat their customers like this. You have made a basic error. You are not the customer, you are the product.'


3. The Website.

Most hosting services have a 'preferred' way for websites to send mail. Find out what this is for your site and plan to set up accordingly. If they can't tell you, you might be advised to carefully re-evaluate how knowledgeable your hosting company really is and whether you should continue relying on them for your business livelihood.


YOUR ZEN CART EMAIL SETTINGS

Working through the various Zen Cart mail transports:

PHP is the default, and is essentially the simplest. Some hosting companies block it or disable it, but if it works, there is no set-up: simply select and use.

Sendmail and sendmail -f are venerable Unix/Linux mail transports, these days they're mostly emulation by more modern mail software, since no server admin in their right mind wants to configure a sendmail mail system. This is pretty much an 'either it works or it doesn't' set-up. The phpMailer class used in Zen Cart v1.3.x uses its own command line flags to invoke sendmail, and many hosting companies do not allow this. They define a fixed implementation, and that's it. This is usually '/usr/sbin/sendmail -t -i' on Linux hosts. We will gloss over the fact that the behaviour of this 'standard' is totally non standard, and in fact undefined. phpMailer uses the '-t' flag, but wants to add a couple more flags to the mix. If this is not allowed, you can't use sendmail.

SMTP/SMTPAUTH is the venerable granddaddy of them all. The original mail transport, used by every internet mail server to interchange mail. As such, it is extremely standardised and cast in stone, and is also the most robust of the transports.

That said, the only auth functionality that the version of phpMailer in ZC v1.3 offers, is PLAIN, so the likes of the Gmail servers are not usable, as they use TLS or SSL encryption. (v1.3.8a has some improvements to aid in this, so YMMV.)

Qmail is a historical relic, sendmail by another name. It's there because it *was* common to use Qmail in place of sendmail at one time. See the sendmail comments above. It might save the day if your host uses Qmail on their server. You'll likely NOT use this one!

In operation all these transport methods except SMTP operate in effect by saying 'I have some mail here' and throwing a bucket full of bytes at the server. It is therefore client-initiated (i.e. your website being the 'client'), and relies on the mail being properly formed and everything being 'just so'.

SMTP is more ponderous: establish a connection, possibly authenticate, initiate a mail send, send a 'To' address, send a 'From' address, send some other header info, send the mail body, say goodbye. At any stage in this the server can object, and issue an appropriate cryptic error message. Thus it is a more 'traceable' and definable transaction than the others. So ideally you should use SMTP for robustness. In practice it is not going to make a huge deal of difference; if it works, stick with what you have, bearing in mind hosting service preferences above.

SMTP also offers the ability to connect to an external mail server, as it is what the internet's mail servers use to communicate with each other. e.g. You could potentially use your company mail server, or your own or ISP or domain registrar mail server to send the mail for your website. Connect to it with SMTP and bingo. In practice this is regarded by many hosting companies with about the same level of enthusiasm as a bubonic plague outbreak, and they actively block the standard SMTP ports to prevent your site doing this.

SENDING AS 'NOBODY'

To add to the woes of the would-be website mailer - you - there is the small matter of the 'Apache user'. On websites running under 'native' PHP, all of the software code is executed or 'run' as a system user. This user is usually imaginatively named 'nobody' or 'www-data'. (Software developers don't get out much). If that were all, this would merely be an exciting conversation piece/stopper you could drop at dinner parties, but unfortunately it has the side effect that all mail originating from the site via PHP or sendmail, or qmail, is sent as coming from this shadowy figure 'nobody'. Or the exciting 'www-data'. It's probably not necessary to point out that Spam filters leap into action as soon as they see any mail from this prolific and widespread pair, with predictable results. What would you do if you opened your letterbox and found a letter from 'nobody'? It is sometimes possible to mitigate this with 'sendmail -f' since the -f parameter forces a 'from' address, or by using SMTP, but not always.

Additionally, when websites send email as 'nobody', it's very common for the hosting company to have the server configured to show the .php script filename in the (normally hidden) raw email headers. This allows them to very quickly determine whether an email that someone claims was spam was sent from a legitimate script or not ... usually they'll simply shut down that script quickly, and tell the account holder that there's a problem, giving them a few days to come up with a fix.

Other websites use Suexec or SuPHP, which run the website code as the website owner rather than 'nobody' and the effect is mitigated. Particularly if the website owner is something like web_admin and has an email mailbox. They come with their own brand of drawbacks, so there is no free lunch, but things are improving over time. Setting an 'Email From' address and 'must send from known domain' may help, and are recommended for any site.


Another small point is that the SMTP standard mandates the use of CR/LF end of line characters in SMTP mail. Sendmail with a -t flag prefers LF. The others, who knows? There's no standard. Set according to your host operating system..

ENCODING AND CHARACTER SETS

You should also be aware that default encoding and character sets are defined in /includes/extra_configures/email_use_8bit.php - EMAIL_ENCODING_METHOD and your main /includes/languages file e.g. english.php - CHARSET. These may need to be changed for your particular hosting set-up, or if you are not using iso-8859-1 as your character set. The question of an encoding setting is complex. The mail servers of the internet nominally send all mail as 7bit data. This means that all 'normal' messages need to be encoded, and this allows you to choose how you would like your message mangled. In practice almost all modern mail servers advertise (via the SMTP login procedure) that they are able to accept 8bit data. Even if this is not the case, the encoding and decoding is handled automatically and as necessary by the mail servers. Best to leave be, if you're experiencing specific problems in this area, feel free to experiment, but it is unlikely to make much difference. What IS important is that it actually BE defined.

The character set used is similar, but if you are not in North America, it could be relevant. If you are using a European or Asian character set it is very important that you set the definition correctly. All sections of the emails are labelled with the character set, and there is no faster way of getting mail labelled as Spam than wrongly defining the character set. Quite apart from that, the result will probably be gibberish. i.e. No wishful thinking. If you set the Charset to 'utf-8', you must be using utf-8 data.

It is inevitable and unfortunate that certain short emails, such as 'password forgotten', especially when sent as HTML emails, appear to mail-server Spam filters as just an image (the logo) and a small amount of text, which they regard as a flag for potential Spam mail.


4. What to do when it doesn't work?

  • Write down any error messages or error numbers, and if you have access to your website logs, check them for any errors relevant to the problem.
  • Switch on email archiving and send some mail. The archiving only takes place after an allegedly successful 'send' so if the email is archived, the website system thinks that it was successfully sent, and the problem lies elsewhere.
  • Give your hosting company details of the time the mail was supposedly sent and ask them to check their mailserver logs.
  • Check whether your domain or your hosts have been blacklisted by visiting http://www.dnsstuff.com and doing a DnsReport on your domain, and check http://www.robtex.com/rbls.html to see if your domain is flagged. Check the mail section of the report, and fix any reported warnings, errors or problems that are in your power to resolve. Engage your hosting company if needed. Refer to Section 1 of this article for more specific details/explanation on Blacklisting and other issues your dnsreport shows you.
  • Ask the intended recipient to check Spam and similar folders. It is also worth checking whether the recipient's email system has a 'good guys' list/setting to 'Whitelist' your site. This tells the mail filters that your domain is expected to send mail to their address, and that it's not Spam.
  • If you're getting 'Spam flagged' emails (you're getting the emails, but they're arriving in your spam folder), check them for the entries in their mail headers, particularly the values for Return-Path, Reply To, and From for strange settings. Note also if they are NOT defined in the headers. Also note any MIME header information and content boundary information for strange settings - a multipart/alternative header with only 1 part in the mail body, or a "text/plain" content header with HTML body.
  • Verify the encoding and character sets used.
  • Most mail clients allow you to view the mail headers and message source, which are very valuable diagnostic tools.
  • Accept that the free web based services change the rules regularly and arbitrarily, and that customer email that went through fine last week can disappear or be flagged as Spam this week.
  • Establish whether your hosts have any emails-per-hour or other limits, and keep them in mind when you send out that 10,000 email Newsletter and mailshot.
  • More help and tips, and testing tools can be found here: http://blog.servergrove.com/2012/04/02/best-practices-to-send-out-emails-with-your-server/

When all else fails, post on the support forum, and we'll see if we can help. Or at least be sympathetic.