How to Make Emails and Not Mess Up: Practical Tips

A developer, who first encountered generating emails, has almost no chance to write an application, that will do it correctly. Around 40% of emails, generated by corporate applications, are violating some form of standard, and due to this, there are problems with delivery and display. There are reasons for this: emails are technically more difficult than the web, and operating emails is regulated by a few hundred standards, as well as an uncountable number of generally accepted (and not as much) practices, whereas the email clients are more varied and unpredictable than browsers. Testing may significantly improve the situation, but materials that are dedicated to testing the email system, are practically non-existent.

Mail.ru regularly interacts with its users by email. In our projects, all the components responsible for generating emails and even individual mailings, are subject to mandatory testing. In this article, we will share our experience (learning from our mistakes).

• What kind of emails are there?
  
• Who is involved in the testing and control process?
  
• Mailing and mail transport
  
• The interface of the email infrastructure and the boundaries of the tested application
  
• Definition of test parameters
  
• Typical structure of a generating application
  
• What and when to test
  
  • Delivery infrastructure
    
  • Generating application
    
  • Structural and typesetting email templates
    
• Basic requirements for checking infrastructure
  
• Authentication requirements
  
• Verification of the generating application
  
  • Requirements for email addresses
    
    • Envelope addresses
      
    • Email header addresses
      
  • Requirements for headers
    
  • Requirements for the structure of the email
    
  • URI requirements
    
  • Email layout requirements
    
• Conducting split tests
  

What kind of emails are there?

The application can generate various types of emails. They can be classified into several categories. By the method of selecting recipients – personal/triggered – selective- group. By appointment: transactions- marketing- service. You can set different requirements for different types of email and apply various testing scenarios.

Triggered personal emails are generated in response to any events, for example, user actions or status changes of system objects. They are generated by the application, and therefore are the most interesting in regards to testing. Triggered emails can be transactional, marketing, and for service use. Selective emails are sent to a dynamic selection of users, that meet some form of a criteria. Group emails are sent to a permanent group of recipients, for example, all users or partners. Selective and group emails are often for marketing use, and sending such emails is started manually or on a schedule.

Transactional emails are generated in the process of a user completing some form of action. Such emails include, for example, invoices, tickets, or delivery status notifications. Transactional emails are always triggered and are meant to carry important information. They should be as simple and compatible as possible, and testing them should be done on a large number of mail clients.

Marketing emails encourage the user to take an action, for example, this can be an offer for a personal discount based on previous purchases. Transactional data can be utilized in these emails, and they can be triggered emails or mass-, periodic or one time. For these emails, efficiency is more important, and the results of the split-test usually determine it. Some aspects of compatibility can be sacrificed for efficiency.

Group marketing emails, for example, messages about seasonal offers, promotions, and sales, are often sent ‘manually’, and are not part of your application, but you can (and should) also apply general testing principles to them.

Also, there may be service emails: notifications generated for the staff, for automated CRM systems, journaling, auditing, or DWH. Such emails are usually triggered emails, meaning that they are also part of the application, and must be tested.

Who is involved in the testing and control process?

1. QA engineer – participates at all stages of the process.
   
2. Network engineer – responsible for configuring network infrastructure and message delivery infrastructure. Network engineer should be involved in planning and infrastructure testing.
   
3. Delivery specialist – person who monitors the deliverability of emails, who also participates in monitoring the technical and administrative parameters of all emails sent, and monitors the progress of the mailing process. He is responsible for ensuring that the sent emails reach the highest percentage of users, and do not end up in spam. For this purpose, the specialist must have specific knowledge and contacts. If there are any problems with the delivery of emails, he is the one who must understand the probable cause and eliminate it; either by eliminating technical obstacles; or changing something in the content of the emails; or try and solve the problem with the support service of the mail provider, to which the emails don’t reach. Such a specialist (if any) should also be involved in coming up with the checklist, and testing infrastructure generating applications and emails. However, the testing process itself should be under the control of the QA service.
   
4. Email-marketer – determines the effectiveness of marketing newsletters. Under his control, the split-testing for the distribution of marketing to the audience occurs. Email-marketer also controls the segmentation of the user base, the composition, and frequency of the sent emails, the visual ‘presentation’ of the email to the user.
   

All of these roles are not necessarily performed by a dedicated employee; the role of the marketer can be performed by one of the product managers, and the role of the delivery-specialist, for example, can be performed by a support employee or a network engineer. In start-up companies, it is highly likely that all of this will have to be dealt with by one person, and they may turn out to be a quality specialist.

Mailing and mail transport

The email structure is like a massive iceberg, and there are two levels in it. There are more than a hundred different standards governing emails, but almost all of them belong to one of these two levels:

The underwater part of an iceberg – network service, the basic protocol of which is the SMTP application protocol defined by RFC 5321. It is responsible for the delivery of emails. A so-called SMTP envelope is formed for the delivery of the email, which includes the addresses of the sender and recipient of the SMTP level. Other network services, such as DNS, are also responsible for delivering the email. The main component of the network infrastructure is the Mail Transfer Agent (MTA). The MTA is responsible for handing the message delivery queue and the delivery process itself to the recipient servers. MTA examples include Postfix, Sendmail, Exim, Microsoft SMTP service.

This underwater part of the iceberg, which includes the MTA, DNS parameters, and other network parameters, we will call the email infrastructure or the message delivery infrastructure.

The tip of the iceberg — the email itself. The basic structure of the email is defined by the standard RFC 5322. The email consists of service headers and one or more data parts. The data may be in a plain text format and/or HTML or even AMP, with inline images or attachments of almost any type.

The interface of the email infrastructure and the boundaries of the tested application

The email infrastructure, as a rule, has one or several interfaces through which an email is sent (when it enters the MTA delivery queue). For example, the SMTP Submission service, the function mail() in PHP, data transfer to an external mail or sendmail application, API for internal or external service (such as GetResponse, SendPulse, or Amazon SES). We will consider these interfaces as part of the infrastructure. It often happens that Application A prepares data for an email and a list of recipients, and then sends it to Application B through its API (for marketing mass mailings, this can be done manually via the user interface- UI), and application B generates a mail message in RFC 5322, and delivers it to the MTA. For application A (and when testing it), application B will be part of the email infrastructure. The API or UI of application B will be for the application A interface of the mail infrastructure. Although for Application B, the situation will be different, and the mail infrastructure for it will be lower-level network applications or protocols.

Definition of test parameters

When testing each application, it is important to select all the mail infrastructures used by it (there may be several of them), and for each infrastructure to single out the interfaces used (there may also be several of them for each infrastructure). For each interface, the composition and format of the data transmitted to it is determined as accurately as possible, e.g. email text in TEXT/HTML, email text in TEXT/PLAIN, email subject, recipient name, recipient address, sender name, sender address (RFC5321.From), the address of the sender of the SMTP convector (RFC5322.mailfrom). Next, a set of requirements is developed for each parameter (representations, encodings, boundary values, etc.), methods for monitoring each of the parameters are determined (how to compare the actual result with the expected one).

Typical structure of a generating application

As a rule, the same product that we are testing is responsible for the generation of emails and data in it. This is usually a server (but sometimes client) application. It defines the structure of the email, part of the service headers, data encapsulation formats, string representations, and text encoding. A simple example of such an application is the script that forms the email and calls the mail() function. The main elements of the application that must be controlled are:

• the code responsible for generating headers and / or email structure, if the email structure is dynamically generated, and / or static email templates describing its structure;
  
• HTML layout of the email (ideally, it is a separate entity or part of the email template / layout, but can be embedded in the application code);
  
• substitution of application data into an email (or into an email template);
  
• integration of the application with the email delivery infrastructure, the correctness of the parameters passed to the infrastructure interface.
  

What and when to test

Whether we like it or not, the entire iceberg should be tested. There are several main components that require testing:

Delivery infrastructure

Emphasis in testing should be done on: email deliverability; correct DNS records, including PTR / FCrDNS, MX and A records; SMTP protocol parameters (HELO, use TLS); email authorization (SPF / DKIM / DMARC); SMTP envelope addresses (if the application does not manage them); correctness of processing the input parameters of the infrastructure interface; tracking, recording and processing of undelivered emails.

It is necessary to test the infrastructure during the initial implementation and every time changes are made to the infrastructure itself (the configuration of the MTA, DNS or network changes) or the interface for sending an email; using a new domain, network, or API; additional testing is required if the characteristics of the emails sent, such as their language, size or numbers are significantly changed. According to experience, the infrastructure tends to change «by itself», so basic tests should be conducted periodically, even if there is no information about any changes.

It is possible and necessary to involve the network engineer and the deliverability-specialist in drawing up the plan and checklists for testing the infrastructure.

Generating Application

The addresses of the SMTP envelope should be monitored (if the application controls them, that is, they are transferred to the interface — envelope-from, envelope-to), the values ​​of the service headers of the email (Date, Message-ID, List-Unsubscribe, Auto-Submitted, etc.). Clause), email authorization (DKIM / DMARC), MIME-encoding (base64, quoted-printable), general correctness of the email format, for example, the absence of non-ASCII characters in the headers, the composition of the data being injected, the correct triggering of triggers, unsubscribing mechanisms, tracking mechanisms writing and collecting statistics (postmaster headers, for example, Feedback-ID or X-Mailru-Msgtype, as well as tracking pixels).

It is necessary to test an application during its development, when all its related components that are responsible for generating and storing data change, with significant changes in email templates, when changing the used infrastructure or interface to it, as well as within the framework of general regressions.

Structural and typesetting email templates (can be part of a generating application or are developed separately)

The structure of the email is checked (Content-Type, Content-Disposition, nesting of Multipart-parts of the email, text encoding, string parameters), the value of the target and displayed headers (From, To, Reply-to, Subject), the way email is displayed in the list of emails and when reading in various interfaces, microformats (for example, that a calendar event is recognized as a calendar event or an air ticket as an air ticket), branding.

Email templates should be tested every time you make at least the slightest changes, as well as separately, for example, in a situation where emails get into the application before the server part is ready.

It is recommended to involve an email marketing specialist and a deliverability specialist to compile a checklist for testing an email template.

Basic requirements for checking infrastructure

The IP address selected for the mail server should be as close as possible to the IP address of the mail server. It is recommended to check it using the whois utility. In particular, the sender's address should not belong to the network, which can be perceived as dynamic; the selected network must have active contacts to which complaints can be sent; the network must be used (have the status ASSIGNED in RIPE) The IP address must have a properly configured PTR record. It can be configured independently through the hosting control panel, or with the help of the service provider. A PTR record must point to the real hostname and still be meaningful, resolved back to the same IP address (so-called FCrDNS check), not remind the name of the dynamic host, and not include a large group of numbers or characters. A good example is mailserver.example.com.

Each domain used in envelope addresses or email headers must have a valid MX record pointing to a host A record, which, at a minimum, can handle undeliverable messages. MX is not allowed to directly refer to an IP address.

Control the passage of SPF, DKIM, DMARC. SPF allows the domain owner to specify in the TXT records a list of servers that are authorized to send emails with return addresses in this domain. It is configured for the address used in the envelope-from (SMTP-envelope) in the section of managing DNS zones of the domain. DKIM provides verification of the authorship of a message or of its originator to a specific domain using digital signature technologies, which is added to the email itself (in its DKIM-Signature header). Typically, a DKIM signature is configured at the MTA (infrastructure) level. DMARC sets the policy of checking incoming mail in a specific domain and actions on emails that do not pass SPF or DKIM authentication. When attempting to violate this policy, a structured report comes along with information about such an attempt. DMARC, as well as SPF, is published as a TXT record in the domain zone.

Check the deliverability of emails to the main postal services — for Russia Mail.Ru, Yandex, Gmail, Microsoft (Hotmail / Outlook.com / Office365), Rambler, nic.ru. In the emails arrived, you need to check the correctness of HELO; the presence and passage of checks PTR / FCrDNS, SPF, DKIM, and DMARC; the validity of the headers and data in them, in particular, the synchronism of the clock in the dates and the correctness of the time zones.


(Registration mail has broken authentication due to freemail address used)

The formation of some parameters, for example, authorization, deliverability, and spam are integrally influenced by all components, but for their control, there are usually separate operational tools — DMARC and FBL reports, postmaster services API, email tracking tools, delivery statistics. Testing should take into account the level of implementation of operational monitoring tools in the company — for example, in the absence of operational control of DMARC reports, the authorization of emails should be regularly tested, whereas in the absence of operational control of deliverability, where and how the emails go, even if there is no development related to sending emails.

To test the infrastructure, you can use specialized services, for example, mail-tester.com, mxtoolbox.com. Detailed infrastructure requirements can be found in this article.


(an example of the broken SPF record)

Authentication requirements

Checking the passage of SPF, DKIM, DMARC is usually possible using the Authentication-Results header on the recipient's server.

Check the validity of the SPF record for syntax, the limit for DNS queries (for example, using mxtoolbox.com). When forming an SPF, all sources of mailings should be taken into account (do not forget CRM systems, all currently utilized delivery infrastructures, including those through which one-time marketing campaigns are conducted). It is recommended to set allowed servers for the domain through the list of networks (‘ip4’ / ‘ip6’). SPF is checked for the sender address from the SMTP envelope. Verify that the SMTP envelope domain (envelope-from) matches the domain from the From header. Common SPF errors are listed in this article (https://medium.com/hackernoon/myths-and-legends-of-spf-d17919a9e817).

Check the DKIM DNS record, the validity and composition of the DKIM Signature. Verify that you are using a DKIM key of at least 1024 bits. Recommended hash mode of DKIM signature: relaxed / relaxed. Make sure that all important headers are signed (From, To, Subject, Date, Message-ID, MIME-Version, Content-Type), that tracking headers (Received, Delivered-To, Return-Path) are not signed, and DKIM is validated for basic mail services. Set up forwarding to one of the mail services to another; DKIM should not «beat» on forwarded emails. Verify that the DKIM signature domain matches the sender domain from the ‘From’ header.

Check DMARC for basic email services. Check for DMARC reports, identify and troubleshoot SPF and DKIM for all IP addresses of your infrastructure.

Verify that messages are delivered to external servers using encryption (TLS). You can sometimes check for TLS by the Received header on the recipient’s server: for example, specifying the ESMTPS protocol or having parameters of the form (version = TLS1_2 cipher = ECDHE-RSA-AES128-GCM-SHA256 bits = 128/128); indicates the presence of TLS.

Verification of the generating application

Email address requirements

Envelope addresses

We begin the verification of the generating application with the addresses in the SMTP envelope.
Envelope addresses are addresses of the email infrastructure level. They are not visible to the user, but they are important for delivery because the address of the envelope determines which mailbox the email goes into.

The recipient address in the envelope (envelope-to, aka RCPT TO:) is the address to which the email will actually be delivered.

• for all emails except for registration, the address must be legally signed and validated for mailing in accordance with administrative requirements.
  
• for newsletters, the address must be «live», addresses to which email cannot be delivered regularly should be marked as inactive, mailings to them should not be made. But some categories of emails (for example, access recovery) may also need to be sent to addresses previously marked as inactive.
  

Sender address in an SMTP envelope (usually called envelope-from, smtp.mailfrom, MAIL FROM: or Return-Path) — messages will be delivered to this address about the inability to deliver an email and automatic replies. This address is not visible to the user. This address is also used for SPF authorization. We verify that:

• The address is available;
  
• It is not the address of the employee and is not redirected to him in order to exclude auto answers, messages about inaccessibility, etc.;
  
• It is processed by a script that will mark addresses of inaccessible users as inactive;
  
• The address can be automatically generated, i.e. unique to each email;
  
• An email to this address should not lead to the generation of any response email, for example, a message about a mailbox overflow.
  

Email header addresses

These addresses are either directly visible to the user or are used when replying to an email.
Sender address (the header From:) is the address and sender name displayed in the list of emails and when reading an email. We verify that:

• This is a human-readable friendly address that identifies the company.
  
• It contains not only an email address but also the name of the sender.
  
• Noreply@ is possible, but only if we want to emphasize that we do not expect to receive a response to the email and it will not be read. It is better to duplicate this idea in the text of the email.
  
• In the presence of non-ASCII characters (for example, Cyrillic), the sender name must be encoded in accordance with MIME, the domain in the presence of non-ASCII characters must be encoded in Punycode
  
• Emails of the same category should have the same address; the use of auto-generated addresses should be avoided. This is due to the fact that ‘From:’ is most often used by people to sort emails by folders using filters.
  
• The address should be different (preferably in different subdomains) for transactional, marketing and urgent emails (such as emails from the support service). This is also due to the fact that the user can mark marketing emails as spam or filter them in an unreadable folder.
  
• The address From: and SMTP envelope address must be in the same domain or in subdomains of the same organizational domain in order to pass the SPF within the DMARC.
  
• The address must be from the organization’s domain. It is unacceptable to use free mail services and other public mail domains in From:, since such mailings will not pass SPF and DKIM authentication within the framework of DMARC.
  

The address Reply-to – ‘manual’ replies will be sent to this address when a user answers an email. It is optional. If it is absent, the address from ‘From:’ is used for the response. Check that ‘Reply-to’:

• It can be auto-generated, i.e. unique to the email (allows you to find out which email the answer came to).
  
• It should not fall into the employee’s mailbox to avoid auto-answer, but ideally should be «wrapped» in CRM.
  
• It can generate a standard CRM auto-answer, but it should not generate anything superfluous, for example, mailbox overflow or «on vocation» messages. When generating auto-responses, measures must be taken to avoid looping, they are listed in RFC 3834.
  
• It can be from any domain that does not necessarily coincide with ‘From:’, but sometimes this scares users when answering.
  
• May be absent, then the From: address performs its functions.
  
• In addition to the address, the name of the sender is indicated.
  

The address To:

• Must contain the recipient's email (otherwise it scares the recipient of the message and bothers the anti-spam).
  
• Ideally, it should contain the name of the recipient. But if the name is unknown or doubtful (for example, the address has not yet been confirmed), it is better not to indicate it (someone may enter someone else's address with a bad name, and the recipient may be offended).
  

Email header requirements

The actual encoding of the text should match the one indicated in the header. It is advisable to use one encoding in all headers and parts of the email. It is recommended that you use UTF-8 as a widely supported one. The encoding is indicated in the Content-Type headers and in the tag of the HTML part.

The From:, Message-ID: and Date: headers should be formed directly in the script for sending the email (and by standards — along with the text of the email) and always in the correct format. If they are absent or incorrectly formed, one of the transit servers can add these headers, which leads to a violation of the integrity of the DKIM signature.

8-bit characters in the headers, including the subject line (Subject) and the names of the attached files (Content-Type and Content-Disposition), should be absent; All non-ASCII characters, including Cyrillic, must be encoded in quoted-printable or base64.


(registration confirmation in weird encoding)

Requirements to the structure of the email

For the HTML part of the email, it is desirable to form an alternative — text (plain) part. It is also necessary to check the conformity and readability of the plain text part of the email (if any) and the general structure of the email.

According to RFC 5322, the length of a line in an email should not exceed 998 8-bit characters. Please note that in UTF-8, a character can occupy several octets. The terminator of lines in an email is a pair of CRLF ( ascii 13, ascii 10), which takes 2 octets. You need to check the correctness of the string terminator, since emails are often sent using a Unix script, and in Unix, the string terminator is a single character — this is an error for email. You should also check if the string terminators break UTF-8 encoded characters: you cannot allow the presence of a string terminator between two octets of the same character, for example, Cyrillic symbol. To avoid such situations, it is necessary to break the text before forming the email or encode the text in base64, base64 usually has fixed line length.

It is necessary to check the correct marking of attachments and inlines — that is, the correctness of the formation of the headers «Content-Disposition: inline», if it is a picture displayed inside the email, or «Content-Disposition: attachment» if file attached is intended for download.

The structure of the email should be as simple as possible: in particular, there should not be more than one multipart part of each type (mixed, alternative, related), and multipart/mixed is used only if the email contains file attachments, multipart/related — if HTML comes with inline images, multipart/alternative — in the presence of plain text and HTML parts. In general, the structure of the email, in the absence of attachments and inline pictures, should look like this:

multipart/alternative
— text/plain
— text/html

The order of the parts is important, text/plain must go BEFORE text/html or multipart/related. This is necessary so that the HTML part is displayed by default, and only if its display is unavailable for some reason, the plain part is displayed.

If there are inline pictures in the email, its structure should look like this:

multipart/alternative
— text/plain
— multipart/related
—— text/html
—— image/… (inline-picture)
—— image/… (inline-picture)

Inline images must have Content-Disposition: inline and be strictly inside the multipart/related part.
In the most difficult case, when there are both inline images and attached files, the email has the following structure:

multipart/mixed
— multipart/alternative
—— text/plain
—— multipart/related
——— text/html
——— image/png
——— image/png
…
— application/octet-string (content-disposition: attachment)
— application/octet-string (content-disposition: attachment)
…
multipart related- and multipart/alternative-parts must be closed before attachments, attachments belong to the external multipart/mixed part)


(registration message with incorrect parts structure)

URI requirements

Any URIs (in src, href attributes, styles, etc.) must contain the protocol and hostname (https://example.com/somepath). Typical errors are the use of relative links (/somepath) and the lack of a protocol (//example.com/somepath), which is unacceptable for emails, because in them, the default protocol may be file://.

• Any service and non-ASCII characters (in particular, Cyrillic) in the URI must be encoded using percent encoding.
  
• A link inserted as text (that is, visible to the user as a URL, and not as a piece of text) should still be marked up with the <а> tag, otherwise, the user will not be able to click on it. Some webmails mark up such links on their own, but this is not standard behavior. In this case, the href address inside A must match the link text, otherwise, the content filter may react to such a link as an attempt to deceive the user. This is especially worth paying attention to when there are «clickers» that track the user's transitions from the email.
  
• It is better to limit oneself to the use of the protocols http://, https:// and mailto:.
  
• With high-security requirements, you should completely abandon the use of http:// in favor of https://.
  
• Non-standard ports should not be used (for example, example.com:8080/somepath), as they may not be accessible to the user.
  
• Clicking on the link inside the HTML part should not lead to any changes in the state of the application (subscription, unsubscribe, cancellation of the order, etc.) without additional confirmation by the user on the page, because some content filtering systems can independently verify the security of such a transition by requesting a page by reference; the mail application can show the preview of the page by the link on hover, and modern browsers can load the page before the user clicks the link to reduce load time (in the web application it is generally not recommended to do any modifying actions on the GET request, all modifying requests must go through POST or PUT).
  
• Clicking on the link in the List-Unsubscribe header, on the contrary, should not require any additional actions from the user, because the unsubscribe by this link is usually done by email program or webmail on behalf of the user. Also, there is a new header List-Unsubscribe-Post introduced by RFC 8058.
  
• You should not expect the user to read the email and follow the link in the same browser in which it initiates the action leading to the sending of the email (for example, registers an account). The link should work in any other browser or mobile device. In particular, the user can open the link without being authorized, or authorized in an account other than the one to which the email was sent.
  
• Because the length of the URI can be limited; you should not use URIs of the ‘data:’ type for large objects. For the same reason, don't use URIs that are too long in hyperlinks.
  
• You must not use external link shorteners, this negatively affects the delivery of emails. It is better if all links point to your domain, this will reduce the potential negative impact of someone else's reputation on the delivery of emails.
  
• Do not place external images on some public services or free hosting, use a reliable hosting service or CDN with good performance and reputation.
  

(invalid image and anchor URI due to missing protocol specification)

Email layout requirements

Why is it so difficult to make up emails?

Email clients in one way or another display user content within their interface. Potentially, this can lead to various security problems — cross-site scripting (XSS, Crossite scripting), interface spoofing, DOM clobbering, user deanonymization / information leakage (for example, the user's IP address or cookies through external requests), etc. ., therefore, any mail service and mail application has some form of protection against each class of attacks. Unfortunately, there is no single approach to organizing this protection. It can be organized through:

• isolated limited frames,
  
• filtering tags and / or attributes,
  
• restrictions on absolute positioning,
  
• prohibition or restriction on the use of block styles (which is critical for adaptive layout),
  
• banning external elements by default (i.e. downloading external images requires user permission) or using a proxy to access them,
  
• converting HTML emails to another intermediate format (for example, Microsoft Exchange / Outlook uses RTF, which can make it extremely difficult to display elements properly in Outlook using conventional methods),
  
• prohibition or restriction on the use of forms or their individual elements.
  

The emails also use specific elements, such as inline images and URI cid://, whose support may be limited. For example, Mozilla Thunderbird does not support cid:// for background images.

Even a correctly formed email can be displayed differently in different interfaces due to the peculiarities of their implementation and filtering of the contents of the email.

If there are errors in the email format, the behavior becomes completely unpredictable. For example, email clients may have different behavior with incorrect URIs, or that incorrect header formatting is handled differently. Also, text encoding auto-detection works differently if it is not specified or is specified incorrectly. Therefore, the email must be viewed in different interfaces: the correct display of the email in one interface does not mean that it is composed correctly (in fact, even the correct display of the email in all interfaces does not guarantee that there will be no problems with the display in the future).



It is necessary to pay attention to the following points:

• Check the text of the email for semantic content, display, absence of typos, syntactic, spelling and lexical errors.
  
• Check the correctness of the substitution of application data in the template or layout of the email.
  
• Check the correctness of the amounts, dates, numbers, items of goods and other information, taking into account the permissible boundary conditions. The dates should have a year (some users enter the box very rarely). A time zone must be present in time. The address must contain a city, and in some cases, it is necessary to indicate the country.
  
• Check the operational status of all links in the email, if any.
  
• In emails sent before confirmation of the address, incl. emails with a confirmation link, there should not be any text controlled from the outside, even the username, otherwise, they can be used for spam (in the field displayed in the email, for example, in the name, spam text is inserted and the address is the indicated address of the victim). For example, if you can send an obscene text on the developer's address on behalf of your service, then there is a problem.
  
• Check for the absence of external images on third-party services. It can affect delivery and leak information about your customers.
  
• Check the availability of counters for sending, delivery, reading email, transitions. Some of them are located in the email itself (for example, the counter-pixel of reading the email), some are tracked by the mailer, but, as a rule, all are available in the mailer’s admin panel.
  
• Check the correctness of the subscription category and the user's unsubscription for this category through the link in the email.
  
• Check how the email looks to:
  
  • Popular web versions of mail for a targeted country: the «big three» for Russia are Mail.Ru, Yandex, Gmail. You can also add Rambler and Outlook.com;
    
  • Mobile applications of the above email providers;
    
  • Standard mobile applications using IMAP, taking into account popular mobile platforms, for at least the iPhone, Pixel (Android reference platform), Samsung (the most common for Android), MIUI (takes the second place in Russia for Android platforms);
    
  • Various desktop browsers: Chrome, Firefox, Edge, Internet Explorer, Opera, etc .;
    
  • Desktop applications (email programs), especially Thunderbird, Outlook and Apple Mail, optionally The Bat! and Opera Mail;
    
  • Popular corporate solutions with a web interface (Exchange, maybe Roundcube, Communigate, Zimbra, SquirrelMail) — for B2B solutions;
    
  • Do not forget to check the layout on both Retina monitors and monitors with lower resolution.
    
• During the check in each case, you need to pay attention to:
  
  • Passing the authorization headers, SPF / DKIM / DMARC.
    
  • Email download speed: it should load quickly, and not take its time.
    
  • Displaying an email in the list of emails: avatar, sender’s name and the subject that falls into the message’s snippet, whether its category was correctly defined (for example, if the order did not fall into the «social networks» category).
    
  • The layout of the email as a whole: if the layout remained consistent, were there any incorrect word breaks, etc., including when scaling and resizing the window.
    
  • Fonts should not be small or hard to read.
    
  • Background images and background colors.
    
  • Matching the brand book.
    
  • Ease of performing the actions implied by the email. For example, if the email contains a confirmation code or other information that may need to be stored somewhere, then it should not only be well-read, it should also be convenient to select and copy it even in the mobile interface.
    
  
  
• Keep track of the overall size of the email (including external images) and that it does not exceed reasonable values. The heavier the load time, the more likely that a person will have a negative reaction to it.
  
• Even emails that have not been amended should be checked periodically, as changes can occur on the side of the postal service, and, for example, «reveal» a previously invisible problem.
  
• Some parameters must be controlled in all tests. For example, problems with passing DKIM authentication can be due to problems in the infrastructure (problems with DNS or the formation of a DKIM signature, time synchronization errors), due to errors in the generating program (sender address is incorrectly formed, incorrect characters in the headers are missing or mandatory From, Date, or Message-ID headers have been duplicated) and due to content errors (incorrect line terminators, lines are too long, incorrect addresses). At the same time, the email may not «beat», and the problem may not appear on any service.
  

Conducting Split-Tests

Marketing research is beyond the scope of this article, but a few key points that significantly affect the quality of emails should be mentioned.

The newsletter has a purpose, so it should entail through its quality, not quantity (which is the opposite for spammers). The newsletter must be segmented. When conducting an advertising campaign, you need to know exactly who gets into the segment sample, why they need the offered product and what they want to convey.

For each mailing list, it is necessary to calculate the CTR of the list of emails — this is the ratio of the number of emails read to the total number that has been sent out. In postmaster.mail.ru, you can see indicators for unique users. If the measurement goes through the counter in the email (pixel), then the absolute number of openings is estimated. CTR <10% is a very low indicator, it is undesirable to carry out such mailing. It should strive for a CTR> 30%. For marketing emails, the clickthrough rate of clickthroughs and the percentage of completed actions («sales») on these links are also of great importance. Be sure to monitor complaints (marking the email as spam). Typically, for one-time mailings, a tenth of a percent is a good indicator, for regular ones, a hundredth of a percent. The critical values, after which the mailing is always interpreted as spam, are indicated here: https://help.mail.ru/developers/mailing_rules/technical.

It is necessary to conduct split testing of various distribution options to obtain optimal performance. Just changing the name of the sender and the subject of the email can increase the CTR by several times and significantly reduce the number of complaints. The number of emails should be statistically significant for evaluating the results (for large projects, usually a few thousand). The final version of the email is sent in several stages for additional measurement of indicators and «warming up» — starting with about 10,000 recipients, with an increase of about an order of the day.

The main idea: emails are part of your application, perhaps one of the most complex and problematic. At the same time, this is often a «blind spot» in terms of testing. I hope that I was able to draw your attention to this issue.

I would like to thank Vladimir Dubrovin (z3apa3a) and Alena Likhacheva (s4ever) for helping me with this article. As well as that, the article utilised sources of Eduard Tyantov (edT) and Alexander Purtov (4Alexander).