DMARC, DKIM and SPF — identity and authentication

A history lesson

A great deal has happened since David H. Crocker published RFC822 in 1982, the foundation of the email we know today. Back then email was used mainly to exchange research material between American universities. The World Wide Web had not been thought of yet.

Everyone trusted everyone else, because messages were only exchanged inside a small, closed group of universities. There was no need to authenticate the validity of the sender or the content.

Over the next 30 years it became clear that the original design — taking everything in a message at face value — does not work on a modern internet. Here the place swarms with bad actors trying to exploit ordinary users’ trust, abusing the names of known brands to get people to click, or to make spam filters misclassify their mail.

The sender of an email

Working with email requires care, not least because of the jargon. That is especially true when talking about the sender of a message, because there is more than one.

Think of an email as an old-fashioned letter. A letter is sent in an envelope and typically has:

As far as the postal service is concerned, only the two addresses on the envelope matter. If the recipient no longer lives there, the letter is returned to the sender address on the envelope.

Once you have received the letter, the envelope is removed and the letter put in your in-tray. From then on you only see the addresses in the letter itself. The envelope no longer matters.

Email works the same way:

To be unambiguous, they are named after the RFC that defines them:

The two RFC5321 addresses are the ones used in the SMTP session, in the MAIL FROM and RCPT TO steps. The RFC5322 addresses are the ones you see in your mail client.

So when mail bounces, it bounces to the RFC5321.From address, not RFC5322.From. That is what lets an email service provider such as Ubivox offer you the ability to send from your own address while still collecting the bounces: RFC5321.From points at an address the provider controls, and RFC5322.From is your own.

The two recipient addresses (RFC5321.To and RFC5322.To) are nearly always the same.

For an email from christian@ubivox.dk to peter@example.com passing through Ubivox, the addresses look roughly like this:

That slightly cryptic RFC5321.From is what lets Ubivox trace a bounce to exactly which recipient and which message it concerns, and handle it.

SPF — Sender Policy Framework

SPF began to be standardised around 2004. It lets the holder of a domain publish a policy for which IP networks may appear in RFC5321.From addresses for that domain. The policy is published as a specially formatted TXT record on the domain.

That made it possible to stop anyone claiming to send from your domain in the RFC5321.From address — spoofing, in everyday language. The effect was limited, though, because in practice no user ever inspects the RFC5321.From address. The only address they see in their mail client is RFC5322.From.

Most often RFC5321.From sits on a domain controlled by your provider, so you do not have to set anything up for SPF to validate. That is the case in Ubivox too.

SPF is specified in RFC7208 — Sender Policy Framework.

Sender ID

To make up for that gap, Sender ID was introduced some years later. Alongside the policy for RFC5321.From in SPF, Sender ID also allowed a policy for RFC5322.From.

Unfortunately the standard allows an SPF policy to be read as a substitute when a Sender ID policy is missing. The unhappy side effect was that your SPF policy could suddenly be interpreted in an unexpected context — a policy intended for RFC5321.From ending up applied to RFC5322.From.

That has produced a great many false positives over the years.

Sender ID never really caught on, though Microsoft chose to implement it across their products, and even today you can run into Sender ID rejections when all you meant to publish was an SPF policy.

Sender ID is all but extinct, and was formally declared obsolete with the publication of RFC7208. Some installations linger, so it is still worth being able to pass a Sender ID check cleanly. There are two ways:

  1. If you publish an SPF policy on your RFC5322.From domain, make sure it also includes your provider. For Ubivox, that means including include:spf.ubivox.com.
  2. Alternatively, publish a separate Sender ID policy. The syntax is the same as SPF, except that where SPF opens with v=spf, Sender ID opens with v=spf2.0/pra. It also requires mapping every system that sends mail where the domain appears in the RFC5322.From address.

If you use Ubivox and something is wrong, you get a warning when you try to send.

Sender ID is specified in RFC4406 — Sender ID: Authenticating E-Mail.

DKIM — DomainKeys Identified Mail

At roughly the same time it became clear that a method was needed to ensure a message’s content had not been altered in transit — and, more importantly, to establish a responsible party or organisation for a given message.

DKIM solved that. It is a cryptographic signature that both confirms the content of the message and establishes an identity. In DKIM’s case the identity is a domain name.

As a sender you generate an RSA key pair, a private and a public key. The public key is published in DNS on the domain belonging to the identity you want to attach to the mail, and is used by third parties to verify the signature made with the private key.

If the content has been altered in transit, the signature is no longer valid and the receiving system can quarantine the message. It is no longer possible for a spammer or a virus to change the content or the links in your email without breaking the signature.

DKIM is specified in RFC6376 — DomainKeys Identified Mail (DKIM) Signatures.

DMARC — Domain-based Message Authentication, Reporting and Conformance

Of course, you could simply strip the DKIM signature out of the message and be back where you started. What was missing was a way to publish a policy on your domain declaring which authentication mechanisms are in use.

Some ten years later the answer arrived, and it comes close to a silver bullet against spoofing. The idea was to tie SPF and DKIM validation together in a new policy where the basis for each has to line up.

What does lining up mean? It means the organisational domain name in the SPF validation (RFC5321.From) has to be the same as the organisational domain name in the DKIM validation, which has to be the same as the organisational domain name in RFC5322.From.

The organisational domain name here is the domain you bought from the registrar. So for email.ubivox.dk, the organisational domain is ubivox.dk.

If everything lines up and either SPF or DKIM validates — or both, depending on your policy — the message passes. Otherwise it is rejected or quarantined, again depending on your policy.

That is the Conformance part of DMARC. The other important part is Reporting.

Your DMARC policy can ask for reports, in various formats, about failed DMARC validations. That gives you an effective way of finding out whether somebody is spoofing you.

DMARC is specified in RFC7489 — Domain-based Message Authentication, Reporting, and Conformance.

The effect on spam filters

All of these mechanisms exist to establish an identity for a message. They are not a shortcut past the spam filters. On the contrary, they make it easier for providers to categorise your mail and maintain an overall reputation for you — which, depending on your sending practice and behaviour, can work for you or against you.

Our experience is that several of the larger providers have tightened their requirements for mail not covered by a DMARC policy, which makes reaching the inbox harder without one.

Before you publish a policy

Before putting any policy into production for a domain, make sure it is complete. A great many systems inside an organisation may send email, and it is the policy author’s job to map all of them.

We have seen many cases where a customer had a policy more or less unknowingly forced on them by another supplier, putting all mail through Ubivox into quarantine overnight, with extensive delivery problems following.

Checklist for full SPF, DKIM and DMARC compliance in Ubivox

Notes on DMARC policies:

A worked example

To make it less abstract, here is a full example. We want to send DMARC-compliant mail from sender@example.com through Ubivox:

  1. Set up a subdomain for Ubivox, for instance email.example.com
  2. Make sure the SPF record on example.com contains include:spf.ubivox.com
  3. Configure a public DKIM key on the domain, for instance ubivox._domainkey.example.com
  4. Configure the private DKIM key in Ubivox
  5. Run with this setup for two or three weeks, or until you are convinced there are no problems
  6. Check whether other systems in the organisation send mail from @example.com. If so, they have to meet the same aligned SPF and DKIM
  7. When you are ready, publish a DMARC policy in a TXT record on _dmarc.example.com, for instance v=DMARC1; p=none;
  8. Afterwards, p=none can be tightened to p=quarantine or p=reject

To switch on the reporting part of the DMARC specification, use the parameters below, replacing the addresses with ones that suit your organisation:

Making sense of the reports can be awkward, as they arrive in specialised, machine-readable formats. It is worth investing in software that can read and process them. Among others:

If you would like help mapping your setup before you publish a policy, write to support — this is the part where a second pair of eyes earns its keep.