MTA-STS: Enforcing TLS for Email Transport

The Problem: Opportunistic TLS Is Not Enough

When two mail servers communicate, they use SMTP. Modern SMTP implementations support STARTTLS — an extension that upgrades a plain-text connection to an encrypted TLS connection. The word "opportunistic" describes the key weakness: STARTTLS is attempted if both servers support it, but if the handshake fails or is blocked, most servers fall back to plain-text delivery rather than refusing to send the message.

This fallback behaviour creates an attack surface. An attacker positioned between two mail servers — on the network or by compromising DNS — can strip the STARTTLS announcement from the server's capabilities list. The sending server sees no TLS offer, falls back to plain text, and the attacker reads the message in transit. This is called a STARTTLS stripping attack or STARTTLS downgrade attack.

MTA-STS (Mail Transfer Agent Strict Transport Security), defined in RFC 8461, solves this by publishing an authenticated policy that says: "You must use TLS to deliver to this domain, and here are the MX hostnames you should be connecting to." Sending servers that support MTA-STS will enforce TLS and refuse to deliver if the TLS connection cannot be established.

How MTA-STS Works

MTA-STS combines two components: a DNS TXT record that signals the policy exists, and an HTTPS policy file that contains the actual policy. Both are required.

Step-by-Step Flow

1. Sending MTA resolves recipient domain's MX records
2. Sending MTA checks for _mta-sts.recipient-domain.com TXT record
3. If found: fetch https://mta-sts.recipient-domain.com/.well-known/mta-sts.txt
4. Parse policy: get MX hostnames and mode (enforce/testing/none)
5. Connect to MX server and perform TLS handshake
6. Verify: server certificate matches an MX hostname in the policy
7. If verification passes: deliver normally
8. If verification fails: DO NOT fall back to plain text; queue/reject per policy mode

The HTTPS fetch is critical. An attacker who poisons DNS cannot redirect a validly served HTTPS policy file without a valid TLS certificate for mta-sts.recipient-domain.com. This is what makes MTA-STS resistant to downgrade attacks.

The MTA-STS DNS TXT Record

Publish a TXT record at _mta-sts.yourdomain.com:

_mta-sts.yourdomain.com  TXT  "v=STSv1; id=20240115T120000;"

Record Fields

The id= value is how sending servers know the policy has changed. When they cache a policy, they also cache the id=. On the next delivery attempt, they fetch the DNS record, compare IDs, and re-fetch the policy file only if the ID has changed.

Important: Whenever you update the policy file, you must also update the id= in the DNS record. A common convention is to use a timestamp: id=20240115T120000.

The MTA-STS Policy File

The policy file is a plain text file served over HTTPS at a fixed URL:

https://mta-sts.yourdomain.com/.well-known/mta-sts.txt

Hosting Requirements

• The subdomain mta-sts.yourdomain.com must have a valid TLS certificate
• The Content-Type header must be text/plain
• The file must be served from the .well-known path exactly
• No redirects (the file must be served directly)

Policy File Format

version: STSv1
mode: enforce
mx: mail.yourdomain.com
mx: *.yourdomain.com
max_age: 604800

Each field is on its own line. CRLF or LF line endings are both accepted.

Policy Fields

Policy Modes

mode: enforce — Sending MTAs that support MTA-STS must establish a valid TLS connection. If TLS fails or the certificate does not match, the message must not be delivered in plain text. The sending server should return a temporary failure (4xx) and retry later.

mode: testing — TLS is attempted and verified, but failures are reported via TLS-RPT rather than causing delivery failure. Use this mode while you are deploying and validating your configuration.

mode: none — The policy is disabled. Use this to signal to senders that a previously published policy is being withdrawn.

MX Hostname Patterns in the Policy

The mx: fields specify which MX server hostnames are expected. A sending MTA will connect to an MX server, verify the TLS certificate's Subject Alternative Names (SANs), and confirm that at least one SAN matches an mx: pattern in the policy.

Wildcard patterns are supported:

mx: *.mail.protection.outlook.com   # Microsoft 365
mx: *.google.com                     # Google Workspace
mx: mail.yourdomain.com             # Specific hostname

Critical: Every MX record for your domain must match at least one mx: pattern. If you add a new MX server without updating the policy, deliveries to that server may fail.

TLS-RPT: Reporting for MTA-STS

TLS-RPT (TLS Reporting, RFC 8460) is a companion specification to MTA-STS that lets you receive reports about TLS connection failures from sending servers. It works similarly to DMARC aggregate reports.

Publishing the TLS-RPT Record

_smtp._tls.yourdomain.com  TXT  "v=TLSRPTv1; rua=mailto:[email protected]"

Alternatively, reports can be sent to an HTTPS endpoint:

v=TLSRPTv1; rua=https://tlsrpt.yourdomain.com/report

What TLS-RPT Reports Contain

• Total successful TLS connections
• Connection failures broken down by failure type
• Certificate validation failures
• Negotiation failures (cipher mismatch, protocol version issues)
• Policy fetch failures (MTA-STS policy file unreachable)

Reports arrive in JSON format, similar to DMARC RUA reports. Tools like Postmark's TLS-RPT Viewer or Valimail can parse and display them.

Deployment Strategy: Start with Testing Mode

Never deploy MTA-STS directly in enforce mode. The correct sequence is:

Phase 1: Set Up Infrastructure

1. Create the mta-sts.yourdomain.com subdomain
2. Obtain a TLS certificate for it (Let's Encrypt works fine)
3. Create the policy file at /.well-known/mta-sts.txt in testing mode
4. Publish the DNS TXT record at _mta-sts.yourdomain.com
5. Publish the TLS-RPT record at _smtp._tls.yourdomain.com

Phase 2: Monitor in Testing Mode (1–4 weeks)

version: STSv1
mode: testing
mx: mail.yourdomain.com
max_age: 86400

During this phase, sending servers report failures but do not block delivery. Collect TLS-RPT reports and look for:

• MX hostname mismatches (policy lists wrong hostnames)
• Certificate errors on your MX servers
• Policy file fetch failures
• Missing MX hostnames from your DNS

Phase 3: Switch to Enforce Mode

Once testing shows no unexpected failures:

1. Update the policy file to mode: enforce
2. Optionally increase max_age to 604800 (1 week) or higher
3. Update the id= in the DNS TXT record

version: STSv1
mode: enforce
mx: mail.yourdomain.com
max_age: 604800

Update the DNS TXT record simultaneously:

_mta-sts.yourdomain.com  TXT  "v=STSv1; id=20240215T080000;"

Common Pitfalls

MX Hostname Not in Policy

# DNS MX records:
10 mail.yourdomain.com
20 backup.yourdomain.com

# Policy file:
mx: mail.yourdomain.com     ← backup.yourdomain.com is MISSING

# Result: Deliveries to backup.yourdomain.com fail TLS verification

Fix: List every MX hostname (or use a wildcard pattern like mx: *.yourdomain.com).

Certificate Not Covering MX Hostname

Your MX servers must have TLS certificates with SANs that match the mx: patterns in your policy. If your MX hostname is mail.yourdomain.com, the certificate must include that as a SAN (not just a Common Name).

Policy File Not Reachable

# Verify policy file is accessible
curl -v https://mta-sts.yourdomain.com/.well-known/mta-sts.txt

# Check Content-Type header
curl -I https://mta-sts.yourdomain.com/.well-known/mta-sts.txt
# Should show: content-type: text/plain

Forgetting to Update id= After Policy Changes

If you update the policy file but not the DNS id=, sending servers that cached your policy will continue using the old version for up to max_age seconds.

MTA-STS vs DANE

DANE (DNS-Based Authentication of Named Entities) also enforces TLS for SMTP but uses TLSA records in DNSSEC-signed zones. The two approaches are complementary:

If your DNS is DNSSEC-signed, DANE provides stronger guarantees. For domains without DNSSEC, MTA-STS is the practical choice.

Summary

MTA-STS closes the opportunistic TLS gap by publishing an authenticated policy that forces sending MTAs to use TLS when delivering to your domain. The combination of a DNS pointer record and an HTTPS policy file makes it resistant to DNS spoofing attacks that could otherwise downgrade connections to plain text.

Deploy in testing mode first, collect TLS-RPT reports to validate your configuration, then move to enforce mode once you are confident all MX hostnames are correctly represented. Combined with SPF, DKIM, DMARC, and BIMI, MTA-STS completes a defence-in-depth approach to email security.