MTA-STS: Enforcing Encrypted Email Transport

The problem: STARTTLS downgrade attacks

Most SMTP traffic today is encrypted using STARTTLS — an opportunistic upgrade mechanism where the sending MTA announces TLS support, and both parties negotiate a TLS session before transferring the message. "Opportunistic" is the key word: if TLS negotiation fails for any reason, SMTP falls back to plaintext delivery. This fallback exists for compatibility but creates a critical vulnerability.

An attacker with a network position between two mail servers (on-path, e.g., a compromised router or a BGP hijack) can intercept the EHLO response from the receiving server and strip the STARTTLS capability announcement. The sending server, seeing no TLS support advertised, delivers the message in plaintext — and the attacker reads or modifies it in transit. This is a STARTTLS downgrade attack, and it has been observed in the wild.

The alternative, SMTP over TLS on port 465 (SMTPS), establishes TLS before any SMTP dialog and cannot be stripped in the same way. However, port 465 is used for mail submission (client to server), not for server-to-server relay (MX delivery). Server-to-server relay uses port 25, where STARTTLS is the standard and downgrade attacks apply.

What MTA-STS does

MTA-STS (SMTP MTA Strict Transport Security, RFC 8461) allows a domain to publish a policy stating that inbound mail delivery must use TLS with a valid certificate matching the receiving MX hostnames. Sending MTAs that support MTA-STS fetch and cache this policy; when delivering mail to that domain, they refuse to fall back to plaintext or accept an invalid certificate.

The mechanism is Trust-on-First-Use (TOFU): the first time a sending MTA fetches the policy, it must do so over HTTPS (authenticated by the web PKI). After that, it caches the policy for the duration specified and re-fetches it when the cache expires. During the cached period, policy enforcement does not require a live HTTPS fetch — the cached copy is used.

MTA-STS is typically deployed alongside TLSRPT (TLS Reporting), which provides a mechanism for sending MTAs to report delivery failures and TLS policy violations back to the domain owner. Without TLSRPT you have no visibility into whether senders are enforcing MTA-STS and whether your MX server's TLS configuration is causing delivery failures.

How it works

MTA-STS requires two published components:

1. DNS TXT record at _mta-sts.example.com:

v=STSv1; id=20240101120000Z

The id field is an arbitrary string (commonly a timestamp) that identifies the current policy version. When a sending MTA checks whether its cached policy is still current, it compares the id in DNS against the id in its cached policy file. If they differ, it re-fetches the policy. This is the mechanism for signaling a policy update — increment the id whenever you update the policy file.

2. HTTPS policy file at https://mta-sts.example.com/.well-known/mta-sts.txt:

version: STSv1
mode: enforce
mx: mail.example.com
mx: mail2.example.com
max_age: 604800

The mx lines must match your actual MX hostnames exactly (wildcards like *.example.com are allowed). The max_age is in seconds. The policy file must be served over HTTPS with a valid certificate — not self-signed, not expired.

Policy modes

• enforce: Sending MTAs must establish a TLS connection with a valid certificate matching the listed MX hostnames. If TLS cannot be established or the certificate does not validate, the sending MTA must not deliver the message and should queue it for retry. This is the target state — it provides real security.
• testing: The policy is evaluated and violations are reported via TLSRPT, but delivery is not blocked. Use this mode to verify your MX server's TLS configuration before moving to enforce. Failures show up in TLSRPT reports without impacting mail flow.
• none: The policy effectively disables MTA-STS. Used to gracefully disable a previously published policy without abrupt removal. Senders that have cached your old policy will see the none mode and stop enforcing.

Setup procedure

1. Verify your MX servers have valid TLS certificates that match the MX hostnames (not just the A record — the certificate must cover the actual MX hostname, e.g., mail.example.com). Check that certificates are auto-renewing.
2. Create the policy file at https://mta-sts.example.com/.well-known/mta-sts.txt. The mta-sts subdomain must resolve and serve HTTPS. Set mode: testing initially.
3. Publish the DNS TXT record at _mta-sts.example.com with a new id value.
4. Set up TLSRPT (see below) so you receive reports.
5. Monitor TLSRPT reports for one to two weeks in testing mode. Any TLS failures from sending MTAs will appear here.
6. Switch to enforce mode once no failures are reported. Update the policy file and increment the id in the DNS record simultaneously.

TLSRPT

TLSRPT (TLS Reporting, RFC 8460) is MTA-STS's companion reporting protocol. It works similarly to DMARC aggregate reports: sending MTAs that support TLSRPT send daily JSON reports about TLS delivery successes and failures to the address you specify.

Publish a TXT record at _smtp._tls.example.com:

v=TLSRPTv1; rua=mailto:tlsrpt@example.com

Reports are sent as gzip-compressed JSON files via email or HTTPS POST. They include: the policy type (MTA-STS or DANE-SMTP), the number of successful TLS sessions, and details on any failures — including the failure reason (expired certificate, hostname mismatch, TLS handshake failure). In testing mode, these reports are your only visibility into whether the configuration is working correctly before you enforce.

The rua address must be a monitored inbox. Reports sent to an unmonitored address give a false sense of security — you will not know when things break.

Common mistakes

• Policy file not served over HTTPS: The entire point of the HTTPS fetch is to authenticate the policy using the web PKI. If https://mta-sts.example.com/.well-known/mta-sts.txt returns an error, cannot be fetched, or redirects to HTTP, sending MTAs will be unable to fetch the policy and may treat this as a policy fetch failure — which in enforce mode means delivery delay, not bypass.
• id field not updated after policy change: If you update the policy file but forget to change the id in the DNS TXT record, sending MTAs with cached copies will not know there is a new policy to fetch. The cache expires eventually, but a change to mode: enforce or a new MX hostname will take up to max_age seconds to propagate to all senders.
• MX hostname in policy does not match certificate: The certificate on your MX server must cover the hostname in the MX record (e.g., mail.example.com), not just the IP's rDNS or the apex domain. A certificate issued for example.com does not cover mail.example.com unless mail.example.com is in the SAN.
• Skipping testing mode: Going straight to enforce without a testing period risks deferring legitimate inbound mail if any MX server has a TLS issue. Always start in testing, always monitor TLSRPT.
• No TLSRPT alongside MTA-STS: Without TLSRPT, you have no signal when mail is deferred due to MTA-STS enforcement failures. Set up TLSRPT before or at the same time as MTA-STS.