SMTP transport security is separate from message authentication and inbox placement. MTA-STS lets a receiving domain publish TLS delivery requirements, TLS-RPT reports observed transport results, and DANE/TLSA authenticates SMTP server credentials through DNSSEC. Each mechanism depends on the sender supporting it and the relevant infrastructure being configured correctly.
This guide concerns a sending mail transfer agent delivering to a recipient domain’s MX servers, normally over TCP 25. The central question is which authority binds that destination to the server the sender reaches: HTTPS and public certification authorities, or DNSSEC and TLSA. We compare the trust models, failure precedence and the responsibilities shared with an external mail provider. The short record examples explain those relationships.
Encryption and Authentication Are Different Decisions
A sender using opportunistic STARTTLS without an enforcing policy can fall back to plaintext if STARTTLS is unavailable or disrupted. A successful TLS negotiation also does not necessarily mean the server’s identity was authenticated. MTA-STS and SMTP DANE add defined authentication and downgrade-resistance conditions; they do not retroactively secure every relay or the stored message.
The recipient publishes its desired inbound policy. The sending MTA discovers and applies that policy when it supports the mechanism. Publishing records for your domain does not automatically make all mail leaving your server follow those rules; outbound delivery depends on the destination’s policy and your sending software’s capabilities and configuration.
MTA-STS uses an HTTPS policy and PKIX certificate validation; TLS-RPT supplies aggregate observations from participating senders; DANE uses DNSSEC-authenticated TLSA associations. Plan who controls each DNS zone, which server presents each certificate, and which system makes the delivery decision before changing records.
MTA-STS: HTTPS Policy and PKIX Trust
A supporting sender with a valid enforce policy must not deliver to a candidate MX that fails the allowed-host match, certificate validation or STARTTLS requirement. It can try another appropriate candidate and treat failure as temporary for later retry. The policy does not guarantee inbox placement, and an incorrect enforce policy can delay legitimate incoming mail.
The TXT record at _mta-sts.example.com announces an instance identifier; it is not the authenticated policy body. That body is fetched from https://mta-sts.example.com/.well-known/mta-sts.txt. In this single-MX testing reference, id marks a revision, mx authorizes a destination name and max_age defines the fetched policy’s cache lifetime. The values are illustrative, not a required deployment schedule.
_mta-sts.example.com. IN TXT "v=STSv1; id=policy01"version: STSv1
mode: testing
mx: mx1.example.com
max_age: 86400
The HTTPS endpoint needs a valid certificate for the policy hostname and must return HTTP 200 directly. Senders must not follow HTTP 3xx redirects. Serve the policy body without login or bot challenges; the MTA-STS max_age cache is separate from ordinary HTTP caching.
MTA-STS checks the selected MX hostname against an allowed policy pattern and against the certificate’s SAN DNS-ID. The certificate must be unexpired and chain to an issuer trusted by the sender. The SMTP banner or an unrelated name announced by the server is not the reference identity, and a CN alone is insufficient. A pattern *.example.com matches mx1.example.com, but not example.com or a.b.example.com.
The HTTPS policy host and the SMTP MX are separate authenticated endpoints. A valid certificate at one does not validate the other. For example, a hosting customer may control the policy website while its provider controls every MX certificate. That division determines who can correct a policy identity error and who can correct the credential actually served to the sender.
Testing, enforce and none express different policy decisions. Testing allows delivery despite an MTA-STS validation failure; enforce prohibits that path; none communicates a controlled opt-out. A mode change is not an instant change of every sender’s view. Publish and verify a changed body before its new TXT id, and retain service for previously cached policies. Representative connection checks and reports, rather than a fixed waiting period, support an enforcement decision.
RFC 8461, the MTA-STS specification defines policy discovery, validation and cache behavior. A valid cached policy continues to apply if fresh discovery fails. With no valid cache, an attacker who blocks initial discovery can prevent this protection from being established; MTA-STS does not eliminate that first-contact risk.
TLS-RPT: Evidence About Transport, Not an Authority
TLS-RPT is published separately at _smtp._tls.example.com, for example v=TLSRPTv1; rua=mailto:tlsrpt@example.com. The standard supports mailto and HTTPS POST destinations and comma-separated URI lists. Prepare the reporting mailbox or service before publishing it. Participating senders can submit aggregate I-JSON reports, including compressed attachments; not every sender reports.
Reports describe policy results, successful and failed session counts, and failure details for sending systems and receiving MX hosts. They do not authorize delivery after failed authentication and are neither per-message traces nor a complete TLS-version inventory. A certificate-host-mismatch can indicate an identity problem but does not uniquely diagnose SNI. Separate what a sender observed from a confirmed cause.
The recommended report window is a full UTC day, and delivery may be delayed. Compare failures with total session volume and affected senders. Failure categories can overlap for one connection attempt, so summing every detail count does not yield unique lost messages. Quiet reports may reflect missing traffic or broken report collection. Validate SMTP-delivered reports using the reporting domain’s DKIM signature; their transport is not guaranteed to remain encrypted when reporting TLS failures.
RFC 8460, SMTP TLS reporting defines the schema and failure categories. Store report identifiers and time ranges to avoid counting duplicate arrivals as new incidents. Handle report attachments as untrusted data, restrict retention and access, and correlate the aggregates with your SMTP delivery logs. A successful TLS session does not prove the message reached an inbox.
DANE/TLSA and DNSSEC Trust Boundaries
DANE authenticates the SMTP server’s certificate or key against usable DNSSEC-validated TLSA data. Secure delivery to the intended recipient domain also requires authenticated MX discovery. These involve two DNS locations: the recipient domain’s MX records and the DNS zone containing each target MX hostname and its TLSA records. Signing the recipient zone alone does not create or authenticate TLSA at an external mail provider’s hostname.
For a direct MX target mx1.example.com on TCP 25, the usual TLSA owner is _25._tcp.mx1.example.com. If the target is mx.provider.example, TLSA belongs under that provider hostname, not under a similarly named record in the customer’s zone. Coordinate with the provider or whoever controls the relevant zone and served certificate. A TLSA record specifies certificate usage, selector, matching type and association data; 3 1 1 means DANE-EE leaf association, SubjectPublicKeyInfo and SHA-256.
Under DANE-EE(3), the DNSSEC-validated association authenticates the leaf key or certificate; PKIX name, expiration and public-CA-chain checks are not applied in the same way. DANE-TA(2) has trust-anchor, chain and name requirements. A usable secure TLSA mismatch prevents delivery through that target; bogus or unresolved DNSSEC is not an excuse to silently disable validation. Authenticated absence of usable TLSA is a different state. Support and enforcement depend on the sending MTA.
RFC 7672, SMTP security with DANE also describes a nuance for hosted domains: an opportunistic sender may use a provider’s valid TLSA even when the recipient MX lookup is insecure. That can protect the chosen provider connection, but MX redirection remains possible and must not be described as authenticated delivery to the intended recipient domain. Validate both discovery and TLSA to protect the complete destination binding.
Apply MTA-STS and DANE with the Correct Precedence
When usable DNSSEC-validated TLSA records apply, DANE authentication takes precedence. A sender implementing MTA-STS must not let successful MTA-STS validation override a DANE failure. MTA-STS is useful where DNSSEC deployment is impractical, but it is not a fallback that authorizes delivery after failed applicable DANE validation. TLS-RPT can report observations for either policy.
These mechanisms protect transport and aid diagnosis; they do not replace SPF, DKIM or DMARC and do not prove a sender’s reputation or inbox placement. Maintain each policy according to its own validation rules. For example, if you publish MTA-STS as well as a DANE-EE association, keep the required PKIX/SAN conditions for senders using MTA-STS rather than assuming DANE changes that separate policy.
The DMARC, RUA/RUF and BIMI guide discusses related domain authentication and reporting. DMARC uses aligned SPF or DKIM results with the visible From domain; it does not establish the trustworthiness of every person or message. Treat those identity controls and SMTP transport validation as different operational responsibilities.
Who Controls the Recipient Domain and Each MX?
The recipient-domain administrator selects MX destinations and controls policy publication. The mail provider may instead control the SMTP listeners, certificates, keys and DNS zone containing each MX hostname. These roles need an explicit agreement: permission to edit the customer’s TXT records does not imply permission to publish TLSA for a provider-owned name or to change its server key.
MTA-STS delegates two checks to the sender: the selected MX must match the policy, and its certificate must authenticate the MX hostname. The policy file does not create DNS MX records. Include primary and backup hosts, their IPv4/IPv6 endpoints and external provider names in the ownership record. A low-traffic backup remains part of the domain’s delivery surface.
DANE adds a second DNS authority boundary: recipient-domain MX discovery and the TLSA zone of the selected server. Record the actual DNSSEC validation result for both, not merely the presence of signatures. An external provider’s usable TLSA after insecure MX discovery may protect the selected connection, but it does not authenticate the original recipient-to-MX choice.
Record which credential each TLSA association authorizes and who can change it. A 3 1 1 association selects the leaf certificate’s SPKI, so renewal with the same key can preserve the digest; replacing the key cannot. This differs from publishing an entire-certificate association. The choice determines which changes require coordinated DNS and server deployment.
When rotation changes the matched material, publish the new TLSA associations alongside the current ones before deploying the new key or certificate. Wait at least the previous TLSA TTL for old-only records to expire, accounting for actual DNSSEC and deployment behavior. Retain old associations while legitimate servers or an appropriate rollback still use them. Do not keep a compromised key authorized merely for routine overlap.
How Certificate and Cache Changes Affect Trust
A sender with a valid cached enforce policy continues to apply it if fresh policy discovery fails. A sender without a valid cache may be unable to establish MTA-STS protection. This distinction is why direct HTTP 200, the policy hostname’s certificate and the actual policy body matter even when the TXT announcement still exists. A redirect or bot challenge can break that acquisition step.
For policy retrieval, the MTA-STS client sends the policy hostname in SNI; for SMTP delivery it sends the selected MX hostname. SMTP servers must not universally require SNI from opportunistic clients and must present a fallback chain when SNI is absent or unsupported. Each listener’s chain must fit its actual credential. A PKIX trust failure and a DANE association failure need different explanations; the SMTP banner is not a substitute for the required identity.
Keep the DS, DNSKEY and signing lifecycles visible for each relevant zone, including provider-operated zones. A successful initial signature is not continuous DNSSEC validation. Practice changes and monitor validation errors before altering production trust. Record which deployed server key each TLSA association represents and who can update it.
For an MX migration, prepare the new endpoint, then publish an HTTPS policy accepting both old and new MX names and update the TXT id. Preserve a working path accepted by cached old policies during the transition. Plan TXT/MX TTLs and previously served max_age values before moving traffic. Keep old MX service available while cached DNS answers remain, and remove obsolete policy entries only after the old path is no longer required.
Separate Transport Trust from Message Authentication
Transport controls can make authentication requirements explicit and expose failures that local logs alone miss. Their benefit is not an assumed reputation boost. Measure queue delays, connection results, policy validation and actual SMTP acceptance separately from inbox placement or spam classification.
Publishing MTA-STS or DANE for your receiving domain protects inbound transport when senders support the mechanism. It does not configure every outbound route on your own MTA. SPF, DKIM and DMARC still need independent setup and monitoring; transport success does not replace domain-authentication or content policy.
A report collector observes the transport decision; it does not make that decision for the sending MTA. A support team still needs SMTP logs to distinguish TLS failures from greylisting, rate limiting, recipient rejection or spam handling. Authenticate report sources and handle attachments as untrusted data. Aggregate session counts cannot identify every individual missing message.
Three Cases That Clarify the Choice
A domain without manageable DNSSEC: MTA-STS can provide an HTTPS/PKIX policy when the MX service and policy host meet its requirements. DNSSEC is not an MTA-STS prerequisite, but sender support and first-discovery risk still limit the protection. A signed customer domain using an external MX: signing the customer zone alone does not authenticate the provider’s TLSA data or control its SMTP key.
A domain publishing both mechanisms: a successful MTA-STS check cannot override failed applicable DANE authentication. A publicly trusted certificate can still fail to match usable secure TLSA records. Resolve that association failure rather than treating PKIX success as permission to deliver. Authenticated absence of TLSA, a DNSSEC lookup error and a real association mismatch are different states, not interchangeable descriptions of “DANE unavailable”.
These cases assume the sender implements the relevant mechanism. They are protocol decision examples, not measurements from a customer deployment. When evaluating a provider, establish who controls recipient MX discovery, policy hosting, the TLSA zone and the server key. Provider-hostname TLSA cannot be replaced by an unrelated record under the customer’s domain.
Document the policy and credential a sender could have cached during a change. DNS TTLs and MTA-STS max_age control different caches; lowering either value now does not invalidate older cached data. To withdraw MTA-STS, publish none with a new id and retain the endpoints until all outstanding policies, including older longer-lived ones, expire. Monitor report collection separately from failure trends; a UTC-day aggregate can arrive later and is not a real-time channel for every sender.
Choose a Model You Can Operate and Verify
For MTA-STS, assign ownership for the HTTPS policy, allowed MX identities, PKIX/SAN requirements and cached-policy changes. For DANE, establish authenticated MX discovery, authority over each relevant TLSA zone, continuous DNSSEC validation and matching actual server credentials. Where both are deployed, maintain their distinct conditions and preserve the rule that MTA-STS cannot clear a DANE failure.
Before implementation, record the owners of MX selection, policy publication, DNSSEC/TLSA, certificate and key deployment, and reporting. Verify every MX path and evaluate testing with representative observations rather than elapsed days alone. This decision establishes transport responsibilities; it does not provide end-to-end message encryption or guaranteed inbox placement.
For the implementation sequence, two-MX policy and worked report example, continue with the MTA-STS and TLS-RPT setup and troubleshooting guide.
For related message-domain authentication and reporting, see the DMARC, RUA/RUF and BIMI guide. Keep the transport-policy responsibilities documented alongside those controls so that certificate and DNS changes do not bypass the normal review and deployment process.
