{"id":1659,"date":"2025-11-10T22:20:41","date_gmt":"2025-11-10T19:20:41","guid":{"rendered":"https:\/\/www.dchost.com\/blog\/so-why-mtls-a-story-about-trust-between-machines\/"},"modified":"2026-11-04T03:32:38","modified_gmt":"2026-11-04T00:32:38","slug":"nginx-caddy-microservices-mtls","status":"publish","type":"post","link":"https:\/\/www.dchost.com\/blog\/en\/nginx-caddy-microservices-mtls\/","title":{"rendered":"mTLS for Microservices with Nginx and Caddy"},"content":{"rendered":"<div class=\"dchost-blog-content-wrapper\"><p>Mutual TLS lets an API gateway verify a calling service\u2019s certificate and proof of possession of its private key. A stolen API token alone need not pass that certificate boundary. The gateway still needs a defined trusted CA set, while the application needs authorization rules for resources and actions.<\/p>\n<p>The client verifies the server\u2019s certificate and hostname; Nginx or Caddy verifies the client credential. These two directions can use different issuers. mTLS does not automatically identify a human operator, prevent misuse of a compromised client key or authorize every request made by a valid certificate.<\/p>\n<p>This guide uses the same CA workflow and Node.js authorization application in both languages. It covers incoming mTLS, a precise identity-header contract, a separate upstream mTLS connection, rotation and revocation. Examples assume maintained OpenSSL 3.x, TLS-enabled Nginx, Caddy 2.11.7 syntax and a supported Node.js release. Replace lab names, paths and network addresses for your environment.<\/p>\n<div id=\"toc_container\" role=\"navigation\" aria-label=\"Table of Contents\" data-nosnippet class=\"toc_transparent no_bullets toc_numbered toc_title_center\"><p class=\"toc_title\">\u0130\u00e7indekiler<\/p><ul class=\"toc_list\"><li><a href=\"#What_mTLS_Authenticates_and_Authorizes\"><span class=\"toc_number toc_depth_1\">1.<\/span> What mTLS Authenticates and Authorizes<\/a><\/li><li><a href=\"#Create_the_Lab_CA_and_Certificates\"><span class=\"toc_number toc_depth_1\">2.<\/span> Create the Lab CA and Certificates<\/a><ul><li><a href=\"#Explicit_OpenSSL_CA_Client_and_Server_Profiles\"><span class=\"toc_number toc_depth_2\">2.1.<\/span> Explicit OpenSSL CA, Client and Server Profiles<\/a><\/li><li><a href=\"#Connect_with_the_Client_Certificate\"><span class=\"toc_number toc_depth_2\">2.2.<\/span> Connect with the Client Certificate<\/a><\/li><\/ul><\/li><li><a href=\"#Require_Client_Verification_on_Nginx\"><span class=\"toc_number toc_depth_1\">3.<\/span> Require Client Verification on Nginx<\/a><ul><li><a href=\"#Complete_Incoming_Nginx_Configuration\"><span class=\"toc_number toc_depth_2\">3.1.<\/span> Complete Incoming Nginx Configuration<\/a><\/li><li><a href=\"#An_Additional_Exact-Serial_Gate\"><span class=\"toc_number toc_depth_2\">3.2.<\/span> An Additional Exact-Serial Gate<\/a><\/li><li><a href=\"#Use_a_Defined_Certificate-Header_Contract\"><span class=\"toc_number toc_depth_2\">3.3.<\/span> Use a Defined Certificate-Header Contract<\/a><\/li><li><a href=\"#Authenticate_Nginx_to_a_Separate_Backend\"><span class=\"toc_number toc_depth_2\">3.4.<\/span> Authenticate Nginx to a Separate Backend<\/a><\/li><li><a href=\"#Record_Verification_and_Authorization_Separately\"><span class=\"toc_number toc_depth_2\">3.5.<\/span> Record Verification and Authorization Separately<\/a><\/li><\/ul><\/li><li><a href=\"#Apply_the_Same_Policy_with_Caddy\"><span class=\"toc_number toc_depth_1\">4.<\/span> Apply the Same Policy with Caddy<\/a><ul><li><a href=\"#Complete_Incoming_Caddy_Configuration\"><span class=\"toc_number toc_depth_2\">4.1.<\/span> Complete Incoming Caddy Configuration<\/a><\/li><li><a href=\"#Authenticate_Caddy_to_a_Separate_Backend\"><span class=\"toc_number toc_depth_2\">4.2.<\/span> Authenticate Caddy to a Separate Backend<\/a><\/li><\/ul><\/li><li><a href=\"#Rotate_Certificates_and_Trust_Deliberately\"><span class=\"toc_number toc_depth_1\">5.<\/span> Rotate Certificates and Trust Deliberately<\/a><ul><li><a href=\"#Plan_Lifetime_and_Renewal\"><span class=\"toc_number toc_depth_2\">5.1.<\/span> Plan Lifetime and Renewal<\/a><\/li><li><a href=\"#Protect_Keys_and_Distribution\"><span class=\"toc_number toc_depth_2\">5.2.<\/span> Protect Keys and Distribution<\/a><\/li><li><a href=\"#Coordinate_CA_Trust_and_Identity_Policy\"><span class=\"toc_number toc_depth_2\">5.3.<\/span> Coordinate CA Trust and Identity Policy<\/a><\/li><li><a href=\"#Validate_and_Reload_Manual_Certificate_Files\"><span class=\"toc_number toc_depth_2\">5.4.<\/span> Validate and Reload Manual Certificate Files<\/a><\/li><li><a href=\"#Enforce_Emergency_Access_Removal\"><span class=\"toc_number toc_depth_2\">5.5.<\/span> Enforce Emergency Access Removal<\/a><\/li><\/ul><\/li><li><a href=\"#Test_Acceptance_Rejection_and_Forwarded_Identity\"><span class=\"toc_number toc_depth_1\">6.<\/span> Test Acceptance, Rejection and Forwarded Identity<\/a><ul><li><a href=\"#Compare_Missing_and_Authorized_Credentials\"><span class=\"toc_number toc_depth_2\">6.1.<\/span> Compare Missing and Authorized Credentials<\/a><\/li><li><a href=\"#Check_Both_Certificate_Directions\"><span class=\"toc_number toc_depth_2\">6.2.<\/span> Check Both Certificate Directions<\/a><\/li><li><a href=\"#Inspect_the_Application_Decision\"><span class=\"toc_number toc_depth_2\">6.3.<\/span> Inspect the Application Decision<\/a><\/li><\/ul><\/li><li><a href=\"#Choose_the_Authorization_Boundary\"><span class=\"toc_number toc_depth_1\">7.<\/span> Choose the Authorization Boundary<\/a><\/li><li><a href=\"#Apply_the_Pattern_to_Webhooks_and_Internal_Calls\"><span class=\"toc_number toc_depth_1\">8.<\/span> Apply the Pattern to Webhooks and Internal Calls<\/a><\/li><li><a href=\"#Use_the_Related_Admin-Panel_Pattern\"><span class=\"toc_number toc_depth_1\">9.<\/span> Use the Related Admin-Panel Pattern<\/a><\/li><li><a href=\"#Diagnose_Configuration_and_Lifecycle_Failures\"><span class=\"toc_number toc_depth_1\">10.<\/span> Diagnose Configuration and Lifecycle Failures<\/a><\/li><li><a href=\"#Verify_the_Complete_Service_Access_Boundary\"><span class=\"toc_number toc_depth_1\">11.<\/span> Verify the Complete Service Access Boundary<\/a><\/li><\/ul><\/div>\n<h2 id=\"section-2\"><span id=\"What_mTLS_Authenticates_and_Authorizes\">What mTLS Authenticates and Authorizes<\/span><\/h2>\n<p>Service APIs, administrative endpoints and partner webhooks can use client certificates when callers support them. First identify which TLS endpoint verifies each peer and where application permissions are enforced. A reachable private network does not itself establish service identity; a valid certificate does not replace resource-level policy.<\/p>\n<p>Server and client certificates are separate credentials and need not come from the same CA. The client never sends its private key; TLS proves possession of that key. This lab uses one private CA for simplicity, while production may need distinct issuers, an offline root and intermediate CAs for separate trust boundaries.<\/p>\n<p>Use distinct service or device keys, verified enrollment and an explicit access inventory. Short-lived certificates reduce the remaining validity window but do not automatically rotate the key or block an exposed credential immediately. Plan effective revocation, application authorization removal and treatment of existing connections before rollout.<\/p>\n<h2 id=\"section-3\"><span id=\"Create_the_Lab_CA_and_Certificates\">Create the Lab CA and Certificates<\/span><\/h2>\n<p>A small OpenSSL CA makes the mechanism inspectable. Preserve the issuance database, serial state and CRLs along with controlled CA-key backups. <a href=\"https:\/\/smallstep.com\/docs\/step-ca\/\" rel=\"nofollow noopener\" target=\"_blank\">step-ca<\/a> can automate enrollment and renewal, but its default passive revocation stops renewal rather than invalidating an issued certificate before expiry. Active enforcement must be configured separately.<\/p>\n<h3><span id=\"Explicit_OpenSSL_CA_Client_and_Server_Profiles\">Explicit OpenSSL CA, Client and Server Profiles<\/span><\/h3>\n<p>The following opens an exclusive new directory and stops if it already exists. Continue CA commands in that directory because <code>dir = .<\/code> uses the working directory. The root, clientAuth leaf and serverAuth\/SAN profiles are explicit. <code>api.example.test<\/code> is a private test name, not a name for public Let\u2019s Encrypt issuance. CA signing asks for the encrypted key\u2019s password and confirmation.<\/p>\n<pre class=\"language-bash line-numbers\"><code class=\"language-bash\">set -eu\numask 077\nmtls_dir=&quot;$HOME\/service-mtls-lab&quot;\nmkdir &quot;$mtls_dir&quot;\ncd &quot;$mtls_dir&quot;\nmkdir certs crl newcerts private\n: &gt; index.txt\nprintf &#039;1000\\n&#039; &gt; serial\nprintf &#039;1000\\n&#039; &gt; crlnumber\ncat &gt; openssl.cnf &lt;&lt;&#039;EOF&#039;\n[ ca ]\ndefault_ca = CA_default\n\n[ CA_default ]\ndir = .\ndatabase = $dir\/index.txt\nnew_certs_dir = $dir\/newcerts\nserial = $dir\/serial\ncrlnumber = $dir\/crlnumber\nprivate_key = $dir\/private\/ca.key.pem\ncertificate = $dir\/certs\/ca.cert.pem\ndefault_md = sha256\ndefault_days = 90\ndefault_crl_days = 7\npolicy = client_policy\nunique_subject = no\ncopy_extensions = none\nx509_extensions = client_cert\ncrl_extensions = crl_ext\n\n[ client_policy ]\ncommonName = supplied\n\n[ req ]\ndistinguished_name = req_dn\nprompt = no\nx509_extensions = root_ca\n\n[ req_dn ]\nCN = Service mTLS Lab CA\n\n[ root_ca ]\nbasicConstraints = critical, CA:TRUE, pathlen:0\nkeyUsage = critical, keyCertSign, cRLSign\nsubjectKeyIdentifier = hash\nauthorityKeyIdentifier = keyid:always\n\n[ client_cert ]\nbasicConstraints = critical, CA:FALSE\nkeyUsage = critical, digitalSignature\nextendedKeyUsage = clientAuth\nsubjectKeyIdentifier = hash\nauthorityKeyIdentifier = keyid:always\n\n[ crl_ext ]\nauthorityKeyIdentifier = keyid:always\n[ server_cert ]\nbasicConstraints = critical, CA:FALSE\nkeyUsage = critical, digitalSignature\nextendedKeyUsage = serverAuth\nsubjectAltName = DNS:api.example.test,DNS:backend.example.test\nsubjectKeyIdentifier = hash\nauthorityKeyIdentifier = keyid:always\nEOF\nopenssl genpkey -algorithm RSA -pkeyopt rsa_keygen_bits:3072 -aes-256-cbc -out private\/ca.key.pem\nopenssl req -new -x509 -config openssl.cnf -key private\/ca.key.pem -days 3650 -sha256 -out certs\/ca.cert.pem\nopenssl genpkey -algorithm RSA -pkeyopt rsa_keygen_bits:2048 -out private\/client.key.pem\nopenssl req -new -key private\/client.key.pem -subj &quot;\/CN=stock-service&quot; -out client.csr.pem\nopenssl ca -config openssl.cnf -extensions client_cert -days 90 -notext -in client.csr.pem -out certs\/client.cert.pem\nopenssl ca -config openssl.cnf -gencrl -out crl\/ca.crl.pem\nopenssl verify -CAfile certs\/ca.cert.pem -purpose sslclient -crl_check -CRLfile crl\/ca.crl.pem certs\/client.cert.pem\nopenssl x509 -in certs\/client.cert.pem -noout -serial -dates\nopenssl genpkey -algorithm RSA -pkeyopt rsa_keygen_bits:2048 -out private\/server.key.pem\nopenssl req -new -key private\/server.key.pem -subj \/CN=api.example.test -out server.csr.pem\nopenssl ca -config openssl.cnf -extensions server_cert -days 90 -notext -in server.csr.pem -out certs\/server.cert.pem\nopenssl verify -CAfile certs\/ca.cert.pem -purpose sslserver -verify_hostname api.example.test certs\/server.cert.pem\n<\/code><\/pre>\n<p>The CA database records the issued client and the initial CRL exists before Nginx references it. The first client serial in this fresh lab is 1000. Verify enrollment rather than trusting a CSR\u2019s self-chosen CN. Install the server leaf\/key, public CA and CRL on the proxy with appropriate permissions; never install the CA private key there. The server\u2019s SAN must match the hostname the client actually verifies.<\/p>\n<h3><span id=\"Connect_with_the_Client_Certificate\">Connect with the Client Certificate<\/span><\/h3>\n<p>The test below assumes the proxy is on the local test machine and listens on 443. Change the &#8211;resolve address when testing another machine. This option overrides name resolution for that request while preserving the URL hostname and SNI for TLS verification. Keep the client private key on the test client.<\/p>\n<pre class=\"language-bash line-numbers\"><code class=\"language-bash\">curl --resolve api.example.test:443:127.0.0.1 --cacert certs\/ca.cert.pem -i https:\/\/api.example.test\/stock\ncurl --resolve api.example.test:443:127.0.0.1 --cacert certs\/ca.cert.pem --cert certs\/client.cert.pem --key private\/client.key.pem -i https:\/\/api.example.test\/stock\n<\/code><\/pre>\n<p>Test before revoking the lab credential: a request without a certificate must not reach the application, and the authorized client must reach the expected \/stock response. Then test an untrusted certificate, a same-CA leaf outside the allowlist and the revoked or deauthorized credential. A successful 200 alone does not establish that the rejection boundary works.<\/p>\n<h2 id=\"section-4\"><span id=\"Require_Client_Verification_on_Nginx\">Require Client Verification on Nginx<\/span><\/h2>\n<p>Nginx terminates the client TLS connection and can forward the verified leaf identity to an application. That forwarding is a second trust decision. The backend must accept identity headers only from the intended proxy and must prevent direct untrusted access. Do not treat an arbitrary X-Client header as proof of TLS authentication.<\/p>\n<h3><span id=\"Complete_Incoming_Nginx_Configuration\">Complete Incoming Nginx Configuration<\/span><\/h3>\n<p>Install certs\/server.cert.pem as \/etc\/mtls\/server.crt, private\/server.key.pem as \/etc\/mtls\/server.key, certs\/ca.cert.pem as \/etc\/mtls\/ca.crt and the initial crl\/ca.crl.pem as \/etc\/mtls\/ca.crl. Grant only the necessary service processes access to private keys. The following goes inside the existing http block, with log_format outside server.<\/p>\n<pre class=\"language-nginx line-numbers\"><code class=\"language-nginx\"># Inside the existing http {} block:\nlog_format service_mtls &#039;$remote_addr &quot;$request&quot; $status &#039;\n                        &#039;verify=&quot;$ssl_client_verify&quot; serial=&quot;$ssl_client_serial&quot;&#039;;\nserver {\n    listen 443 ssl;\n    server_name api.example.test;\n    ssl_certificate \/etc\/mtls\/server.crt;\n    ssl_certificate_key \/etc\/mtls\/server.key;\n    ssl_client_certificate \/etc\/mtls\/ca.crt;\n    ssl_verify_client on;\n    ssl_verify_depth 1;\n    ssl_crl \/etc\/mtls\/ca.crl;\n    ssl_protocols TLSv1.2 TLSv1.3;\n    ssl_session_cache off;\n    ssl_session_tickets off;\n    access_log \/var\/log\/nginx\/service_mtls.log service_mtls;\n    location \/ {\n        proxy_pass http:\/\/127.0.0.1:3000;\n        proxy_set_header X-Client-Verify $ssl_client_verify;\n        proxy_set_header X-Client-Cert-Encoding uri-pem;\n        proxy_set_header X-Client-Cert $ssl_client_escaped_cert;\n    }\n}\n<\/code><\/pre>\n<p>This whole-host configuration requires a verified client for every route, including \/healthz. Verification depth 1 fits the direct-root lab; reassess trust and required CRLs for intermediate chains. Session cache and tickets are disabled here to simplify fresh-handshake tests. If you enable reuse, assess its revocation behavior separately from established application connections.<\/p>\n<h3><span id=\"An_Additional_Exact-Serial_Gate\">An Additional Exact-Serial Gate<\/span><\/h3>\n<p>As an optional proxy gate under this one dedicated issuer, map the actual verified serial to an allow flag. The example serial 1000 is only the first lab client. This literal match avoids an unanchored subject regex that would also accept a longer, unintended CN or organization name.<\/p>\n<pre class=\"language-nginx line-numbers\"><code class=\"language-nginx\"># Optional additional gate, inside http {}; this single CA only:\nmap &quot;$ssl_client_verify:$ssl_client_serial&quot; $client_allowed {\n    default 0;\n    &quot;SUCCESS:1000&quot; 1;\n}\n# Inside each protected location before proxy_pass:\n# if ($client_allowed = 0) { return 403; }\n<\/code><\/pre>\n<p>Place the map at http scope and enable its illustrated if-return check in each protected location. It does not enforce anything by itself. If multiple issuers are trusted, use an issuer-qualified identity rather than a serial alone. Standard Nginx does not provide the invented <code>$ssl_client_s_dn_cn<\/code> variable or a general set of SAN variables assumed by some examples; parse a supported certificate representation when needed.<\/p>\n<h3><span id=\"Use_a_Defined_Certificate-Header_Contract\">Use a Defined Certificate-Header Contract<\/span><\/h3>\n<p>Nginx supplies URL-escaped PEM using <code>$ssl_client_escaped_cert<\/code>; Caddy below supplies base64 DER. Both overwrite the verification, encoding and certificate headers. The application decodes the declared format and authorizes the exact SHA-256 leaf fingerprint. Do not send multiline PEM directly in an HTTP header or confuse Nginx\u2019s SHA-1 fingerprint variable with a SHA-256 application identity.<\/p>\n<pre class=\"language-bash line-numbers\"><code class=\"language-bash\">        proxy_set_header X-Client-Verify $ssl_client_verify;\n        proxy_set_header X-Client-Cert-Encoding uri-pem;\n        proxy_set_header X-Client-Cert $ssl_client_escaped_cert;\n<\/code><\/pre>\n<p>Create the controlled allowlist below and install it as \/etc\/service-auth\/allowed-clients.json. The caller must not be able to edit it. Save the standalone Node.js example as app.mjs and run it through your service manager. It uses no Express dependency. Only the trusted proxy may reach the listener: loopback blocks remote access, but does not protect against untrusted local processes. Use an appropriately permissioned Unix socket, process\/network isolation or separately authenticated transport when that distinction matters.<\/p>\n<pre class=\"language-python line-numbers\"><code class=\"language-python\">node --input-type=module - certs\/client.cert.pem &gt; allowed-clients.json &lt;&lt;&#039;JS&#039;\nimport { readFileSync } from &#039;node:fs&#039;;\nimport { X509Certificate } from &#039;node:crypto&#039;;\nconst cert = new X509Certificate(readFileSync(process.argv[2]));\nconsole.log(JSON.stringify({ [cert.fingerprint256]: &#039;stock-service&#039; }, null, 2));\nJS\n\n<\/code><\/pre>\n<pre class=\"language-python line-numbers\"><code class=\"language-python\">\/\/ Save as app.mjs. Only the trusted local proxy may reach this listener.\nimport { createServer } from &#039;node:http&#039;;\nimport { readFileSync } from &#039;node:fs&#039;;\nimport { X509Certificate } from &#039;node:crypto&#039;;\n\ncreateServer((req, res) =&gt; {\n  const reply = (status, body) =&gt; {\n    res.writeHead(status, { &#039;Content-Type&#039;: &#039;application\/json&#039; });\n    res.end(JSON.stringify(body));\n  };\n  if (req.headers[&#039;x-client-verify&#039;] !== &#039;SUCCESS&#039;) {\n    return reply(401, { error: &#039;Client authentication required&#039; });\n  }\n  let cert;\n  try {\n    const value = req.headers[&#039;x-client-cert&#039;];\n    const encoding = req.headers[&#039;x-client-cert-encoding&#039;];\n    if (typeof value !== &#039;string&#039; || value.length &gt; 24000) throw new Error();\n    if (encoding === &#039;uri-pem&#039;) {\n      cert = new X509Certificate(decodeURIComponent(value));\n    } else if (encoding === &#039;base64-der&#039; &amp;&amp; \/^[A-Za-z0-9+\/]+={0,2}$\/.test(value)) {\n      cert = new X509Certificate(Buffer.from(value, &#039;base64&#039;));\n    } else {\n      throw new Error();\n    }\n  } catch {\n    return reply(401, { error: &#039;Invalid identity encoding&#039; });\n  }\n  let allowed;\n  try {\n    allowed = JSON.parse(readFileSync(&#039;\/etc\/service-auth\/allowed-clients.json&#039;, &#039;utf8&#039;));\n    if (!allowed || typeof allowed !== &#039;object&#039; || Array.isArray(allowed)) throw new Error();\n  } catch {\n    return reply(503, { error: &#039;Authorization policy unavailable&#039; });\n  }\n  const service = Object.hasOwn(allowed, cert.fingerprint256)\n    ? allowed[cert.fingerprint256] : undefined;\n  if (service !== &#039;stock-service&#039;) return reply(403, { error: &#039;Not authorized&#039; });\n  let path;\n  try { path = new URL(req.url, &#039;http:\/\/localhost&#039;).pathname; }\n  catch { return reply(400, { error: &#039;Invalid request target&#039; }); }\n  if (req.method === &#039;GET&#039; &amp;&amp; path === &#039;\/stock&#039;) {\n    return reply(200, { service, status: &#039;authorized&#039; });\n  }\n  if (req.method === &#039;GET&#039; &amp;&amp; path === &#039;\/healthz&#039;) return reply(200, { status: &#039;ok&#039; });\n  return reply(404, { error: &#039;Not found&#039; });\n}).listen(3000, &#039;127.0.0.1&#039;);\n<\/code><\/pre>\n<p>X509Certificate parsing does not independently validate the TLS chain; the application relies on the authenticated proxy boundary. This small example reads the policy on each request and fails closed if it is unavailable. For high traffic, use a controlled policy cache or authorization service with the same rejection behavior. Certificate renewal changes the fingerprint and requires a policy transition. Large certificate headers may exceed proxy or application header limits; test the actual leaf size.<\/p>\n<h3><span id=\"Authenticate_Nginx_to_a_Separate_Backend\">Authenticate Nginx to a Separate Backend<\/span><\/h3>\n<p>A remote backend introduces a new TLS connection. Prepare a backend server certificate matching backend.example.test, the correct backend CA trust and a separate gateway client certificate with clientAuth. The backend must require and specifically authorize that gateway credential, then accept forwarded original-client identity only through that trusted connection.<\/p>\n<pre class=\"language-nginx line-numbers\"><code class=\"language-nginx\"># Replace the incoming server&#039;s location \/ with this example.\n# Requires separately issued gateway client credentials and backend trust.\nlocation \/ {\n    proxy_pass https:\/\/10.0.0.12:8443;\n    proxy_ssl_server_name on;\n    proxy_ssl_name backend.example.test;\n    proxy_ssl_trusted_certificate \/etc\/mtls\/backend-ca.crt;\n    proxy_ssl_verify on;\n    proxy_ssl_verify_depth 1;\n    proxy_ssl_certificate \/etc\/mtls\/gateway-client.crt;\n    proxy_ssl_certificate_key \/etc\/mtls\/gateway-client.key;\n    proxy_ssl_protocols TLSv1.2 TLSv1.3;\n        proxy_set_header X-Client-Verify $ssl_client_verify;\n        proxy_set_header X-Client-Cert-Encoding uri-pem;\n        proxy_set_header X-Client-Cert $ssl_client_escaped_cert;\n}\n<\/code><\/pre>\n<p>Replace the incoming server\u2019s location block with this example; retain incoming TLS verification. proxy_ssl_verify is explicitly enabled because its default is off. proxy_ssl_name sets the name to verify and proxy_ssl_server_name enables SNI. The gateway client credential identifies Nginx at the backend, while the forwarded header represents the original caller. A correct CA does not compensate for the wrong server name.<\/p>\n<h3><span id=\"Record_Verification_and_Authorization_Separately\">Record Verification and Authorization Separately<\/span><\/h3>\n<p>Record the proxy\u2019s client-verification result and serial alongside the application\u2019s authorization decision. The serial or fingerprint links to a credential inventory; shared keys weaken attribution to an individual service instance. Do not routinely log full certificate headers or private material.<\/p>\n<pre class=\"language-nginx line-numbers\"><code class=\"language-nginx\"># Already included above; do not define the format twice.\n# log_format belongs in http {}, access_log in the relevant server {}.\nlog_format service_mtls &#039;$remote_addr &quot;$request&quot; $status &#039;\n                        &#039;verify=&quot;$ssl_client_verify&quot; serial=&quot;$ssl_client_serial&quot;&#039;;\naccess_log \/var\/log\/nginx\/service_mtls.log service_mtls;\n<\/code><\/pre>\n<p>Use restricted retention and access for identity-related logs. A TLS verification success followed by application 403 can be correct: the chain is trusted but the leaf is not authorized. Trace correlation helps diagnose that distinction without treating a request identifier as authentication.<\/p>\n<h2 id=\"section-5\"><span id=\"Apply_the_Same_Policy_with_Caddy\">Apply the Same Policy with Caddy<\/span><\/h2>\n<p>Caddy applies the same certificate boundary and sends a representation the same Node.js application accepts. The example uses manually supplied server certificate files; their renewal is your responsibility. Automatic HTTPS for a suitable public hostname is a separate deployment choice and does not make private .local or .test names publicly issuable.<\/p>\n<h3><span id=\"Complete_Incoming_Caddy_Configuration\">Complete Incoming Caddy Configuration<\/span><\/h3>\n<p>This example uses the current trust_pool file syntax for Caddy 2.11.7. It loads the public CA trust and requires verified clients. The certificate and key are the same manually issued server files used by Nginx. All identity header_up directives belong inside reverse_proxy.<\/p>\n<pre class=\"language-bash line-numbers\"><code class=\"language-bash\">https:\/\/api.example.test {\n    tls \/etc\/mtls\/server.crt \/etc\/mtls\/server.key {\n        client_auth {\n            mode require_and_verify\n            trust_pool file {\n                pem_file \/etc\/mtls\/ca.crt\n            }\n        }\n    }\n    reverse_proxy 127.0.0.1:3000 {\n        header_up X-Client-Verify SUCCESS\n        header_up X-Client-Cert-Encoding base64-der\n        header_up X-Client-Cert {tls_client_certificate_der_base64}\n    }\n}\n<\/code><\/pre>\n<p>The actual placeholder <code>{tls_client_certificate_der_base64}<\/code> produces a single-line certificate encoding. The literal SUCCESS header is justified by mandatory require_and_verify for the whole site; there is no assumed tls_client_verified placeholder. A path-specific handle cannot remove a TLS requirement. Keep health routes protected or use a separate network-restricted monitoring listener that cannot bypass the application boundary.<\/p>\n<h3><span id=\"Authenticate_Caddy_to_a_Separate_Backend\">Authenticate Caddy to a Separate Backend<\/span><\/h3>\n<p>To connect Caddy to the same separately protected backend, replace its reverse_proxy block with the following. Provision the gateway credential and backend trust first. The backend must identify the gateway as an authorized proxy before trusting its headers about the original client.<\/p>\n<pre class=\"language-bash line-numbers\"><code class=\"language-bash\"># Replace reverse_proxy inside the incoming mTLS site with this block.\n# The backend must require and authorize the gateway client certificate.\nreverse_proxy https:\/\/10.0.0.12:8443 {\n    header_up X-Client-Verify SUCCESS\n    header_up X-Client-Cert-Encoding base64-der\n    header_up X-Client-Cert {tls_client_certificate_der_base64}\n    transport http {\n        tls_server_name backend.example.test\n        tls_trust_pool file {\n            pem_file \/etc\/mtls\/backend-ca.crt\n        }\n        tls_client_auth \/etc\/mtls\/gateway-client.crt \/etc\/mtls\/gateway-client.key\n    }\n}\n<\/code><\/pre>\n<p><code>tls_trust_pool<\/code> supplies backend server trust, <code>tls_server_name<\/code> selects the verified name and <code>tls_client_auth<\/code> loads the gateway client certificate and key. <a href=\"https:\/\/caddyserver.com\/docs\/caddyfile\/directives\/tls#client-authentication\" rel=\"nofollow noopener\" target=\"_blank\">Caddy TLS client-authentication documentation<\/a> describes the incoming side; the reverse_proxy transport has its own directive names. Do not substitute unsupported root_ca or client_certificate spellings or disable backend verification.<\/p>\n<h2 id=\"section-6\"><span id=\"Rotate_Certificates_and_Trust_Deliberately\">Rotate Certificates and Trust Deliberately<\/span><\/h2>\n<p>Track leaf certificates, issuing CAs, policy identities and the connections that consume them separately. A successful file copy is not proof of a successful rotation. Validate the new path and the intended rejection of obsolete credentials at each relevant proxy and application boundary.<\/p>\n<h3><span id=\"Plan_Lifetime_and_Renewal\">Plan Lifetime and Renewal<\/span><\/h3>\n<p>The lab uses 90-day leaves; production lifetime should fit exposure and reliable renewal capacity. <a href=\"https:\/\/smallstep.com\/docs\/step-ca\/\" rel=\"nofollow noopener\" target=\"_blank\">step-ca<\/a> or another internal PKI can automate parts of that process. Explicitly rotate keys when needed: issuing a new certificate with the old public key is certificate renewal, not key replacement. Stopping renewal leaves a still-valid credential usable until another control removes its authority or it expires.<\/p>\n<h3><span id=\"Protect_Keys_and_Distribution\">Protect Keys and Distribution<\/span><\/h3>\n<p>Keep the CA signing key off proxies and applications. Protect client and gateway keys for the exact runtime that needs them, use controlled read-only mounts where practical and avoid broad file permissions. Prefer generating client keys at their intended service and sending only an approved CSR for signing.<\/p>\n<h3><span id=\"Coordinate_CA_Trust_and_Identity_Policy\">Coordinate CA Trust and Identity Policy<\/span><\/h3>\n<p>For a planned CA transition, stage only the required new trust alongside the old valid trust, then move credentials and verify before removing the old issuer. Multiple issuers can reuse a serial number, so revisit serial-only gates. Keep the application\u2019s exact-leaf fingerprint policy aligned with the certificates actually deployed.<\/p>\n<pre class=\"language-bash line-numbers\"><code class=\"language-bash\">cat new-ca.crt old-ca.crt &gt; clients-ca-bundle.crt\n<\/code><\/pre>\n<p>This bundle command assumes two already authenticated public CA files; it does not generate CAs or deploy trust. A similar controlled overlap can be needed for clients verifying server chains. CA compromise requires removal of the compromised authority, not a routine indefinite overlap with it. Test both accepted and rejected identities after the change.<\/p>\n<h3><span id=\"Validate_and_Reload_Manual_Certificate_Files\">Validate and Reload Manual Certificate Files<\/span><\/h3>\n<p>For Nginx, validate the configuration and perform the supported service reload. For manually loaded Caddy files, use <code>caddy validate --config \/etc\/caddy\/Caddyfile --adapter caddyfile<\/code> followed by <code>caddy reload --config \/etc\/caddy\/Caddyfile --adapter caddyfile --force<\/code>. Reload uses the protected, enabled admin API. &#8211;force can reload manual certificate files even if the config text is unchanged. Do not assume production Caddy watches arbitrary certificate files automatically. Validate the newly presented credential after loading it.<\/p>\n<h3><span id=\"Enforce_Emergency_Access_Removal\">Enforce Emergency Access Removal<\/span><\/h3>\n<p>For an exposed key, remove its SHA-256 entry through an atomic update of the controlled application policy; an empty <code>{}<\/code> policy denies all services. Also revoke the certificate in the CA database. The CA commands below produce a new CRL; securely install that public file at Nginx\u2019s \/etc\/mtls\/ca.crl, test, reload and verify rejection on a new connection. Refresh before this profile\u2019s seven-day nextUpdate even if no new revocations occur.<\/p>\n<pre class=\"language-bash line-numbers\"><code class=\"language-bash\">cd &quot;$HOME\/service-mtls-lab&quot;\nopenssl ca -config openssl.cnf -revoke certs\/client.cert.pem -crl_reason keyCompromise\nopenssl ca -config openssl.cnf -gencrl -out crl\/ca.crl.pem\nopenssl crl -in crl\/ca.crl.pem -noout -lastupdate -nextupdate\n<\/code><\/pre>\n<p>Stock Caddy\u2019s basic trust_pool does not consume that Nginx CRL automatically. Use a supported additional certificate verifier or enforce current authorization in the application. In this example policy removal rejects the next request while the TLS chain can still verify. Existing TLS\/WebSocket connections and application sessions require separate incident handling. Short lifetime alone is not immediate revocation.<\/p>\n<h2 id=\"section-7\"><span id=\"Test_Acceptance_Rejection_and_Forwarded_Identity\">Test Acceptance, Rejection and Forwarded Identity<\/span><\/h2>\n<p>Test the intended positive path and deliberate negative paths through both proxies. Record which layer rejects the request: TLS trust, certificate policy, application authorization or backend server validation. Keep private fixture keys out of logs and use credentials created for testing.<\/p>\n<h3><span id=\"Compare_Missing_and_Authorized_Credentials\">Compare Missing and Authorized Credentials<\/span><\/h3>\n<p>A request without a client certificate must not reach the protected application. Use the correct hostname and CA trust so a client-side server-trust failure is not mistaken for proof of client verification. Depending on the proxy and failure stage, rejection can occur before an HTTP response.<\/p>\n<pre class=\"language-bash line-numbers\"><code class=\"language-bash\">curl --resolve api.example.test:443:127.0.0.1 --cacert certs\/ca.cert.pem -i https:\/\/api.example.test\/stock\n<\/code><\/pre>\n<p>Before revocation, the authorized lab credential should receive the application\u2019s expected \/stock result. Test an untrusted client and a different valid same-CA leaf too; the latter must be denied by the application allowlist. Do not predict a single status for every certificate failure.<\/p>\n<pre class=\"language-bash line-numbers\"><code class=\"language-bash\">curl --resolve api.example.test:443:127.0.0.1 --cacert certs\/ca.cert.pem --cert certs\/client.cert.pem --key private\/client.key.pem -i https:\/\/api.example.test\/stock\n<\/code><\/pre>\n<p>After CA revocation and CRL deployment, test a fresh Nginx connection with the original files. For Caddy\u2019s basic configuration, also remove the original SHA-256 policy entry and confirm application rejection. An already accepted long-running session has different lifecycle rules from these fresh request tests.<\/p>\n<pre class=\"language-bash line-numbers\"><code class=\"language-bash\"># Use the same credential after revocation\/deauthorization; it must be rejected.\ncurl --resolve api.example.test:443:127.0.0.1 --cacert certs\/ca.cert.pem --cert certs\/client.cert.pem --key private\/client.key.pem -i https:\/\/api.example.test\/stock\n<\/code><\/pre>\n<h3><span id=\"Check_Both_Certificate_Directions\">Check Both Certificate Directions<\/span><\/h3>\n<p>Inspect the actual issuer, chain, lifetime, purpose and verified hostname on the failing connection. <a href=\"https:\/\/nginx.org\/en\/docs\/http\/ngx_http_ssl_module.html#ssl_client_certificate\" rel=\"nofollow noopener\" target=\"_blank\">Nginx client-certificate documentation<\/a> explains CA trust, chain depth and CRL handling. An intermediate deployment needs the intended trust and required CRLs, not simply an arbitrary larger depth. Retain server verification while testing.<\/p>\n<h3><span id=\"Inspect_the_Application_Decision\">Inspect the Application Decision<\/span><\/h3>\n<p>The sample \/stock response provides a limited confirmation of the authorized service mapping. Send fake incoming identity headers with a valid client and verify that the proxy replaces them; send a different valid leaf and verify that it cannot select the approved identity. Avoid exposing general certificate-debug routes publicly.<\/p>\n<pre class=\"language-bash line-numbers\"><code class=\"language-bash\">{\n  &quot;service&quot;: &quot;stock-service&quot;,\n  &quot;status&quot;: &quot;authorized&quot;\n}\n<\/code><\/pre>\n<h2 id=\"section-8\"><span id=\"Choose_the_Authorization_Boundary\">Choose the Authorization Boundary<\/span><\/h2>\n<p>Define who approves certificate enrollment, who assigns service permissions and how urgent removal reaches each enforcement point. Those are separate responsibilities. An operationally complete design includes monitoring for policy-file failure, CRL expiry and failed trust deployment as well as successful issuance.<\/p>\n<p>The gateway can enforce broad admission and the application can enforce resource-level rules. This example pins an exact SHA-256 leaf fingerprint to stock-service; it does not grant authority based on a suggestive CN substring. More complex policy may use a CA-controlled identity and an authorization service, provided the complete trust path is verified.<\/p>\n<p>Keep an inventory of service, environment, issuer, serial, leaf fingerprint, expiration and ownership. Stable service names aid operations, but requested names must be authorized by enrollment policy. A SAN URI such as a SPIFFE ID is not validated merely by appearing in a certificate; the issuing system and consuming policy must implement its intended trust rules.<\/p>\n<p>Monitor verification failures separately from policy denials, and correlate them with the selected route and backend. Do not log private keys or unrestricted raw certificate headers. Expiry and nextUpdate alerts need a responsible owner and enough lead time to repair a failed distribution.<\/p>\n<p>A public token-authenticated API and a private mTLS API can coexist under deliberate boundaries. TLS policy is chosen before the HTTP path is known. A separate hostname is often easier to reason about; if using optional verification with path-level enforcement, inventory every protected route and alternative backend path.<\/p>\n<h2 id=\"section-9\"><span id=\"Apply_the_Pattern_to_Webhooks_and_Internal_Calls\">Apply the Pattern to Webhooks and Internal Calls<\/span><\/h2>\n<p>A partner webhook can use mTLS if the sender supports client credentials. Payload signatures, replay protection and rate controls address additional properties; mTLS does not provide all of them automatically. Bind the verified sender identity to the allowed webhook actions and manage the sender\u2019s credential lifecycle.<\/p>\n<p>For internal service calls, each hop needs its own configured server verification and client identity when mTLS is required. A gateway\u2019s upstream certificate identifies the gateway, while original-caller identity is delegated through an explicitly trusted mechanism. Do not describe one verified ingress handshake as automatic end-to-end authentication for every backend.<\/p>\n<h2 id=\"section-10\"><span id=\"Use_the_Related_Admin-Panel_Pattern\">Use the Related Admin-Panel Pattern<\/span><\/h2>\n<p><a href=\"https:\/\/www.dchost.com\/blog\/en\/nginx-mtls-admin-panel\/\">Protecting admin panels with Nginx mTLS<\/a> covers the related user\/device access pattern, including whole-host versus path checks, an initial CRL and controlled recovery. Keep application authentication and roles there as well. A health or rescue route inside a mandatory TLS host cannot bypass its certificate handshake requirement.<\/p>\n<h2 id=\"section-11\"><span id=\"Diagnose_Configuration_and_Lifecycle_Failures\">Diagnose Configuration and Lifecycle Failures<\/span><\/h2>\n<p>Begin diagnosis by locating the failing hop and whether it reached HTTP. A server-name error, missing client certificate, revoked leaf, unavailable policy and deliberate application 403 call for different fixes. Preserve the actual error rather than weakening verification to make a request pass.<\/p>\n<p>Check issuer trust, the presented chain, verification depth, purpose constraints, certificate\/CRL dates and the allowlist identity. The lab uses explicit CA:FALSE and separate clientAuth\/serverAuth leaf profiles. Missing an EKU extension is not identical to an explicit restriction; inspect the actual certificate and validator policy instead of assuming every failure is the CA file.<\/p>\n<p>For an upstream failure, compare the actual backend SAN with proxy_ssl_name or tls_server_name and verify the intended CA chain. The gateway credential must also be accepted by the backend\u2019s client policy. These are separate validation directions, and a successful TCP connection proves neither.<\/p>\n<p>If identity fields are missing, verify the exact variables, header placement and encoding contract. Nginx\u2019s escaped PEM and Caddy\u2019s DER base64 are decoded differently by the sample app. Repeating header directives at a lower Nginx level affects inheritance, so preserve the entire required header set when changing a location. Test certificate size against real header limits.<\/p>\n<p>Connection reuse can reduce handshake work, but measure the actual latency and capacity impact. It also means authorization and revocation need a defined policy for existing connections and sessions. Do not promise negligible impact for every workload or continuous certificate revalidation for every message on one established connection.<\/p>\n<h2 id=\"section-12\"><span id=\"Verify_the_Complete_Service_Access_Boundary\">Verify the Complete Service Access Boundary<\/span><\/h2>\n<p>The completed boundary admits the intended service and rejects missing, untrusted and unauthorized credentials at the appropriate layer. Nginx and Caddy feed the same explicit application identity policy through different documented certificate encodings. A separate backend connection verifies the backend server and authorizes the gateway client before trusting delegated caller information.<\/p>\n<p>Repeat acceptance and rejection tests after CA, certificate, proxy, DNS, network or application-route changes. Check that the new leaf is authorized and the old one is no longer accepted. Keep CRL distribution, current application policy and open-session handling visible during rotations and incidents.<\/p>\n<p>Reliable mTLS depends on enrollment, key protection, trust, authorization and revocation continuing to work together. Document the operational owner for each step and verify the actual traffic paths. A correct initial handshake is the beginning of that lifecycle, not its completion.<\/p>\n<\/div>","protected":false},"excerpt":{"rendered":"<p>Use one verified identity and application policy across both proxies, with explicit certificate profiles, safe encoding, upstream mTLS and access removal.<\/p>\n","protected":false},"author":1,"featured_media":8154,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[184],"tags":[],"class_list":["post-1659","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-technology"],"_links":{"self":[{"href":"https:\/\/www.dchost.com\/blog\/en\/wp-json\/wp\/v2\/posts\/1659","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.dchost.com\/blog\/en\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.dchost.com\/blog\/en\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.dchost.com\/blog\/en\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.dchost.com\/blog\/en\/wp-json\/wp\/v2\/comments?post=1659"}],"version-history":[{"count":2,"href":"https:\/\/www.dchost.com\/blog\/en\/wp-json\/wp\/v2\/posts\/1659\/revisions"}],"predecessor-version":[{"id":8156,"href":"https:\/\/www.dchost.com\/blog\/en\/wp-json\/wp\/v2\/posts\/1659\/revisions\/8156"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.dchost.com\/blog\/en\/wp-json\/wp\/v2\/media\/8154"}],"wp:attachment":[{"href":"https:\/\/www.dchost.com\/blog\/en\/wp-json\/wp\/v2\/media?parent=1659"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.dchost.com\/blog\/en\/wp-json\/wp\/v2\/categories?post=1659"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.dchost.com\/blog\/en\/wp-json\/wp\/v2\/tags?post=1659"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}