HAProxy SSL certificate expired: total TLS failure and how to catch it first

Every client connecting to the affected frontend gets a TLS handshake failure. Not a percentage of clients, not a slow degradation: all of them, immediately, from the moment the certificate’s notAfter timestamp passes. Browsers show NET::ERR_CERT_DATE_INVALID, API clients throw certificate validation errors, and HAProxy itself is perfectly healthy, running, and passing every process-liveness check you have.

The mechanism is trivial, the blast radius is total, and HAProxy gives you no built-in metric, counter, or log warning that a certificate is about to expire. If you are reading this during an incident, skip to Quick checks and Fixes. If you are reading it afterward, the Prevention section is the part that matters.

What this means

When HAProxy terminates TLS on a frontend (bind ... ssl crt /etc/haproxy/certs/example.pem), it serves the loaded certificate to every client during the handshake. Clients validate the chain, including the expiry date. Once the certificate expires, validation fails on the client side and the connection dies before any HTTP request exists.

Three properties make this failure mode nasty:

• It is binary. One second before expiry everything works. One second after, nothing does. There is no brown-out period.
• HAProxy’s own stats look almost normal. The process is up, backends are healthy, queue depth is zero. The only signals are a collapsing frontend session rate and a rise in request errors (ereq) and handshake-failure entries in the logs. There is no “certificate expired” counter.
• One SAN certificate can take down many domains at once. A multi-domain certificate covering 30 hostnames expires once, and all 30 frontends fail simultaneously. The blast radius is the certificate’s SAN list, not the frontend count.

flowchart TD
  A[Renewal automation fails silently] --> B[Cert on disk and in memory ages toward notAfter]
  B --> C[notAfter passes]
  C --> D[All TLS handshakes on the frontend fail]
  D --> E[Frontend session rate collapses]
  D --> F[ereq rises and logs fill with handshake errors]
  G[External expiry check at 30/7/1 days] -.->|catches it first| B

The same failure applies to inter-node TLS: if HAProxy connects to backends over TLS with certificate verification enabled, an expired backend certificate makes HAProxy refuse those connections, and the backend appears down even though the application is fine.

Common causes

Quick checks

All read-only and safe to run during an incident.

# 1. What certificate is HAProxy actually serving right now?
echo | openssl s_client -connect 127.0.0.1:443 -servername example.com 2>/dev/null | \
  openssl x509 -noout -subject -issuer -enddate

# 2. What does HAProxy think it has loaded? (runtime API, lists all certs in memory)
echo "show ssl cert" | socat unix-connect:/var/run/haproxy.sock stdio

# 3. Expiry for one specific certificate as HAProxy sees it
echo "show ssl cert /etc/haproxy/certs/example.pem" | socat unix-connect:/var/run/haproxy.sock stdio

# 4. Expiry of the file on disk (what a reload would load)
openssl x509 -in /etc/haproxy/certs/example.pem -noout -enddate

# 5. Compare (3) and (4). If (3) shows an older date than (4), the renewal
#    landed on disk but never reached the running process.

# 6. Is the frontend collapsing? Session rate and request errors per frontend
echo "show stat" | socat unix-connect:/var/run/haproxy.sock stdio | \
  awk -F, '$2 == "FRONTEND" {print $1": rate="$34" ereq="$13" req_rate="$47}'

Two things to note. First, show ssl cert identifies certificates by full filesystem path, not a short name. If you configured crt /etc/haproxy/certs/, each file under that directory is registered by its full path. Second, on HAProxy 2.4 and later, show ssl cert also lists certificates used on backend server lines, so check those too if you terminate TLS to backends.

How to diagnose it

1. Confirm the served certificate is the problem. Run check 1 against the public listener. If notAfter is in the past, you have your answer. If the date is fine but the subject or issuer is wrong, you have an SNI or wrong-certificate problem instead. Go to the SNI fix below.
2. Compare memory vs. disk. If the on-disk PEM (check 4) is valid but the in-memory cert (check 3) is expired, the renewal worked and only the deploy hook failed. This is the best-case incident: the fix is loading the file, not getting a new certificate.
3. If disk is also expired, check the renewal automation. Look at the renewal tool’s logs (certbot’s /var/log/letsencrypt/, acme.sh logs, or your systemd timer’s journal). Common silent failure: the ACME HTTP-01 challenge needs port 80, but HAProxy binds port 80 and the standalone challenge server cannot start, or a firewall blocks the challenge path. The renewal has often been failing for weeks before expiry while the timer kept running.
4. Verify scope. If this is a SAN certificate, enumerate the SAN list (openssl x509 -text | grep -A1 "Subject Alternative Name") so you know every hostname that is down. If HAProxy fronts many certificates, run check 3 across all of them to see whether other certs are also near expiry.
5. After the fix, verify from the client side. Re-run check 1 for each affected hostname and confirm the new notAfter and the expected subject. Then watch ereq and the frontend session rate return to baseline.
6. Check for a stale-cert edge case. There are documented cases (HAProxy issue #1700, affecting HAProxy 2.5.5) where HAProxy keeps serving an older in-memory certificate even after a reload, with the correct file on disk. If checks 3 and 4 still disagree after a reload, a full restart clears it. A restart drops all existing connections, so treat it as disruptive and plan for it.

Metrics and signals to monitor

Fixes

Hot-update the certificate without a reload

If the valid PEM is already on disk, load it into the running process via the runtime API. HAProxy 2.1+ supports updating an existing certificate in memory without touching traffic:

# Stage the new certificate (transaction-style update)
echo -e "set ssl cert /etc/haproxy/certs/example.pem <<\n$(cat /etc/haproxy/certs/example.pem)\n" | \
  socat unix-connect:/var/run/haproxy.sock stdio

# Commit it
echo "commit ssl cert /etc/haproxy/certs/example.pem" | socat unix-connect:/var/run/haproxy.sock stdio

# Verify the new notAfter
echo "show ssl cert /etc/haproxy/certs/example.pem" | socat unix-connect:/var/run/haproxy.sock stdio

Constraints that bite people:

• The certificate name is the full filesystem path exactly as HAProxy registered it, not a bare filename.
• You can only set ssl cert over a certificate already referenced by the configuration. For a certificate HAProxy has never loaded, use new ssl cert (HAProxy 2.2+).
• Runtime API changes live in memory only. They are lost when the process stops. The file on disk must be correct too, or the next restart or reload brings the expired certificate back.
• Make sure the PEM contains the full chain (leaf plus intermediates). A missing or silently dropped intermediate causes validation failures on some clients, especially mobile, even with a valid expiry date. On HAProxy versions affected by issue #2818 (observed through 3.0.5, fixed in later 3.0.x/3.1.x releases), set ssl cert silently drops the intermediate CA if its PEM block uses CRLF (\r\n) line endings — common with certificates from the DigiCert API. Normalise the PEM to Unix line endings before uploading.

Renew the certificate, then load it

If the file on disk is also expired, renew first, then deploy. Whatever ACME client you use, the deploy hook must (a) assemble the combined PEM in the exact layout HAProxy expects (cert, intermediates, private key, in the order your existing files use), (b) write it to the configured path, and (c) load it via set ssl cert/commit ssl cert or trigger a reload. Test the deploy hook, not just the renewal. Most expiry outages are deploy-hook failures, not CA failures.

If a reload is unavoidable

systemctl reload haproxy loads the new certificate and drains old connections gracefully. Do not use restart unless you have to: restart drops every established connection. If a reload does not clear a stale in-memory certificate, a restart is the last resort and should be treated as a disruptive action.

SNI misconfiguration after rotation

If clients receive the default certificate instead of the one matching their hostname, the SNI mapping is wrong, not the certificate. Check that the new certificate is registered under the path your bind ... ssl crt directive expects, and test each hostname explicitly with openssl s_client -servername <host>. A rotated SAN certificate with a changed SAN list can silently stop matching hostnames that used to be covered.

Backend (inter-node) certificate expiry

If HAProxy verifies backend certificates and one expires, HAProxy refuses those connections: econ rises, servers go DOWN, and the backend looks broken while the application is healthy. Fix the backend certificate, and monitor backend cert expiry the same way as frontend certs.

Prevention

• External expiry check on every certificate. HAProxy has no expiry metric and will not warn you. Scrape show ssl cert (or parse PEM files with openssl x509 -enddate) on a schedule and alert on days remaining. Thresholds that work: create a plan item at 30 days, a ticket at 7 days, page inside 24 hours.
• Alert on renewal runs, not just expiry. The real failure is a renewal job that stopped working 60 days before expiry. Alert if the renewal tool has not logged a successful check within its expected interval.
• Test the full chain after every rotation. openssl s_client per hostname, checking subject, issuer, chain completeness, and notAfter. Automate this as a post-rotation verification step.
• Prefer runtime cert updates over reloads. set ssl cert/commit ssl cert avoids the counter resets, stick-table flushes, cold connection pools, and SSL session cache invalidation that come with every reload. Keep the disk file authoritative, because runtime changes are memory-only.
• Inventory SAN coverage. Know which hostnames ride on which certificate. The expiry of one SAN cert should never surprise you with its blast radius.
• Watch ereq and handshake failures after every cert change. Any spike within minutes of a rotation means broken clients. Roll back or fix immediately rather than discovering it from support tickets.
• Shorten the manual loop. With public CA maximum certificate validity shrinking (the CA/Browser Forum has ratified a reduction to 47 days), manual renewal is no longer a viable operating model. Automate renewal and deployment end to end.

How Netdata helps

• Frontend session rate per frontend, so a TLS-driven traffic cliff on one frontend is visible in seconds and distinguishable from a global outage.
• ereq per frontend, which is where TLS handshake failures land in HAProxy’s stats; correlating an ereq spike with a recent cert rotation is the fastest confirmation.
• Frontend vs. backend 5xx breakdown, so you can rule out the backend and stop debugging the application during what is purely a TLS event.
• Reload detection via Uptime_sec, letting you correlate “cert renewed on disk” with “cert actually loaded” and catch deploy-hook failures.
• ML-based anomaly detection on session rates, which flags the cliff even on frontends you forgot to threshold.
• Because HAProxy exposes no expiry metric itself, pair Netdata’s HAProxy signals with any external certificate-expiry probe and alert on the 30/7/1-day ladder described above.

Related guides

• HAProxy health checks green but the application is broken: when UP does not mean healthy
• HAProxy 502 Bad Gateway: backend connection and response failures
• HAProxy 503 Service Unavailable: no server is available to handle this request
• HAProxy health check L4TOUT and L4CON: server marked DOWN and unreachable
• HAProxy backend losing servers: active server count and cascade risk
• How HAProxy actually works in production: a mental model for operators