HAProxy TLS handshake storm: new-connection floods that saturate the event loop

Traffic looks normal at the TCP layer. Connections are arriving, the frontend is accepting them, but latency is climbing, some clients time out before ever sending a request, and backend servers look healthy. When you check show info, Idle_pct is near zero and SslFrontendKeyRate is far above anything you have tested. You are in a TLS handshake storm: HAProxy is burning every CPU cycle on cryptographic handshake work and the event loop has nothing left for the rest of the pipeline.

This failure mode is deceptive because HAProxy does not crash, does not reject connections at the socket level, and does not log a specific error. It simply cannot complete handshakes fast enough. Existing keep-alive connections often continue working while every new connection stalls, which makes the symptom look like an intermittent client-side or network problem.

The signal set is small and well-defined, and one of the most common root causes (a broken or undersized session cache forcing full handshakes for every client) can be fixed without adding any hardware.

What this means

TLS handshakes are the most CPU-intensive operation HAProxy performs. A full handshake with modern key exchange (ECDHE) costs orders of magnitude more CPU than proxying an already-established connection. Session resumption (session IDs or tickets) lets a returning client skip most of that cost. When a flood of new, unique connections arrives, or when resumption silently stops working, every connection pays the full price.

Because HAProxy is event-driven, handshake CPU work competes directly with everything else the loop does: accepting connections, parsing requests, connecting to backends, and forwarding responses. When the loop saturates, the failure propagates in a specific order:

flowchart TD
  A[Surge of new TLS connections] --> B[SslFrontendKeyRate spikes]
  C[Session cache broken or too small] --> B
  B --> D[Event loop CPU saturated: Idle_pct near 0]
  D --> E[accept and backend connects starved]
  E --> F[ctime spikes, kernel accept queue overflows]
  D --> G[New handshakes stall and time out]
  G --> H[req_rate falls, clients retry, more handshakes]
  H --> B

Note the feedback loop: clients whose handshakes time out retry, generating yet more full handshakes. Storms sustain themselves until the incoming rate drops or you restore cheap resumption.

Two caveats before you trust the signals:

• Busy-polling: if busy-polling is enabled, Idle_pct stays near zero by design and is meaningless. Use show activity run-queue depth or system CPU instead.
• Reloads: the in-process session cache is invalidated on every reload, so SslFrontendKeyRate spikes briefly after each one. Do not confuse a post-reload warmup with a storm.

Common causes

Quick checks

All of these are read-only against the runtime socket.

# 1. Handshake rate and event loop headroom: the two defining signals
echo "show info" | socat unix-connect:/var/run/haproxy.sock stdio | \
  grep -E "^(SslFrontendKeyRate|SslRate|Idle_pct|CurrConns|ConnRate|SessRate):"

# 2. Session cache effectiveness (hit rate = (Lookups - Misses) / Lookups)
echo "show info" | socat unix-connect:/var/run/haproxy.sock stdio | \
  grep -E "^Ssl(Cache|FrontendSessionReuse)"

# 3. Is the loop starving backend connects? (ctime is a rolling 1024-sample average)
echo "show stat" | socat unix-connect:/var/run/haproxy.sock stdio | \
  awk -F, '$2 == "BACKEND" {print $1": ctime="$60"ms rtime="$61"ms"}'

# 4. Are new connections being dropped before HAProxy sees them?
nstat -az | grep -i listen

# 5. Per-thread skew (busy-polling or suspected hot thread)
echo "show activity" | socat unix-connect:/var/run/haproxy.sock stdio

# 6. What is holding sessions? Look for floods of handshaking connections
echo "show sess" | socat unix-connect:/var/run/haproxy.sock stdio | head -50

# 7. Process uptime, to rule out a post-reload cache warmup
echo "show info" | socat unix-connect:/var/run/haproxy.sock stdio | grep -E "^(Uptime_sec|Stopping):"

How to diagnose it

1. Confirm CPU saturation is real and TLS-driven. Idle_pct below 20% means the loop is under stress; below 5% it is saturated and latency will spike. If busy-polling is on, skip Idle_pct entirely and check show activity run-queue depth. Correlate with SslFrontendKeyRate: if the key rate is at or above your tested per-core capacity, TLS is the consumer.

2. Check the session cache first. Compute hit rate as (SslCacheLookups - SslCacheMisses) / SslCacheLookups. A low hit rate with high key rate means nearly every connection is paying for a full handshake. SslFrontendSessionReuse_pct tells the same story directly. If resumption is broken, fixing it restores capacity immediately, often without scaling anything.

3. Rule out the reload artifact. If Uptime_sec is small or you reloaded recently, the in-process cache was just flushed and the key rate spike is warmup, not a storm. Shared ticket keys on disk mitigate this; a purely in-memory cache does not survive a reload.

4. Characterize the incoming connections. Compare ConnRate with SessRate. A large gap means many connections never become sessions (handshakes abandoned, denials, or pre-accept losses). Check ListenOverflows/ListenDrops via nstat: when the loop is too busy to call accept(), the kernel silently drops SYNs and HAProxy sees nothing. If the gap is large, also inspect show sess for source-IP concentration to distinguish a flash crowd from an attack.

5. Confirm the blast radius on existing traffic. Check backend ctime (spikes when the loop cannot service backend connects in time), req_rate (falling means successful traffic is degrading), and frontend 5xx. If rtime is normal and backends are healthy, the problem is entirely on the frontend TLS path, which is the signature of this pattern.

6. Check for thread skew with nbthread > 1. Idle_pct is an average across threads. show activity reveals whether one thread is doing most of the handshake work while others idle, which changes the fix (distribution problem, not capacity problem).

7. Decide: cache fix, rate shedding, or capacity. If hit rate is broken, fix resumption. If resumption is healthy and volume genuinely exceeds capacity, shed load or scale.

Metrics and signals to monitor

Fixes

Restore session resumption

If the cache hit rate is the problem, this is the highest-leverage fix and requires no new capacity.

• Check tune.ssl.cachesize. If it is set too small, entries are evicted before clients return and every connection does a full handshake. Setting it to 0 disables the cache entirely, which guarantees this failure mode under new-connection load. The default value is 20000 entries unless overridden at build time.
• Check tune.ssl.lifetime. A very short session lifetime invalidates entries before clients can reuse them.
• Survive reloads. The in-process cache flushes on every reload. Storing ticket keys on disk (shared across processes) lets the new process honor tickets issued by the old one and avoids a full-handshake warmup spike after every config change.
• Know the TLS 1.3 caveat. no-tls-tickets only affects TLS 1.2 and below; TLS 1.3 sends tickets regardless. Do not assume ticket suppression where it does not apply.
• Be aware of version-specific resumption bugs. Session ID or ticket resumption has had version-specific regressions in the TLS code across HAProxy releases; if SslFrontendSessionReuse_pct collapses with no config change, check the changelog for your exact HAProxy and TLS library version.

Shed or cap the incoming load

When resumption is healthy and the volume itself exceeds capacity:

• Rate-limit new SSL sessions. HAProxy supports a global SSL session rate limit (maxsslrate): SSL listeners stop accepting new connections when the limit is reached, which protects the loop for already-established sessions. This trades new-client failures for existing-client survival, which is usually the right trade during a storm.
• Block abusive sources. If show sess or stick tables show source concentration, per-IP conn_rate limits via a stick table cut off handshake floods at the source. Watch table utilization: a full table silently stops tracking new sources.
• Terminate TLS further upstream. CDN or cloud-LB termination absorbs geographically distributed handshake CPU before it reaches your HAProxy tier. This is an architecture change, not an incident fix, but it is the standard answer for recurring storms.

Add CPU capacity or fix distribution

• Scale horizontally. More HAProxy instances behind the upstream LB is the cleanest capacity add; handshake cost scales with new-connection rate, not with total connections.
• Use threads, not processes. If you are on a legacy nbproc deployment, modern guidance is nbthread with a single process. There are production reports of TLS performance regressions when migrating from multi-process to multi-threaded setups with very high thread counts, so load-test the target topology before relying on it during a storm.
• Fix hot-thread skew. If show activity shows one thread doing most accepts, SO_REUSEPORT on listeners helps distribute new connections across threads.

What not to do first

Do not restart HAProxy. A restart (or reload) flushes the session cache and forces every connected client into a fresh full handshake, which makes the storm strictly worse. If the loop is saturated, a reload also risks losing visibility right when you need it.

Prevention

• Load-test handshake capacity, not just throughput. Establish your measured SslFrontendKeyRate ceiling per core and alert when sustained rates approach it. Throughput tests with reused connections hide TLS costs completely.
• Alert on the composite, not a single metric. Key rate above baseline AND (Idle_pct < 5% OR rising run-queue under busy-polling) AND degrading req_rate, persisting more than 2 minutes, with an Uptime_sec > 600 gate to suppress post-reload warmup.
• Trend the cache hit rate. A gradual decline in SslFrontendSessionReuse_pct is a silent capacity leak. By the time Idle_pct drops, the margin is already gone.
• Handle counter resets. All these counters reset on reload. Use Uptime_sec to gate rate alerts so a reload does not produce phantom drops or spikes.
• Practice the reload path. Frequent reloads (dynamic service discovery) mean frequent cache flushes. If your deploy pipeline reloads HAProxy often, shared on-disk ticket keys are mandatory, not optional.

How Netdata helps

• Netdata collects the HAProxy show info and stats CSV signals that define this pattern: SslFrontendKeyRate, Idle_pct, session rates, and per-backend ctime, at per-second granularity, so the storm onset and the feedback loop are visible as they develop.
• Correlating key rate against Idle_pct on one dashboard separates TLS CPU saturation from connection-count saturation (scur/slim), which need different fixes.
• Cache hit rate derived from SslCacheLookups/SslCacheMisses alongside the key rate shows instantly whether you are in a “fix resumption” incident or an “add capacity” incident.
• Reload detection via Uptime_sec resets lets you distinguish a post-reload cache warmup spike from a genuine external storm.
• Per-backend ctime trending next to frontend TLS metrics confirms when loop starvation starts leaking into backend connectivity.
• Historical retention lets you baseline your tested handshake ceiling and alert on approach, before clients feel it.

Related guides

• HAProxy 502 Bad Gateway: backend connection and response failures
• HAProxy 503 Service Unavailable: no server is available to handle this request
• HAProxy 504 Gateway Timeout: timeout server and the backend timeout cascade
• HAProxy backend losing servers: active server count and cascade risk
• HAProxy backend queue building (qcur): requests waiting for a free server slot
• HAProxy connect time (ctime) high: network latency and backend accept-queue overflow
• HAProxy connection errors (econ): backend connections refused, timed out, or reset
• HAProxy scur approaching slim: the concurrent-session saturation signal
• HAProxy 5xx delta: telling HAProxy-generated errors from backend errors
• HAProxy health checks green but the application is broken: when UP does not mean healthy
• HAProxy health check L4TOUT and L4CON: server marked DOWN and unreachable
• HAProxy health check L7STS: server DOWN on the wrong HTTP status