HAProxy connection reuse dropping: http-reuse, pooling, and handshake overhead

Backend latency crept up, CPU on the HAProxy host is climbing, and ctime went from zero to measurable milliseconds, but the backends are healthy and the network looks fine. Before you blame the application or the network, check the one ratio almost nobody graphs: connection reuse, reuse / (connect + reuse) on the backend. If it fell off a cliff, HAProxy stopped reusing pooled backend connections and is now paying a full TCP (and possibly TLS) handshake for every request.

Nothing errors and nothing pages on its own. HAProxy just works harder per request until ephemeral ports or CPU run out, and by then the symptom (latency, econ spikes, TLS CPU saturation) points everywhere except the actual cause.

What this means

HAProxy maintains a pool of idle, previously used backend connections per server. When a request needs a backend connection, HAProxy first tries to pull one from the pool (reuse); only if the pool is empty or the connection is unusable does it open a new one (connect). Both are cumulative counters in the stats CSV, per backend and per server.

The reuse ratio tells you what fraction of backend connection demands were satisfied from the pool:

pool utilization = reuse / (connect + reuse)

A healthy HTTP backend with keep-alive typically sits well above 50%, often much higher. A sudden drop means requests that used to ride pooled connections now each open a fresh socket. The usual trigger is configuration drift on either side: a backend switched HTTP version, disabled keep-alive, or started sending Connection: close, and nothing alerted because nothing technically failed.

The consequences compound:

1. ctime appears. With reuse working, connect time is zero for most requests. Nonzero ctime showing up on a backend that historically read zero is the fingerprint of broken pooling.
2. SslBackendKeyRate spikes. If backend connections are TLS, every new connection pays a full handshake. Backend TLS handshakes are CPU spent inside HAProxy’s event loop.
3. Ephemeral port pressure rises. Each new backend connection consumes an ephemeral port on the HAProxy host. With roughly 28K ports in the default local range and a 60s TIME_WAIT, the sustained new-connection rate caps out near 450/s. Past that, connect() fails and you get econ spikes with no backend-side failure at all.
4. Latency grows. Each request now includes connect time plus handshake time it did not have before.

flowchart TD
  A[Backend config drift:
HTTP version change, keep-alive off,
Connection: close header] --> B[Pool starves:
idle_conn_cur falls toward 0]
  B --> C[reuse ratio drops]
  C --> D[ctime becomes nonzero]
  C --> E[SslBackendKeyRate spikes
if backend TLS]
  C --> F[ephemeral port churn rises]
  F --> G[econ spikes with no
backend-side failure]
  D --> H[request latency grows]
  E --> H
  G --> H

Common causes

Quick checks

All read-only. Run against the stats socket.

# Backend-level connect vs reuse counters
echo "show stat" | socat unix-connect:/var/run/haproxy.sock stdio | \
  awk -F, '$2 == "BACKEND" {print $1": connect="$85" reuse="$86}'

# Per-server idle and in-use pool gauges
echo "show stat" | socat unix-connect:/var/run/haproxy.sock stdio | \
  awk -F, '$2 != "FRONTEND" && $2 != "BACKEND" {print $1"/"$2": idle="$96" safe="$97" used="$98" need_est="$99}'

# Connect time: nonzero on a backend that used to read 0 is the fingerprint
echo "show stat" | socat unix-connect:/var/run/haproxy.sock stdio | \
  awk -F, '$2 == "BACKEND" {print $1": ctime="$60"ms qtime="$59"ms rtime="$61"ms"}'

# Backend TLS handshake rate (only relevant if server lines use ssl)
echo "show info" | socat unix-connect:/var/run/haproxy.sock stdio | \
  grep -E "^(SslBackendKeyRate|SslFrontendKeyRate|Idle_pct):"

# Connection errors: are new connects failing yet?
echo "show stat" | socat unix-connect:/var/run/haproxy.sock stdio | \
  awk -F, '$2 != "FRONTEND" {print $1"/"$2": econ="$14}'

# Ephemeral port and TIME_WAIT pressure on the HAProxy host
ss -s
cat /proc/net/sockstat

# Confirm what the backend actually sends
curl -sv -o /dev/null http://<frontend>/ 2>&1 | grep -i "^< connection"

Field positions are stable in the CSV format but worth confirming against the management documentation for your version before scripting against them.

connect and reuse are cumulative counters that reset on every reload. To compute the ratio you need deltas over an interval, or a monitoring system that handles counter resets.

How to diagnose it

1. Confirm the drop. Compute reuse / (connect + reuse) from deltas over two intervals a few minutes apart, per backend. Compare against your historical baseline. A drop from, say, 80% to 10% with steady request rate is the incident.

2. Localize it. Break the counters down per server. If every server in the backend dropped simultaneously, the cause is backend-wide: an application deploy, a shared config change, or an HAProxy-side directive. If only one server dropped, look at that server’s config or version.

3. Check the timeline against reloads and deploys. A reuse drop exactly at an HAProxy reload points at the HAProxy config (http-reuse, pool-max-conn, server-line changes). A drop at a backend deploy with no HAProxy reload points at the application: keep-alive disabled, HTTP version change, or a framework/proxy in front of the app now emitting Connection: close.

4. Verify the response headers. curl -sv through the frontend and against a backend directly. If the direct response carries Connection: close, HAProxy is behaving correctly: it cannot pool a connection the server refuses to keep alive. The fix belongs on the backend.

5. Check pool state. idle_conn_cur near zero while traffic is flowing means connections are never returned to the pool (server closes them) or the pool is disabled. used_conn_cur high with idle_conn_cur near zero can also mean connections are simply always busy, which is saturation, not broken reuse. Correlate with per-server scur/slim and qcur before concluding pooling is broken.

6. Assess the blast radius. Check SslBackendKeyRate (if backend TLS), ctime, and host-level TIME_WAIT counts. If TIME_WAIT is climbing toward the ephemeral range ceiling, you are on the clock: when ports run out, econ starts and requests fail.

7. For deeper pool introspection, show servers conn <backend> on the runtime API shows per-server idle and used connection detail, including per-thread idle breakdown (idle_per_thr) in recent versions. Use it for live debugging, not for continuous polling.

Metrics and signals to monitor

Fixes

Backend sends Connection: close or disabled keep-alive

This is the most common root cause and it is not fixable in HAProxy. Fix the backend: re-enable keep-alive in the application server or the intermediary (nginx, ingress, framework server) that started closing connections. Verify with curl -sv that responses no longer carry Connection: close. The reuse ratio recovers on its own once connections can be pooled again.

HAProxy config drifted

Set http-reuse explicitly in the backend or defaults section instead of relying on inheritance. safe reuses idle connections only for requests HAProxy considers safe to replay; aggressive and always reuse progressively more aggressively; never disables reuse entirely. Check that nothing set pool-max-conn 0 (which disables the idle pool) and that no server-line or backend change removed pooling. Reload and watch reuse recover within minutes.

Tradeoff: http-reuse always maximizes reuse but changes semantics around which requests may share a connection, and the default value of http-reuse has changed across major versions. Read the configuration manual for your version before switching from safe.

Ephemeral port exhaustion already in progress

This is triage while you fix the root cause:

• Restoring connection reuse is the real fix. Each reused connection is one fewer port in TIME_WAIT.
• Widening net.ipv4.ip_local_port_range and enabling net.ipv4.tcp_tw_reuse buys headroom but does not fix churn. These are sysctl changes applied on the live host; test them during a low-traffic window if you are unsure of the impact on your network path.
• Watch econ: if it is already rising, you are at the cliff, and the connect failure ratio econ / (connect + reuse) tells you how far over.

Pool disabled deliberately and load grew

Some deployments disable reuse to work around a backend bug, then forget. If the workaround predates the current team, re-test with reuse enabled on one backend and watch idle_conn_cur, ctime, and eresp. Old backend bugs that motivated http-reuse never are often long fixed.

Prevention

• Graph the reuse ratio per backend. This is the whole lesson. The counters exist in the stats CSV; almost nobody computes the ratio. Alert on a sustained drop from baseline, not an absolute threshold, since healthy ratios vary by workload.
• Alert on nonzero ctime where it used to be zero. Cheap, high-signal proxy for pooling breakdown.
• Track SslBackendKeyRate on any backend that uses TLS. A spike with steady request rate means handshake overhead returned.
• Monitor host TIME_WAIT and ephemeral range utilization on HAProxy hosts with high backend churn.
• Treat backend connection semantics as a contract. If your platform ships application deploys that can change keep-alive behaviour or HTTP versions, note it in the deploy checklist. The reuse ratio will catch it, but only if someone is watching.
• Set http-reuse explicitly in the config rather than relying on defaults, so a defaults-section edit or version upgrade does not silently change pooling for every backend.

How Netdata helps

• Netdata collects the HAProxy stats CSV fields including connect, reuse, and the per-server pool gauges, so the reuse ratio and idle_conn_cur are graphed per backend and per server without hand-rolled socket polling.
• Counter resets on reload are handled automatically, so the ratio computed from cumulative connect/reuse counters does not produce phantom drops at every config reload.
• ctime, qtime, and rtime are charted alongside the pool signals on the same backend, which makes the “reuse dropped, ctime appeared, latency grew” correlation visible in one view instead of three terminals.
• Backend TLS handshake rate and Idle_pct from show info sit next to the CSV metrics, so CPU cost from lost reuse is attributable to handshakes rather than guessed at.
• Host-level socket and TIME_WAIT metrics from the same agent close the loop on ephemeral port pressure without a separate monitoring stack.

Related guides

• HAProxy connect time (ctime) high: network latency and backend accept-queue overflow
• HAProxy connection errors (econ): backend connections refused, timed out, or reset
• HAProxy 502 Bad Gateway: backend connection and response failures
• HAProxy scur approaching slim: the concurrent-session saturation signal
• HAProxy backend queue building (qcur): requests waiting for a free server slot
• HAProxy 5xx delta: telling HAProxy-generated errors from backend errors