HAProxy ephemeral port exhaustion: TIME_WAIT and backend connection churn

Your backends are healthy. Every server is UP, health checks pass, the application team sees nothing wrong on their side. Yet HAProxy is logging failed backend connections, econ counters are climbing, and clients see intermittent 502s or stalls. On the HAProxy host, tens of thousands of sockets sit in TIME_WAIT.

This is ephemeral port exhaustion on the backend-facing side of HAProxy. Every new TCP connection from HAProxy to a backend consumes one local ephemeral port. When the connection closes, that port is held in TIME_WAIT for 60 seconds before it can be reused. If the rate of new backend connections is high enough, the port range empties and connect() starts failing with EADDRNOTAVAIL. HAProxy reports these as connection errors even though nothing is wrong with the backend.

The trap is that every signal points at the backend except the one that matters. This guide covers the math behind the cliff, how to confirm the diagnosis in minutes, and the fixes in order of preference.

What this means

A TCP connection is uniquely identified by the four-tuple: source IP, source port, destination IP, destination port. From the HAProxy host, connections to a given backend share three of those four values: HAProxy’s source IP and the backend’s IP:port. The only varying component is the source port, drawn from net.ipv4.ip_local_port_range.

The default range on Linux is 32768 to 60999, roughly 28,000 ports. When HAProxy closes a backend connection (which it does on every request if connection reuse is not working), the socket enters TIME_WAIT for a fixed 60 seconds. This duration is hardcoded in the kernel (TCP_TIMEWAIT_LEN) and is not tunable.

That gives a hard ceiling on sustained new connections to a single backend IP:port:

• ~28,000 ports / 60 seconds = ~450 to 470 new connections per second, sustained.
• Below that rate, ports return to the pool as fast as you consume them.
• Above that rate, the pool drains. Once no free port exists for the four-tuple, connect() fails immediately with EADDRNOTAVAIL.

This is a cliff edge, not gradual degradation. Everything works at 440 new connections per second and breaks at 470. Near the boundary the failures are intermittent because ports free up in a rolling window, which makes the symptom look like flaky networking.

flowchart TD
  A[High request rate] --> B[No backend connection reuse]
  B --> C[New TCP connection per request]
  C --> D[Socket closed: enters TIME_WAIT for 60s]
  D --> E{New conn rate > ports / 60s?}
  E -- No --> F[Steady state, ports recycle]
  E -- Yes --> G[Ephemeral port pool drains]
  G --> H[connect fails: EADDRNOTAVAIL]
  H --> I[econ spikes, 502s to clients]
  I -. misleading .-> J[Backends look healthy, no backend-side failure]

Things that shrink the ceiling further:

• Fewer destination four-tuples. The port pool is per destination IP:port. One server on one port means all churn hits one 28K-port budget. Ten servers on distinct IPs each get their own budget.
• Health checks. TCP health checks also open and close connections from HAProxy to each server. Frequent checks across many servers add their own TIME_WAIT accumulation on top of traffic.
• A narrowed port range. Some hardening guides suggest shrinking ip_local_port_range. If yours was narrowed, the ceiling drops proportionally.

Common causes

Quick checks

All read-only, safe to run during an incident.

# Socket summary on the HAProxy host
ss -s

# Count TIME_WAIT sockets directly
ss -tan state time-wait | wc -l

# Check the ephemeral port range
sysctl net.ipv4.ip_local_port_range

# More detail: sockets in use, TIME_WAIT, orphan count
cat /proc/net/sockstat

The stats CSV columns shift between HAProxy versions, so resolve them from the header instead of hardcoding column numbers:

# econ (connection errors) per backend/server
echo "show stat" | socat unix-connect:/var/run/haproxy.sock stdio | awk -F, '
  NR==1 { for (i=1;i<=NF;i++) { gsub(/[# ]/,"",$i); h[$i]=i } next }
  $2 != "FRONTEND" { print $1"/"$2": econ="$(h["econ"]) }'

# New-connection vs reused-connection counters per backend
echo "show stat" | socat unix-connect:/var/run/haproxy.sock stdio | awk -F, '
  NR==1 { for (i=1;i<=NF;i++) { gsub(/[# ]/,"",$i); h[$i]=i } next }
  $2 == "BACKEND" { print $1": connect="$(h["connect"])" reuse="$(h["reuse"]) }'

# Idle pooled and in-use connections per server
echo "show stat" | socat unix-connect:/var/run/haproxy.sock stdio | awk -F, '
  NR==1 { for (i=1;i<=NF;i++) { gsub(/[# ]/,"",$i); h[$i]=i } next }
  $2 != "FRONTEND" && $2 != "BACKEND" { print $1"/"$2": idle="$(h["idle_conn_cur"])" used="$(h["used_conn_cur"]) }'

The connect, reuse, and idle_conn_cur fields were introduced with the connection pool in HAProxy 1.9; if the header lacks them, use show servers conn below instead.

# Current and idle backend connections per server (HAProxy 2.2+)
echo "show servers conn" | socat unix-connect:/var/run/haproxy.sock stdio

The one-line diagnosis: high and growing TIME_WAIT on the HAProxy host, plus a connect counter rate close to the request rate, plus econ incrementing, while every backend health check is green. That combination is ephemeral port exhaustion until proven otherwise.

How to diagnose it

1. Confirm the econ spike is real and scope it. Pull econ per backend and per server from the stats socket twice, 60 seconds apart, and compute the rate. Note whether the errors hit all servers in a backend equally (port exhaustion affects the host, so it hits every destination) or just one server (more likely a server-side problem).

2. Check the backend side for absence of failure. Backend servers show no connection spikes, no SYN backlog overflow, no application errors. Econ spikes with no backend-side failure are the signature of this failure mode. If backends also show distress, you likely have a different problem; see the related guide on connection errors.

3. Count TIME_WAIT on the HAProxy host. ss -s shows the total. If it is in the tens of thousands and tracks your traffic rate, you are at or near the ceiling. Compare against the port range from sysctl net.ipv4.ip_local_port_range; the range size is your budget.

4. Compute the churn rate. Take two samples of the backend connect counter 60 seconds apart. The delta divided by 60 is your sustained new-backend-connection rate. Compare against range_size / 60. If the ratio is above roughly 0.7, you are one traffic spike from the cliff.

5. Check whether reuse is working at all. The reuse ratio is reuse / (connect + reuse). If connect dominates, every request pays a new TCP connection. Look for why: no http-reuse directive, backends closing connections, or HTTP/1.0-style responses. Also check ctime: with working reuse it averages near zero because most requests skip connection setup. A sudden appearance of nonzero ctime at constant traffic means reuse broke down.

6. Rule out health check churn. If TIME_WAIT is high but traffic churn is low, count your health checks: number of servers multiplied by check frequency. TCP checks closed from the HAProxy side leave TIME_WAIT sockets on the HAProxy host. High-frequency tcp-check across many servers can generate thousands of TIME_WAIT sockets by itself.

7. Check HAProxy logs for source bind failures. When the port pool is dry, HAProxy logs a failure to bind a source address for the outgoing connect. The log message is Cannot bind to source address before connect(). If you see that, the diagnosis is confirmed.

One version-specific hazard: on some older HAProxy versions (the report was 2.0.14 on kernel 4.18), port exhaustion has been observed to trigger a pathological state where HAProxy stops closing file descriptors for dead connections, accumulating tens of thousands of CLOSE_WAIT sockets (45,329 in the reported case) and pinning all cores at 100% CPU (upstream issue #579). If CLOSE_WAIT piles up alongside TIME_WAIT during an exhaustion event, check your HAProxy version against that issue.

Metrics and signals to monitor

Fixes

In order of preference. The first fix removes the root cause; the others raise the ceiling.

Enable backend connection reuse with http-reuse

This is the real fix. If connections are reused, they never close, never enter TIME_WAIT, and never consume a fresh port. One persistent connection per active request stream replaces the per-request connect/close cycle.

In HAProxy 3.x, http-reuse defaults to safe, which reuses connections conservatively. For backends on a stable path (typical reverse-proxy-to-application setups), http-reuse always is the recommended mode and maximizes reuse:

backend app_servers
    http-reuse always
    server app1 10.0.1.11:8080 check maxconn 200
    server app2 10.0.1.12:8080 check maxconn 200

Connection pooling is bounded by pool-max-conn (default -1, meaning unlimited) and pool-purge-delay (default 5s).

If your backends are why reuse fails (they send Connection: close, speak HTTP/1.0, or keepalive is disabled), fix the backend. Reuse is a leading indicator of this class of drift: when a backend change silently kills keepalive, the reuse ratio drops and churn returns before anyone notices why.

Note: connection reuse has had version-specific bugs. http-reuse always was reported unreliable on HAProxy 2.2.9, 2.2.19, and 2.4.9 (upstream issue #1472). Verify on your version with the connect/reuse counters after enabling it, not just from the config.

Widen the ephemeral port range

This raises the ceiling but does not remove the churn:

# Inspect current range
sysctl net.ipv4.ip_local_port_range

# Widen it (runtime, does not survive reboot)
sysctl -w net.ipv4.ip_local_port_range="10240 65535"

Persist it in /etc/sysctl.conf or a drop-in under /etc/sysctl.d/. Widening from ~28K to ~55K ports roughly doubles the sustained ceiling to ~900 new connections per second per destination. Mind the lower bound: do not overlap ports your own listeners or other local services bind.

Add source IPs or let HAProxy manage port ranges

The port budget is per four-tuple, so more source IPs multiply the budget:

• Bind server lines to multiple local source IPs, spreading connections across several source-IP budgets.
• Use source <ip>:<port_low>-<port_high> on server lines so HAProxy manages port allocation itself across a defined range, which can use the range more efficiently than the kernel allocator.
• source 0.0.0.0 usesrc clientip preserves client IPs as source addresses, which also spreads four-tuples, but requires the backend network to route client-IP return traffic back through HAProxy (typically TPROXY or HAProxy as the default gateway). Only use this if that network setup already exists.

tcp_tw_reuse, cautiously

net.ipv4.tcp_tw_reuse lets the kernel reuse a TIME_WAIT socket for a new outbound connection when TCP timestamps prove it safe. Points to know:

• On modern kernels the default is 2 (loopback only). Setting it to 1 enables it globally for outbound connections. HAProxy’s backend connections are outbound (client role), which is the side where this is considered relatively safe.
• It only works when TCP timestamps are enabled on both ends, which is the Linux default.
• Do not confuse it with tcp_tw_recycle, which was removed from the kernel entirely in Linux 4.12 because it broke clients behind NAT. Any guide recommending tcp_tw_recycle is outdated and dangerous.

Treat tcp_tw_reuse=1 as a mitigation while you roll out http-reuse, not as the primary fix.

Reduce health check churn

If health checks are a meaningful share of TIME_WAIT:

• HAProxy 3.2 added check-reuse-pool, which lets health checks draw from the idle connection pool instead of opening a fresh connection each cycle.
• HTTP health checks over reused connections generate less churn than raw TCP checks that connect and close.
• If check frequency is higher than operationally needed, lowering it directly lowers churn.

Prevention

• Monitor the churn rate as a leading indicator. Alert when the backend connect rate exceeds ~70% of port_range / 60 sustained. This fires before the cliff.
• Alert on reuse ratio drops. reuse / (connect + reuse) falling after a deploy is the early warning that a backend change killed keepalive. Most teams never watch this ratio and discover the problem at the port-exhaustion cliff instead.
• Track system TIME_WAIT. Collect ss -s or /proc/net/sockstat on HAProxy hosts. TIME_WAIT tracking traffic linearly is the slow-burn version of this incident.
• Load test the ceiling, not just the app. Include the HAProxy host’s per-destination connection ceiling in capacity models. The app may handle 2,000 req/s while the port budget caps at 450 conn/s per server without reuse.
• Set sane per-server maxconn. It will not prevent port exhaustion by itself, but bounding concurrency per server bounds how fast connections cycle.

How Netdata helps

• Backend connect vs reuse rates per backend and server, collected every second, so you see the churn rate and the reuse ratio directly instead of deriving them from two manual socat samples during an incident.
• econ per server in real time, letting you confirm the “errors on HAProxy, silence on backends” signature in one view instead of cross-checking two systems by hand.
• System-level socket stats from the HAProxy host, including TIME_WAIT counts, correlated on the same dashboard as HAProxy’s connect rate, which is the correlation that confirms this diagnosis.
• ctime trending, where a shift from near-zero to nonzero tells you connection reuse broke down before ports run out.
• Anomaly detection on the connect counter rate, catching the slow traffic-growth drift toward the ~450/s-per-server ceiling weeks before it becomes an incident.

Related guides

• HAProxy connection errors (econ): backend connections refused, timed out, or reset
• HAProxy connect time (ctime) high: network latency and backend accept-queue overflow
• HAProxy 502 Bad Gateway: backend connection and response failures
• HAProxy 5xx delta: telling HAProxy-generated errors from backend errors
• HAProxy backend queue building (qcur): requests waiting for a free server slot
• HAProxy scur approaching slim: the concurrent-session saturation signal
• HAProxy health check L4TOUT and L4CON: server marked DOWN and unreachable