HAProxy maxconn reached: new connections queued and rejected

HAProxy is refusing work. Clients see connection timeouts or resets, load tests fail at a suspiciously round concurrency number, and yet the HAProxy process looks healthy: CPU is fine, memory is fine, the process is up, health checks pass. The stats tell the real story: CurrConns is flat at Maxconn, or scur is pinned at slim on a frontend, and it does not move.

The global or per-frontend maxconn limit has been reached, so HAProxy stops accepting new connections. Those connections pile up in the kernel’s listen backlog, and once the backlog fills, SYNs are dropped silently. From HAProxy’s perspective nothing is wrong, because the connections never reached user space. From the client’s perspective the service is down.

The dangerous property of this failure is that it is a cliff edge, not gradual degradation. HAProxy can serve existing connections at full speed with a healthy Idle_pct while refusing every new connection. If your dashboards only show CPU, request rate, and backend 5xx, this failure is invisible until the tickets arrive.

What this means

HAProxy enforces connection limits at four independent levels: global, frontend, backend, and per-server. When the global or frontend limit is hit, new connections cannot be accepted. They wait in the kernel accept queue (sized by net.core.somaxconn and the SYN backlog). When that queue overflows, the kernel drops SYNs silently. HAProxy never sees them, so no HAProxy error counter increments for the dropped attempts.

flowchart LR
  client[New client connections] --> backlog[Kernel listen backlog]
  backlog -->|accept| haproxy[HAProxy event loop]
  haproxy -->|CurrConns below Maxconn| sessions[Active sessions]
  haproxy -.->|CurrConns == Maxconn: stop accepting| backlog
  backlog -.->|backlog full| drop[SYNs dropped silently]
  drop -.->|client sees| timeout[Connection timeout]

Two things follow. First, the rejection is invisible in HAProxy’s own metrics unless you know where to look (ConnRate vs SessRate gap, kernel ListenOverflows). Second, the fix depends entirely on why the slots are occupied: genuine traffic surge, idle connections holding slots, or a limit set too low for the workload.

Common causes

Quick checks

All read-only. Assumes the runtime socket is at /var/run/haproxy.sock; adjust if your config puts it elsewhere.

# 1. Global saturation: is CurrConns pinned at Maxconn?
echo "show info" | socat unix-connect:/var/run/haproxy.sock stdio | \
  grep -E "^(CurrConns|Maxconn|Maxsock|Ulimit-n|ConnRate|SessRate|Idle_pct|Stopping):"

# 2. Per-proxy saturation: which frontend/backend/server is at its limit?
echo "show stat" | socat unix-connect:/var/run/haproxy.sock stdio | \
  awk -F, '{if($7>0) print $1"/"$2": scur="$5" slim="$7" pct="int($5/$7*100)"%"}'

# 3. Surge or idle pile-up? Session rate and request rate per frontend.
echo "show stat" | socat unix-connect:/var/run/haproxy.sock stdio | \
  awk -F, '$2 == "FRONTEND" {print $1": rate="$34" req_rate="$47}'

# 4. Are connections queueing behind backend/server limits?
echo "show stat" | socat unix-connect:/var/run/haproxy.sock stdio | \
  awk -F, '$2 != "FRONTEND" {print $1"/"$2": qcur="$3" qmax="$4}'

# 5. Kernel accept queue overflowing (connections dropped before HAProxy sees them)?
nstat -az | grep -i listen
ss -tln | grep -E ":(80|443) "

# 6. Who is holding the slots?
echo "show sess" | socat unix-connect:/var/run/haproxy.sock stdio | head -50

# 7. FD headroom on the process itself
HPID=$(pgrep -x haproxy | head -1)
echo "FDs used: $(ls /proc/$HPID/fd 2>/dev/null | wc -l)"
grep "Max open files" /proc/$HPID/limits

How to diagnose it

1. Confirm the cliff. If CurrConns is flat at Maxconn (or a frontend scur is flat at slim) and does not fluctuate, you are at the limit. A flat line at exactly the configured number is the signature. If nothing is at a limit, this is not a maxconn problem; look elsewhere.

2. Identify which level is throttling. Check all four: global (CurrConns/Maxconn in show info), frontend, backend, and per-server (scur/slim per row in the CSV). A per-server limit can bottleneck traffic while the global limit has plenty of headroom, and per-server exhaustion shows up as qcur growth rather than accept stalls.

3. Separate surge from pile-up. This decides the fix:

   • High scur + high req_rate: genuine load. HAProxy is doing real work for every connection. You need more capacity or a higher limit.
   • High scur + low req_rate + low bin/bout + qcur zero: connections are open but not carrying traffic. Idle keep-alives, long-lived streams, a client-side connection leak, or a Slowloris-style pattern. Raising maxconn only buys time.

4. Inspect the sessions. show sess shows what is holding slots: source IPs, connection age, state. A pile-up dominated by one or a few source IPs is a misbehaving client or an attack. A pile-up of aged connections in an idle state points at keep-alive timeouts.

5. Check for the invisible drops. ListenOverflows/ListenDrops incrementing in nstat -az means the kernel backlog is overflowing and SYNs are being dropped before HAProxy can count them. These are system-wide counters, so other listeners contribute, but a rising counter correlated with your incident window is strong evidence.

6. Quantify the rejection rate. The gap between ConnRate and SessRate in show info approximates how many arriving connections never become sessions. In healthy operation the ratio is close to 1.0.

7. Rule out FD exhaustion as the real ceiling. If Maxsock (HAProxy’s computed FD ceiling) is close to Ulimit-n, the practical limit is file descriptors, not the configured maxconn. Each proxied connection needs roughly two FDs (client side plus server side) plus listeners, stats, log sockets, and health checks. Check /proc/<pid>/fd against /proc/<pid>/limits directly.

8. Rule out a reload storm. pgrep -c haproxy should show 1-2 processes (3 in master-worker during a reload). Many PIDs means old processes are draining and still holding connections and FDs, shrinking effective capacity. Check Stopping: 1 and whether hard-stop-after is configured.

Metrics and signals to monitor

Note on Idle_pct: if busy-polling is enabled, Idle_pct sits near zero by design and tells you nothing. Use show activity run-queue depth or system CPU instead.

Fixes

If it is an idle-connection pile-up

Lower keep-alive and client timeouts. Reducing timeout http-keep-alive frees idle keep-alive slots sooner. timeout http-request and timeout client bound how long a connection can sit without completing a request, which is also the primary Slowloris mitigation. Tradeoff: aggressive timeouts can cut off legitimate slow clients on mobile networks or long-polling endpoints.

Identify and constrain the offenders. If show sess shows slot consumption concentrated in a few source IPs, use a stick table to rate-limit connections per source IP. Tradeoff: NAT gateways and corporate proxies legitimately aggregate many users behind one IP.

Fix the client. A client-side connection leak (new connection per request, never closed, or a broken retry loop) will defeat any limit you set. If one client or one user-agent dominates show sess, that is your root cause.

If it is a genuine surge

Scale or shed load. More HAProxy capacity, or reject cheaper at the edge. Raising maxconn alone does not create CPU or backend capacity; if backends are already the constraint, you are just moving the queue.

If maxconn is simply set too low

Raise maxconn with the matching resources. This is not a one-line change:

• File descriptors must follow. Each proxied connection consumes roughly two FDs plus overhead for listeners, stats, logs, and health checks. The process ulimit -n (or the service manager’s LimitNOFILE) must cover maxconn * 2 + overhead with at least 20% headroom. If HAProxy computed its own maxconn from the FD limit at startup, raising maxconn in the config without raising the FD limit will fail or be capped.
• Memory must follow. Budget roughly 32KB per connection for buffers (two times tune.bufsize, default 16KB) plus around 20KB of per-connection metadata. 100k connections is on the order of 3-5GB before stick tables, SSL cache, and certificates.
• Reload, do not restart. A restart drops all existing connections. Use a reload (SIGUSR2 / systemctl reload) so the old process drains.

If the kernel backlog is the visible symptom

Raising net.core.somaxconn and the SYN backlog lets the kernel absorb bursts, but it only deepens the waiting room. If HAProxy is at maxconn, deeper backlogs mean clients wait longer before timing out instead of getting a fast refusal. Treat backlog tuning as burst absorption, not as the fix.

If a reload storm is shrinking capacity

Reduce reload frequency (batch config changes), make sure hard-stop-after is set so draining processes cannot linger on long-lived connections indefinitely, and count HAProxy PIDs as a monitored signal.

Prevention

• Alert on the ratio, not the number. scur/slim and CurrConns/Maxconn as percentages at every level: global, frontend, backend, server. Warning at 75-80%, critical at 90%. Absolute thresholds are meaningless across instance sizes.
• Track the rejection gap. Alert on SessRate/ConnRate falling below baseline and on ListenOverflows/ListenDrops accelerating. These catch the invisible drops that HAProxy’s own error counters miss.
• Baseline the scur vs req_rate relationship. If you know what normal concurrency looks like per request rate, pile-ups stand out immediately.
• Size limits deliberately. Set global, frontend, and per-server maxconn intentionally, with FD and memory headroom documented next to them. An auto-computed or inherited limit you have never looked at is an incident waiting for a traffic spike.
• Set hard-stop-after so reloads cannot strand connections and FDs in draining processes.
• Watch smax trends. If the historical peak creeps toward the limit over weeks, you have a runway problem you can fix on a schedule instead of at 3 a.m.

How Netdata helps

• Netdata collects HAProxy stats over the runtime socket, so scur/slim per proxy and CurrConns/Maxconn globally are graphed continuously. A flat line pinned at the limit is visible at a glance, including which level (global, frontend, backend, server) is throttling.
• Plotting session rate against request rate on the same dashboard makes the surge-vs-pile-up distinction immediate: the two curves diverge when connections stop carrying traffic.
• Correlating HAProxy saturation with node-level network stack metrics surfaces ListenOverflows/ListenDrops rising in the same window, which is how you catch the SYN drops HAProxy itself never counts.
• Ratio-based alerts on scur/slim fire at 80-90% utilization, before clients start timing out, instead of after.
• Per-second collection catches short saturation spikes between scrape intervals that longer-interval polling averages away.
• qcur and qtime alongside scur/slim show whether the limit being hit is admission-side (frontend/global) or server-side, which changes the fix.

Related guides

• HAProxy scur approaching slim: the concurrent-session saturation signal
• HAProxy maxconn hierarchy: global, frontend, backend, and per-server limits
• HAProxy backend queue building (qcur): requests waiting for a free server slot
• HAProxy Slowloris and idle-connection pile-up: high scur, low request rate
• HAProxy monitoring checklist: the signals every production proxy needs
• How HAProxy actually works in production: a mental model for operators
• HAProxy monitoring maturity model: from survival to expert
• HAProxy health check L4TOUT and L4CON: server marked DOWN and unreachable
• HAProxy health check L7STS: server DOWN on the wrong HTTP status
• HAProxy server flapping: rise, fall, and health checks oscillating UP and DOWN
• HAProxy health checks green but the application is broken: when UP does not mean healthy
• HAProxy backend losing servers: active server count and cascade risk