HAProxy scur approaching slim: the concurrent-session saturation signal

Your alert says scur is at 85% of slim on a frontend, or CurrConns is closing on Maxconn in show info. HAProxy is still up. Traffic is still flowing. But you are running out of concurrent-session headroom, and what happens next depends entirely on which of the four independent limits you are about to hit.

The danger is that the system still appears responsive: HAProxy is alive and healthy, but clients are about to see queuing delays or 503 rejections. The scur/slim ratio is the warning, if you read it correctly.

Two rules matter most: always evaluate this as a ratio, never an absolute count, and always check all four levels, because global, frontend, backend, and per-server limits throttle independently.

What this means

scur is the current number of concurrent sessions. slim is the configured session limit for that row. Both appear in the stats CSV on FRONTEND, BACKEND, and SERVER rows, and the global equivalents (CurrConns and Maxconn) appear in show info.

Each level enforces its own ceiling:

flowchart TD
  A[Global: CurrConns vs Maxconn in show info] --> B[Frontend: scur vs slim]
  B --> C[Backend: scur vs slim]
  C --> D[Server: scur vs slim]
  B -. at limit .-> E[new connections queued or rejected]
  C -. at limit .-> F[requests enter backend queue: qcur rises]
  D -. at limit .-> G[requests queue per-server, other servers take load]
  A -. at limit .-> H[accept stalls, hard ceiling]

Key behaviors per level:

• Global (CurrConns / Maxconn): the hard ceiling for the whole process. Maxconn is set by the maxconn global directive. There is no useful built-in default: when global maxconn is not explicitly set, HAProxy derives its connection ceiling from the build-time SYSTEM_MAXCONN (a minimum of 100 if unset) and the system file descriptor limit (ulimit -n), whichever is lower. A single proxied request consumes two connections (client side and server side), so effective proxying capacity is roughly half of Maxconn minus overhead.
• Frontend (scur / slim): if no frontend maxconn is set, slim inherits the global maxconn, which means all frontends can silently share one limit. A frontend at its limit rejects or queues new connections to that service even when other frontends have headroom.
• Backend (scur / slim): the backend-level limit reflects the fullconn parameter, which has an auto-calculated default based on the sum of frontend maxconn values. When reached, requests enter the backend queue and qcur rises.
• Server (scur / slim): reflects the maxconn parameter on the server line. The default is 0, which means the server inherits the global maxconn as its ceiling rather than having an independent limit. When a server’s limit is reached, requests queue per-server and load shifts to the remaining servers.

One subtlety: scur counts sessions (connections), not requests. With http-reuse and keep-alive, a server at scur = 10 may be handling far more than 10 concurrent requests. With HTTP/2 coalescing to backends, backend scur can look low while frontend scur is high. That is normal multiplexing, not a discrepancy.

Common causes

Quick checks

All read-only, safe to run during an incident.

# Ratio across every row with a nonzero limit (FRONTEND, BACKEND, SERVER)
echo "show stat" | socat unix-connect:/var/run/haproxy.sock stdio | \
  awk -F, '{if($7>0) print $1"/"$2": scur="$5" slim="$7" pct="int($5/$7*100)"%"}'

# Global level
echo "show info" | socat unix-connect:/var/run/haproxy.sock stdio | grep -E "^(CurrConns|Maxconn|Maxsock):"

# Is queuing already happening?
echo "show stat" | socat unix-connect:/var/run/haproxy.sock stdio | \
  awk -F, '$2 != "FRONTEND" {print $1"/"$2": qcur="$3" qmax="$4}'

# Are the sessions doing work, or idle?
echo "show stat" | socat unix-connect:/var/run/haproxy.sock stdio | \
  awk -F, '$2 == "FRONTEND" {print $1": rate="$34" req_rate="$47}'

# Are old processes inflating FD usage during/after reloads?
pgrep -c haproxy

# System-level FD pressure on the active process
HPID=$(pgrep -x haproxy | head -1)
echo "FDs used: $(ls /proc/$HPID/fd 2>/dev/null | wc -l)"
grep 'Max open files' /proc/$HPID/limits

How to diagnose it

1. Identify which level is saturating. Run the ratio command above. If only one SERVER row is pinned, this is a per-server bottleneck. If a FRONTEND row is hot but backends are idle, the frontend limit is the constraint. If CurrConns approaches Maxconn, the global ceiling is near.
2. Decide if the sessions are real work. High scur with high req_rate is a genuine surge: scale or raise limits. High scur with low req_rate means connections are being held without doing work: idle keep-alives, long-lived streams, a connection leak, or a Slowloris-style pattern. Use show sess to inspect what is holding slots.
3. Check whether queuing or rejection has started. Nonzero qcur means requests are already waiting. Rising frontend hrsp_5xx (specifically 503s) means admission is failing. If maxqueue is set to a positive value and the queue is full, additional requests are rejected with 503. With maxqueue at 0 (default), the queue grows unbounded and users wait instead.
4. Rule out the reload artifact. During a reload, old-process sessions still hold system file descriptors but do not appear in the new process’s stats. If you are saturating system FDs while scur looks low, count HAProxy PIDs and check Stopping: 1 on old processes.
5. Find the driver. If backends are slow (rtime rising), connections pile up because each request holds a slot longer. Fixing backend latency releases connection slots faster than any HAProxy-side change.

Metrics and signals to monitor

Alert as ratios. “scur > 1000” is meaningless without knowing slim; “scur > 80% of slim” works on every instance regardless of size. A reasonable escalation ladder: warning at 75-80% (headroom shrinking), critical at 90-95% (queuing imminent or active), page when the ratio exceeds 95% AND qcur > 0 or frontend 503s are actually occurring, sustained for several minutes.

Fixes

Genuine traffic surge

Raise the limit, with eyes open about the cost. Global maxconn requires file descriptor headroom: budget roughly maxconn x 2 + listeners + stats + logs + health checks + peers against ulimit -n, and remember each active connection also consumes memory (up to two buffers of tune.bufsize plus per-session metadata overhead). If the host cannot absorb a higher maxconn, scale out more HAProxy instances instead. Raising the limit past what CPU and memory can serve just moves the saturation signal from scur/slim to Idle_pct.

Slow backends holding connections

The connection pile-up is a symptom, not the disease. Check per-server rtime, find the slow backend or its dependency, and fix that. As a pressure valve, reducing timeout http-keep-alive frees idle keep-alive connections sooner. Reducing per-server maxconn deliberately creates backpressure (queuing at HAProxy) instead of letting slow servers drown in concurrency they cannot handle.

Idle connections and Slowloris patterns

High scur with near-zero req_rate and near-zero throughput is connections consuming slots without doing work. Tighten timeout http-request so clients that never send a complete request are dropped. Use stick-table based per-source-IP rate or connection tracking to identify concentration. Note that WebSocket, SSE, and gRPC streaming legitimately produce this signature; do not tune timeouts blindly on frontends that serve long-lived traffic.

Per-server bottleneck

If one server’s scur pins at slim while global capacity is fine, check whether its maxconn matches what the application can actually serve, and whether sticky sessions or the balancing algorithm are funneling disproportionate traffic to it. If slim is 0 (unlimited, the default), consider setting a per-server maxconn deliberately: an explicit limit with visible queuing is far easier to operate than silent application-side overload.

Reload storms

Old processes in soft-stop hold FDs and memory but their sessions do not count in the new process’s stats, so system-level FD pressure can climb while scur looks calm. Set hard-stop-after so draining processes are force-killed after a bounded window, and reduce reload frequency by batching configuration changes.

Prevention

• Alert on the ratio at all four levels, not just the global one. A per-server or per-frontend limit can bottleneck while CurrConns is at 30% of Maxconn.
• Track smax against slim per frontend and per server. A peak that keeps creeping toward the limit is your capacity runway shrinking.
• Size with the FD and memory math up front. maxconn is not free: two FDs and up to two buffers per connection, plus overhead. Keep at least 20% headroom at both the HAProxy and OS level.
• Plan for failure concentration. After losing one backend server, the survivors should still handle peak traffic without saturating their scur/slim. That is the N+1 test.
• Estimate runway explicitly: time to saturation is roughly (Maxconn - CurrConns) / session arrival rate. Track peak CurrConns over weeks, not just the current value.
• Handle counter resets and reload semantics in your monitoring. Stats reset on reload and old-process sessions are invisible to the new process, so blend HAProxy stats with system-level FD counts.

How Netdata helps

• Ratio, not count: Netdata collects scur and slim per frontend, backend, and server plus global CurrConns/Maxconn, so saturation is visible as a percentage at every throttling level, not just the global one.
• Queue correlation: pairing scur/slim with qcur and qtime on the same dashboard shows the exact moment saturation turned into user-facing latency, which is what separates “tight” from “on fire.”
• Load characterization: session rate, request rate, and bytes in/out alongside scur make it obvious whether a rising ratio is real traffic or idle connections piling up.
• Error confirmation: frontend hrsp_5xx next to the saturation ratio confirms whether admission is actually failing (503s) or just close to it.
• Reload awareness: uptime and process-level visibility help explain the FD-vs-stats discrepancy during reloads, so you do not chase a phantom.
• Per-second granularity: saturation events during bursts are short; per-second collection catches spikes that minute-interval scraping smooths away.

Related guides

• How HAProxy actually works in production: a mental model for operators
• HAProxy monitoring checklist: the signals every production proxy needs
• HAProxy monitoring maturity model: from survival to expert