HAProxy backend losing servers: active server count and cascade risk

A backend can lose half its servers and HAProxy will still report the BACKEND aggregate row as UP. As long as one server is available, the rollup status says traffic is being served, and status-based dashboards stay green. Meanwhile the surviving servers are absorbing redistributed load, their session counts are climbing, and you are one health check failure away from a full 503 outage for that backend.

This is the leading edge of the backend collapse cascade: initial server failure, traffic redistribution, survivor overload, survivor health check failures, more redistribution, total collapse. The operators who catch it early are watching the active server count and the load on survivors, not the aggregate status.

This article covers how to read the act/bck counts and per-state server rows, how to judge whether a partial loss is dangerous for your specific pool, and what to do before the cascade completes.

What this means

On the BACKEND row of show stat, two fields describe how much of the pool is actually serving traffic:

• act (CSV field 19): number of active servers currently UP in the backend.
• bck (CSV field 20): number of backup servers currently UP.

Both only count servers HAProxy considers available. A server in DOWN, MAINT, or DRAIN does not appear in either number. To get the full picture, including how many servers are DOWN versus intentionally removed, you have to count the individual server rows by their status field. That distinction matters operationally: a backend with 6 configured servers showing act=3 is a very different situation if the 3 missing servers are DOWN (failure) versus MAINT (planned work someone forgot to close out).

The cascade mechanism works like this: servers go DOWN, HAProxy redistributes their traffic to survivors, survivors slow down or hit their per-server maxconn, their health checks start timing out or the queue grows, more servers get marked DOWN, and the loop accelerates. The BACKEND status only flips to DOWN when the last available server goes, at which point every request to that backend gets a 503 (or a TCP reset in TCP mode). Everything before that flip is invisible if you only alert on status.

flowchart TD
  A[Server fails health check] --> B[HAProxy marks it DOWN]
  B --> C[act decreases, traffic redistributes]
  C --> D[Survivor scur rises, rtime increases]
  D --> E{Survivors overloaded?}
  E -- no --> F[Degraded but stable: fix the failed server]
  E -- yes --> G[qcur grows, survivor checks flap]
  G --> H[More servers marked DOWN]
  H --> C
  H --> I[act reaches 0: BACKEND DOWN, 503 for all requests]

The key decision point is “survivors overloaded?”. Losing 50% of a pool that was running at 20% utilization is a ticket. Losing 50% of a pool that was running at 70% utilization is the first move of an outage.

Common causes

Quick checks

All of these are read-only and safe to run during an incident. They assume the admin stats socket at /var/run/haproxy.sock; adjust the path for your deployment. The field numbers below use HAProxy’s 0-indexed CSV positions, so act (field 19) is $20 in awk.

# Active and backup server counts per backend
echo "show stat" | socat unix-connect:/var/run/haproxy.sock stdio | \
  awk -F, '$2 == "BACKEND" {print $1": active="$20" backup="$21}'

This is the fastest single check. Compare active against how many servers the backend should have. If you do not know the expected count, the next command gives you the full breakdown.

# Per-state server counts per backend (UP, DOWN, MAINT, DRAIN, transitions)
echo "show stat" | socat unix-connect:/var/run/haproxy.sock stdio | \
  awk -F, '$2 != "FRONTEND" && $2 != "BACKEND" {backends[$1]++; states[$1","$18]++} END {for (b in backends) {printf "%s: total=%d", b, backends[b]; for (k in states) {split(k,a,","); if(a[1]==b) printf " %s=%d", a[2], states[k]}; print ""}}'

# Individual server status, check failures, and last check result
echo "show stat" | socat unix-connect:/var/run/haproxy.sock stdio | \
  awk -F, '$2 != "FRONTEND" && $2 != "BACKEND" {print $1"/"$2": status="$18" chkfail="$22" chkdown="$23" lastchg="$24"s check_status="$37}'

check_status tells you why a server is DOWN: L4TOUT (TCP timeout), L4CON (connection refused), L7STS (wrong HTTP status), L7TOUT (response timeout). lastchg is seconds since the last status change, which tells you whether this is a fresh failure or a long-standing one.

# Are the survivors overloaded? Per-server session utilization
echo "show stat" | socat unix-connect:/var/run/haproxy.sock stdio | \
  awk -F, '$2 != "FRONTEND" && $2 != "BACKEND" {if($7>0) print $1"/"$2": scur="$5" slim="$7" pct="int($5/$7*100)"%"}'

# Queue and latency on the backend
echo "show stat" | socat unix-connect:/var/run/haproxy.sock stdio | \
  awk -F, '$2 == "BACKEND" {print $1": qcur="$3" qmax="$4}'

# Retries and redispatches: is HAProxy masking instability?
echo "show stat" | socat unix-connect:/var/run/haproxy.sock stdio | \
  awk -F, '$2 == "BACKEND" {print $1": wretr="$16" wredis="$17}'

How to diagnose it

1. Establish expected versus actual server count. Run the act/bck check and the per-state count. You need three numbers: total configured servers, servers in each state, and how many are actually serving. If the gap is MAINT or DRAIN, find out who put them there and when, because forgotten maintenance is a capacity risk, not a failure.

2. Classify the failure from check_status. L4TOUT/L4CON means HAProxy cannot reach the server at the TCP level (network, dead process, firewall). L7STS means the server is reachable but the health endpoint returns the wrong status (application-level). L7TOUT means the server accepts connections but responds too slowly, often the first sign of overload rather than a hard failure.

3. Check the timing pattern. Simultaneous chkdown across servers with recent lastchg points at a shared cause: network partition, shared dependency, or a deploy. Staggered failures in sequence point at a cascade already in progress.

4. Measure survivor load. Pull per-server scur/slim for the remaining UP servers, plus backend qcur and qtime. This is the step that determines severity. If survivors are above roughly 80% of their session limit, or qcur is nonzero and growing, the backend is at capacity and the next failure is the outage.

5. Check whether HAProxy is already compensating. Rising wretr/wredis means connections to servers are failing and being retried or redispatched. Users may see nothing yet, but the retry counters tell you the pool is unstable before errors become visible. High retries with rising 5xx means compensation is failing.

6. Find the first failure. In a cascade, the first server that went DOWN usually holds the root cause. The later failures are often just overload. Compare lastchg values to identify which server failed first, and investigate that server’s application and its dependencies.

7. Confirm or rule out the shared dependency. If rtime was climbing on all servers before the first failure, suspect the database, cache, or downstream API rather than the servers themselves. Servers drowning in slow dependency responses will fail health checks (L7TOUT) even though their own process is healthy.

Metrics and signals to monitor

The escalation rule of thumb: fewer than 50% of non-MAINT/DRAIN servers active is high cascade risk if survivors are loaded, and one server remaining means any single failure is a total outage. But the 50% line is not universal. An overprovisioned pool running at 15% utilization can lose half its servers and be boring. Always pair the count with survivor scur/slim before deciding severity.

Fixes

Restore the failed servers

The correct fix is almost always to bring the failed servers back, not to route around their absence. Work from the check_status classification: TCP-level failures get network and process investigation on the server; L7STS gets application investigation. If a deploy is in progress and servers are failing as they receive new code, roll back first and diagnose after.

Stop flapping servers from churning the pool

If a server is oscillating UP/DOWN, each transition redistributes traffic and each recovery dumps load back onto a server that may not be ready. Put genuinely broken servers into MAINT via the runtime API (set server <backend>/<server> state maint) so the pool stabilizes while you investigate. This is a deliberate, reversible operator action that takes the server out of rotation immediately; document who did it and why, because MAINT servers are invisible to DOWN alerts. Restore with set server <backend>/<server> state ready.

Relieve the survivors while you work

If survivors are near saturation, reduce their load rather than adding more: shed nonessential traffic at the frontend if you have ACL-based controls, or shift traffic to another backend or site if one exists. Do not raise per-server maxconn as a reflex; the per-server limit exists to protect the application, and raising it on already-overloaded servers converts queuing at HAProxy into collapse at the application.

Use backup servers and capacity-aware routing

If your backends define backup servers, HAProxy uses them only when all active servers are DOWN by default; option allbackups changes that to use all backups simultaneously. For earlier, capacity-based failover, the nbsrv(<backend>) sample fetch returns the number of available servers and can drive an ACL that switches traffic before the pool empties, for example routing to an overflow backend when nbsrv falls to 2 or below. Both patterns are described in the official HAProxy documentation and blog; test them before you need them, and note that stick-table persistence can keep sessions pinned to a backup server even after the active servers recover.

The all-down hazard

When every server in a backend is DOWN, HAProxy returns 503s, but the frontend keeps accepting connections and sessions can pile up in queues and connection state. A known issue (haproxy/haproxy#1946) documents severe CPU spikes under high connection volume in exactly this state, recovering as soon as one server comes back. If a backend is fully down under load, consider draining traffic away from the frontend, not just waiting for servers to return.

Prevention

• Alert on the count, not the status. BACKEND status UP/DOWN alone misses everything before total outage. Alert when act drops below a defined floor per backend, and page when the composite condition holds: more than 50% of non-MAINT/DRAIN servers DOWN, survivors overloaded (scur/slim > 80% or qcur > 0), persisting for more than a couple of minutes.
• Size for N+1 at minimum. The capacity question to answer per backend: with one server removed, can the rest serve peak traffic without queuing? If not, your redundancy is nominal.
• Set per-server maxconn deliberately. It is what turns survivor overload into visible queuing (qcur) instead of silent application collapse, and it makes the cascade detectable in HAProxy’s own metrics.
• Make health checks verify the real application. TCP-only checks or trivial /health endpoints let broken servers stay in rotation, which makes server count changes abrupt when they finally happen. Checks that exercise real dependencies fail earlier and more informatively.
• Track MAINT and DRAIN duration. Servers parked in MAINT for weeks silently shrink every pool they belong to. Audit them on a schedule.
• Watch retries as the early-warning channel. A backend whose wretr/wredis counters climb for days before a server finally goes DOWN gave you the warning; most teams just never look.

How Netdata helps

• Netdata collects the HAProxy stats CSV continuously, so act/bck per backend and per-server UP/DOWN state become time series rather than point-in-time show stat snapshots. You can see the pool eroding, not just discover it after the fact.
• Per-server scur/slim, qcur, and rtime are charted alongside the server counts, which makes the key correlation (count down, survivor load up) a single glance instead of two manual queries during an incident.
• Retry and redispatch counters (wretr/wredis) are surfaced as rates, so the “HAProxy is masking instability” phase of a cascade is visible in a dashboard rather than only in a runtime socket dump.
• Health check transition signals (chkdown, chkfail, status changes) let you reconstruct the failure sequence during post-incident review: which server went first, and how long the cascade took.
• Alerting on ratios (survivors above a percentage of their session limit, active servers below a fraction of the pool) matches how cascade risk actually behaves, and Netdata’s per-second collection catches the fast transitions that minute-resolution polling misses.

Related guides

• HAProxy backend queue building (qcur): requests waiting for a free server slot
• HAProxy scur approaching slim: the concurrent-session saturation signal
• HAProxy health check L4TOUT and L4CON: server marked DOWN and unreachable
• HAProxy health check L7STS: server DOWN on the wrong HTTP status
• How HAProxy actually works in production: a mental model for operators
• HAProxy maxconn hierarchy: global, frontend, backend, and per-server limits
• HAProxy maxconn reached: new connections queued and rejected
• HAProxy monitoring checklist: the signals every production proxy needs
• HAProxy monitoring maturity model: from survival to expert
• HAProxy Slowloris and idle-connection pile-up: high scur, low request rate