HAProxy monitoring checklist: the signals every production proxy needs

Most HAProxy outages are visible in its own stats long before users notice. The problem is that the stats socket exposes dozens of fields, and teams usually wire up the five metrics their dashboard template shipped with and stop there. Then a reload storm, a retry storm, or a silent stick-table overflow teaches them what they were missing.

This checklist organizes the signals that catch production failures into four maturity levels: survival, operational, mature, and expert. Each level builds on the previous one. You do not need to reach expert on day one, but you should know which level you are at, because that determines which failure modes are invisible to you.

Everything here is collectable from the runtime API (stats socket), the CSV stats, or the host OS. No agents inside HAProxy are required.

How to use this checklist

Work down the levels in order. For each signal, confirm three things: you are collecting it, you are alerting on the right condition, and you have verified the alert fires (reload counters reset, thresholds tuned as ratios, not absolutes).

The canonical collection pattern:

# Process and event-loop liveness
echo "show info" | socat unix-connect:/var/run/haproxy.sock stdio | head -1

# CSV stats: per-proxy rows (FRONTEND, BACKEND, and per-server)
echo "show stat" | socat unix-connect:/var/run/haproxy.sock stdio

# Loaded certificate list (HAProxy does not expose expiry; check cert files on disk)
echo "show ssl cert" | socat unix-connect:/var/run/haproxy.sock stdio

# FD usage vs limit for the running worker
HPID=$(pgrep -x haproxy | head -1)
echo "FDs used: $(ls /proc/$HPID/fd 2>/dev/null | wc -l)"
grep 'Max open files' /proc/$HPID/limits

The stats socket path varies by deployment (/var/run/haproxy.sock is a common default, not a guarantee). All socket commands above are read-only and safe on a production process. Do not run runtime commands that change state (disable server, set maxconn, shutdown session) as part of a monitoring check.

flowchart TD
  L1["Level 1 - Survival: is it up and serving?"] --> L2["Level 2 - Operational: is it healthy?"]
  L2 --> L3["Level 3 - Mature: is it degrading?"]
  L3 --> L4["Level 4 - Expert: what is it hiding?"]

Level 1: survival

The absolute minimum. If any of these fire, users are already affected.

Two liveness subtleties that bite in practice:

• Master-worker mode. The master process stays alive even while workers restart. Monitor the worker, not the master. After a reload, multiple PIDs are normal; only one is the active worker.
• Cold start gating. Gate availability alerts on Uptime_sec so restarts and reloads do not page you. A backend that is DOWN during the first minutes of process life may just be waiting for health checks to pass their rise threshold.

Level 2: operational

What a competent production team should have. This is where you start detecting problems before users do.

Three things to internalize at this level:

• wretr/wredis are pre-failure signals. High retries with zero user-visible errors means the compensation layer is working. High retries with rising errors means it just stopped working. The outage feels sudden only if you were not watching the retries.
• qtime is an average; qcur is now. A brief queue burst can be diluted across the 1024-sample window and never show in qtime. Alert on qcur for real-time visibility, use qtime for latency budgeting.
• maxconn is a hierarchy, not a number. Limits are enforced independently at global, frontend, backend, and per-server levels. A per-server slim can bottleneck a backend while the global limit has plenty of headroom. Monitor all four.

Level 3: mature

Full coverage of internals and the distinctions that determine where you look first during an incident.

The busy-polling caveat is mandatory reading at this level: when busy-polling is enabled, Idle_pct sits near zero by design because the event loop never sleeps. Alerting on Idle_pct in that mode pages you forever. Use show activity run-queue depth or system CPU instead. Similarly, Idle_pct is an average across threads; one hot thread can hide inside a healthy average, which is what show activity per-thread counters are for.

On certificates: expect a brief SslCacheMisses spike after every reload, because the in-process session cache is invalidated. Do not alert on the transient.

Level 4: expert

The signals teams add after learning the hard way.

• Kernel accept queue overflow. ListenOverflows and ListenDrops in /proc/net/netstat (or nstat -az). When the kernel queue overflows, SYNs are dropped silently. HAProxy sees nothing because the connections never reached user space. Client timeouts with perfectly healthy HAProxy metrics is the signature. Check the ConnRate vs SessRate gap in show info as a secondary hint.
• Per-thread skew. show activity exposes per-thread counters (introduced in HAProxy 1.9). With nbthread > 1, one saturated thread caps total throughput while average Idle_pct looks fine.
• Connection pool utilization. CSV connect vs reuse on BACKEND rows, plus idle_conn_cur. A sudden drop in the reuse ratio means something broke pooling (backend started sending Connection: close, HTTP version change). It shows up as rising ctime and SslBackendKeyRate before anyone notices the cause.
• Stick-table utilization. show table (not in CSV). Eviction under default settings is silent; with nopurge, new entries are rejected. Either way, rate limiting and persistence quietly stop working for affected clients. Above 80% used deserves a ticket.
• DNS resolver health. show resolvers for server-template / service-discovery deployments. Resolver failures silently route to stale IPs while server status may still read UP on cached results.
• Peer replication health. show peers. Broken peer sync means stick tables diverge: rate limiting works on one node and not the other.
• Allocator pressure. PoolFailed in show info, detail via show pools. Normally zero; any nonzero value means HAProxy failed to allocate internal objects and dropped work.
• Dropped logs. DroppedLogs in show info. Losing logs during an incident cripples the investigation. UDP syslog drops are silent at the sender; this counter is the only local signal.
• Tail latency from logs. qtime/ctime/rtime/ttime are averages over 1024 samples. P99 requires parsing per-request timing fields from HAProxy logs. A backend failing fast shows low rtime with rising 5xx, which looks great on a latency dashboard.

Rules that apply at every level

• Alert on ratios, not absolutes. scur > 1000 is meaningless without knowing slim. scur > 80% of slim works on every instance regardless of size. Same for 5xx: express it as a fraction of req_rate.
• Handle counter resets. Every cumulative CSV counter (hrsp_5xx, econ, wretr, bin…) resets to zero on reload. Use Uptime_sec drops or the Stopping flag to detect reloads, and make sure your monitoring treats the reset as a reload, not an incident.
• Health checks are not traffic. A server can pass health checks while returning 500 on every real request (the health endpoint is fine, the app is not). Always pair server status with per-server hrsp_5xx. If frontend 5xx approximately equals backend 5xx while all servers are UP, the application is broken, not HAProxy.
• Do not confuse session rate with request rate. With keep-alive and HTTP/2 multiplexing, one session carries many requests. req_rate / rate is the reuse ratio; a drop toward 1:1 means every request now pays full connection setup cost.

How Netdata helps

Netdata’s HAProxy collector scrapes the stats socket and CSV stats at per-second granularity, which matters specifically for the signals in this checklist:

• Per-proxy and per-server charts for scur/slim, qcur, and status, so the maxconn hierarchy and per-server saturation are visible instead of averaged away.
• Frontend vs backend 5xx side by side, making the HAProxy-generated error delta a visual comparison rather than a manual computation.
• wretr/wredis and econ/eresp as first-class series, so retry-masked instability and protocol-level failures show up before they become user-visible errors.
• Idle_pct trending next to SslFrontendKeyRate, which is the correlation that confirms (or rules out) TLS as the CPU consumer.
• Counter-reset-aware rate computation, so reloads do not produce phantom 5xx drops or spikes.
• Host-level correlation with FD usage, /proc/net/netstat listen overflows, and process counts, covering the kernel-side and reload-storm signals the CSV stats cannot see.

Related guides

• How HAProxy actually works in production: a mental model for operators
• HAProxy monitoring maturity model: from survival to expert
• HAProxy maxconn reached: new connections queued and rejected
• HAProxy scur approaching slim: the concurrent-session saturation signal
• HAProxy maxconn hierarchy: global, frontend, backend, and per-server limits
• HAProxy backend queue building (qcur): requests waiting for a free server slot
• HAProxy Slowloris and idle-connection pile-up: high scur, low request rate
• HAProxy health check L4TOUT and L4CON: server marked DOWN and unreachable
• HAProxy health check L7STS: server DOWN on the wrong HTTP status
• HAProxy server flapping: rise, fall, and health checks oscillating UP and DOWN
• HAProxy health checks green but the application is broken: when UP does not mean healthy
• HAProxy backend losing servers: active server count and cascade risk