HAProxy Idle_pct dropping: event-loop CPU saturation and the busy-polling trap

Your HAProxy latency is climbing, nothing is queuing on the backends, and the one metric that looks wrong is Idle_pct sliding toward zero. Or worse: Idle_pct has been pinned at zero for weeks and you only just noticed because someone finally asked what it meant. Both situations are common, and they have very different explanations.

Idle_pct is HAProxy’s self-reported measure of event-loop headroom: the share of time the loop spends waiting in poll() versus processing events. It measures HAProxy CPU headroom, not system CPU. A host at 50% system CPU with Idle_pct at 80% is fine. A host at 50% system CPU with Idle_pct at 10% means HAProxy itself is saturated. That distinction is the entire point of the metric, and it is also where the traps live.

What this means

HAProxy multiplexes potentially hundreds of thousands of connections across a small number of threads using the kernel’s most efficient poller (epoll on Linux). When the loop has spare capacity, it spends most of its time blocked in poll() waiting for the next event. When the loop is the bottleneck, it goes straight from one event to the next and poll() returns immediately. Idle_pct is the ratio of polling time to total time: near 100% means almost no load, near 0% means the loop never sleeps.

The degradation curve is gradual then cliff. Latency rises smoothly as idle time shrinks, then spikes sharply below ~5%. Two caveats invalidate most naive alerting on this metric:

1. Busy-polling trap: with busy-polling enabled, the poller always uses a null timeout, so the loop never waits in poll() and Idle_pct sits near zero by design, regardless of actual load. The metric is meaningless in this mode. Use show activity run-queue depth or system CPU instead.
2. Thread skew: Idle_pct is averaged across all threads. One thread at 0% and three threads at 80% average to a comfortable-looking 60% while a single hot thread silently caps your throughput. show activity exposes the per-thread picture.

flowchart TD
    A[Idle_pct low or dropping] --> B{busy-polling enabled?}
    B -->|yes| C[Idle_pct meaningless
use show activity run-queue
and system CPU]
    B -->|no| D{show activity:
per-thread skew?}
    D -->|one hot thread| E[Thread skew
check accept distribution
SO_REUSEPORT, nbthread]
    D -->|all threads loaded| F[Real CPU saturation
find the CPU consumer:
TLS, ACLs, Lua, compression]
    F --> G[SslFrontendKeyRate high?]
    C --> H{run-queue rising?}
    H -->|yes| F
    H -->|no| I[Not actually saturated]

Common causes

Quick checks

All commands are read-only. Adjust the socket path if yours differs.

# 1. Current Idle_pct and the key corroborating counters
echo "show info" | socat unix-connect:/var/run/haproxy.sock stdio | \
  grep -E "^(Idle_pct|Run_queue|Tasks|SslFrontendKeyRate|SslRate|CurrConns|rate):"

# 2. Per-thread activity: hot-thread skew, run-queue depth, CPU steal
echo "show activity" | socat unix-connect:/var/run/haproxy.sock stdio

# 3. Is busy-polling configured? (check before trusting Idle_pct at all)
grep -rn "busy-polling" /etc/haproxy/

# 4. TLS pressure: cache lookups vs misses
echo "show info" | socat unix-connect:/var/run/haproxy.sock stdio | \
  grep -E "^Ssl(Cache|Rate|Frontend)"

# 5. System CPU per thread, to spot one pinned core (thread skew at the OS level)
top -H -p "$(pgrep -x -d, haproxy)"

Notes on the checks:

• Check 3 first. If busy-polling is in the config, stop interpreting Idle_pct entirely. It is ignored by the select and poll pollers, so on those pollers the metric still behaves normally; on epoll (the Linux default) the loop never sleeps by design.
• Check 2 caveat: show activity counters are 32-bit and can wrap on long-running, high-traffic processes. Read them as deltas over short windows, not as lifetime totals. The command exists since HAProxy 1.9; since 2.7r1 the output adds a first column with the total/average for all threads, with per-thread values in square brackets. The individual counters are not documented in the manual; the source code is the authoritative reference for their semantics.
• Check 5 confirms at OS level what show activity shows internally: one haproxy thread pinned at 100% of a core while others idle is thread skew, not global saturation.

How to diagnose it

1. Rule out the busy-polling artifact before anything else. grep -rn "busy-polling" /etc/haproxy/. If it is set, Idle_pct is near zero by design, and any alert on it will page you forever for nothing. Switch to show activity run-queue depth and system CPU as the saturation signals for this instance.

2. Confirm the process is genuinely loaded. show info gives you Idle_pct, Run_queue, and Tasks together. Run_queue is the number of runnable tasks waiting for a thread; it should be near zero in normal operation. Idle_pct below ~20% with Run_queue persistently above zero is real saturation. Low Idle_pct with a flat zero Run_queue is suspicious and points at the busy-polling or measurement artifacts instead.

3. Check for thread skew. Run show activity and compare per-thread counters (loop iterations, accept events, poll-wait timing, run-queue depth). With nbthread > 1, the average in show info can hide one saturated thread, and a single hot thread limits overall throughput even when every other thread has headroom.

4. Identify the CPU consumer. The dominant CPU cost in HAProxy is TLS handshakes. If SslFrontendKeyRate is well above baseline and Idle_pct drops proportionally, TLS is the cause. Check SslCacheLookups vs SslCacheMisses: a high miss rate means most connections pay full handshake cost, often because the session cache is too small or ticket keys rotated. If TLS is not the driver, look at the per-request work: complex ACL or regex evaluation, compression, and Lua scripts all consume event-loop CPU in proportion to req_rate.

5. Corroborate with traffic and latency signals. Real saturation shows up elsewhere: ctime spiking because the loop is too busy to establish backend connections, ConnRate exceeding SessRate (pre-accept losses), and in the kernel, ListenOverflows/ListenDrops in /proc/net/netstat climbing because the loop cannot call accept() fast enough. If Idle_pct is low but every one of these is quiet, question the measurement before you act on it.

6. Decide capacity vs incident. A slow multi-week decline in peak Idle_pct is a capacity problem. A sudden drop correlated with a traffic spike, a config change, or a deploy is an incident: a TLS session-cache failure, a new regex ACL, or a Lua script path being exercised for the first time.

Metrics and signals to monitor

Fixes

TLS handshake storm

Fix session reuse first, because it is the cheapest win: a broken session cache forces every connection through a full handshake, and restoring reuse immediately returns capacity. Check SslCacheMisses / SslCacheLookups; expect a brief miss spike after every reload since the in-process cache is invalidated. Sharing ticket keys on disk across processes mitigates this. If the storm is a genuine surge of new unique clients, scale out or add capacity: handshake cost scales with new connection rate, not total connections.

Expensive per-request CPU work

Profile the config for costly constructs: regex-heavy ACL chains evaluated per request, compression of already-compressed content, and Lua scripts in the request path. The tradeoff is direct: compression trades CPU for bandwidth, and in a CPU-constrained instance disabling it is the correct call. If the load is irreducible, raise nbthread to use spare cores (verify with per-core system CPU that cores are actually free first).

Thread skew

One hot thread means the load is not distributing. SO_REUSEPORT helps distribute accepts across threads; verify nbthread matches the available core count. If you recently raised nbthread, confirm the instance was actually reloaded with the new value.

Busy-polling misconfiguration

If busy-polling was enabled without understanding it, review whether you need it at all. It trades CPU for latency by never letting the poller sleep; it is disabled by default for a reason. The documented operational hazards: improperly bound threads can heavily conflict, producing worse performance and high CPU-stolen values in show info; do not run the process on the same processor as network interrupts; avoid running it on multiple CPU threads sharing the same core. If you keep it, rewrite your Idle_pct alerting to use show activity run-queue depth instead.

Do not reach for a restart as a first move: nothing in this failure mode is fixed by restarting, and a reload resets every counter you were using to diagnose it.

Prevention

• Alert on the right signal for your mode. Idle_pct below ~20% sustained is a ticket-level warning, below ~5% is saturation, but only when busy-polling is off. With busy-polling, alert on show activity run-queue depth and system CPU. Encoding the wrong one guarantees either pages-for-nothing or silence during a real event.
• Trend peak Idle_pct, not averages. The daily peak value is your capacity signal. Plan upgrades when peak drops below ~30%; by the time you are below ~20% at peak you are already in the risk zone.
• Monitor the drivers, not just the symptom. SslFrontendKeyRate against baseline, SslCacheMisses / SslCacheLookups ratio, and req_rate trend tell you why headroom is shrinking before it is gone.
• Watch per-thread distribution on every multi-thread instance. Add show activity to routine collection so skew is visible as it develops, not during the incident. Counters reset on reload and wrap at 32 bits on long-lived processes.
• Treat busy-polling as a documented decision. If you enable it, write down the CPU-affinity constraints (no sharing with network interrupts, no co-locating threads on the same core) in the same change, and pin thread bindings deliberately.

How Netdata helps

• Netdata collects Idle_pct alongside Run_queue, Tasks, and the TLS signals (SslFrontendKeyRate, session-cache lookups/misses) from the HAProxy stats socket at per-second resolution, so a dropping idle percentage is immediately correlated with the driver instead of requiring three manual socket queries during an incident.
• Per-second collection matters specifically here: Idle_pct dips are short, and a low-frequency poller averages away the exact dips that precede latency events.
• Correlating Idle_pct with ctime and SessRate/ConnRate on the same dashboard separates real event-loop saturation (all moving together) from measurement artifacts (idle low, everything else calm).
• System-level per-core CPU from the same agent lets you confirm thread skew (one pinned core) without switching tools.
• Counter-reset handling on reload means your Idle_pct and TLS trends do not phantom-drop every time the config reloads.

Related guides

• HAProxy 502 Bad Gateway: backend connection and response failures
• HAProxy 503 Service Unavailable: no server is available to handle this request
• HAProxy 504 Gateway Timeout: timeout server and the backend timeout cascade
• HAProxy backend losing servers: active server count and cascade risk
• HAProxy backend queue building (qcur): requests waiting for a free server slot
• HAProxy connect time (ctime) high: network latency and backend accept-queue overflow
• HAProxy connection errors (econ): backend connections refused, timed out, or reset
• HAProxy scur approaching slim: the concurrent-session saturation signal
• HAProxy 5xx delta: telling HAProxy-generated errors from backend errors
• HAProxy health checks green but the application is broken: when UP does not mean healthy
• HAProxy health check L4TOUT and L4CON: server marked DOWN and unreachable
• HAProxy health check L7STS: server DOWN on the wrong HTTP status