HAProxy PoolFailed: buffer starvation and memory-allocator failures

PoolFailed in show info counts failed internal memory-pool allocations since process start. It is normally zero. Any nonzero value means HAProxy tried to allocate a buffer, connection, or session object and could not get one. Work was dropped or stalled because of it.

The tricky part is how this surfaces. Buffer starvation rarely looks like a memory problem from the outside. It looks like mysterious latency: rtime and backend health are fine, CPU has headroom, the network is clean, and yet clients see stalls and sporadic failures. PoolFailed is one of the few direct signals that HAProxy itself is refusing work because it ran out of an internal resource.

What this means

HAProxy pre-allocates memory in pools for high-frequency objects: connection buffers (two per connection, request and response, each sized by tune.bufsize, default 16384 bytes), sessions, connections, and related metadata (roughly 20KB of overhead per connection on top of the buffers). These pools sit between HAProxy and the system allocator so the hot path does not hit malloc() on every event.

A pool allocation can fail for three broad reasons:

1. The buffers are all checked out. Under extreme concurrency with large payloads, every buffer in the pool is in use. New or existing connections stall waiting for one to be released.
2. A configured memory limit is reached. If the -m <MB> startup flag is set, HAProxy stops allocating at that ceiling (shown as Memmax_MB in show info) even if the host has free RAM.
3. The system cannot give HAProxy more memory. Host-level memory pressure means the underlying allocation fails. Left unchecked, the kernel OOM killer is the next step. HAProxy has no built-in memory limit unless one is set at startup with -m.

The failure cascade:

flowchart TD
  A[Traffic surge or large payloads] --> B[All pool buffers checked out]
  B --> C{More memory available?}
  C -->|No: Memmax reached or host OOM pressure| D[Allocation fails - PoolFailed increments]
  C -->|Yes, but slowly| E[Connections stall waiting for buffers]
  D --> F[Work dropped or sessions fail]
  E --> G[Mysterious latency, no CPU/network cause]
  F --> H[If unchecked: kernel OOM killer]

The latency-only path (E) is what makes this painful: nothing in the usual error counters moves. The dropped-work path (D) is where PoolFailed finally gives you a counter to look at.

Common causes

Quick checks

All read-only. Adjust the socket path to your deployment.

# PoolFailed and the memory summary in one shot
echo "show info" | socat unix-connect:/var/run/haproxy.sock stdio | \
  grep -E "^(PoolFailed|PoolAlloc_MB|PoolUsed_MB|Memmax_MB|CurrConns|Maxconn|Idle_pct|Stopping):"

# Per-pool breakdown: which object type is exhausted
echo "show pools" | socat unix-connect:/var/run/haproxy.sock stdio

What to look for:

• PoolFailed: should be zero. Any nonzero value is real.
• Memmax_MB: 0 means no configured limit; the ceiling is then host memory and the OOM killer.
• PoolUsed_MB vs PoolAlloc_MB: used is what is actively checked out; allocated is what the pools have taken from the OS. Allocated can legitimately sit above used because pools cache freed objects.
• show pools: per-pool size, count, and failure counts. The pool with failures is the resource that ran dry, which tells you whether you are out of buffers or out of session/connection objects.

# Host-level memory and OOM evidence
free -m
dmesg -T | grep -i -E "out of memory|oom-kill" | tail -20

# HAProxy resident memory and process count (reload storm check)
HPID=$(pgrep -x haproxy | head -1)
grep -E "VmRSS" /proc/$HPID/status
pgrep -c haproxy

A process count above 2-3 with Stopping: 1 on old processes means old workers are still draining and still holding their pools. Total HAProxy memory is the sum across all processes, and counters like PoolFailed reset on each reload, so a fresh process can show zero while the system is still tight.

How to diagnose it

1. Confirm the failure is current. Sample PoolFailed twice, a minute apart. A nonzero but static value is history from before a reload. A moving value is an active allocation failure.

2. Identify the failing pool. Run show pools and find the pool line with failures. Buffer-pool failures point at concurrency and payload size. Session or connection object failures point at connection count and per-connection metadata.

3. Check the configured ceiling. If Memmax_MB is nonzero and PoolUsed_MB is riding just under it, the limit is the bottleneck regardless of free host RAM. If Memmax_MB is 0, the effective ceiling is host memory.

4. Check host memory pressure. free -m and HAProxy RSS tell you whether the system can give HAProxy more. dmesg shows whether the OOM killer has already been involved. With -m unset, HAProxy allocates until the kernel intervenes.

5. Estimate expected memory per connection. Each connection costs roughly 2 x tune.bufsize + ~20KB. Multiply by CurrConns (and by process count if old draining processes exist). If that estimate is close to Memmax_MB or to available host memory, you have a sizing problem, not a leak.

6. Rule out the lookalikes. Check Idle_pct (unless busy-polling is enabled, in which case it is meaningless; use show activity instead) and backend rtime. Healthy CPU and backends plus spiking latency plus a moving PoolFailed points at buffer starvation.

Metrics and signals to monitor

Fixes

Add memory headroom or raise the ceiling

If the -m limit is set and the workload legitimately needs more, raise it. If the host is the constraint, add RAM or reduce co-located consumers. Setting -m on a shared host is still the right call: it makes HAProxy fail allocations (visible in PoolFailed) instead of growing until the OOM killer picks a victim, which may not be HAProxy.

Reduce per-connection memory

tune.bufsize (default 16384) is paid twice per connection. Lowering it reduces the footprint linearly but risks parse failures for large headers, which surface as ereq and 400s. Raising it for big-header workloads must be budgeted: 2 x bufsize x peak CurrConns plus metadata. Treat any bufsize change as a capacity decision, not just a protocol one. Very large SSL records may not fit in a 16KB buffer; some deployments raise it to 17-18KB for that reason, which raises per-connection cost further.

Control buffer pool growth explicitly

tune.buffers.limit caps the total number of buffers HAProxy will allocate globally (default 0, unlimited), and tune.buffers.reserve pre-allocates buffers kept aside for allocation-failure conditions (default 4 per thread) so in-flight sessions can finish. Together they convert an unbounded failure mode (grow until OOM) into a bounded one (fail early, visibly, and recoverably).

Fix the reload storm

If multiple draining processes are the memory consumer, set hard-stop-after so old workers are force-killed after a bounded drain, reduce reload frequency, and use master-worker mode. Without hard-stop-after (default: no limit), a single long-lived connection can keep an old process and its pools alive indefinitely.

Do not flush pools on a live process as a fix

show pools is safe and does not flush anything. Sending SIGQUIT to force pool flushing on a busy production process is disruptive and has been associated with crashes on loaded instances. Diagnose with show pools; fix with sizing and limits, not with signals.

Allocator-level notes

On HAProxy 2.4 and later, periodic malloc_trim() behavior can be disabled with no-memory-trimming in the global section. The documented failure mode is watchdog kills triggered by malloc_trim() scanning large fragmented arenas on systems with tens to hundreds of thousands of concurrent connections, leaving the process unable to pass traffic; if you hit that, no-memory-trimming is the documented workaround.

Prevention

• Alert on any nonzero PoolFailed rate. It should always be zero. This is a Level 4 signal in the HAProxy monitoring maturity model for a reason: it is the earliest direct evidence of memory-allocator failure.
• Budget memory before raising concurrency. Any increase to maxconn, tune.bufsize, or stick-table size is a memory commitment. Estimate CurrConns x (2 x bufsize + ~20KB) against the -m limit and host RAM before deploying.
• Set -m on shared hosts. Bounded, visible allocation failures beat an unbounded march to the OOM killer.
• Keep draining-process headroom in mind during reloads. Total HAProxy memory during a reload window is old process plus new process.
• Trend PoolUsed_MB against CurrConns. Divergence between the two over weeks is either a leak or an unbudgeted consumer such as growing stick tables.

How Netdata helps

• Netdata collects the show info fields continuously, so PoolFailed, PoolUsed_MB, Memmax_MB, and CurrConns are graphed together at per-second resolution instead of sampled by hand during an incident.
• Correlating PoolFailed increments against CurrConns peaks answers the first diagnostic question immediately: concurrency-driven starvation or a fixed ceiling?
• System-level memory, per-process RSS, and OOM events are collected on the same host and timeline, so host pressure and HAProxy pool pressure appear in one view.
• Reload detection via process uptime and counter resets prevents misreading a post-reload zero as “problem solved.”
• Alerting on a nonzero PoolFailed delta catches buffer starvation during the latency-only phase, before sessions start failing.

Related guides

• HAProxy 502 Bad Gateway: backend connection and response failures
• HAProxy 503 Service Unavailable: no server is available to handle this request
• HAProxy 504 Gateway Timeout: timeout server and the backend timeout cascade
• HAProxy backend losing servers: active server count and cascade risk
• HAProxy backend queue building (qcur): requests waiting for a free server slot
• HAProxy compression overhead: comp_byp, CPU cost, and when compression backfires
• HAProxy connect time (ctime) high: network latency and backend accept-queue overflow
• HAProxy connection errors (econ): backend connections refused, timed out, or reset
• HAProxy scur approaching slim: the concurrent-session saturation signal
• HAProxy 5xx delta: telling HAProxy-generated errors from backend errors
• HAProxy health checks green but the application is broken: when UP does not mean healthy
• HAProxy health check L4TOUT and L4CON: server marked DOWN and unreachable