$ guides / haproxy / haproxy-memory-growth-oom ▌

Operations Guides

HAProxy memory growth and OOM: connection buffers, pools, and memory ceilings

The symptom usually arrives as a dead process, not a slow one. HAProxy was proxying fine, then the PID changed, connections dropped for a few seconds, and dmesg shows the OOM killer picked haproxy as the victim. Or, before it gets that far, RSS climbs day over day and nobody knows whether it is a leak or just load.

HAProxy has no internal memory ceiling unless you set one. It allocates what the workload demands, holds most of it in internal pools for reuse, and keeps going until the kernel OOM killer intervenes. Memory is a cliff-edge resource: fine, fine, fine, dead.

The good news: HAProxy memory consumption is dominated by one predictable term, current connections, and the allocator exposes counters that warn you before the cliff.

What this means

Every established connection costs roughly 2 x tune.bufsize (16KB each by default, so about 32KB) for the request and response buffers, plus around 20KB of connection and session metadata. Call it about 50KB per connection. At 10,000 concurrent connections that is roughly 500MB before anything else. CurrConns from show info times the per-connection cost is the dominant term in any memory estimate.

On top of that:

Stick tables consume memory proportional to their configured size and entry data types. Tables without a sane expire fill with stale entries.
The SSL session cache and certificates add steady-state memory, larger with big cert bundles.
DNS cache and pool caching add smaller amounts.

The second thing to understand is pool behavior. HAProxy’s allocator keeps released objects in internal pools for reuse instead of returning them to the OS. After a traffic spike, RSS stays high: PoolAlloc_MB (allocated from the OS) sits above PoolUsed_MB (actively in use). This looks like a leak and usually is not. A real leak shows PoolUsed_MB itself climbing without a matching climb in CurrConns or table usage.

When memory actually runs out, the PoolFailed counter in show info increments: HAProxy tried to allocate a buffer or session and failed. Any nonzero PoolFailed means requests were dropped or stalled for lack of memory.

Common causes

Cause	What it looks like	First thing to check
Connection count growth	RSS tracks `CurrConns`; step changes with traffic	`CurrConns` trend vs RSS trend
Long-lived connections (WebSocket, SSE, slow clients)	High `scur`, low `req_rate`, low throughput	`show sess` for session age and state
Stick tables without `expire`	Memory grows with unique clients, never shrinks	`show table` used vs size
Pool retention after a spike	RSS jumped once, now flat; `PoolUsed_MB` stable	`show info` pool fields; not a leak
Reload storm	Multiple haproxy PIDs, each holding memory	`pgrep -c haproxy`
Real leak or version bug	`PoolUsed_MB` climbs with flat traffic	`show pools` to find the growing pool

Quick checks

All read-only and safe to run during an incident.

# Connections and memory in one shot
echo "show info" | socat unix-connect:/var/run/haproxy.sock stdio | \
  grep -E "^(CurrConns|Maxconn|Memmax_MB|PoolAlloc_MB|PoolUsed_MB|PoolFailed|Uptime_sec|Stopping):"

# Per-pool breakdown: which pool is large
echo "show pools" | socat unix-connect:/var/run/haproxy.sock stdio

# Stick table usage
echo "show table" | socat unix-connect:/var/run/haproxy.sock stdio

# Process count (reload storm check)
pgrep -c haproxy

# RSS of the worker
HPID=$(pgrep -x haproxy | head -1)
grep VmRSS /proc/$HPID/status

# Kernel OOM evidence
dmesg -T | grep -iE "out of memory|killed process"

Two things to note in the show info output. Memmax_MB is the configured memory ceiling; 0 means unlimited. PoolFailed should always be zero.

How to diagnose it

Confirm whether an OOM kill actually happened. Check dmesg for the OOM killer naming haproxy, and check Uptime_sec in show info: a recently restarted process with no deploy record is a crash or kill. If there is no OOM record, you are looking at growth, not a kill, and you have time.
Compute the expected connection memory. Take CurrConns, multiply by roughly 50KB (2 x tune.bufsize plus about 20KB metadata; more if you raised tune.bufsize). Compare against RSS. If expected connection memory explains most of RSS, this is workload, not a leak. The question becomes why connection count is high, which is a different investigation: see scur approaching slim.
Check for idle connection pile-up. High CurrConns with low req_rate means connections are being held, not worked. Look at timeout client, timeout http-request, and timeout http-keep-alive, and inspect show sess for long-lived or stuck sessions. WebSocket and streaming deployments should expect this shape; everyone else should not.
Check the pool fields. If PoolAlloc_MB is high but PoolUsed_MB is low and stable, the allocator is holding memory from a past spike. That is normal retention. If PoolUsed_MB itself climbs while CurrConns is flat, run show pools twice a few minutes apart and find which pool is growing. That pool name points at the subsystem.
Check stick tables. show table reports used vs size per table. A table pinned near 100% with no expire configured (or a very long one) is steady memory that grows with unique client keys. Also note: with default settings, full tables evict expired entries silently; with nopurge, new entries are rejected. Neither failure mode shows up in memory counters directly.
Count processes. If pgrep -c haproxy returns more than 2 or 3, old soft-stopped workers are holding memory. Check Stopping: 1 on old processes and whether hard-stop-after is configured. During a reload storm, total HAProxy memory is the sum across all PIDs, and system-level RSS keeps climbing even though each process looks normal.
Only then suspect a leak. Flat traffic, flat CurrConns, flat tables, single process, and still-climbing PoolUsed_MB is the real leak signature. Capture show pools output, note your HAProxy version, and check the changelog before upgrading in place.

Metrics and signals to monitor

Signal	Why it matters	Warning sign
`CurrConns` (show info)	Dominant memory term at ~50KB per connection	Climbing without a traffic explanation
RSS of worker PID	What the OOM killer actually sees	Sustained upward trend across days
`PoolUsed_MB`	Memory actively in use by HAProxy objects	Rising while `CurrConns` is flat
`PoolAlloc_MB` vs `PoolUsed_MB`	Separates pool retention from real growth	Alloc » Used after a spike is normal; Used rising is not
`PoolFailed`	Allocation failures, dropped or stalled work	Any nonzero value
`Memmax_MB`	The configured ceiling, 0 means none	0 on hosts running other services
`show table` used/size	Stick table memory and eviction risk	used > 80% of size, no `expire`
`Stopping` + PID count	Reload storms holding memory in old workers	More than 2-3 PIDs, lingering soft-stop
dmesg OOM events	Confirms the kill and the victim	haproxy named in OOM killer output

Fixes

Reduce the per-connection footprint

The per-connection cost scales directly with tune.bufsize. If you raised it for large headers or big cookies, you raised memory per connection proportionally: at tune.bufsize 32768, buffers alone are about 64KB per connection. Do not lower bufsize blindly; too small causes request parse failures on large headers (ereq, 400s). Set it for your actual header sizes and account for the memory cost in capacity math.

Reap idle connections

If CurrConns is inflated by idle sessions, tighten the timeouts that hold them: timeout http-keep-alive, timeout client, and timeout http-request for slow senders. This frees buffers and file descriptors without touching capacity. Tradeoff: aggressive keep-alive timeouts cost clients reconnections and TLS handshakes, so lower them stepwise and watch SslFrontendKeyRate and cli_abrt.

Give stick tables an expiry

Every stick table should have an expire that matches the tracking window. A rate-limit table tracking 10-second rates does not need hour-long entry lifetimes. Right-size size as well; a table sized for 1M entries reserves for 1M entries. Tradeoff: shorter expiry evicts entries that are still legitimate if your window was too tight, so watch dreq behavior after the change.

Set a memory ceiling

If HAProxy shares a host with anything else, give it a ceiling. The ceiling is set at startup with the -m <MB> flag and shows up as Memmax_MB in show info.

When the ceiling is hit, HAProxy fails allocations (PoolFailed increments) and stalls or drops work instead of growing until the OOM killer fires. That converts an unbounded cliff into bounded, observable degradation. Tradeoff: you are choosing “HAProxy degrades under memory pressure” over “HAProxy survives until the kernel kills something, possibly something else.” On a shared host that is usually the right choice; on a dedicated host you may prefer headroom and an alert instead. An OS-level cgroup or systemd memory limit is a viable alternative, but the OOM killer inside the cgroup will terminate the process rather than degrade it.

Size the ceiling from your own math: expected peak CurrConns x per-connection cost, plus stick table configured sizes, plus SSL cache, plus headroom for pool retention after spikes.

Stop reload storms

If old workers linger in soft-stop, set hard-stop-after so draining processes are force-killed after a bounded window, and reduce reload frequency by batching config changes. Without hard-stop-after, a single WebSocket connection can pin an old worker, and its memory, indefinitely. Tradeoff: clients still connected when the timer fires get a hard disconnect.

Handle the aftermath of an OOM kill

If the OOM killer already fired, the process is gone and connections dropped. Restart it, then treat it as a capacity incident. Pull the numbers from steps 2 through 6, because without a ceiling and without trending on CurrConns and RSS, it will happen again at the next traffic peak.

Prevention

Do the capacity math once, write it down. Peak expected CurrConns x ~50KB, plus stick tables, plus SSL cache, plus 30% headroom. Compare against host RAM. This is a 10-minute exercise that prevents the 3 a.m. version.
Set a ceiling on shared hosts. -m (visible as Memmax_MB) so HAProxy degrades observably instead of triggering the kernel OOM lottery.
Alert on PoolFailed > 0. It is normally zero and it is the earliest in-process signal that memory pressure is dropping work.
Trend PoolUsed_MB and RSS against CurrConns. Divergence between them is the leak detector.
Expire every stick table. No table without an expire matching its tracking purpose.
Bound reloads. hard-stop-after set, reload frequency kept low, PID count monitored.
Gate on connection saturation upstream. Memory follows connections, so the scur/slim ratio is a leading indicator for memory too.

How Netdata helps

Netdata collects the show info fields that matter here, including CurrConns, PoolAlloc_MB, PoolUsed_MB, and PoolFailed, so the connection-to-memory relationship is visible on one dashboard instead of in ad-hoc socket commands.
Per-second RSS and system memory charts let you overlay HAProxy process memory against CurrConns to spot the divergence that separates leaks from pool retention.
Alerting on PoolFailed going nonzero and on RSS trend deviations catches allocator pressure before the OOM killer does.
Reload storms show up as process count changes and Uptime_sec resets, which Netdata tracks alongside memory, making the multi-PID memory pile-up obvious.
Correlating HAProxy memory signals with system-level OOM events and stick table utilization shortens the path from “RSS is climbing” to the specific term driving it.

The Netdata solution

HAProxy load balancer monitoring with Netdata

Netdata monitors HAProxy with per-second frontend, backend, and queue metrics plus ML-powered anomaly detection. Correlate maxconn saturation, queue buildup, health-check cascades, 5xx attribution, and file-descriptor exhaustion against the backend and host signals behind them, so you catch the incidents in these runbooks before they page anyone.

See load balancer monitoring → Start monitoring free