HAProxy reload storm: lingering processes leaking file descriptors and memory

You log into the HAProxy host because memory and file descriptor usage have been climbing for days, and you find a dozen haproxy processes running. Only one is serving traffic. The rest are old workers stuck in soft-stop, each holding open connections, connection buffers, and file descriptors it will never release. Traffic looks fine, per-process metrics look fine, and the host is still slowly running out of resources.

Every config reload spawns a new HAProxy worker while the old one enters a graceful drain. If connections on the old worker never close, the old worker never exits. Reload often enough and the drain backlog grows faster than it clears. The signature: pgrep -c haproxy well above 2-3, Stopping: 1 on old workers, and system-level FD and memory usage rising while HAProxy’s own per-process metrics look normal.

This is most common in Kubernetes Ingress Controller deployments and other environments with automated config generation: every endpoint change, Ingress update, or service discovery event can trigger a reload. A few reloads per day is harmless. A few per minute, combined with long-lived connections, is a slow leak that ends in FD exhaustion or an OOM kill of the worker actually serving traffic.

What this means

HAProxy reloads are designed to be seamless. In master-worker mode, the master receives SIGUSR2 (for example via systemctl reload haproxy), spawns a new worker with the new configuration, and tells the old worker to soft-stop: stop accepting new connections, finish draining existing ones, then exit. The old worker sets Stopping: 1 in show info and waits.

The design assumption is that connections close. A short-lived HTTP request drains in milliseconds. WebSockets, Server-Sent Events, gRPC streams, long-polling, and keepalive connections with active traffic can keep the old worker alive indefinitely. There is no default time limit on the drain: hard-stop-after is unset by default, so a single idle WebSocket can pin an old process for days.

Each lingering worker holds roughly two file descriptors per proxied connection plus listener and health check sockets, and its full connection buffer and memory footprint. The new worker’s stats show a small, healthy process. The sum across all processes is what consumes the host.

flowchart TD
    A[Config change triggers reload] --> B[New worker starts with fresh counters]
    B --> C[Old worker receives soft-stop signal]
    C --> D{Connections drained?}
    D -->|yes| E[Old worker exits, resources freed]
    D -->|no: long-lived or idle connections| F[Old worker lingers with Stopping: 1]
    F --> G[Holds FDs, buffers, memory]
    A2[Next reload repeats the cycle] --> B
    G --> H[Accumulating processes exhaust host FDs or memory]

Two side effects make this worse. First, every reload resets all cumulative stats counters to zero, so rate-based dashboards show gaps or phantom drops. Second, lingering workers keep running health checks against backends, adding probe traffic that no longer corresponds to real routing decisions.

Common causes

Quick checks

All read-only and safe to run during an incident.

# 1. Count haproxy processes. More than 2-3 persistent PIDs is abnormal.
pgrep -c haproxy
ps -eo pid,etimes,rss,cmd | grep [h]aproxy

# 2. Check whether a worker is in soft-stop
echo "show info" | socat unix-connect:/var/run/haproxy.sock stdio | grep -E "^(Stopping|Uptime_sec|CurrConns):"

# 3. FD usage per haproxy process
for p in $(pgrep -x haproxy); do
  echo "pid $p: $(ls /proc/$p/fd 2>/dev/null | wc -l) fds, limit: $(awk '/Max open files/{print $4}' /proc/$p/limits)"
done

# 4. Memory per haproxy process
ps -o pid,rss,cmd -C haproxy --sort=-rss

# 5. What is holding connections open on a draining worker
echo "show sess" | socat unix-connect:/var/run/haproxy.sock stdio | head -50

# 6. How often reloads are happening
journalctl -u haproxy --since "6 hours ago" | grep -ciE "reload|stopping|new worker"

Caveat on checks 2 and 5: after a reload, the stats socket is typically taken over by the new process, so if show info answers, it is the new worker answering. To inspect old workers, use /proc/<pid> data, ss -p filtered by PID, or per-process sockets if your deployment exposes them.

# Old-worker connection state via ss (no stats socket needed)
for p in $(pgrep -x haproxy); do
  echo "pid $p: $(ss -tanp 2>/dev/null | grep -c "pid=$p,") established sockets"
done

How to diagnose it

1. Confirm the pattern. pgrep -c haproxy should be 1 (standalone) or 2 (master plus one active worker) in steady state. A count of 3+ that persists for more than a few minutes, or grows over hours, confirms lingering processes. Check etimes from ps: old workers with ages matching past reload times are the leak.

2. Confirm the drain state. On any worker you can still query, Stopping: 1 confirms soft-stop. For old workers you cannot query, infer from age and process count.

3. Identify what is pinning connections. Use show sess (if reachable) or ss -tanp against the old PID. Sessions in ESTABLISHED state with ages in the hours range, especially on ports serving WebSocket, streaming, or TCP-mode proxies, are the blockers. If sessions show regular traffic, client keepalives may be keeping them alive.

4. Measure the reload rate. Count reload events in the last hour from journald or the ingress controller logs. Compare the reload interval to the longest observed drain time: if reloads arrive faster than drains complete, the PID count can only grow.

5. Quantify the resource damage. Sum FDs across all haproxy PIDs against the per-process ulimit and the system limit (/proc/sys/fs/file-max). Sum RSS across PIDs against host memory. Note that during every reload, old and new process FDs coexist, so peak FD usage during a reload window is roughly double steady state even in a healthy setup.

6. Check the monitoring side effects. Look at your time series for counter resets: sudden drops to zero in cumulative metrics like hrsp_5xx, econ, bin. If your monitoring does not handle resets, reload storms also show up as phantom rate spikes and data gaps. A drop in Uptime_sec is the reliable reload detector.

7. Rule out the upstream trigger. In Kubernetes, correlate reloads with endpoint churn: frequent pod rescheduling, a flapping readiness probe, or a CRD/Ingress being updated in a loop can drive continuous reloads. Tuning HAProxy without fixing the trigger only slows the leak.

Metrics and signals to monitor

Fixes

Bound the drain time with hard-stop-after

Set hard-stop-after in the global section. This is the definitive fix for indefinite lingering: after the configured time from receiving the soft-stop signal, the old process is force-killed regardless of remaining connections.

global
    hard-stop-after 60s

The value is a tradeoff: whatever you set is the maximum connection lifetime that survives a reload. HTTP workloads commonly use 30 seconds to 5 minutes. Workloads proxying legitimately long-lived connections need more: GitLab’s production infrastructure, where SSH keepalive connections blocked drains, settled on 30 minutes (hard-stop-after 30m in their public load-balancer configuration). If you serve WebSockets with multi-hour sessions, either the connections get cut at hard-stop-after or the processes linger; there is no third option.

Each old worker starts its own hard-stop-after timer when it receives its own soft-stop signal, so during rapid successive reloads multiple draining workers count down in parallel; the directive has existed since HAProxy 1.7. If you set it and old processes still linger past the window, check the version: a confirmed bug in 2.9.1 caused lingering processes with high CPU on reload when TLS traffic was present, fixed in 2.9.2.

Cap worker generations in master-worker mode

In master-worker mode, mworker-max-reloads limits how many reloads a worker can survive. Once a worker has lived through more reloads than the limit, it receives SIGTERM.

global
    master-worker
    mworker-max-reloads 10

This complements hard-stop-after: the reload-count cap catches workers that would otherwise survive across many generations even with modest per-reload drain times. If you are not running master-worker mode at all, enable it; without it, each reload forks an independent process tree and accumulation is worse.

Reduce reload frequency

This is the root-cause fix when the trigger is automated config management:

• Batch changes. If config is regenerated on every service discovery event, debounce it: accumulate changes and reload at most once per N seconds.
• Use the runtime API instead of reloads for dynamic backends. Server address changes, weight changes, and server add/remove can be applied at runtime through the stats socket (add server, set server addr, set server ... state) without a reload, without counter resets, and without new processes. In Kubernetes Ingress, prefer controller configurations that use dynamic server updates rather than full reloads.
• Fix the churn source. Flapping readiness probes, rapidly rescheduling pods, or an Ingress object being updated in a loop will drive reload storms no matter how well HAProxy is configured.

Runtime commands that mutate state without a reload include add server, del server, set server ... addr/weight/state, set maxconn server/frontend, map and ACL updates, and set ssl cert / commit ssl cert for in-memory certificate rotation (add/del server are available since HAProxy 2.4). Whether those commands avoid a reload in a given deployment depends on the controller issuing them, not on the proxy.

Clear the existing backlog carefully

To reclaim resources now, kill old draining workers with SIGTERM. This drops every connection still held by that worker immediately; clients on those sessions see hard disconnects.

# DISRUPTIVE: drops all connections still held by old workers.
# Identify old workers first (all PIDs except the current active worker),
# then terminate them one at a time while watching client impact.
kill -TERM <old_worker_pid>

Do not kill the active worker or the master. If you are unsure which PID is active, the one bound to the listener ports (ss -tlnp | grep haproxy) and answering the stats socket is current.

If reloads stop working entirely

A known issue in HAProxy 3.2.x master-worker mode: if a new worker fails during startup (for example a missing map file), the master can stop processing subsequent SIGUSR2 reload signals, leaving reloads silently ineffective until a full restart. If systemctl reload seems to do nothing, check for failed worker startups in the logs and plan a restart at a safe window. A restart drops all connections, so schedule it deliberately.

Prevention

• Set hard-stop-after everywhere. Treat it as a required global directive, not an optional tuning knob. Size it to the longest connection lifetime you are willing to protect.
• Alert on process count, not just process existence. A PID count above 2-3 sustained for more than a few minutes is a ticket-level condition; catch it days before FD or memory exhaustion forces the issue.
• Gate reload frequency. In automation that generates HAProxy configs, enforce a minimum reload interval and emit a metric for reloads per hour so the trigger is visible.
• Prefer runtime API updates for dynamic state. Backend membership changes do not need process restarts; reserve reloads for structural config changes.
• Handle counter resets in monitoring. Use Uptime_sec drops to detect reloads and make rate calculations reset-aware, so reload storms do not also corrupt your other alerts.
• Capacity-plan for the reload window. Even in a healthy setup, FD and memory usage roughly double during a drain. Keep steady-state headroom above that: peak FD usage should stay under about 60% of ulimit.
• Be aware of old-process health checks. Lingering workers continue probing backends. If backend health check load looks anomalous, correlate with PID count before blaming the backends.

How Netdata helps

• Per-process visibility. Netdata charts per-process FD counts, memory, and CPU from /proc, which is exactly the view this problem needs: the sum across all haproxy PIDs, not just the active worker’s self-reported stats.
• HAProxy stats collection with reload detection. Netdata collects the stats CSV and show info fields (including CurrConns, Maxsock, Stopping, Uptime_sec) at one-second granularity, so reloads show up as Uptime_sec drops you can correlate against PID count and system resource charts.
• Correlation in one timeline. The diagnostic sequence above (PID count up, system FDs up, active-worker metrics flat, Uptime_sec resetting) crosses process-level, HAProxy-level, and system-level signals. Seeing them on the same timeline turns a multi-command investigation into a glance.
• Alerting on the leading indicator. Alerting on haproxy process count and aggregate FD usage catches the storm while it is still a capacity trend, not an accept() failure cliff.
• Post-fix verification. After setting hard-stop-after or reducing reload frequency, the PID count and FD trend lines are the direct proof the fix worked.

Related guides

• HAProxy 502 Bad Gateway: backend connection and response failures
• HAProxy 503 Service Unavailable: no server is available to handle this request
• HAProxy 504 Gateway Timeout: timeout server and the backend timeout cascade
• HAProxy backend losing servers: active server count and cascade risk
• HAProxy backend queue building (qcur): requests waiting for a free server slot
• HAProxy compression overhead: comp_byp, CPU cost, and when compression backfires
• HAProxy connect time (ctime) high: network latency and backend accept-queue overflow
• HAProxy connection errors (econ): backend connections refused, timed out, or reset
• HAProxy scur approaching slim: the concurrent-session saturation signal
• HAProxy 5xx delta: telling HAProxy-generated errors from backend errors
• HAProxy health checks green but the application is broken: when UP does not mean healthy
• HAProxy health check L4TOUT and L4CON: server marked DOWN and unreachable