HAProxy stuck in soft-stop: draining connections and hard-stop-after

You reloaded HAProxy minutes ago, but pgrep haproxy still shows two, three, maybe a dozen PIDs. The old process is not serving new traffic, but it refuses to die. Memory and file descriptor usage on the host keep climbing with every reload, and your monitoring loses counter history each time it happens.

The old process is in soft-stop: it has unbound from its listeners and is waiting for its existing connections to close before exiting. A drain of a few seconds to a couple of minutes is normal. A process draining for more than 10 minutes is stuck, and the cause is almost always long-lived connections (WebSocket, Server-Sent Events, gRPC streaming, or plain TCP keepalive sessions) that have no reason to close on their own.

Without hard-stop-after, HAProxy waits forever. A single idle WebSocket can pin an old process, and its memory and file descriptors, indefinitely.

What this means

A graceful reload or shutdown sends the old HAProxy process a soft-stop signal (SIGUSR1). The process stops accepting new connections, unbinds from its listening sockets, and continues processing only the connections it already holds. When the last connection closes, the process exits cleanly.

The show info field Stopping is the canonical signal: Stopping: 0 means normal operation, Stopping: 1 means soft-stop in progress. Brief is expected. Lingering is the symptom.

stateDiagram-v2
  [*] --> Active : serving traffic
  Active --> SoftStop : SIGUSR1 (reload / graceful stop)
  SoftStop --> Draining : Stopping=1, no new connections
  Draining --> Exited : last connection closes
  Draining --> Stuck : long-lived connection never closes
  Stuck --> Killed : hard-stop-after expires
  Stuck --> Stuck : no hard-stop-after (default: forever)

A stuck process serves no new traffic but still costs you: it holds file descriptors (two per proxied connection, plus listeners and health check sockets), it holds memory (connection buffers, stick tables, SSL session cache), and the reload that orphaned it reset all cumulative stats counters on the new process, creating monitoring gaps. Stack several reloads on top of each other and you get the reload storm pattern: multiple draining processes accumulating until the host runs out of FDs or memory.

Common causes

Quick checks

These are all read-only.

# How many HAProxy processes are running?
pgrep -c haproxy

# Is the process in soft-stop? Stopping: 1 = draining
echo "show info" | socat unix-connect:/var/run/haproxy.sock stdio | grep "^Stopping:"

# How long has this process been up? A drainer with a large Uptime_sec
# relative to your last reload has been stuck since that reload.
echo "show info" | socat unix-connect:/var/run/haproxy.sock stdio | grep "^Uptime_sec:"

# What connections is the draining process still holding?
echo "show sess" | socat unix-connect:/var/run/haproxy.sock stdio | head -50

# How many connections is it holding?
echo "show info" | socat unix-connect:/var/run/haproxy.sock stdio | grep "^CurrConns:"

# System-level FD usage of the old process
HPID=<old-haproxy-pid>
ls /proc/$HPID/fd | wc -l

# Is hard-stop-after configured at all?
grep -n "hard-stop-after" /etc/haproxy/haproxy.cfg

After a reload, each process may have its own stats socket (for example, one per worker PID in master-worker mode). Make sure you are querying the draining process, not the new active worker. The new worker shows Stopping: 0 and near-zero counters, which can hide the problem.

How to diagnose it

1. Confirm the state. Check Stopping: 1 in show info for the old process and note Uptime_sec. If the process has been in soft-stop for more than 10 minutes, treat it as stuck rather than draining.

2. Count the processes. pgrep -c haproxy. More than two or three concurrent processes (one active worker plus one drainer) points to a reload storm: reloads happening faster than drains complete.

3. Identify what is holding the drainer open. Run show sess against the old process’s stats socket and look at what remains. Long-lived sessions stand out: WebSocket upgrades, SSE endpoints, gRPC streams, or TCP-mode sessions with high age and no recent activity. If CurrConns on the drainer is small but nonzero and static, those are your pinning connections.

4. Check the config. If hard-stop-after is absent from both the global and defaults sections, the drain timeout is infinity. That is the default, and it is the root configuration gap in nearly every stuck soft-stop incident.

5. Check the reload trigger. If processes are accumulating, find what is reloading HAProxy: config management, service discovery, certificate rotation, an ingress controller. Reloads faster than drains will accumulate processes no matter what the drain timeout is.

Metrics and signals to monitor

Fixes

Set hard-stop-after (the actual fix)

Add a drain bound in the global section:

global
    hard-stop-after 60s

After the timeout, the draining process is force-killed even if connections remain. Without this directive the timeout is infinite, which is exactly the bug you are hitting.

The tradeoff is the tuning knob: any client still connected when the timer fires gets a hard disconnect. For request/response HTTP traffic, drains finish in seconds and hard-stop-after 30s to 60s is conservative. For WebSocket, SSE, or gRPC streaming workloads, connections legitimately live for minutes or hours, so decide which is worse: killing those clients on every reload, or letting old processes linger. A common compromise is a larger window (5 to 15 minutes) combined with client-side reconnect logic, which well-behaved streaming clients should have anyway. Do not set this to a few seconds unless your workload is strictly short-lived requests.

Spread the close with close-spread-time

close-spread-time spreads the closing of idle and active connections over a time window during soft-stop instead of closing them all at once, which softens the reconnect thundering herd on the new process. It must be set lower than hard-stop-after, otherwise connections will not finish closing before the hard stop kills them. The directive is present in HAProxy 2.2 and later.

Reduce reload frequency

If the trigger is service discovery or an ingress controller reacting to every pod event, batch or debounce updates. Every reload resets cumulative counters on the new process, invalidates the in-process SSL session cache, and cold-starts backend connection pools. Reloads are cheap, not free.

Use master-worker mode

In master-worker mode the master orchestrates worker lifecycle, and old workers enter soft-stop under its supervision. It does not remove the need for hard-stop-after, but it makes process accounting and signals (Stopping, per-worker stats sockets) easier to reason about than unmanaged fork-per-reload setups.

Do not SIGKILL as the routine answer

Killing the stuck process with kill -9 frees resources immediately but hard-drops every connection it holds and bypasses any cleanup. As a one-off during an incident it is acceptable when the process has been stuck for hours and the host is running out of FDs or memory. As a standing practice it just hides the missing hard-stop-after configuration.

Prevention

• Always set hard-stop-after in global config. Treat the absence of this directive as a misconfiguration on any host that serves long-lived connections. The default (infinity) is only safe for pure short-request workloads.
• Alert on drain duration. Alert when any process reports Stopping: 1 for more than 10 minutes. Brief drains are normal; lingering ones are not.
• Alert on process count. pgrep -c haproxy above 2-3 sustained indicates reloads outpacing drains.
• Track reload frequency. Correlate Uptime_sec resets with your deployment pipeline. If reloads happen more often than every few minutes, fix the trigger before tuning the drain.
• Handle counter resets in monitoring. All cumulative counters reset on reload. Rate calculations must tolerate resets, or every reload will produce phantom metric drops.
• Size client expectations. If you serve WebSocket or SSE, document that reloads will disconnect clients within the hard-stop-after window and verify clients reconnect cleanly.

How Netdata helps

• Netdata collects show info fields including Stopping and Uptime_sec per process, so a draining process and its age are visible without SSHing in during an incident.
• Process-level charts (FD count, memory) per HAProxy PID make a stuck drainer obvious: the new worker’s counters reset while the old process’s resource usage stays flat and nonzero.
• Correlating drain duration with reload events (visible as Uptime_sec resets) tells you whether you have one pinned connection or a systemic reload storm.
• Alerting on “process in soft-stop longer than N minutes” and on HAProxy process count turns this from a 3 a.m. discovery into a ticket during business hours.
• Connection and session charts on the draining instance show whether CurrConns is shrinking (healthy drain) or static (pinned by long-lived connections), which decides whether you wait or tune hard-stop-after.

Related guides

• HAProxy scur approaching slim: the concurrent-session saturation signal
• HAProxy connection reuse dropping: http-reuse, pooling, and handshake overhead
• HAProxy ephemeral port exhaustion: TIME_WAIT and backend connection churn
• HAProxy backend losing servers: active server count and cascade risk
• HAProxy 502 Bad Gateway: backend connection and response failures
• HAProxy 503 Service Unavailable: no server is available to handle this request
• HAProxy 504 Gateway Timeout: timeout server and the backend timeout cascade
• HAProxy backend queue building (qcur): requests waiting for a free server slot
• HAProxy connect time (ctime) high: network latency and backend accept-queue overflow
• HAProxy connection errors (econ): backend connections refused, timed out, or reset
• HAProxy compression overhead: comp_byp, CPU cost, and when compression backfires
• HAProxy 5xx delta: telling HAProxy-generated errors from backend errors