HAProxy Too many open files: file descriptor exhaustion and refused connections

HAProxy is refusing new connections. Clients see connection timeouts or resets, health checks start failing for no application reason, and the logs show Too many open files. The process is alive and CPU looks fine, which makes this confusing the first time you hit it: nothing is overloaded in the usual sense. The proxy has run out of file descriptors.

FD exhaustion is a hard cliff. When the process hits its limit, accept() fails immediately with EMFILE. There is no queue, no graceful degradation, no backpressure. New connections are dropped at the syscall boundary. At the same time, HAProxy can no longer open backend sockets, health check sockets, or log sockets, so the failure spreads well beyond new client connections.

What this means

Every connection HAProxy proxies consumes file descriptors:

• Two FDs per proxied connection: one for the client side, one for the server side.
• Listeners: one per bound frontend address.
• Overhead FDs: the stats socket, log sockets, health check sockets, peer connections, and internal pipes.

The process-wide ceiling is the RLIMIT_NOFILE the HAProxy process inherited at startup (visible as Max open files in /proc/<pid>/limits, and as Ulimit-n and Maxsock in show info). HAProxy derives its Maxconn from this limit at startup, so a low ulimit -n silently caps your connection capacity even if you never set maxconn yourself.

When the kernel refuses to hand out another FD, two errnos matter:

• EMFILE: the per-process limit was reached. This is the common case.
• ENFILE: the system-wide kernel limit (fs.file-max) was reached. Rarer, but check it on shared hosts.

flowchart TD
  A[Connections accumulate] --> B[FD count reaches process limit]
  B --> C[accept returns EMFILE]
  B --> D[Cannot open backend sockets]
  B --> E[Cannot open health check or log sockets]
  C --> F[New client connections dropped]
  D --> G[503s and connection errors on existing work]
  E --> H[Servers flap or logs go silent]

The dangerous property of this failure is that the trigger and the limit are set at different times. The limit was fixed at process startup. The trigger is whatever pushed FD consumption over the line weeks later: traffic growth, a reload storm, or a slow build-up of long-lived connections.

Common causes

Quick checks

All read-only and safe to run during an incident.

# 1. HAProxy's own view: limit and current connections
echo "show info" | socat unix-connect:/var/run/haproxy.sock stdio | \
  grep -E "^(Ulimit-n|Maxsock|Maxconn|CurrConns|ConnRate|SessRate):"

# 2. Actual FD usage from the kernel (authoritative)
HPID=$(pgrep -x haproxy | head -1)
echo "FDs used: $(ls /proc/$HPID/fd | wc -l)"
grep 'Max open files' /proc/$HPID/limits

# 3. How many HAProxy processes are running right now?
pgrep -c haproxy

# 4. Are any processes stuck draining?
echo "show info" | socat unix-connect:/var/run/haproxy.sock stdio | grep "^Stopping:"

# 5. Confirm the errno in the logs
journalctl -u haproxy --since "30 min ago" | grep -i "too many open files"

# 6. System-wide limit (only if you suspect ENFILE, not EMFILE)
cat /proc/sys/fs/file-max

Two interpretations worth memorizing:

• FDs used within a few percent of Max open files: per-process exhaustion (EMFILE). The limit or the consumption is wrong.
• ConnRate noticeably higher than SessRate in show info: connections are arriving but not becoming sessions. During FD exhaustion this gap widens because accept() is failing.

If the stats socket itself stops responding during the incident, that is also consistent with FD exhaustion: HAProxy may be unable to accept on the admin socket either. Fall back on /proc and the logs.

How to diagnose it

1. Confirm exhaustion, not something else. Compare ls /proc/$HPID/fd | wc -l against the soft limit in /proc/$HPID/limits. If usage is at the ceiling and logs show Too many open files, you have confirmation. If FDs are well below the limit, your refused connections have another cause: see the related guides on 503s and connection errors.

2. Determine which limit was hit. Per-process (EMFILE) is the norm. If dmesg or kernel logs point at the system-wide table, check fs.file-max and total system FD usage instead. On a dedicated HAProxy host this is rarely the binding constraint.

3. Attribute the FDs. For the active worker, estimate expected usage: roughly CurrConns x 2 plus listeners, stats, log, and health check sockets. If actual FD usage is far above that, something is holding FDs that is not current proxied traffic. show fd on the runtime socket dumps every open FD with its state and is the right tool for chasing that down.

4. Check for a reload storm. pgrep -c haproxy returning more than 2-3 means old processes are still draining. Each old worker holds its full FD set until its connections close. During a reload, old and new process FDs both count against system limits, and busy hosts can double their FD footprint for the drain duration. Check Stopping: 1 on old workers and look at what triggered the reloads (config management, service discovery, ingress controller).

5. Check the consumption trend. If there is no storm and usage grew steadily, pull the trend for scur/slim and FD count. A slow climb with matching traffic growth means you outgrew the limit. High scur with low req_rate and near-zero qcur means idle or long-lived connections are pinning slots: WebSockets, SSE, or clients with very long keep-alive timeouts.

6. If HAProxy refuses to start. A startup failure like “Cannot raise FD limit to N, limit is M” means the config (or the auto-computed value) wants more FDs than the process is allowed. This bites after runtime or orchestrator upgrades that silently lower LimitNOFILE; some container runtime versions have regressed the default container FD limit from over a million down to 1024 soft. Fix the unit or container spec, not HAProxy.

Metrics and signals to monitor

Alert on FD usage above 80% of the process limit. Below that threshold you still have room to react; above it, a single traffic burst or one extra reload can push you over the cliff in seconds.

Fixes

Raise the process FD limit

If consumption is legitimate, raise RLIMIT_NOFILE for the HAProxy process. Under systemd, set LimitNOFILE= in the unit (a drop-in under /etc/systemd/system/haproxy.service.d/ is the clean way); in containers, set ulimits in the container spec. Size it deliberately: maxconn x 2 + listeners + stats + log + health check + peer FDs, plus headroom for a reload (old and new process coexist during drain). Then restart HAProxy; the new limit only takes effect at process start.

Tradeoff: raising the limit without understanding why consumption grew just delays the next cliff. Do the diagnosis first.

Align maxconn with the FD limit

If you explicitly set maxconn, keep it comfortably below what the FD budget supports. Remember that HAProxy auto-derives Maxconn from the ulimit at startup when you do not set it, so an environment change that lowers the ulimit silently lowers your capacity. After any change, verify the effective values in show info (Maxconn, Maxsock) rather than trusting the config file.

Drain or kill stuck old processes

If a reload storm is the cause, the immediate fix is to stop reloading and let old workers drain. If workers are stuck because of long-lived connections, configure hard-stop-after so draining workers are force-killed after a bounded time. Without it, a single WebSocket connection can pin an old worker (and its entire FD set) indefinitely. Reducing reload frequency (batching config changes, debouncing service-discovery updates) removes the root cause.

Tradeoff: hard-stop-after disconnects clients on the draining worker when it fires. Pick a value that covers your longest legitimate connection.

Reclaim FDs from idle connections

If consumption is idle connections, lower timeout client and timeout http-keep-alive so dead sessions close sooner. For WebSocket or streaming frontends, size maxconn and the FD limit for the expected concurrent long-lived connections, since those FDs are held for hours by design.

Prevention

• Budget FDs explicitly. For every HAProxy host, record the arithmetic: ulimit >= maxconn x 2 + overhead + reload margin. Put the number in the config or unit file as a comment so the next person does not have to reverse-engineer it.
• Alert at 80% FD usage. This is a cliff-edge resource; you want the ticket while there is still runway, not the page after accept() starts failing.
• Watch the process count. Alert on more than 2-3 HAProxy PIDs sustained. Reload storms are the most common way FD exhaustion appears “suddenly” on a host whose traffic did not change.
• Set hard-stop-after. Bounded drain time means bounded FD double-counting during reloads.
• Pin the FD limit in your deployment spec. Whether systemd or containers, make LimitNOFILE / ulimits explicit so a runtime or package upgrade cannot silently shrink it.
• Load-test the limit, not just the traffic. Verify in staging that maxconn connections actually fit inside the FD budget with reloads happening.

How Netdata helps

• Netdata collects per-process FD usage from /proc alongside HAProxy’s own Maxsock, Maxconn, and CurrConns, so you can see consumption approaching the limit on one dashboard instead of correlating by hand during an incident.
• Per-second granularity on ConnRate vs SessRate makes the acceptance gap visible the moment accept() starts failing, rather than after clients report timeouts.
• Process-count and uptime tracking surfaces reload storms: you see the extra workers and the counter resets that explain a sudden jump in system FD usage.
• Session metrics (scur, slim, qcur) let you separate genuine load from idle-connection pile-up when deciding whether to raise limits or tighten timeouts.
• Alerts on FD-usage ratio (used vs limit) give you the 80% early warning this failure mode demands.

Related guides

• HAProxy 502 Bad Gateway: backend connection and response failures
• HAProxy 503 Service Unavailable: no server is available to handle this request
• HAProxy 504 Gateway Timeout: timeout server and the backend timeout cascade
• HAProxy backend losing servers: active server count and cascade risk
• HAProxy backend queue building (qcur): requests waiting for a free server slot
• HAProxy compression overhead: comp_byp, CPU cost, and when compression backfires
• HAProxy connect time (ctime) high: network latency and backend accept-queue overflow
• HAProxy connection errors (econ): backend connections refused, timed out, or reset
• HAProxy scur approaching slim: the concurrent-session saturation signal
• HAProxy 5xx delta: telling HAProxy-generated errors from backend errors
• HAProxy health checks green but the application is broken: when UP does not mean healthy
• HAProxy health check L4TOUT and L4CON: server marked DOWN and unreachable