HAProxy kernel accept-queue overflow: silent SYN drops and somaxconn

Clients report connection timeouts. You open the HAProxy stats page and everything looks fine: session rate is normal, no 5xx spike, servers are UP, Idle_pct is healthy. The network team sees nothing. The clients insist the load balancer is dropping them.

Both sides are right. The kernel is silently dropping SYN packets in the accept queue before they ever reach HAProxy’s event loop. From HAProxy’s perspective, those connections never existed. There is no log line, no error counter, no stat field that increments. The only evidence lives in two kernel counters, ListenOverflows and ListenDrops, that almost nobody graphs.

This guide shows how to confirm it in under five minutes, how to size the fix, and how to monitor the right signals so it never becomes invisible again.

What this means

When a client opens a TCP connection to HAProxy, the path through the kernel looks like this:

flowchart LR
  C[Client SYN] --> SQ[SYN queue]
  SQ --> AQ[Accept queue
capped by somaxconn]
  AQ -->|accept| HA[HAProxy event loop]
  AQ -.->|queue full| DROP[SYN silently dropped
ListenOverflows +1]
  DROP --> CT[Client sees timeout]
  HA --> HB[HAProxy stats
show nothing wrong]

A new connection first lands in the SYN queue while the three-way handshake completes. Once the handshake finishes, the kernel moves the connection into the accept queue for the listening socket, where it waits for HAProxy to call accept(). The size of that accept queue is the backlog value the application passes to listen(), and here is the part that bites operators: if the backlog is larger than net.core.somaxconn, the kernel silently truncates it to somaxconn. No error, no log, no warning in haproxy -c.

When the accept queue is full, the kernel drops incoming SYNs (or the completed handshakes waiting in the queue, depending on the state). The client retransmits, eventually gives up, and reports a connection timeout. HAProxy never sees any of it.

Three things make the queue fill up:

• The queue is too small for the burst. somaxconn (and net.ipv4.tcp_max_syn_backlog) is sized below the peak connection arrival rate.
• HAProxy cannot call accept() fast enough. The event loop is saturated (check Idle_pct) or there are too few threads, so completed connections pile up in the kernel waiting to be accepted.
• A genuine connection flood. DDoS, a misbehaving client retrying aggressively, or a thundering herd after an upstream outage.

One more trap: net.core.somaxconn is namespaced. If HAProxy runs in a container with its own network namespace, the container gets the kernel default, not the host’s tuned value. Tuning the host sysctl does nothing for the container’s sockets.

Common causes

Quick checks

All of these are read-only and safe to run during an incident.

# 1. The smoking gun: listen overflow and drop counters
nstat -az | grep -i listen
# Look for TcpExtListenOverflows and TcpExtListenDrops.
# These are cumulative; take two samples a few seconds apart to get a rate.

# 2. Current accept-queue depth on HAProxy's listen sockets
ss -tln | grep -E ":(80|443|8404) "
# Recv-Q = connections currently waiting in the accept queue.
# Send-Q = the effective backlog (after kernel truncation to somaxconn).
# Recv-Q persistently near Send-Q means the queue is filling.

# 3. Effective backlog for a specific port, with process info
ss -plnt sport = :443

# 4. What the kernel cap actually is
cat /proc/sys/net/core/somaxconn
cat /proc/sys/net/ipv4/tcp_max_syn_backlog

# 5. If HAProxy runs in a container, check the namespace it actually uses
nsenter -t $(pgrep -x haproxy | head -1) -n cat /proc/sys/net/core/somaxconn

# 6. Can the event loop keep up? (proxy side)
echo "show info" | socat unix-connect:/var/run/haproxy.sock stdio | grep -E "^(Idle_pct|ConnRate|SessRate|CurrConns|Maxconn):"

# 7. Per-thread skew
echo "show activity" | socat unix-connect:/var/run/haproxy.sock stdio

Two things to keep in mind while reading these. First, ListenOverflows and ListenDrops are per-network-namespace counters, not per-socket and not HAProxy-specific. If the host (or namespace) runs other listening services, they contribute too; use ss Recv-Q per port to confirm HAProxy’s sockets are the ones filling. Second, SYN cookies (enabled by default via net.ipv4.tcp_syncookies=1) protect the SYN queue from SYN floods but do not prevent accept-queue overflow. The ListenDrops counter still increments with SYN cookies on, so “syncookies are enabled” is not a reason this cannot be happening.

How to diagnose it

1. Confirm the symptom is pre-accept loss. Take two nstat -az | grep -i listen samples 10 seconds apart while clients report timeouts. If ListenOverflows or ListenDrops is incrementing at a nonzero rate, you have kernel-level drops. If both are flat, this article is not your problem; look at HAProxy connection errors (econ) or HAProxy 503 Service Unavailable instead.

2. Confirm it is HAProxy’s sockets. Run ss -tln against HAProxy’s ports. If Recv-Q is hovering at or near Send-Q during the incident window, HAProxy’s accept queue is the one overflowing. Send-Q also tells you the effective backlog after kernel truncation, which is often smaller than you think.

3. Determine whether the queue is too small or the consumer is too slow. Check Idle_pct in show info. If Idle_pct is below ~20% (and busy-polling is not enabled), the event loop is saturated and cannot accept fast enough; raising somaxconn will only buy a bigger buffer that drains too slowly. If Idle_pct is healthy and the queue still overflows during bursts, the queue is genuinely undersized for the arrival rate.

4. Check the ConnRate vs SessRate gap. In show info, ConnRate counts connections arriving and SessRate counts sessions established. A persistent gap means connections are being lost or rejected before becoming sessions. The gap is not exclusively caused by accept-queue overflow (ACL denials and rate limits also contribute, via dcon/dses on frontend rows), but combined with incrementing ListenDrops it corroborates pre-accept loss.

5. Check for thread skew. If nbthread > 1, run show activity and compare per-thread counters such as loop iterations and run-queue depth. One thread running far hotter than the others is a distribution problem; SO_REUSEPORT (via HAProxy’s shards bind option, available since 2.5) or thread tuning addresses it, not a bigger queue.

6. Rule out a flood. If ConnRate is far above baseline, you may be absorbing a connection flood rather than a sizing problem. Check stick-table conn_rate per source if you track it, and frontend dcon/dses for TCP-level denials.

Metrics and signals to monitor

Note the mirror-image failure on the other side: HAProxy’s own ctime (connect time to backends) rises when a backend’s accept queue overflows. Same kernel mechanism, opposite direction. If your symptom is high ctime rather than client-side timeouts, see HAProxy connect time (ctime) high.

Fixes

Raise net.core.somaxconn (and tcp_max_syn_backlog)

If the queue is undersized for your peak connection arrival rate and the event loop has headroom, raise the ceiling:

# Runtime (takes effect for sockets whose backlog exceeds the old cap;
# a HAProxy reload may be needed for the new value to apply to existing listeners)
sysctl -w net.core.somaxconn=65535
sysctl -w net.ipv4.tcp_max_syn_backlog=65535

Persist both in /etc/sysctl.conf or a drop-in under /etc/sysctl.d/. Sizing guidance: the queue absorbs bursts, so size it for the peak connection arrival rate multiplied by the worst-case accept latency you can tolerate, with margin. Modern kernels (5.4+) default to 4096; older kernels defaulted to 128, and many inherited configs still carry that. There is no meaningful cost to a larger cap: memory is only consumed by connections actually queued.

Two caveats. If HAProxy runs in a container, set the sysctl inside the container’s network namespace (or via the orchestrator’s sysctl support), because the namespace does not inherit the host value. And remember the truncation is silent: whatever backlog HAProxy requests, the effective value is min(backlog, somaxconn). Verify the result with ss -tln Send-Q after the change.

Add accept capacity: nbthread and SO_REUSEPORT

If the event loop cannot accept fast enough, a bigger queue just delays the drop. Spread the accept work:

• Raise nbthread to use more cores, if the host has spare CPU and Idle_pct shows saturation.
• Use shards on the bind line (HAProxy 2.5+) to create multiple listening sockets on the same address with SO_REUSEPORT. Each shard gets its own kernel accept queue, multiplying total queue depth and distributing accepts across threads. Verify with ss -tln that you now see multiple listeners.
• nbproc is the legacy approach and is deprecated in favor of threads plus shards; do not build new deployments on it.

Reduce the burst at the source

If a flood is the cause, queue tuning treats the symptom. Options include rate limiting new connections per source IP with a stick table (conn_rate), fixing a retry-storming client, or absorbing the burst at an upstream layer. During an active flood, raising somaxconn still helps legitimate clients get through, but do not mistake that for the fix.

Do not restart HAProxy first

A restart will drain the queue and make the symptom briefly disappear, which is why this failure so often gets “fixed” without being diagnosed. It also costs you the nstat correlation window and resets all HAProxy counters. Check the counters first, then act.

Prevention

• Graph ListenOverflows and ListenDrops on every HAProxy host. Alert on any sustained increment. This single panel converts the invisible failure into a 30-second diagnosis.
• Alert on Recv-Q near Send-Q for HAProxy listen sockets as a leading indicator, before drops begin.
• Graph ConnRate and SessRate together. A widening gap, correlated with dcon/dses and ListenDrops, tells you where connections are dying.
• Size somaxconn and tcp_max_syn_backlog deliberately for peak connection rate, in the same sysctl drop-in, in every environment including container namespaces, and verify with ss Send-Q after every deploy.
• Watch Idle_pct (or show activity run-queue depth under busy-polling) so you know whether accept capacity is a queue problem or a consumer problem before the next burst.
• Load-test the accept path, not just request throughput. A burst of N new connections per second is a different workload than N requests over keep-alive sessions, and only the former exercises this queue.

How Netdata helps

• Netdata collects the TCP extended counters from /proc/net/netstat, including ListenOverflows and ListenDrops, per second and per node, so the silent drop becomes a visible, alertable timeseries instead of a counter nobody reads.
• The HAProxy collector pulls show info and stats CSV over the runtime socket, putting ConnRate, SessRate, Idle_pct, scur/slim, and dcon/dses on the same dashboard as the kernel counters, which is exactly the correlation this diagnosis needs.
• Anomaly detection on ConnRate and session rate surfaces the burst that triggered the overflow, and anomaly flags on ListenDrops catch the first increments before clients notice.
• Because Netdata monitors the host and HAProxy together, you can see in one view whether the event loop was saturated (Idle_pct collapsing) or idle (queue purely undersized) at the moment drops started, which determines which fix to apply.

Related guides

• HAProxy scur approaching slim: the concurrent-session saturation signal
• HAProxy backend queue building (qcur): requests waiting for a free server slot
• HAProxy connect time (ctime) high: network latency and backend accept-queue overflow
• HAProxy connection errors (econ): backend connections refused, timed out, or reset
• HAProxy 503 Service Unavailable: no server is available to handle this request
• HAProxy 504 Gateway Timeout: timeout server and the backend timeout cascade