HAProxy Slowloris and idle-connection pile-up: high scur, low request rate

Your HAProxy frontend shows scur pinned at or near slim, but req_rate is a fraction of what those connections should be producing. Throughput (bin/bout) is near zero, backend queue depth (qcur) is zero because backends are doing nothing, and ereq with 408 responses is climbing. HAProxy is not down. It is not slow. It is full: thousands of connections are holding session slots open and sending data so slowly they may as well be dead.

This is the Slowloris / idle-connection pile-up pattern. It can be a deliberate attack, a misconfigured client fleet, mobile clients on terrible networks, or (the trap) completely legitimate long-lived traffic like WebSocket, SSE, or gRPC streams that look identical in the aggregate stats. The diagnostic work is telling these apart before you start blocking things.

The pattern matters because it sits one step away from a real outage. When scur reaches slim, new connections queue or get rejected, and healthy clients start seeing timeouts and 503s even though backends are idle. See HAProxy maxconn reached for that follow-on failure mode.

What this means

Every proxied connection consumes resources on the HAProxy host: roughly two file descriptors (client side plus, once dispatched, server side), two buffers of tune.bufsize (default 16KB each), and about 20KB of connection metadata. A connection that opened, sent a partial request, then went quiet consumes all of that while producing nothing. When thousands pile up, the global or per-frontend maxconn becomes the ceiling and admission of new connections stops.

The triage signal is the asymmetry in the stats: frontend sessions are high, but the work those sessions generate (requests, bytes, backend connections, queue depth) is absent. A genuine traffic surge raises everything together. This pattern raises only scur.

flowchart TD
  A[scur near slim, req_rate low] --> B{bin/bout growing?}
  B -->|yes, bytes flowing| C[Legit long-lived traffic
WebSocket, SSE, gRPC, uploads]
  B -->|no, near-zero bytes| D{ereq / 408 elevated?}
  D -->|yes| E[Slowloris or stalled clients
hold-and-dribble connections]
  D -->|no| F[Idle keep-alive pile-up
or pre-connect probes]
  E --> G[show sess: source IP distribution]
  G -->|few IPs, high count| H[Attack or broken client fleet]
  G -->|many IPs, low count each| I[Slow clients, mobile networks]

Common causes

Quick checks

All commands below are read-only against the runtime API.

# Per-frontend and global session saturation
echo "show stat" | socat unix-connect:/var/run/haproxy.sock stdio | \
  awk -F, '{if($7>0) print $1"/"$2": scur="$5" slim="$7" pct="int($5/$7*100)"%"}'

# Request rate vs session rate (the divergence is the signature)
echo "show stat" | socat unix-connect:/var/run/haproxy.sock stdio | \
  awk -F, '$2 == "FRONTEND" {print $1": rate="$34" req_rate="$47" ereq="$13" bin="$9" bout="$10}'

# Backend queue depth: zero confirms backends are idle
echo "show stat" | socat unix-connect:/var/run/haproxy.sock stdio | \
  awk -F, '$2 == "BACKEND" {print $1": qcur="$3" scur="$5}'

# Inspect live sessions: who holds the slots, what state are they in
echo "show sess" | socat unix-connect:/var/run/haproxy.sock stdio | less

# Global picture
echo "show info" | socat unix-connect:/var/run/haproxy.sock stdio | \
  grep -E "^(CurrConns|Maxconn|ConnRate|SessRate|Idle_pct):"

Two samples of bin/bout a minute apart tell you whether bytes are flowing at all. ereq and req_tot deltas over the same window quantify the 408 rate relative to completed requests.

How to diagnose it

1. Confirm the signature. scur above 80% of slim, req_rate far below its baseline for that scur, qcur zero on all backends, bin/bout rates near zero. If all four hold, this is an idle-connection pile-up, not a backend-saturation incident.

2. Rule out legitimate long-lived traffic first. This is the step people skip, and it is how you end up blocking your own WebSocket users. Take two show stat samples: if bin/bout deltas are proportional to the session count, those connections are working, not stalled. Legitimate streams also do not produce elevated 408s or ereq. If your app serves SSE or gRPC on this frontend, compare against a baseline from a known-good period before concluding anything.

3. Attribute connections to sources. Run show sess and aggregate by client address; the exact output format is intentionally undocumented and varies between versions, so parse the first line per stream for the source IP. A handful of IPs holding hundreds of sessions each is an attack or a broken client. Thousands of IPs with one session each points to network conditions or client behavior.

4. Check session state and age. In show sess, look at what each connection is doing: waiting for request headers, mid-body, idle in keep-alive. Connections stuck waiting for a complete request for many seconds are the classic slow-dribble signature. Idle keep-alive connections with no prior request point at probes or pre-connecting browsers.

5. Check the logs for termination codes. cR-- means the client side timed out waiting for a complete request (HAProxy emitted a 408). A rising share of cR terminations confirms connections are expiring on timeout http-request (or its fallback, timeout client) rather than completing.

6. Quantify concentration with a stick table. If you already track per-source rates, show table <name> shows which sources hold elevated conn_rate or http_req_rate. If you do not, add one (below) and let it populate; within a minute you will see whether the pile-up is concentrated or diffuse.

7. Check what is not the problem. Idle_pct healthy, rtime normal, econ flat, servers UP. Backends and CPU are fine; the bottleneck is purely admission on the frontend. Do not restart anything.

Metrics and signals to monitor

Fixes

Fixes group into two buckets: expire stalled connections faster, and stop abusive sources from opening unlimited new ones. Apply them in this order, and only after step 2 of the diagnosis ruled out legitimate long-lived traffic.

Bound request completion time

timeout http-request caps how long HAProxy waits for a complete HTTP request before returning a 408 and freeing the slot. If unset, it falls back to timeout client, which is usually far too generous for this purpose.

frontend http-in
    timeout http-request 10s

By default this covers the request headers. To also cover the request body (slow POST attacks), add option http-buffer-request, which makes HAProxy wait for the full body before forwarding and puts the body under the same timeout. The tradeoff is memory: buffered bodies consume buffer space per connection, so size this with tune.bufsize and expected max body size in mind.

Do not set timeout http-request to single-digit seconds blindly. Slow clients on real networks will start eating 408s. Ten seconds is a common starting point for internet-facing frontends; watch the 408 rate after the change.

Two version caveats worth knowing. Older HAProxy releases handled timeout http-request differently for HTTP/2 streams than for HTTP/1, and in some releases the timeout could surface as a 504 instead of a 408. Newer releases also apply the timeout to idle HTTP/2 connection lifetime via GOAWAY, not just per-stream waits. Test against your version before rolling to production.

Tighten idle connection timeouts

timeout client governs how long an idle client connection (including idle keep-alive between requests) may sit. If the pile-up is idle keep-alive connections rather than incomplete requests, lowering this reclaims slots. timeout http-keep-alive controls the idle window specifically between requests. Lowering either disconnects slow-but-legitimate clients sooner, so change incrementally and watch cli_abrt and 408 rates.

Rate limit per source IP

A stick table with per-source request and connection rates lets you cap abusive sources without touching global limits:

frontend http-in
    stick-table type ip size 1m expire 10m store conn_rate(10s),http_req_rate(10s)
    http-request track-sc0 src
    http-request deny deny_status 429 if { sc_conn_rate(0) gt 50 }
    http-request deny deny_status 429 if { sc_http_req_rate(0) gt 100 }

Tune thresholds to your real per-user baseline; NAT gateways and corporate proxies concentrate many users into one IP and will trip aggressive limits. Monitor the table itself with show table: when a table fills, eviction is silent and rate limiting quietly stops working for new sources, which defeats the limiter mid-attack.

What not to do

Do not raise maxconn as the primary response. It buys time by absorbing more idle connections, but each one still costs FDs and memory, and the pile-up will grow to fill the new ceiling. Use a maxconn bump only as a temporary bridge while timeouts and rate limiting take effect. Do not restart HAProxy: you drop all the good connections along with the bad, and the bad ones reconnect in seconds.

Prevention

• Set timeout http-request explicitly on every internet-facing frontend. Do not rely on the timeout client fallback; the two timeouts exist for different purposes.
• Decide where long-lived traffic lives. If you serve WebSocket, SSE, or gRPC, isolate it on a dedicated frontend or listener with its own timeout profile and its own maxconn budget, so stream traffic and request/response traffic do not compete for the same session pool.
• Size maxconn deliberately at all four levels (global, frontend, backend, per-server) and monitor each ratio independently, as covered in the monitoring checklist.
• Baseline the ratio of req_rate to rate per frontend. The divergence is your earliest detection signal, and alerting on it requires knowing what normal looks like for that service.
• Keep per-source stick tables warm on exposed frontends even in peacetime, so attribution data exists the moment a pile-up starts, and alert on table utilization above 80%.
• Watch log termination codes continuously. Aggregate cR share is a quieter, earlier signal than any counter on the stats socket.

How Netdata helps

• Session saturation with context: Netdata charts scur against slim per frontend and backend over time, so you can see the pile-up build as a ratchet rather than a spike, and distinguish it from a normal traffic ramp.
• The divergence chart: plotting rate, req_rate, and bin/bout together on one dashboard makes the signature (sessions up, requests and bytes flat) visible in one glance instead of three CSV queries.
• Error correlation: ereq and 4xx response counts alongside session counts let you confirm the 408 component of the pattern without log parsing during the incident.
• Backend alibi: qcur, rtime, and per-server status on the same screen confirm backends are idle, ruling out a saturation incident and keeping you from chasing the wrong layer.
• Anomaly detection on request rate: ML-based anomaly flags on req_rate catch the “requests fell off a cliff while sessions climbed” divergence even before threshold alerts fire.

Related guides

• How HAProxy actually works in production: a mental model for operators
• HAProxy maxconn reached: new connections queued and rejected
• HAProxy monitoring checklist: the signals every production proxy needs
• HAProxy monitoring maturity model: from survival to expert