HAProxy 504 Gateway Timeout: timeout server and the backend timeout cascade

Your frontend error rate jumps. Clients report “504 Gateway Timeout.” HAProxy looks alive: the process is running, servers are UP, health checks are green. Yet requests are dying on a timer.

This is the backend timeout cascade, and the 504 is its final symptom. The servers are reachable (TCP accepts succeed), but the application behind them has slowed past the timeout server budget. HAProxy holds the client connection, waits, gives up, and generates a 504 itself.

The key distinction: a 504 from HAProxy is almost never a connectivity problem. It is a “server accepted the connection but never answered in time” problem. That separates it from 503 (no server available, saturation, or queue overflow) and 502 (connect refused or a broken/truncated response). Getting that distinction right is the difference between debugging the network and debugging the application.

What this means

timeout server is the maximum inactivity time HAProxy tolerates on the server side of a proxied connection. In HTTP mode its most important role is the first phase of the response: the time the server takes to send response headers. That interval directly represents how long the backend application spent processing the request. When the timer fires, HAProxy aborts the server-side connection and returns 504 to the client. If timeout server is never set, the timeout is effectively infinite and HAProxy warns at startup, so a hung backend can hold connections forever.

The cascade builds like this: the application (or its database, cache, or downstream API) slows down. Each slow request holds a server connection slot longer. Servers stay UP because TCP accepts succeed and health checks may still pass. New requests have no free slot, so they queue (qcur rises, qtime grows). Response time (rtime) climbs toward timeout server. Once enough requests cross the threshold, timeout server starts firing and 504s appear on the frontend. The queue keeps filling because the backend is not recovering. Left alone, this degrades into the backend collapse pattern as health checks start failing under load.

flowchart TD
  A[Backend dependency slows] --> B[App threads held, rtime climbs]
  B --> C[Server slots stay busy]
  C --> D[qcur rises, requests queue]
  D --> E[Requests exceed timeout server]
  E --> F[HAProxy emits 504]
  C --> G[Health checks still pass, servers UP]
  G --> D
  F --> H[If unchecked: health checks fail, backend collapse]

Two related timers produce different errors, so do not confuse them. timeout connect governs establishing the TCP connection to the server; when it fires you typically see 503 or 504 with rising econ. timeout http-request governs how long HAProxy waits for the client to send a complete request, and produces a 408, not a 504. If you are seeing 408s, you are in idle-connection territory, not backend-timeout territory.

Common causes

Quick checks

# Per-server and per-backend response time, queue depth, and errors
echo "show stat" | socat unix-connect:/var/run/haproxy.sock stdio | \
  awk -F, '$2 != "FRONTEND" {print $1"/"$2": status="$18" qcur="$3" econ="$14" eresp="$15" rtime="$61"ms 5xx="$44}'

# Session utilization at every level (rule out saturation)
echo "show stat" | socat unix-connect:/var/run/haproxy.sock stdio | \
  awk -F, '{if($7>0) print $1"/"$2": scur="$5" slim="$7" pct="int($5/$7*100)"%"}'

# Frontend vs backend 5xx: is HAProxy generating the errors?
echo "show stat" | socat unix-connect:/var/run/haproxy.sock stdio | \
  awk -F, '{print $1"/"$2": 5xx="$44}'

# Retries and redispatches: is HAProxy masking instability?
echo "show stat" | socat unix-connect:/var/run/haproxy.sock stdio | \
  awk -F, '$2 != "FRONTEND" {print $1"/"$2": wretr="$16" wredis="$17}'

Then confirm in the logs. With option httplog, the termination flags tell you exactly which timer fired:

• sH means timeout server fired before the server returned response headers. This is the classic 504 signature and the most common anomaly; the manual describes it as typically caused by server or database saturation.
• sD means the server stopped sending or acknowledging data mid-response, during the data phase. Different failure: headers arrived, then the transfer stalled.
• sC means timeout connect fired; the connection to the server never completed. In HTTP mode this can surface as 503 or 504.

The log timing fields confirm the mechanism: Tr is the time spent waiting for the server to send a full response (excluding data transfer). When 504’d requests show Tr values at or just past timeout server, the backend simply did not answer in time.

How to diagnose it

1. Confirm HAProxy is the one generating the 504s. Compare frontend hrsp_5xx rate against the sum of backend hrsp_5xx rates. If frontend 5xx is well above backend 5xx, HAProxy is synthesizing the errors (timeouts, no-server 503s). If they are nearly equal, the backends are returning the errors themselves and this is an application incident, not a timeout incident.

2. Read the termination flags. Grep your HAProxy logs for sH vs sD vs sC. sH means the server never sent headers in time; sD means the response stalled mid-body; sC means you are actually looking at a connect problem. Do not tune timeout server for an sC problem.

3. Check per-server rtime. Is every server in the backend slow, or just one? All servers slow points to a shared dependency (database, cache, downstream API). One server slow points to that host: GC pauses, disk, a bad deploy on a canary.

4. Verify econ is low. The timeout cascade signature is “servers UP, econ low, rtime climbing.” If econ is rising with ctime near timeout connect, you have a reachability or SYN-queue problem instead. If econ is low but eresp is rising, servers are dying mid-response, which is closer to 502 territory.

5. Assess the queue. Nonzero, growing qcur with rising qtime means every server slot is held by slow requests. Express qtime against your timeout budget: if timeout server is 30s and qtime is 15s, half of a request’s budget is gone before it reaches a server, so even healthy rtime values will tip it into 504.

6. Rule out saturation. Check scur/slim at all four levels (global, frontend, backend, per-server). If scur is pinned at slim with normal rtime, the bottleneck is admission, not speed; that is the maxconn exhaustion pattern, and raising timeout server will make it worse by holding doomed requests longer.

7. Size the timeout against reality. Pull Tr for successful slow requests from the logs and compare the distribution’s tail against timeout server. If your legitimate p99 is 25s and timeout server is 30s, any minor degradation produces 504s. The timeout should sit above real p99 with headroom, not at the mean.

8. Investigate the backend dependency. The proxy has told you everything it can: the answer to “why is rtime climbing” lives in the application, its database, or its downstream calls. Shift to those signals.

Metrics and signals to monitor

Fixes

Slow backend or dependency

This is the majority case and the fix is not in HAProxy. Reduce load or restore the dependency: kill the expensive query, scale the app tier, shed traffic. HAProxy-side, per-server maxconn set to what the application can actually handle creates backpressure and converts a silent pile-up into visible queuing earlier, which is easier to alert on than 504s. If some servers are healthy and some are hung, put the hung ones in MAINT via the runtime API to stop HAProxy feeding them while you investigate.

Timeout sized wrong

If legitimate requests genuinely take longer than timeout server (report generation, large exports, slow search), raise the timeout for those routes rather than globally. http-request set-timeout, introduced in HAProxy 2.4, lets you adjust timeout server per request via ACLs, so you can give /report 120s while keeping the default at 30s. The tradeoff of any increase: slow requests hold server slots and frontend connections longer, so the queue grows deeper before errors surface. Raising a timeout to mask a slow backend trades 504s for resource exhaustion. Treat timeout server as an SLO enforcement mechanism, not a dial to make errors disappear.

WebSocket and tunnel traffic

Once a connection upgrades to a tunnel, timeout tunnel supersedes both timeout client and timeout server. If your 504s are confined to WebSocket connections that established successfully, tune timeout tunnel, not timeout server. A precedence issue reported against HAProxy 2.2.x (haproxy issue #2280) shows that backend-level timeout tunnel does not always override defaults for WSS connections, so verify that your tunnel timeout is actually applied.

One more caveat: slow uploads

There is a long-standing report of timeout server firing mid-upload at exactly its configured value even while data is actively flowing from a slow client, producing intermittent 504s on large uploads (haproxy issue #949). The reported workaround was raising timeout server substantially. If your 504s correlate with large request bodies from slow clients and the termination flag is sD, this pattern is a candidate explanation.

Prevention

• Set timeouts explicitly, everywhere. An unset timeout server is infinite, with only a startup warning to tell you. Pick a value from measured p99 latency plus headroom, not a round number copied from a default config. If you still have srvtimeout, contimeout, or clitimeout in old configs, replace them with timeout server, timeout connect, and timeout client; the old directives are deprecated.
• Alert on the cascade, not just the 504. Page when servers are UP, qcur > 0, rtime > 2x baseline, and frontend 5xx exceeds about 1% of requests, sustained for 5 minutes with a traffic floor. That catches the cascade minutes before users do.
• Watch the leading signals as tickets. Per-server rtime deviation, nonzero qtime, and any sustained wretr/wredis rate all precede the first 504. None of them require an error to fire.
• Keep log timing fields and termination flags. The stats CSV gives you rolling 1024-sample averages, which smooth out exactly the tail behavior that produces 504s. Per-request Tr and termination flags in logs are where the proof lives. Also monitor DroppedLogs; losing logs mid-incident removes your best evidence.
• Handle counter resets. hrsp_5xx, econ, and friends reset on every reload. Gate alerts on Uptime_sec so a reload does not look like a recovery or a fresh spike.

How Netdata helps

• Netdata collects the HAProxy stats CSV continuously, so per-server rtime, qcur, econ, eresp, and scur/slim are charted together rather than sampled by hand during the incident.
• The frontend vs backend 5xx split is visible on separate charts, making the “is HAProxy generating these errors?” question a glance instead of a computation.
• Correlating rtime climbing against flat econ and UP server status on one dashboard surfaces the timeout cascade signature directly, and distinguishes it from saturation (scur at slim) and connect failures (econ rising).
• Anomaly detection on per-server rtime flags the single slow server in a pool before the average moves enough to notice.
• Because counters reset on reload, continuous collection with proper counter handling avoids the phantom drops and spikes that mislead rate-based diagnosis.

Related guides

• HAProxy backend queue building (qcur): requests waiting for a free server slot
• HAProxy scur approaching slim: the concurrent-session saturation signal
• HAProxy health checks green but the application is broken: when UP does not mean healthy
• HAProxy health check L4TOUT and L4CON: server marked DOWN and unreachable
• HAProxy health check L7STS: server DOWN on the wrong HTTP status
• How HAProxy actually works in production: a mental model for operators
• HAProxy maxconn hierarchy: global, frontend, backend, and per-server limits
• HAProxy maxconn reached: new connections queued and rejected
• HAProxy monitoring checklist: the signals every production proxy needs
• HAProxy monitoring maturity model: from survival to expert
• HAProxy server flapping: rise, fall, and health checks oscillating UP and DOWN
• HAProxy Slowloris and idle-connection pile-up: high scur, low request rate