HAProxy 5xx delta: telling HAProxy-generated errors from backend errors

Your frontend hrsp_5xx counter is climbing and users are seeing errors. The first question that decides the next hour of your incident response: is HAProxy generating these errors, or is it passing through errors your backend servers produced?

HAProxy counts 5xx responses in both places. Frontend rows count every 5xx the client saw, including responses HAProxy fabricated itself. Backend and server rows count only 5xx responses that actually came back from a server. The difference between the two answers “is the backend failing or is HAProxy failing?” without walking every backend one by one.

This is an expert-level signal (see the maturity model), but the mechanics are simple: subtract, then interpret the sign of the result.

What this means

Three failure modes produce 5xx responses that clients see, and only one of them involves your application:

• 503 Service Unavailable generated by HAProxy: no server was available to handle the request. All servers DOWN, per-server or per-backend maxconn reached with a full queue, or a maxqueue limit exceeded.
• 504 Gateway Timeout generated by HAProxy: timeout server (or the queue wait) expired before the backend responded. The backend may be hung, not dead.
• 502 Bad Gateway generated by HAProxy: the connection to the server failed, or the server returned an empty, invalid, or incomplete response.

All three increment frontend hrsp_5xx. None of them increment backend hrsp_5xx, because no valid 5xx response ever came back from a server. By contrast, when your application returns a 500, HAProxy passes it through and both counters increment.

So:

• Delta > 0 (frontend above backend sum): HAProxy is the source. Capacity, connectivity, health, or timeout problem in the proxy layer.
• Delta near zero: the application is the source. Every 5xx the client saw came from a real server response.
• Delta < 0 (backend sum above frontend): retries or aborted transfers are hiding responses. See the caveats below.

flowchart TD
  A[Frontend hrsp_5xx rising] --> B[Compute delta:
frontend 5xx rate minus
sum of backend 5xx rates]
  B --> C{Sign of delta}
  C -->|Positive| D[HAProxy-generated errors:
503 no server, 504 timeout,
502 connect failure]
  C -->|Near zero| E[Backend-returned errors:
application 500s passed through]
  C -->|Negative| F[Retries succeeding or
aborted transfers masking responses]
  D --> G[Check server status,
qcur, scur/slim, econ, rtime]
  E --> H[Check per-server hrsp_5xx
and application logs]
  F --> I[Check wretr, wredis,
cli_abrt, srv_abrt]

Common causes

Quick checks

All commands are read-only against the stats socket. Adjust the socket path if yours differs.

# 1. Pull hrsp_5xx for every row (frontend, backend, server)
echo "show stat" | socat unix-connect:/var/run/haproxy.sock stdio | \
  awk -F, '{print $1"/"$2": 5xx="$44}'

# 2. Backend aggregate health status (503 source check)
echo "show stat" | socat unix-connect:/var/run/haproxy.sock stdio | \
  awk -F, '$2 == "BACKEND" {print $1": "$18}'

# 3. Connection and response errors (502 source check)
echo "show stat" | socat unix-connect:/var/run/haproxy.sock stdio | \
  awk -F, '$2 != "FRONTEND" {print $1"/"$2": econ="$14" eresp="$15}'

# 4. Saturation and queue depth (503 source check)
echo "show stat" | socat unix-connect:/var/run/haproxy.sock stdio | \
  awk -F, '{if($7>0) print $1"/"$2": scur="$5" slim="$7" qcur="$3}'

# 5. Response time (504 source check, rolling avg over last 1024 requests)
echo "show stat" | socat unix-connect:/var/run/haproxy.sock stdio | \
  awk -F, '$2 != "FRONTEND" {print $1"/"$2": rtime="$61"ms"}'

# 6. Retries, redispatches, and client aborts (negative-delta check)
echo "show stat" | socat unix-connect:/var/run/haproxy.sock stdio | \
  awk -F, '$2 != "FRONTEND" {print $1"/"$2": wretr="$16" wredis="$17" cli_abrt="$50}'

hrsp_5xx is a cumulative counter that resets on reload. Take two samples a known interval apart and work with rates, not raw values. A single snapshot tells you nothing about direction.

How to diagnose it

1. Compute the delta as a rate. For each frontend, take the frontend hrsp_5xx rate and subtract the sum of the hrsp_5xx rates of the backends it routes to. Worked example: frontend web shows 40 5xx/s. Its backends app and api show 8/s and 2/s. Delta = 40 - 10 = 30/s of HAProxy-generated errors.

2. If the delta is positive, classify by status code. You need logs or the stats page response-code breakdown for this step: 503 points at availability (servers DOWN, maxconn, queue overflow), 504 points at timeouts, 502 points at connect failures or invalid responses. Then jump to the matching deep-dive: 503 no server available, 504 timeout cascade, or 502 connect and response failures.

3. For 503s, check the routing table first. Any server not UP stops receiving traffic. If the whole backend is DOWN, every request gets a fabricated 503. If only some servers are DOWN, check whether survivors are saturating: backend losing servers and scur approaching slim.

4. For 504s, confirm the timeout cascade signature. Servers UP, econ low, rtime rising past half of timeout server, qcur growing. The backend is hanging, not refusing. See backend queue building.

5. For 502s, separate connect failures from broken responses. Rising econ with low ctime is immediate refusal; rising econ after high ctime is a connect timeout. Rising eresp is the server closing or truncating mid-response. See ctime high.

6. If the delta is near zero, stop looking at HAProxy. Every 5xx came from a server. Check per-server hrsp_5xx to see whether one server is sick or all of them are. If all servers are UP and healthy by checks while returning 500s, you are in the health check green, app broken pattern. Engage the application side.

7. If the delta is negative, account for masking. When HAProxy retries a request and the retry succeeds, the backend counter can record the failed attempt while the client never sees a 5xx. Client disconnects mid-request produce the same asymmetry: the server sent its response, but nothing was delivered. Check wretr, wredis, and cli_abrt before concluding the numbers are broken.

Metrics and signals to monitor

Fixes

Delta positive: errors HAProxy generated

Servers DOWN or missing. Restore capacity: fix the failing servers, or correct the health check if it is marking healthy servers DOWN (wrong port, wrong expected status). If a deploy is flapping servers, consider MAINT on the offenders to stop redistribution churn while you roll back. Tradeoff: forcing a sick server UP floods it again; confirm the root cause first.

Saturation. If scur is pinned at slim, either raise the relevant maxconn (remember the hierarchy: global, frontend, backend, per-server, and each level throttles independently) or reduce connection hold time by fixing slow backends. Raising maxconn without checking memory and file descriptor headroom just moves the cliff. See the maxconn hierarchy.

Timeouts. A 504 delta means backends exceed timeout server. The durable fix is backend latency; the temporary fix is raising the timeout, which trades user-facing errors for user-facing slowness and holds connection slots longer, worsening saturation. Treat timeout increases as a bridge, not a fix.

Connect failures. Rising econ means the network path or the server’s accept behavior is broken: refused connections, full SYN backlog, security group changes, or ephemeral port exhaustion on the HAProxy host under high churn with poor connection reuse.

Delta near zero: errors the application returned

HAProxy is doing its job. Identify whether one server or all servers produce the 5xx (per-server hrsp_5xx). A single bad server argues for pulling it; a uniform distribution argues for an application-level bug, dependency failure, or bad deploy. Note the fast-failure trap: a server returning instant 500s shows low rtime, so latency dashboards look healthy while error rate climbs.

Delta negative: masked failures

Do not “fix” the counters. Reduce retry storms by stabilizing the backends that trigger wretr/wredis, and treat elevated retries as the early warning they are: HAProxy is compensating for instability that will become user-visible the moment retries stop succeeding.

Prevention

• Alert on the delta, not just the totals. A ticket-level alert on a sustained positive frontend-minus-backend 5xx delta catches proxy-layer faults (capacity, health, timeouts) independently of application error budgets.
• Handle counter resets. hrsp_5xx resets on every reload. Rate calculations must tolerate resets, and Uptime_sec from show info is the reliable way to detect one.
• Watch retries continuously. wretr/wredis nonzero means HAProxy is hiding a problem from your users and from your error metrics at the same time.
• Keep health checks honest. A delta near zero during an outage usually means health checks pass while the app fails. Health checks should exercise the dependencies real requests use.
• Log per-request status and termination codes. Stats counters tell you the delta exists; logs tell you which requests and which termination states produced it. Protect the log pipeline: DroppedLogs in show info incrementing during an incident is the worst time to lose visibility.

How Netdata helps

• Netdata collects HAProxy stats per frontend, backend, and server, so frontend and backend hrsp_5xx series are available side by side and the delta is visible without manual socket polling.
• Per-second collection catches short 503 bursts from brief maxconn saturation that interval scrapes average away.
• The delta becomes actionable when correlated on one dashboard with server status, qcur, scur/slim, econ, eresp, and rtime: the signals that disambiguate 503 from 504 from 502.
• Retry and redispatch counters (wretr, wredis) charted next to the 5xx delta explain negative or shrinking deltas and surface masked backend instability before it becomes user-visible.
• Anomaly detection on the delta itself flags the moment HAProxy starts generating errors, even when absolute error counts are still small.

Related guides

• HAProxy 502 Bad Gateway: backend connection and response failures
• HAProxy 503 Service Unavailable: no server is available to handle this request
• HAProxy 504 Gateway Timeout: timeout server and the backend timeout cascade
• HAProxy backend losing servers: active server count and cascade risk
• HAProxy backend queue building (qcur): requests waiting for a free server slot
• HAProxy connect time (ctime) high: network latency and backend accept-queue overflow
• HAProxy scur approaching slim: the concurrent-session saturation signal
• HAProxy health checks green but the application is broken: when UP does not mean healthy
• HAProxy maxconn hierarchy: global, frontend, backend, and per-server limits
• How HAProxy actually works in production: a mental model for operators