HAProxy health check L7STS: server DOWN on the wrong HTTP status

A backend server flips to DOWN in the stats page, last_chk shows L7STS, and the log line says something like Layer7 wrong status, code: 503, info: "Service Unavailable". The server process is running. The port is open. You can curl it by hand and get a response. But HAProxy has pulled it out of rotation and traffic is concentrating on the survivors.

L7STS means: the TCP connection worked, the HTTP response arrived in time, and the status code was not the one the health check expects. The failure is at the application layer, not the network layer. That narrows the search space considerably, but only if you know what status HAProxy was actually expecting.

This page covers what L7STS means, how it differs from the other last_chk codes, why the server did not go DOWN on the first failed check, and the small set of causes behind almost every L7STS incident in practice. For the wider health-checking picture, see how HAProxy actually works in production.

What this means

HAProxy’s health check engine runs independently of traffic. It probes each server on a schedule and classifies the result into a short code visible in the last_chk stats field, in the stats web UI, and in log lines when a state change happens. The layer-7 codes you will see most often:

L7STS is the interesting one operationally because both ends did their jobs at the protocol level. The kernel accepted the connection, the application read the request and wrote a complete HTTP response, and HAProxy parsed it successfully. The application simply answered with a status the check is configured to reject. Compare:

• L7RSP points at a protocol mismatch or a broken intermediary: the server answered, but not with parseable HTTP.
• L7TOUT points at a hung application or an overloaded event loop: the request went in, nothing came back before the timeout.
• L4TOUT / L4CON never got to HTTP at all: TCP timeout or connection refused.

The key fact about L7STS is that the “expected” status is a matter of configuration, and the default is broader than most operators assume. When no http-check expect directive is configured, HAProxy treats any 2xx or 3xx response as healthy. Anything else, including 401, 403, 404, 405, and every 5xx, produces L7STS.

Common causes

Two of these deserve emphasis because they generate the most confusion:

A 403 on the health endpoint. The server is up, the endpoint exists, but something in the request is missing: a Host header the virtual host routing needs, a token, a source-IP allowlist entry. HAProxy sends a minimal request by default. If your application or an upstream filter gates on any of that, you get L7STS/403 while manual tests with full headers pass.

The default check request itself. When you configure option httpchk with no method or URI, the documented defaults are an OPTIONS request to /. If your application does not answer OPTIONS / with a 2xx or 3xx, for example because it returns 405 for unsupported methods, the check fails on a healthy application. Explicitly configuring the method and URI (option httpchk GET /healthz) removes this whole class of surprise.

Quick checks

All of these are read-only.

# Per-server status, check counters, and last check result in one pass
echo "show stat" | socat unix-connect:/var/run/haproxy.sock stdio | \
  awk -F, '$2 != "FRONTEND" && $2 != "BACKEND" {print $1"/"$2": status="$18" chkfail="$22" chkdown="$23" lastchg="$24"s last_chk="$57}'

The last_chk field carries the detail: L7STS/500 is not the same investigation as L7STS/404. The log line emitted at the state change also includes the code and the reason string.

# Reproduce the check by hand from the HAProxy host
curl -sv -o /dev/null -w "%{http_code}\n" http://<server-ip>:<port>/<check-uri>

Run this exactly as the check runs it: same URI, and if your config sets a Host header or specific method in the check, replicate those. A mismatch between what curl sends and what HAProxy sends is the whole story in a large fraction of L7STS incidents.

# Watch the check arrive from the application side
# (web server access log on the backend)
tail -f /var/log/nginx/access.log | grep healthz

If the check request never appears in the application log, something between HAProxy and the app is answering or dropping it. If it appears with an unexpected method or path, the check configuration is not what you think it is.

# Confirm what HAProxy is actually configured to send
haproxy -c -f /etc/haproxy/haproxy.cfg && grep -E "httpchk|http-check|option.*check" /etc/haproxy/haproxy.cfg

Look for http-check expect lines. If none exist, the expected set is 2xx and 3xx by default.

# Is the application failing real traffic too?
echo "show stat" | socat unix-connect:/var/run/haproxy.sock stdio | \
  awk -F, '$2 != "FRONTEND" {print $1"/"$2": 5xx="$44}'

Per-server hrsp_5xx is the cross-check. Elevated 5xx on the same server means the health check is accurately reporting a broken application. Flat 5xx with L7STS means the check endpoint or the check request is the problem, not the app.

How to diagnose it

1. Read the exact code. Get last_chk for the affected server and note the status code after the slash. L7STS/503 and L7STS/404 are different incidents.

2. Check the transition state. If the server shows DOWN 2/3 or UP 1/3, it is mid-transition under the rise/fall counters, not fully committed to a state. More on this below.

3. Correlate with real traffic. Compare per-server hrsp_5xx and frontend versus backend 5xx. If backend 5xx is elevated on the same server, the application is broken and the health check is doing its job. If frontend and backend 5xx are both quiet, the check is failing in isolation.

4. Reproduce the request manually. From the HAProxy host, send the exact request the check sends: same method, URI, HTTP version, and any configured headers. The status you get back should match the L7STS code. If it does not, the difference between your manual request and HAProxy’s request is the bug.

5. Check the application log for the check request. Confirm the method, path, and response the server logged. This catches config drift: a deploy that moved the health endpoint, added auth, or started rejecting the check’s HTTP version.

6. Check timing. A server that alternates between L7OK and L7STS with small lastchg values is flapping: intermittently returning a bad status, usually under load. That points at resource exhaustion or a dependency that fails intermittently, not at a config error.

flowchart TD
  A[Health check fires] --> B{TCP connect OK?}
  B -- no --> C[L4TOUT / L4CON]
  B -- yes --> D{Response before timeout?}
  D -- no --> E[L7TOUT]
  D -- yes --> F{Parseable HTTP?}
  F -- no --> G[L7RSP]
  F -- yes --> H{Status expected?}
  H -- yes --> I[L7OK]
  H -- no --> J[L7STS: check the status code after the slash]

Why the server did not go DOWN immediately

HAProxy does not mark a server DOWN on a single failed check. The fall parameter sets how many consecutive failed checks are required before the status flips; the usual default is fall 3. Likewise rise sets consecutive successes needed to come back UP. While the counters are in progress, the stats show transitional states: DOWN 2/3 means two of the three required failures have happened; UP 1/3 means one of the required successes on the way back.

This has two operational consequences:

• Detection delay. With fall 3 and a 2-second check interval, a hard failure takes roughly 4 to 6 seconds to pull the server. During that window, traffic still flows to the failing server. If your checks run on a slower interval, the window is proportionally longer.
• chkfail without DOWN. chkfail increments on every failed check, but the server only transitions after fall consecutive failures. A server with a steadily rising chkfail and a stable UP status is failing intermittently below the threshold. That is flapping territory and worth investigating before it tips over. chkdown counts the actual transitions to DOWN, and lastchg tells you how long the current state has held.

A fast-moving L7STS incident therefore shows up in counters before it shows up in status. Watching chkfail rate and last_chk content gives you earlier warning than watching UP/DOWN alone.

Metrics and signals to monitor

The cascade angle matters: every server pulled by L7STS pushes its share of traffic onto the survivors. If survivors were already warm, their scur climbs toward slim, queuing starts, and their response times degrade, which can tip their own health checks over. One bad check configuration can take down a whole backend this way. See HAProxy backend queue building (qcur) and scur approaching slim for the follow-on signals.

Fixes

If the application is genuinely broken. The check is working as designed. Fix the application or its failing dependency; HAProxy will bring the server back automatically after rise consecutive passes. Do not loosen the check to silence the alarm. If flapping is oscillating the server in and out of rotation and making things worse, put it in MAINT via the runtime API while you work, and watch survivor load.

If the check endpoint or request is wrong. Make the check explicit instead of relying on defaults. Configure option httpchk with a specific method and URI that the application actually serves with a 2xx, and align http-check expect with what that endpoint returns. If the backend requires a Host header or a specific HTTP version, set it in the check. The general shape:

backend app
    option httpchk GET /healthz
    http-check expect status 200
    server app1 10.0.0.11:8080 check inter 2s fall 3 rise 2

Validate with haproxy -c -f before reloading, and reproduce the exact request with curl first so you know what status to expect.

If the expectation is too strict. An endpoint that legitimately returns a redirect or a 204 will fail a naive http-check expect status 200. Either widen the expectation to match what the endpoint returns by design, or point the check at an endpoint whose semantics you control.

If the check is fine but the endpoint is trivial. This is the deeper design problem. A /health endpoint that returns a static 200 proves almost nothing: the app can be returning 500 on every real request while the check stays green. The inverse failure also exists, where the trivial endpoint breaks while the app is fine. The durable fix is a health endpoint that exercises the same critical dependencies real requests use, at a cost the check interval can sustain. This “health check green, app broken” blind spot is one of the most common postmortem findings. It is also why an HTTP check beats a bare TCP check: TCP only proves the port accepts connections, which is exactly the failure L7STS exists to catch. The tradeoff is that a deeper check costs more per probe and can fail for dependency reasons that do not affect all request paths equally.

Prevention

• Make check requests explicit. Always configure method, URI, and expected status. Never rely on the default request shape against an application you do not fully control.
• Alert on chkfail rate and last_chk content, not just UP/DOWN. You get minutes of warning instead of learning about it when the server leaves rotation.
• Track per-server hrsp_5xx next to health status. This pair distinguishes “check lies” from “app broken” in one glance.
• Log the status code on state changes and keep those lines. L7STS/404 in the log at 03:12 is the whole diagnosis.
• Review rise/fall and inter deliberately. Faster detection means less traffic sent to a failing server, but also more sensitivity to transient blips. Know your detection window: interval times fall.
• Test the check after every deploy that touches routing, auth, or the health endpoint. A one-line curl from the HAProxy host, replicating the check request, catches most L7STS regressions before the counters do.

How Netdata helps

Netdata collects the HAProxy stats CSV continuously, so the signals above are already correlated on one dashboard instead of requiring manual socket queries during an incident:

• Per-server health status and check counters (status, chkfail, chkdown, lastchg) plotted over time, so flapping and transition patterns are visible rather than inferred from snapshots.
• Per-server hrsp_5xx alongside health state, which is the fastest way to separate a genuine application failure from a check misconfiguration.
• Frontend versus backend 5xx breakdown, exposing the HAProxy-generated error delta when servers leave rotation and 503s start.
• Survivor saturation signals (scur/slim, qcur) on the same timeline as server DOWN events, so you can see a cascade forming while there is still time to act.
• Check-related counters at per-second granularity, catching short flaps that minute-resolution polling averages away.

Related guides

• HAProxy backend queue building (qcur): requests waiting for a free server slot
• HAProxy scur approaching slim: the concurrent-session saturation signal
• How HAProxy actually works in production: a mental model for operators
• HAProxy maxconn hierarchy: global, frontend, backend, and per-server limits
• HAProxy maxconn reached: new connections queued and rejected
• HAProxy monitoring checklist: the signals every production proxy needs
• HAProxy monitoring maturity model: from survival to expert