HAProxy server flapping: rise, fall, and health checks oscillating UP and DOWN

A backend server that alternates between UP and DOWN is worse than one that is cleanly down. Every transition redistributes traffic: when the server goes DOWN its connections move to the survivors, when it comes back UP it absorbs a fresh share of load before it has warmed up, and every cycle costs retries, redispatches, and connection churn. If the flap rate is high enough, the oscillation itself becomes the incident.

The signature is easy to spot in the stats: chkdown keeps incrementing, lastchg (seconds since the last status change) never grows large, and the status field cycles through UP, DOWN 1/3, DOWN, UP 1/3 and back. Meanwhile chkfail climbs even while the server still shows UP, because individual checks are failing below the fall threshold.

This guide covers how to confirm flapping, how to read the health check counters to find the cause, how to stop the churn safely while you investigate, and how to tune rise, fall, inter, and the check timeout so a borderline server stops oscillating.

What this means

HAProxy’s health check engine runs independently of traffic. Each server is probed every inter milliseconds (default 2000). The state transitions are threshold-based:

• A DOWN server must pass rise consecutive checks (default 2) before it returns to service.
• An UP server must fail fall consecutive checks (default 3) before it is marked DOWN.

The transitional states appear in the stats as UP 1/3 or DOWN 2/3 style values: the fraction shows progress through the rise/fall counter. Flapping means the server lives on the boundary: it fails fall checks, goes DOWN, the load lifts off it (or the transient fault clears), it passes rise checks, comes back UP, receives traffic again, and fails again. Each cycle is a few inter periods long, so with defaults a full flap can complete in under 20 seconds.

flowchart TD
  UP["UP (serving traffic)"]
  TDN["DOWN n/fall (transitional)"]
  DOWN["DOWN (no traffic)"]
  TUP["UP n/rise (transitional)"]
  UP -->|"fall consecutive check failures"| TDN
  TDN -->|"threshold reached"| DOWN
  TDN -->|"a check passes"| UP
  DOWN -->|"checks pass"| TUP
  TUP -->|"rise consecutive successes"| UP
  TUP -->|"a check fails"| DOWN

Why this matters operationally:

• Traffic churn. Every DOWN transition redistributes the server’s share of traffic to survivors. If the survivors are near capacity, the extra load pushes their response times up, which can make their own checks slower. This is how one flapping server starts a collapse cascade.
• Retry noise. Requests in flight when the server drops produce wretr and wredis events and latency outliers, even when users never see an error.
• Counter semantics hide the real failure rate. chkfail only increments for checks that fail while the server is UP. Once the server is DOWN, further failed checks do not move chkfail. A flapping server can therefore look less broken than it is if you only watch chkfail. chkdown is the counter that tells the truth about oscillation.

Common causes

Quick checks

All of these are read-only against the runtime socket. Column numbers below match the standard show stat CSV field order; the first line of show stat output is a header row you can use to verify positions on your version.

# Status, check counters, and last change time for every server
echo "show stat" | socat unix-connect:/var/run/haproxy.sock stdio | \
  awk -F, '$2 != "FRONTEND" && $2 != "BACKEND" {print $1"/"$2": status="$18" chkfail="$22" chkdown="$23" lastchg="$24"s last_chk="$57}'

# Detailed per-server state (operational vs configured state)
echo "show servers state" | socat unix-connect:/var/run/haproxy.sock stdio

# Check latency: is check_duration near the check timeout?
echo "show stat" | socat unix-connect:/var/run/haproxy.sock stdio | \
  awk -F, '$2 != "FRONTEND" && $2 != "BACKEND" {print $1"/"$2": check_duration="$39"ms"}'

# Is the flapping server also slow or saturated on real traffic?
echo "show stat" | socat unix-connect:/var/run/haproxy.sock stdio | \
  awk -F, '$2 != "FRONTEND" && $2 != "BACKEND" {print $1"/"$2": scur="$5" slim="$7" rtime="$61"ms"}'

# Retry and redispatch pressure caused by the flapping
echo "show stat" | socat unix-connect:/var/run/haproxy.sock stdio | \
  awk -F, '$2 != "FRONTEND" {print $1"/"$2": wretr="$16" wredis="$17}'

Two things to look at immediately in the first command’s output:

• lastchg resetting repeatedly. If lastchg never climbs past a few tens of seconds, the server keeps changing state. A stable server has a lastchg that grows for hours or days.
• last_chk result codes. These tell you the failure mode directly: L4TOUT (TCP connect timeout), L4CON (connection refused), L6TOUT (TLS timeout), L7TOUT (response timeout), L7STS (wrong HTTP status), L7RSP (unexpected response content), L7OK (passing). The flap between an OK code and a specific failure code is your root-cause pointer. If you want just the bare code, check_status (column 37) carries it without the longer description.

How to diagnose it

1. Confirm it is flapping, not just down. Take two samples of status, chkfail, chkdown, and lastchg a minute apart. chkdown incrementing between samples confirms UP to DOWN transitions are happening right now. chkfail incrementing while status stays UP means the server is failing individual checks below the fall threshold and is one bad stretch away from another transition.

2. Read last_chk and classify the failure. Timeout codes (L4TOUT, L6TOUT, L7TOUT) point at latency: the check itself, the network path, or an overloaded server. L4CON points at the server refusing connections (process down or at a connection limit). L7STS/L7RSP point at the application answering but answering wrong.

3. Compare check_duration to the effective check timeout. If timeout check is set, that is the read timeout for checks. If it is not set, inter doubles as the whole check timeout, so a server whose checks occasionally take longer than inter will produce spurious failures. A check_duration hovering near the timeout is the classic borderline case.

4. Correlate flap timing with load. Look at the server’s rtime and scur/slim around the transitions. If the server flaps only during traffic peaks and its real-request latency climbs right before each DOWN, it is overloaded, and the health check is correctly detecting that. The fix is capacity or per-server maxconn, not check tuning.

5. Rule out the network. If last_chk shows L4TOUT but the server’s own metrics look idle, check ctime on the backend (rising connect time suggests congestion), and look for retransmits on the HAProxy host with ss -ti. If several servers behind the same network path flap together, suspect the path, not the servers. Simultaneous multi-server flapping is almost always network or a shared dependency.

6. Inspect the health endpoint itself. For HTTP checks, hit the exact check URL from the HAProxy host repeatedly and time it. If the endpoint touches the database or a cache, it can fail while the application still serves cached or simple requests fine. If it is a trivial static response, the opposite problem applies: the server can pass checks while real requests fail (see Health Check Green, App Broken).

7. Stop the churn before it hurts the survivors. If the flapping is causing visible traffic churn (rising wredis, survivor scur climbing), put the server in maintenance mode while you investigate:

# Disruptive for that server: stops all new traffic to it. Existing sessions finish.
echo "set server <backend>/<server> state maint" | socat unix-connect:/var/run/haproxy.sock stdio

MAINT stops the redistribution oscillation completely. The survivors get a stable traffic share and you get a quiet system to debug on. Do not forget the server is in MAINT; a forgotten MAINT server is silent capacity loss. Bring it back with state ready when done.

Metrics and signals to monitor

Fixes

Stop the churn first (MAINT)

If the flapping server is actively destabilizing the backend, put it in MAINT via the runtime API as shown above. This is a traffic-affecting action for that server, so do it deliberately, and track how long it stays in MAINT. It converts an oscillating problem into a stable one you can reason about.

Check timeout too tight: set timeout check explicitly

If check_duration is close to the timeout, the common misconfiguration is a low inter set for fast detection with no separate timeout check. Because inter then serves as the entire check timeout, you have made the detection interval and the failure threshold the same number: any check that merely runs slow counts as failed. Set timeout check to a value comfortably above the p99 of check_duration but below inter, so a slow check fails but a normal one never does. Tradeoff: longer check timeouts slow down detection of genuine failures slightly.

Rise/fall too tight: widen the thresholds

Defaults (rise 2, fall 3, inter 2000) mean a server goes DOWN after roughly 6 seconds of consecutive failure and returns after roughly 4 seconds of success. For a server with jittery but recoverable behavior, that is a hair trigger. Raising fall to 5-10 (or spacing checks with a larger inter) makes a DOWN decision require sustained failure, and raising rise makes re-admission require sustained health. Tradeoff: you accept slower detection of real failures and traffic hitting a genuinely dead server for longer. Tune against your actual failure-detection budget, not against a desire to make the flapping counter quiet.

If many servers share the same check schedule and hardware, spread-checks in the global section adds jitter to check timing and avoids resonance effects where checks synchronize and pile up.

Server genuinely overloaded: fix capacity, not the check

If the server flaps because it is overloaded under its traffic share, health check tuning only masks it. Options: give the server a per-server maxconn so HAProxy queues instead of overwhelming it (see HAProxy maxconn hierarchy), add capacity, or fix whatever makes it slow (rtime decomposition, application profiling). In this case flapping was the health check working as designed.

Intermittent network: fix the path

L4TOUT/L4CON with idle servers and rising ctime means the path between HAProxy and the backend is dropping or delaying packets. Check retransmits on the HAProxy host (ss -ti), the interface error counters, and any middlebox (firewall connection tracking tables are a classic culprit for intermittent refusal). No amount of rise/fall tuning fixes a lossy path; it only changes how loudly HAProxy complains about it.

Passive checks amplifying the problem

If the server uses observe layer4 or observe layer7, real traffic errors feed into the health state via error-limit (default 10) and on-error. The default on-error fail-check simulates a failed check and forces the fast interval, which can accelerate a flap under a burst of real errors. If passive checks are causing oscillation, consider relaxing error-limit or removing observe until the underlying instability is fixed.

Prevention

• Set timeout check explicitly on every checked server so check timeout is decoupled from inter. Never let a fast inter silently become a short timeout.
• Choose fall and rise from your detection budget, and make rise strict enough that a server is demonstrably stable before it gets traffic back. A server that just recovered should prove it.
• Baseline chkdown and lastchg per server and alert on any chkdown increment on servers that are normally stable for weeks. Flapping alone warrants an alert; it becomes urgent when survivors approach saturation.
• Alert on check_duration approaching the check timeout. That is the earliest possible warning: the server is borderline before a single transition happens.
• Watch the blast radius. During any flap, track survivor scur/slim and backend qcur. If one server’s DOWN transitions push survivors over 80%, the real fix is headroom, and you are one flap away from a cascade (see HAProxy backend queue building).
• Make health endpoints check what real traffic needs, but keep them fast. A health endpoint that depends on a slow downstream will flap when the downstream does.

How Netdata helps

• Per-server chkfail, chkdown, and lastchg as time series turns flapping from a log-grep exercise into a visible oscillation pattern: you see the flap frequency and whether it is accelerating.
• check_duration trended against the configured timeout surfaces the borderline server days before the first DOWN transition, which is the cheapest time to fix it.
• Correlating chkdown with wretr/wredis and survivor scur shows the cost of each flap: whether the redistribution is being absorbed cleanly or pushing the backend toward queuing.
• lastchg never growing is a simple, robust alert condition for oscillation that does not depend on thresholds you have to tune per service.
• Backend-level correlation (qcur, rtime, 5xx) against flap events distinguishes “server is broken” from “server is the first victim of a shared dependency problem”.

Related guides

• HAProxy backend queue building (qcur): requests waiting for a free server slot
• HAProxy scur approaching slim: the concurrent-session saturation signal
• How HAProxy actually works in production: a mental model for operators
• HAProxy maxconn hierarchy: global, frontend, backend, and per-server limits
• HAProxy maxconn reached: new connections queued and rejected
• HAProxy monitoring checklist: the signals every production proxy needs
• HAProxy monitoring maturity model: from survival to expert
• HAProxy Slowloris and idle-connection pile-up: high scur, low request rate