HAProxy retries and redispatches (wretr/wredis): the backend instability nobody watches

Your dashboards are clean. No 5xx spike, latency averages look normal, every backend server is UP. Then the backend falls over in what feels like seconds, and the postmortem shows the application was flapping for six hours before the outage. The warning was there the whole time, sitting in two counters almost nobody charts: wretr and wredis.

These counters record HAProxy’s built-in compensation mechanism at work. When a connection to a backend server fails, HAProxy does not immediately return an error to the client. It retries. If option redispatch is set, it can also give up on the original server and dispatch the request to a different one. From the client’s perspective the request succeeds, possibly a bit slower. From the proxy’s perspective, something in your backend just failed and got papered over.

That papering-over is exactly why retries matter. High retries with zero user-visible errors means HAProxy is successfully masking backend instability. High retries with rising errors means the masking is failing and user impact has begun. The transition between those two states is what makes outages feel sudden: the failure was visible in the retry counters for hours before it reached the error counters. For the broader mental model of how requests flow through HAProxy, see how HAProxy works in production.

What this means

Two cumulative counters on BACKEND and SERVER rows in the stats CSV:

• wretr (CSV index 15, awk field $16): the number of times HAProxy retried a connection to the same server after a failure.
• wredis (CSV index 16, awk field $17): the number of times HAProxy redispatched a request to a different server than the one originally selected.

wredis only moves if option redispatch is enabled. Without it, HAProxy burns all of its retry budget on the same server and then errors out. The retries directive controls how many attempts HAProxy makes before giving up; the default is 3.

There are two operational states that matter:

Normally both counters are zero or near-zero. During a rolling deploy you will see brief, expected bumps as servers restart. Sustained retries above roughly 1% of total requests means significant backend instability, whether or not anyone has complained yet.

flowchart TD
  A[Backend intermittent failure] --> B[Connect or response fails]
  B --> C{Retry budget left?}
  C -->|yes, same server| D[wretr +1]
  C -->|redispatch enabled| E[wredis +1, try another server]
  D --> F{Recovered?}
  E --> F
  F -->|yes| G[Client sees success, extra latency]
  F -->|no, retries exhausted| H[5xx returned to client]
  G --> I[hrsp_5xx flat: compensation working]
  H --> J[hrsp_5xx rising: compensation failing]

Common causes

Quick checks

All of these are read-only against the stats socket.

# Retry and redispatch counters per backend and server
echo "show stat" | socat unix-connect:/var/run/haproxy.sock stdio | \
  awk -F, '$2 != "FRONTEND" {print $1"/"$2": wretr="$16" wredis="$17}'

# Correlate with connection errors and response errors
echo "show stat" | socat unix-connect:/var/run/haproxy.sock stdio | \
  awk -F, '$2 != "FRONTEND" {print $1"/"$2": econ="$14" eresp="$15}'

# Which servers are flapping: status, check failures, time since last change
echo "show stat" | socat unix-connect:/var/run/haproxy.sock stdio | \
  awk -F, '$2 != "FRONTEND" && $2 != "BACKEND" {print $1"/"$2": status="$18" chkfail="$22" chkdown="$23" lastchg="$24"s"}'

# Is the backend queueing as a side effect of slow, retried requests?
echo "show stat" | socat unix-connect:/var/run/haproxy.sock stdio | \
  awk -F, '$2 != "FRONTEND" {print $1"/"$2": qcur="$3" qtime="$59"ms rtime="$61"ms"}'

# Confirm whether redispatch is even possible in the running config
grep -E "redispatch|retries|retry-on" /etc/haproxy/haproxy.cfg

Two things to note when reading these numbers. First, wretr and wredis are cumulative and reset to zero on every reload, so compute deltas or rates before comparing anything. Second, the retry-to-request ratio is what matters: divide the retry delta by the request delta over the same window. Absolute counts are meaningless across backends with different traffic volumes.

How to diagnose it

1. Establish the rate, not the count. Sample wretr and wredis twice, 60 seconds apart, and compute the delta per server. Compare against the request volume delta for the same backend. Anything sustained above ~1% of requests is significant instability.

2. Identify the blast radius. Are retries concentrated on one server, or spread across the whole backend? One server points at that host (restarts, OOM, saturation). The whole backend points at something shared: network path, a common dependency causing timeouts, or per-server maxconn everywhere.

3. Pair with econ and eresp. wretr rising with econ means HAProxy cannot connect: refusals, timeouts, resets. wretr rising with eresp means it connects but the response breaks. These are different failure modes with different fixes. The distinction is covered in the signal correlation rules of the HAProxy monitoring checklist.

4. Check the flapping evidence. chkfail incrementing on a server that still shows UP means health checks are failing below the fall threshold. Real traffic is hitting a server that is already failing probes. last_chk tells you how the check is failing (connection refused vs timeout vs wrong status).

5. Check whether compensation is holding. Compare frontend hrsp_5xx against backend hrsp_5xx. If the frontend error rate is still flat while retries climb, HAProxy is absorbing the problem and you have time. If frontend 5xx is climbing with the retry rate, the retry budget is being exhausted and the incident has already started for users.

6. Check for retry-amplified latency. Retries add delay. Watch qtime and rtime: if they are climbing alongside retries, the compensation mechanism itself is degrading the user experience even while it hides errors. Retried requests against an already-slow backend also hold connection slots longer, which feeds the queue. See HAProxy backend queue building (qcur) for that failure mode.

7. Verify the retry configuration matches your assumptions. Confirm option redispatch is actually set if you expect wredis to move, and check the configured retries count. A backend where you assumed redispatch was active but is not will show wretr climbing and errors arriving earlier than expected.

Metrics and signals to monitor

Fixes

There is no “fix the retries” knob. The counters are the messenger. Fixes target the cause.

Backend restarting or crashing

Fix the restart loop: memory limits, application crash cause, deploy health. This is the most common root cause of a sustained wretr trend on a single server. Do not tune HAProxy retry settings to hide it; you are trading warning time for a harder failure later.

Per-server maxconn too low for the workload

If retries cluster on servers sitting at scur = slim, HAProxy is refusing new connections to full servers and retrying. Raise per-server maxconn to what the application can actually sustain, or add capacity. Raising maxconn without understanding backend capacity trades backpressure for overload. See the maxconn hierarchy guide.

Retry configuration not doing what you think

Several configuration behaviors routinely surprise operators:

• Without option redispatch, there is no failover to another server. HAProxy retries the same server up to the retries count, then errors. If you expect wredis to be nonzero and it is pinned at zero during failures, this is why.
• Redispatch timing is tunable. The optional interval argument to option redispatch controls when the redispatch happens within the retry sequence. A positive value P redispatches on every Pth retry, a negative value -N redispatches on the Nth retry before the last, and the default -1 preserves the historical behavior of redispatching only on the final retry. option redispatch 1 therefore redispatches on every retry; 0 disables redispatch.
• retry-on replaces the default, it does not extend it. On HAProxy 2.0 and later, setting retry-on 503 504 without including a connection-failure condition such as conn-failure means connection failures stop being retried. If wretr mysteriously drops to zero after a config change while econ keeps rising, check for this.
• Non-idempotent requests are not retried by default. Layer-7 retries do not fire for methods like POST unless you explicitly allow it, because replaying a POST can duplicate side effects. That is the safe default. In the configuration manual’s own terms, retry-on defaults to conn-failure only, so layer-7 conditions such as empty-response, junk-response, response-timeout, or any status-code keyword never fire for a request whose method has side effects unless an operator opts in; the manual explicitly warns against retry-on values beyond conn-failure and none unless the application has replay protection.

Network instability to backends

If retries spread evenly across servers with elevated ctime, the path is the problem: packet loss, congestion, an overloaded middlebox. Fix the path, not the proxy.

Rolling deployments

Do not “fix” expected retry bursts during deploys; suppress alert noise instead. Deploy windows should be annotated so a retry spike during a rollout is evaluated differently from the same spike at 3 a.m.

Prevention

• Chart the counters, not just the errors. Add wretr and wredis rates per backend to the standard HAProxy dashboard. This is a Level 3 maturity signal for a reason: it converts a sudden outage into a multi-hour warning. See the monitoring maturity model.
• Alert on the ratio, with compensation state. Alert when retry rate exceeds ~1% of requests sustained. Escalate severity when the retry rate is high AND frontend 5xx is rising: that is compensation failing.
• Baseline around deploys. Know your normal retry envelope during rollouts so deviations stand out.
• Test the config assumption once. Trigger a controlled backend failure and confirm wredis actually increments. Many teams discover mid-incident that option redispatch was never set.
• Correlate with capacity signals. Retries that track per-server scur/slim saturation are a capacity problem wearing an instability costume. See scur approaching slim and backend losing servers.
• Handle counter resets in your monitoring. Every reload zeroes wretr and wredis. Rate calculations must detect resets or you will see phantom drops and phantom spikes. Uptime_sec from show info is the standard reset detector.

How Netdata helps

Netdata’s HAProxy collector pulls the stats CSV continuously, so the retry story is visible without manual socket queries:

• Per-server and per-backend wretr/wredis rates rendered as deltas, so cumulative counter resets on reload do not corrupt the trend.
• Side-by-side correlation with econ, eresp, and per-server status, which is how you split “cannot connect” from “response broke” without leaving the dashboard.
• Retry rate next to hrsp_5xx on the same backend, making the compensation-working vs compensation-failing distinction a visual one.
• Retry rate overlaid with qtime/rtime, exposing the latency tax that compensation charges even when it hides every error.
• Per-second granularity that catches short flapping bursts that minute-resolution scrapers average away.

Related guides

• HAProxy monitoring checklist: the signals every production proxy needs
• HAProxy monitoring maturity model: from survival to expert
• How HAProxy actually works in production: a mental model for operators
• HAProxy 504 Gateway Timeout: timeout server and the backend timeout cascade
• HAProxy backend queue building (qcur): requests waiting for a free server slot
• HAProxy backend losing servers: active server count and cascade risk
• HAProxy health checks green but the application is broken: when UP does not mean healthy
• HAProxy health check L4TOUT and L4CON: server marked DOWN and unreachable
• HAProxy maxconn hierarchy: global, frontend, backend, and per-server limits
• HAProxy scur approaching slim: the concurrent-session saturation signal