HAProxy DNS resolver failure: stale IPs and silently misrouted traffic

Your backend servers report UP. Health checks pass. The stats page is green. And traffic is going to IP addresses that belong to pods deleted an hour ago, or to a node that now runs something else entirely.

This is the HAProxy DNS resolver failure mode, and it only bites deployments that resolve backends dynamically: server-template with Consul DNS, Kubernetes headless services, SRV records, or any setup where server addresses come from DNS instead of static config lines. When the resolver times out or a TTL expires without a successful re-resolution, HAProxy keeps routing to the last IPs it knew about. There is no stats CSV counter for this. show resolvers is the only direct view into it, and most teams never look.

The failure is silent in a specific, nasty way: the server entry can read UP (the health check may still pass if the stale IP now hosts something that answers TCP), while requests go to decommissioned or wrong endpoints. If the stale IP is genuinely unreachable, econ starts climbing, but by then you are debugging “backend connectivity” instead of the actual cause.

What this means

HAProxy’s internal DNS resolver periodically re-resolves the hostnames behind server-template entries and updates server slots with the results. When resolution works, slots track the live topology of your service. When it fails, nothing visible happens:

• The server slot keeps its last resolved IP.
• The server status does not change just because resolution failed.
• No error counter in the CSV stats moves.
• Health checks may keep passing if the stale IP still answers, which makes the dashboard look completely healthy.

The trigger is usually mundane: the DNS server (CoreDNS, Consul DNS, a local resolver) is slow, overloaded, or briefly unreachable; a response times out; a TTL expires during the gap. HAProxy rides on cached data. If the cache goes stale while your orchestrator reassigns those IPs, traffic lands somewhere wrong. With SRV-based discovery, you can also end up with obsolete slots that linger longer than expected before being cleaned up.

flowchart TD
  A[DNS query due] --> B{Resolver responds?}
  B -- valid response --> C[Server slots updated to current IPs]
  B -- timeout or error --> D[HAProxy keeps last known IPs]
  D --> E{Stale IP still reachable?}
  E -- yes --> F[Server reads UP, traffic silently misrouted]
  E -- no --> G[econ spikes, servers eventually flap DOWN]
  C --> H[Routing correct]

Common causes

Quick checks

All read-only, safe to run during an incident.

# 1. Resolver state: the only direct view into DNS health
echo "show resolvers" | socat unix-connect:/var/run/haproxy.sock stdio

# 2. What IPs does HAProxy currently hold for each server slot?
echo "show servers state" | socat unix-connect:/var/run/haproxy.sock stdio

# 3. Connection errors: do they correlate with resolution gaps?
echo "show stat" | socat unix-connect:/var/run/haproxy.sock stdio | \
  awk -F, '$2 != "FRONTEND" {print $1"/"$2": status="$18" econ="$14" lastchg="$24"s"}'

# 4. Independent DNS answer for the same name (run from the HAProxy host)
dig +short SRV _myservice._tcp.service.consul
# or for A records:
dig +short myservice.default.svc.cluster.local

On the show resolvers output, look at the per-nameserver counters. sent tells you HAProxy is trying. valid tells you it is getting usable answers. A growing timeout counter means queries leave and nothing comes back in time. The remaining counters break out other failure shapes (errors, refusals, invalid or truncated responses); the exact set varies by version, but the question you want answered is simple: are valid answers arriving at the rate queries are sent?

How to diagnose it

1. Confirm the deployment is actually DNS-dependent. Look for server-template lines with a resolvers argument in the config. If your backends are static server lines with literal IPs, this failure mode does not apply to you.

2. Snapshot the resolver. Run show resolvers twice, a minute apart. If sent increments but valid does not, resolution is broken right now. If both increment normally, the current window is fine and you are looking at a past gap (check when econ started).

3. Diff HAProxy’s IPs against live DNS. For each dynamic backend, compare the addresses in show servers state with what dig returns for the same name from the HAProxy host. Any address HAProxy holds that DNS no longer returns is a stale IP being routed to.

4. Correlate with econ and server status. If stale IPs are unreachable, econ climbs on the affected slots and health checks eventually flap them DOWN with timeouts or connection refusals. If stale IPs are reachable but wrong, status stays UP and the only evidence is misrouted traffic: wrong responses, errors logged on some other service, or clients hitting an unexpected tenant or version.

5. Check the timeline against your orchestrator. Ask when pods, tasks, or instances were replaced. If the replacement time falls inside a resolver gap (a window where valid stalled), you have your causal chain.

6. Check the HAProxy version. Some resolver bugs are version-specific: silent loss of DNS-driven server updates under heavy event load was reported on HAProxy 3.0.0 through 3.0.4 and fixed in 3.0.5 (the fix was also backported to the 2.9 branch), and HAProxy 2.2.x had a documented stale-IP persistence bug with Consul DNS after scale-down/scale-up events. If you are on an affected version, the fix is an upgrade, not a config tweak.

Metrics and signals to monitor

Note what is absent: there is no CSV stats field for resolution failures. Anything you want on a dashboard has to come from periodically collecting show resolvers and computing deltas.

Fixes

Restore resolution now

Fix the DNS path itself: restart or scale the overloaded resolver (CoreDNS, Consul agent), repair the network path to it, or point the resolvers section at a healthy nameserver. Once valid answers flow again, HAProxy re-resolves on its normal cycle and slots converge back to current IPs. You do not need to restart HAProxy for this.

Purge stale state after the fix

If load-server-state-from-file is in play, old IPs can survive a reload via the state file even after DNS is healthy. After any reload, compare the addresses in show servers state against live DNS answers and remove or recreate stale slots if old IPs reappear.

Tune for resilience to DNS outages

• init-addr none on template-managed servers lets HAProxy start and accept traffic before the first successful resolution, which matters in container environments where DNS may lag process startup.
• Hold settings: hold obsolete controls how long a server that disappears from DNS answers is kept before removal. For SRV-based discovery, obsolete entries trigger a slot state change; tune this deliberately rather than leaving defaults. Long hold values ride out resolver outages but keep dead IPs longer; short values converge fast but amplify flapping. There is no universally right number; size it against how fast your orchestrator actually reassigns IPs.
• Do not assume hold valid overrides the DNS TTL. The TTL from the DNS answer gates how often re-resolution happens; hold settings shape what HAProxy does when answers stop. Operators routinely misread these as a cache TTL.

Upgrade if you are on an affected version

If your symptoms match a known resolver regression (updates silently dropped at scale on HAProxy 3.0.0–3.0.4, fixed in 3.0.5 and backported to 2.9; stale IPs sticking after Consul scale events on 2.2.x builds), config changes will not fix it. Move to a patched release in your branch.

Prevention

• Collect show resolvers continuously. Almost nobody does, which is exactly why this failure goes undetected for hours. Treat a rising timeout counter or a falling valid/sent ratio as a ticket-severity alert in any server-template deployment.
• Alert on IP divergence. Periodically diff show servers state addresses against live DNS answers for the same names. Any HAProxy-held IP absent from DNS for longer than one re-resolution cycle plus TTL is a stale-route alarm.
• Watch econ on template-managed slots separately from static backends. A nonzero sustained rate there, with healthy backends, is a classic stale-IP signature.
• Size your DNS for the query load. Short TTLs on heavily templated backends multiply query volume. Make sure CoreDNS or Consul DNS can absorb it, or lengthen TTLs and accept slower convergence.
• Rehearse the comparison. During a normal deploy, run the IP diff once so the on-call knows what “matching” looks like before they have to spot “not matching” at 3 a.m.

How Netdata helps

• Netdata collects HAProxy stats continuously, so econ, per-server status, and active server counts are already time-series; a stale-IP event shows up as an econ step-change you can align with deploy timelines instead of a one-off snapshot.
• Correlating econ with server status and lastchg in one view separates “backend actually down” from “HAProxy routing to an IP that stopped existing,” which is the core ambiguity in this failure.
• Because resolver health has no CSV counter, pairing Netdata’s HAProxy charts with a small collector for show resolvers closes the one blind spot this failure mode exploits.
• Counter resets on reload are handled, so the econ history around a reload-and-stale-state event stays readable instead of turning into phantom drops.

Related guides

• HAProxy 503 Service Unavailable: no server is available to handle this request
• HAProxy 502 Bad Gateway: backend connection and response failures
• HAProxy connection errors (econ): backend connections refused, timed out, or reset
• HAProxy backend losing servers: active server count and cascade risk
• HAProxy counter resets on reload: phantom drops and spikes in your metrics
• HAProxy connect time (ctime) high: network latency and backend accept-queue overflow