$ guides / haproxy / haproxy-connection-errors-econ ▌

Operations Guides

HAProxy connection errors (econ): backend connections refused, timed out, or reset

Your HAProxy backend shows a rising econ counter. Clients may not see errors yet, because HAProxy retries failed connections and redispatches to other servers, but the counter means HAProxy is trying to open TCP connections to a backend server and failing. The connection is being refused, timing out, or getting reset.

econ is one of the highest-signal error counters HAProxy exposes, and one of the most misread. It is cumulative, it only increments on new connection attempts (so connection reuse can mask a real problem for hours), and it lumps three very different failure modes into one number. Telling “the server process is dead” apart from “a firewall is silently dropping SYNs” requires correlating econ with ctime and a few other signals.

What this means

Every time HAProxy dispatches a request to a backend, it either reuses an idle connection from its pool or opens a new TCP connection to a server. econ (field 13 in the stats CSV, exposed on BACKEND and SERVER rows) counts requests that hit an error during that connect phase: the SYN was refused (ECONNREFUSED), the handshake never completed before timeout connect fired, or the connection was reset during establishment.

Three properties that matter when reading it:

It is cumulative. A raw value of 40,000 means nothing without a rate. Compute deltas over time, and express the rate as a failure ratio: econ_rate / (connect_rate + reuse_rate). A sustained ratio above 1% warrants investigation. Brief spikes during rolling restarts are expected.
Connection reuse masks it. With http-reuse working, most requests never open a new connection, so econ can read zero for long stretches even while a server intermittently refuses connections. A server that fails 1 in 50 connects looks silent when 98% of traffic rides pooled connections.
Backend-level econ is a superset. The BACKEND row sums all servers plus connection errors not attributable to any server (for example, the backend having no active servers). A backend econ spike with clean per-server rows points at a backend-level condition, not a dying server.

HAProxy compensates for connection failures with retries (retries, default 3) and, if option redispatch is enabled, by sending the request to a different server. Those show up as wretr and wredis. This is why econ often rises before users see errors: the failure is real but absorbed. When retries stop absorbing it, the failure becomes a 502. See HAProxy 502 Bad Gateway.

Common causes

Cause	What it looks like	First thing to check
Server process down or not listening	`econ` rising on one server, `ctime` near zero (instant refusal)	Is the process up and listening on the expected port on that host?
Server overloaded, SYN/accept queue full	`econ` rising with `ctime` climbing toward `timeout connect`	Backend host CPU/load and its listen backlog
Firewall or security group blocking	`ctime` at `timeout connect` (DROP) or near zero (REJECT)	Reachability from the HAProxy host to that server:port
Network partition or routing change	Multiple servers in one backend failing at once, rising `ctime`	Whether failures cluster by subnet/AZ
Ephemeral port exhaustion on HAProxy	`econ` spikes across backends with healthy servers, high TIME_WAIT count	`ss -s` / `/proc/net/sockstat` on the HAProxy host
Stale DNS / wrong resolved IP	`econ` rising after deploys or instance replacement, health checks also failing	`show resolvers`, resolved IP vs actual inventory
Rolling deployment restarts	Brief `econ` and `wretr` bursts per server, self-resolving	Deploy timeline; usually benign

Two of these are the most commonly missed:

Ephemeral port exhaustion is a HAProxy-side failure that looks like a backend failure. Each new backend connection consumes an ephemeral port on the HAProxy host. With the default Linux range of roughly 28,000 ports and connections stuck in TIME_WAIT, sustained new-connection rates in the low hundreds per second can exhaust the range. New connects then fail locally and increment econ while every backend is perfectly healthy. The tell: econ spread across multiple unrelated backends, plus a large TIME_WAIT count on the HAProxy host.

A refused connection and a timed-out connection are different incidents. Refusal (low ctime) means something actively rejected the SYN: no listener, or a firewall sending RST. Timeout (ctime at timeout connect) means the SYN was dropped or the server’s accept queue overflowed: packet loss, a DROP rule, or a server too busy to complete handshakes. The fix paths diverge completely, so establish which one you have before changing anything.

Quick checks

All read-only. Run from the HAProxy host unless noted.

# econ per backend and per server (take two samples 60s apart and diff)
echo "show stat" | socat unix-connect:/var/run/haproxy.sock stdio | \
  awk -F, '$2 != "FRONTEND" {print $1"/"$2": econ="$14}'

# ctime: near-zero = refusal, near timeout connect = timeout
echo "show stat" | socat unix-connect:/var/run/haproxy.sock stdio | \
  awk -F, '$2 != "FRONTEND" {print $1"/"$2": ctime="$60"ms"}'

# New connections vs reuse (HAProxy 2.4+): low reuse means econ samples more of your traffic
echo "show stat" | socat unix-connect:/var/run/haproxy.sock stdio | \
  awk -F, '$2 == "BACKEND" {print $1": connect="$85" reuse="$86}'

# Retries and redispatches: is HAProxy currently absorbing the failures?
echo "show stat" | socat unix-connect:/var/run/haproxy.sock stdio | \
  awk -F, '$2 != "FRONTEND" {print $1"/"$2": wretr="$16" wredis="$17}'

# Server health state
echo "show stat" | socat unix-connect:/var/run/haproxy.sock stdio | \
  awk -F, '$2 != "FRONTEND" && $2 != "BACKEND" {print $1"/"$2": status="$18}'

# Ephemeral port / TIME_WAIT pressure on the HAProxy host
ss -s
cat /proc/net/sockstat

Test a specific suspect server from the HAProxy host:

# Is the port actually reachable and how long does connect take?
time curl -sv --max-time 5 http://<server-ip>:<port>/ -o /dev/null

A Connection refused in milliseconds confirms refusal; a hang until --max-time confirms the timeout path. If curl succeeds consistently while econ rises, suspect ephemeral port exhaustion or an intermittent SYN-queue overflow rather than a dead server.

How to diagnose it

Work through this in order. Each step narrows the failure class.

flowchart TD
  A[econ rate rising] --> B{ctime pattern}
  B -->|near zero| C[Immediate refusal]
  B -->|near timeout connect| D[Connect timeout]
  B -->|mixed| E[Intermittent overload]
  C --> C1{Process listening on server?}
  C1 -->|no| C2[Restart app / fix listener]
  C1 -->|yes| C3[Firewall REJECT or wrong port]
  D --> D1{Multiple backends affected?}
  D1 -->|yes| D2[Check HAProxy host: ephemeral ports, network path]
  D1 -->|no| D3[Server SYN queue full or firewall DROP]
  E --> E1[Backend accept-queue overflow under load]

Quantify the failure ratio. Pull econ, connect, and reuse twice, 60 seconds apart. Compute the delta rates and econ / (connect + reuse). Above 1% sustained is a real problem. Occasional single increments are likely noise.
Scope it. Is econ rising on one server, all servers in one backend, or multiple backends? One server points at that host. One backend points at a shared dependency or network segment. Multiple unrelated backends point at the HAProxy host itself (ephemeral ports, local network stack).
Classify with ctime. Near-zero ctime with rising econ is refusal. ctime hovering near your configured timeout connect is a timeout. This is the single most diagnostic correlation in the stats.
Check server status and wretr/wredis. If the server flaps between UP and DOWN (chkdown incrementing, small lastchg), health checks are seeing the same failure your traffic is. Rising wretr means HAProxy is compensating; flat frontend errors mean compensation is working so far.
Rule out the HAProxy host. Check ss -s and /proc/net/sockstat for TIME_WAIT volume, and the connect counter rate. High connection churn plus broad econ strongly suggests ephemeral port exhaustion. This is a cliff: once the range is full, every new connect fails locally regardless of backend health.
Test the path directly. From the HAProxy host, curl the suspect server repeatedly. Consistent refusal: dead listener or REJECT rule. Consistent hang: DROP rule or partition. Intermittent failure under load: SYN-queue overflow on the server. Consistent success while econ climbs: back to step 5, the problem is local to HAProxy.
If servers are dynamically resolved, check show resolvers and verify the IPs in show servers state match current inventory. Stale DNS sends connections to decommissioned addresses, which shows up as econ with no real server failure.

Metrics and signals to monitor

Signal	Why it matters	Warning sign
`econ` rate, per server and backend	The failure itself	Sustained `econ / (connect + reuse)` > 1%
`ctime`	Distinguishes refusal from timeout	Climbing toward `timeout connect`
`connect` vs `reuse`	Reuse masks econ; dropping reuse increases exposure	Sudden drop in reuse ratio
`wretr` / `wredis`	HAProxy absorbing failures before users see them	Any sustained nonzero rate
Server `status`, `chkdown`, `lastchg`	Whether health checks agree with traffic	Flapping (small `lastchg`, rising `chkdown`)
TIME_WAIT / socket stats on HAProxy host	Ephemeral port exhaustion detection	Tens of thousands in TIME_WAIT
Frontend `hrsp_5xx`	Whether failures are surfacing as 502s	Rising while `econ` rises

Fixes

Group by what step 3 told you.

Immediate refusal: server down or wrong target

Restart or fix the backend process if it is dead or listening on the wrong port/interface. Verify with ss -tln on the server that the port is bound to an address HAProxy can reach.
Fix the firewall or security group if it is sending RST. A REJECT rule at least fails fast; confirm the rule matches the intended policy.
Correct the server address if DNS or config points HAProxy at the wrong IP or port. For dynamic backends, fix the resolver or registration source.

Connect timeout: drops, backlog, or partition

Relieve the overloaded server. A SYN/accept queue that overflows under load produces exactly this signature. Reduce load, add capacity, or scale the pool. Check HAProxy backend losing servers if health checks are also flapping.
Fix the network path if a partition or routing change is dropping SYNs. Failures clustered by subnet or availability zone are the fingerprint.
Widen a DROP-only firewall rule to the intended allowlist. DROP rules turn every misconfiguration into a slow timeout cascade instead of a fast refusal.

HAProxy-side: ephemeral port exhaustion

Enable or fix connection reuse (http-reuse). This is the structural fix: it cuts the new-connection rate to backends by an order of magnitude, which directly relieves port pressure.
Reduce connection churn sources. A backend that sends Connection: close or has keep-alive disabled forces a new connection per request. Fix the backend, not just HAProxy.
Widen net.ipv4.ip_local_port_range and consider net.ipv4.tcp_tw_reuse as tactical relief. These raise the ceiling but do not fix the churn.

If users are already seeing errors

Verify retries and option redispatch are configured so single-server failures get absorbed while you fix the root cause. Gotcha: if you set retry-on explicitly, connection failures are only retried if you include conn-failure or all-retryable-errors; the default TCP-connect retry is not implied.
Put a flapping server in MAINT to stop redispatch churn while you investigate it. This takes the server out of rotation; traffic it was serving moves to the remaining servers, so confirm they have headroom first.

Prevention

Alert on the ratio, not the counter. econ / (connect + reuse) > 1% sustained, per backend. Guard for deploy windows.
Alert on wretr/wredis as a leading indicator. Retries are the canary: they show backend instability hours before it becomes user-visible.
Monitor reuse ratio and HAProxy-host socket stats so ephemeral port exhaustion and reuse breakdown are caught before econ spikes.
Set per-server maxconn to what the application actually handles, so overload queues in HAProxy (visible, measurable) instead of overflowing the server’s SYN backlog (shows up as timeouts). See HAProxy maxconn hierarchy.
Size health checks to detect this failure class. L4 checks catch dead listeners fast; see HAProxy health check L4TOUT and L4CON for the check-side view of the same failures.

How Netdata helps

Netdata collects the HAProxy stats continuously, so econ, ctime, connect, reuse, wretr, and wredis are graphed per server and per backend at per-second resolution, no manual socat polling needed.
Plotting econ next to ctime on the same dashboard makes the refusal-vs-timeout classification instant instead of a two-terminal exercise.
The failure ratio econ / (connect + reuse) can be alerted on directly, with the 1% sustained threshold, rather than raw counters that mean nothing without context.
Rising wretr/wredis with flat frontend 5xx shows up as a visible divergence, surfacing absorbed backend instability before it becomes an incident.
Host-level socket and TIME_WAIT metrics from the same agent let you rule ephemeral port exhaustion in or out without SSHing to the box.

The Netdata solution

HAProxy load balancer monitoring with Netdata

Netdata monitors HAProxy with per-second frontend, backend, and queue metrics plus ML-powered anomaly detection. Correlate maxconn saturation, queue buildup, health-check cascades, 5xx attribution, and file-descriptor exhaustion against the backend and host signals behind them, so you catch the incidents in these runbooks before they page anyone.

See load balancer monitoring → Start monitoring free