HAProxy connect time (ctime) high: network latency and backend accept-queue overflow

Your HAProxy backend shows ctime climbing from a steady 1-2ms to tens or hundreds of milliseconds. Total request latency (ttime) rises with it, and if the trend continues, connect timeouts (econ increments, 502/504s on the frontend) are next. The connect phase of the HAProxy-to-backend path is degrading, and it almost always means one of three things: the network path is congested or losing packets, the backend’s kernel accept queue is overflowing, or the backend is too CPU-starved to answer the handshake promptly.

There is a fourth, quieter cause operators routinely miss: if you run http-reuse, ctime should sit at or near zero for almost all requests because connections come from the pool. A sudden appearance of nonzero ctime on a proxy that used to show zero is not a latency problem at all. It is connection reuse breaking down, and every request is now paying for a fresh TCP handshake.

What this means

ctime is the average time, in milliseconds, for HAProxy to establish a TCP connection to a backend server: SYN out to connection established. It is a rolling average over the last 1024 requests, reported per backend and per server in the stats CSV (0-based field index 59).

Two consequences follow:

• It blends network round-trip time with the server’s accept-side processing. A slow SYN-ACK, a full accept queue, and a congested path all look identical in ctime alone.
• The rolling window smooths spikes. A brief burst of slow connects gets diluted across 1024 samples, so ctime lags real-time events. Watch econ deltas and per-request Tc in logs for the sharp edge.

Baseline expectations: in the same datacenter, ctime should be under 1-5ms. Cross-datacenter values are legitimately higher but should be stable. Any change beyond roughly 2x baseline deserves investigation, and if ctime exceeds about 50% of timeout connect, your margin before connect failures is thin.

Common causes

The last row is a different incident. Zero ctime plus econ means the connection was refused or reset immediately, or the path is black-holed before any handshake started: a dead server, a firewall, or ephemeral-port exhaustion on the HAProxy host. Do not let a mixed picture pull you into the wrong investigation.

Quick checks

These are all read-only. Run them before changing anything.

# 1. Current ctime per server and per backend (0-based CSV index 59, so awk $60)
echo "show stat" | socat unix-connect:/var/run/haproxy.sock stdio | \
  awk -F, '$1 != "# pxname" && $2 != "FRONTEND" {print $1"/"$2": ctime="$60"ms"}'

# 2. Connection errors on the same rows (cumulative, so sample twice for a delta)
echo "show stat" | socat unix-connect:/var/run/haproxy.sock stdio | \
  awk -F, '$1 != "# pxname" && $2 != "FRONTEND" {print $1"/"$2": econ="$14}'

# 3. Connection reuse vs new connects (is the pool still working?)
echo "show stat" | socat unix-connect:/var/run/haproxy.sock stdio | \
  awk -F, '$2 == "BACKEND" {print $1": connect="$82" reuse="$83}'

The connect and reuse CSV fields (0-based indexes 81 and 82) only exist on HAProxy versions that report connection-reuse stats (1.8 and later). On older builds these columns are absent and the awk output will be empty.

# 4. Queue time and response time, to place the connect delay in context
echo "show stat" | socat unix-connect:/var/run/haproxy.sock stdio | \
  awk -F, '$2 == "BACKEND" {print $1": qtime="$59"ms ctime="$60"ms rtime="$61"ms ttime="$62"ms"}'

# 5. Event loop headroom on HAProxy itself (a saturated loop delays its own connects)
echo "show info" | socat unix-connect:/var/run/haproxy.sock stdio | grep -E "^(Idle_pct|Run_queue):"

# 6. Retransmissions on the HAProxy host (packet loss on the backend path)
ss -ti | grep -A1 -E "ESTAB" | grep -c retrans

On the suspect backend host:

# Kernel accept-queue overflow counters (should be zero or static)
nstat -az | grep -i listen

# Listen socket queue depth: Recv-Q near or above the backlog is an overflow in progress
ss -tln | grep -E ":(<backend-port>) "

# CPU starvation check
uptime
top -b -n1 | head -15

How to diagnose it

Work the fork in this order. Each step either confirms a cause or eliminates it.

flowchart TD
  A[ctime rising] --> B{econ also rising?}
  B -->|No| C[Handshake still completes, just slow]
  C --> C1{All servers or one?}
  C1 -->|All| C2[Network path congestion or loss]
  C1 -->|One/few| C3[Backend accept-queue overflow or CPU starvation]
  B -->|Yes| D{What is ctime value?}
  D -->|High then econ| E[Connect timeouts: network loss or overloaded backend kernel]
  D -->|Zero plus econ| F[Immediate refusal: server down, firewall, or ephemeral ports]
  A --> G{Was ctime ~0 before with http-reuse?}
  G -->|Yes| H[Reuse broke down - check connect vs reuse ratio]

1. Split timeout from refusal. Take two econ samples a few seconds apart and compare with ctime. High ctime followed by econ growth means the handshake started but could not finish inside timeout connect: network loss or an overloaded backend kernel. econ growth with ctime pinned at zero means immediate refusal: the backend is not listening, a firewall is sending RST, or HAProxy cannot allocate an ephemeral port. Only the first branch belongs to this article; for the second, check server status and last_chk (L4CON/L4TOUT codes) and see the health-check failure guides linked below.

2. Check whether reuse broke. If the backend has http-reuse configured, compare the connect and reuse cumulative counters against historical values. A healthy pool shows reuse / (connect + reuse) well above zero and ctime near zero. If reuse flatlined while connect accelerates, something invalidated the pool: the backend started sending Connection: close, an HTTP version or keep-alive change on the application side, or an upgrade on either end. This is a configuration-drift incident, and the elevated ctime is a symptom, not the disease. Also expect SslBackendKeyRate in show info to rise if backend TLS is in use, since every new connection re-handshakes.

3. Determine scope: all servers or one. Per-server ctime comparison is decisive. All servers elevated together points at the shared network path between HAProxy and the backend subnet. One or two servers elevated points at those hosts.

4. If all servers: prove network loss. On the HAProxy host, inspect backend-facing connections with ss -ti and watch the retransmission counters. Retransmits climbing means the path is dropping packets. Also check Idle_pct on HAProxy itself: a saturated event loop (Idle_pct below ~20%) delays its own connect processing, and ctime inflates even though the network is fine. If busy-polling is enabled, Idle_pct is meaningless by design; use show activity run-queue depth instead.

5. If one server: check its accept queue. On the backend host, run nstat -az | grep -i listen. ListenOverflows or ListenDrops incrementing means the application’s accept queue is full: the kernel has established connections waiting for the app to call accept(), and under overflow it delays or drops further handshake progress. New connections from HAProxy sit in the handshake longer, which is exactly what ctime measures. Confirm with ss -tln: Recv-Q at or above the socket’s backlog on the listen socket is the overflow happening right now.

6. Check the backend’s CPU. A host at saturation delays SYN-ACK and accept() processing. High load average relative to core count, plus high ctime plus high rtime on that server, is the signature. Here ctime is the leading edge of a general overload problem, and rtime will be the bigger number.

7. Place the delay in the total-time budget. ttime decomposes roughly into qtime + ctime + rtime + transfer. If ctime is 100ms but ttime is 5s, the connect delay is real but not your main user-facing problem; fix it, but investigate rtime first.

Metrics and signals to monitor

Fixes

Network congestion or packet loss

Fix the path, not the proxy. Identify the lossy segment with retransmission evidence (ss -ti) and interface error counters, then escalate to whoever owns the link. Short-term, HAProxy retries (retries directive, option redispatch) mask single-loss events from users, but every retry adds latency and every redispatch hides the problem a little longer. Watch wretr/wredis while you treat the network: rising retries with flat user-visible errors means the masking is working but the disease is progressing.

Backend accept-queue overflow

The overflow is on the backend host, so the fixes live there:

• Increase the application’s listen backlog, and make sure the kernel ceiling allows it: net.core.somaxconn on the backend host truncates the backlog the application asks for. Raising the app backlog without raising somaxconn does nothing.
• If the application is simply not calling accept() fast enough (thread pool exhausted, event loop busy), the backlog raise buys time but the real fix is application concurrency or CPU. The rtime on that server will tell you which one you have.

Tradeoff: a larger backlog delays the onset of overflow but also delays the backpressure signal. Do not size it to hide a chronically slow app.

CPU-starved backend

Relieve the host: scale out the backend, shed load, or fix whatever is consuming CPU. From the HAProxy side, lowering that server’s maxconn caps how much work HAProxy will park on it, which reduces both its accept-queue pressure and its queueing. The cost is that requests spill to other servers or into the backend queue (qcur), so only do this if the other servers have headroom.

http-reuse breakdown

Find what invalidated the pool. Common triggers: the application began emitting Connection: close, a deploy changed keep-alive behavior or HTTP version, or the http-reuse mode no longer matches the traffic pattern. Restoring reuse returns ctime to near zero and simultaneously relieves backend connection churn, TLS handshake CPU (SslBackendKeyRate), and ephemeral-port pressure. Verify with the connect/reuse ratio recovering, not just with ctime dropping.

Safety margin on timeout connect

If ctime legitimately sits above 50% of timeout connect (for example cross-region backends), raise timeout connect deliberately rather than discovering the cliff during the next congestion event. This is a margin change, not a fix for the latency itself.

Prevention

• Track ctime as a ratio of timeout connect, not an absolute number. The same 50ms is noise on a 5s timeout and an emergency on a 100ms one.
• Alert on the reuse ratio, not just on ctime. The drop in reuse / (connect + reuse) fires before users feel the extra handshake latency, and it catches configuration drift that ctime-only alerting only notices after the fact.
• Collect ListenOverflows/ListenDrops on every backend host, not just on HAProxy. Backend accept-queue overflow is invisible from HAProxy’s own metrics except as elevated ctime, by which point it has been happening for a while.
• Baseline per-server ctime. A single server drifting upward while peers stay flat is an early host-degradation signal. Averages across the backend hide it.
• Handle counter resets. All cumulative counters (econ, connect, reuse) reset on reload. Use Uptime_sec to detect resets so delta-based alerts do not misfire.

How Netdata helps

• Netdata collects HAProxy per-backend and per-server timing metrics including ctime, qtime, rtime, and ttime, so you can see the connect phase degrade in the context of the full latency decomposition without hand-polling the stats socket.
• Connection-error and retry counters (econ, wretr, wredis) are charted alongside the timing signals, which makes the timeout-versus-refusal fork a visual comparison instead of two SSH sessions and a stopwatch.
• On the backend hosts, Netdata’s system-level charts surface TCP listen-overflow counters and CPU saturation next to HAProxy’s view, letting you confirm an accept-queue overflow on the server at the same timestamp where ctime jumped on the proxy.
• Connection-pool behavior (connect vs reuse) is tracked over time, so a reuse breakdown shows up as a clear ratio change rather than a mystery rise in connect time.
• Per-second collection granularity catches the sharp edge of econ deltas that HAProxy’s 1024-sample rolling averages smooth away.

Related guides

• HAProxy 504 Gateway Timeout: timeout server and the backend timeout cascade
• HAProxy backend queue building (qcur): requests waiting for a free server slot
• HAProxy backend losing servers: active server count and cascade risk
• HAProxy health check L4TOUT and L4CON: server marked DOWN and unreachable
• HAProxy maxconn reached: new connections queued and rejected
• HAProxy monitoring checklist: the signals every production proxy needs