HAProxy backend queue building (qcur): requests waiting for a free server slot

You are looking at HAProxy stats and qcur is nonzero on a backend, or worse, climbing. Clients have not seen errors yet, but latency is drifting up. That instinct that something is wrong is correct: qcur is the earliest signal that backend capacity is exhausted, and it shows up before any error counter moves.

qcur is the instantaneous count of connections sitting in a queue because every server that could take them is already at its maxconn limit. Those requests are not being served. They are parked, consuming their client’s timeout budget, waiting for a slot to free up. If the queue grows faster than slots free up, the outcome is queue timeouts and 503s.

This article covers what the queue actually is, how to read qcur alongside qmax, qtime, and qlimit, how to find out why it is growing, and what to change depending on the cause. For the bigger picture of how the queue fits into HAProxy’s request pipeline, see How HAProxy actually works in production.

What this means

Every request HAProxy proxies needs a connection slot on a backend server. Slots are bounded by maxconn, which HAProxy enforces at four independent levels: global, frontend, backend, and per-server. When the level that applies to a request is full, the request is not rejected immediately. It enters a queue and waits.

There are two queue scopes:

• Per-server queue: requests waiting for a specific server to free a slot. Session persistence (cookie or stick-table based) pins requests to a server, so one hot server can build a queue while its peers sit idle.
• Per-backend queue: requests waiting because the backend as a whole has no free slot anywhere.

By default, maxqueue is 0, which means the queue is unbounded. Requests wait until a slot frees up or until a timeout fires. The relevant timeout is timeout queue, and if you have not set it explicitly, it inherits the value of timeout connect. That default surprises people: a 10-second connect timeout means requests can sit in the queue for 10 seconds before HAProxy drops them with a 503. When maxqueue is set to a positive value and the queue reaches that depth, new requests no longer wait. Per the configuration manual, overflowed requests are redispatched to other servers, which breaks persistence.

The important operational property: queue buildup is a leading indicator. It precedes latency complaints, it precedes 5xx errors, and it precedes backend collapse. A backend with qcur oscillating between 0 and 20 during peaks is one traffic bump or one failed server away from an incident.

flowchart LR
  R[Incoming request] --> S{Free server slot?}
  S -->|yes| C[Connect or reuse connection]
  S -->|no| Q[Queue: qcur increments]
  Q --> W{Slot frees before timeout?}
  W -->|yes| C
  W -->|no, timeout queue expires| E[503 to client]
  Q --> M{maxqueue reached?}
  M -->|yes| RD[Redispatch to another server]
  C --> B[Backend server]

Common causes

Quick checks

All of these are read-only against the runtime API socket. Adjust the socket path for your deployment.

# Current and peak queue depth per backend and per server
echo "show stat" | socat unix-connect:/var/run/haproxy.sock stdio | \
  awk -F, '$2 != "FRONTEND" {print $1"/"$2": qcur="$3" qmax="$4}'

# Queue time (ms, rolling average over last 1024 requests)
echo "show stat" | socat unix-connect:/var/run/haproxy.sock stdio | \
  awk -F, '$2 != "FRONTEND" {print $1"/"$2": qtime="$59"ms"}'

# Session utilization: are servers actually at their maxconn?
echo "show stat" | socat unix-connect:/var/run/haproxy.sock stdio | \
  awk -F, '$2 != "FRONTEND" && $7>0 {print $1"/"$2": scur="$5" slim="$7" pct="int($5/$7*100)"%"}'

# Server status and how long since the last state change
echo "show stat" | socat unix-connect:/var/run/haproxy.sock stdio | \
  awk -F, '$2 != "FRONTEND" && $2 != "BACKEND" {print $1"/"$2": status="$18" lastchg="$24"s"}'

# Active server count per backend
echo "show stat" | socat unix-connect:/var/run/haproxy.sock stdio | \
  awk -F, '$2 == "BACKEND" {print $1": active="$20" backup="$21}'

# Response time per server (are slow servers causing the pile-up?)
echo "show stat" | socat unix-connect:/var/run/haproxy.sock stdio | \
  awk -F, '$2 != "FRONTEND" {print $1"/"$2": rtime="$61"ms"}'

# Is HAProxy itself saturated, or just the backends?
echo "show info" | socat unix-connect:/var/run/haproxy.sock stdio | grep -E "^(Idle_pct|CurrConns|Maxconn|Uptime_sec):"

How to diagnose it

1. Confirm the queue is real and scoped. Run the first command above. Note whether qcur is nonzero on the BACKEND row, on individual SERVER rows, or both. Backend-level queue with idle servers underneath means requests are pinned (persistence) or the balancing algorithm is not spreading load. A queue on exactly one server is a hot-server problem, not a capacity problem.

2. Check whether slots are actually full. A queue only forms when scur has reached slim somewhere. If scur/slim is at 100% on all servers, the queue is legitimate backpressure: you need more slots or faster turnover. If servers are queuing well below their slim, something else is throttling dispatch, so check the backend-level slim and global Maxconn too. Remember maxconn is enforced at four levels independently; see HAProxy maxconn reached for the global case.

3. Determine whether the problem is arrival rate or service time. Compare current req_rate against the baseline for this time of day. If arrival rate is normal but the queue formed anyway, look at rtime: slower responses hold slots longer, so the same request rate consumes more concurrency. This is the backend timeout cascade pattern: rtime up, qcur up, econ low, servers still UP. The fix is on the backend, not in HAProxy.

4. Check for lost capacity. A single server going DOWN shifts its share of traffic onto survivors. Look at status and act. If a server recently went DOWN (lastchg small) and the queue appeared at the same time, you are watching a capacity-concentration event. If the survivors are near their limits, one more failure cascades.

5. Rule out a reload artifact. Check Uptime_sec. Right after a reload, the backend connection pool is cold, every request pays a fresh connect (brief ctime spike), and stick tables start empty unless you sync them via peers. A short-lived queue spike in the first minute after a reload is expected; a queue that persists past warmup is not.

6. Decide what the queue is doing to clients. Read qtime. If qtime is a few milliseconds and the queue drains between bursts, the backpressure mechanism is working as designed. If qtime is a meaningful fraction of timeout queue (or timeout connect if timeout queue is unset), clients are about to start seeing 503s.

Metrics and signals to monitor

Alert on qcur as a TICKET-level signal when it is sustained nonzero, and treat qmax trending upward over weeks as a PLAN-level capacity signal. Page when the queue is producing 503s: frontend 503 rate above a small percentage of req_rate with qcur > 0, held for a few minutes. The full severity model for these signals is in the HAProxy monitoring checklist.

Fixes

Backend is genuinely undersized

If arrival rate is normal, servers are healthy, rtime is at baseline, and scur/slim is pegged, you simply need more capacity.

• Add servers to the backend. The durable fix. Size for N+1: with one server removed, the rest should handle peak without queuing.
• Raise per-server maxconn. Only if the application behind the server actually has headroom. Raising maxconn past what the app can serve converts an orderly HAProxy queue into an uncontrolled pile-up inside the application, which is harder to observe and recover from. Set per-server maxconn at or below the backend’s own connection limit.
• Improve connection reuse. With http-reuse enabled, HAProxy sends new requests over idle persistent backend connections instead of opening fresh ones, so each slot carries more work. If reuse is collapsing (backend sending Connection: close, HTTP version downgrade), fixing it recovers capacity without adding anything.

Backend got slower

If rtime is climbing, the queue is a symptom. HAProxy-side changes only buy time.

• Fix the backend latency. Database regression, GC pauses, saturated downstream dependency. Per-server rtime comparison tells you whether it is one server or the whole pool.
• Set timeout queue explicitly. If it is unset, it inherits timeout connect, which is often far longer than you want clients waiting for a slot. A shorter queue timeout fails fast and sheds load instead of letting an unbounded queue amplify the slowdown. Tradeoff: queued requests become 503s sooner, so you are choosing fast failure over slow success.
• Set maxqueue as a circuit breaker. A bounded queue plus redispatch keeps one saturated server from accumulating an infinite wait line. The documented tradeoff: redispatch breaks persistence, so sticky-session users land on a different server. If persistence is load-bearing, weigh that carefully.

Lost capacity or hot server

• Restore or replace the failed server. If a server is flapping, put it in MAINT to stop traffic oscillating onto a sick instance.
• For a persistence-driven hot server, check whether a small set of clients (a NAT gateway, a single heavy account) is pinned to one server. Rebalancing the stickiness key or the balancing algorithm addresses the skew; raising capacity does not.

What not to do

Do not reload or restart HAProxy to clear a queue. The queued connections belong to clients, the queue rebuilds in seconds if the capacity problem is real, and the reload resets your stats counters, drops warm backend connections, and destroys the evidence you need for diagnosis.

Prevention

• Monitor both queue scopes continuously. Per-backend qcur alone hides the single-hot-server case; per-server qcur alone hides whole-pool saturation. Collect both, plus qmax and qtime. Queue monitoring sits at Level 3 of the HAProxy monitoring maturity model: if you are not watching it, you are flying without your earliest saturation signal.
• Set timeout queue deliberately. Do not inherit timeout connect by accident. Pick a value that reflects how long a client should reasonably wait for a slot.
• Decide on a queue policy. Unbounded (maxqueue 0) means slow degradation then timeout errors. Bounded means fast failure and redispatch with a persistence tradeoff. Either is defensible; accidental is not.
• Capacity-plan with N+1. Track qmax and peak scur/slim over weeks. If losing one server would push survivors into queueing at peak, you are already short.
• Treat persistent qcur as a ticket, not background noise. In a correctly sized system, steady-state queue depth is zero.

How Netdata helps

• Per-second qcur, qmax, and qtime collection per backend and per server, so short queue bursts that a 60-second scrape interval would average away are still visible.
• Queue depth correlated with slot utilization (scur/slim) on the same dashboard, which answers the first diagnostic question (are slots actually full?) without switching tools.
• Queue signals next to rtime and server status, so you can tell at a glance whether the queue is a capacity problem, a slowness problem, or a lost-server problem.
• Anomaly detection on qcur and qtime catches the slow drift (queue depth creeping up week over week) that fixed zero-threshold alerts miss until it becomes an incident.
• Frontend 5xx alongside queue metrics, making the queue-to-503 progression visible as a single story rather than two unrelated alerts.

Related guides

• How HAProxy actually works in production: a mental model for operators
• HAProxy maxconn reached: new connections queued and rejected
• HAProxy monitoring checklist: the signals every production proxy needs
• HAProxy monitoring maturity model: from survival to expert