How HAProxy actually works in production: a mental model for operators

Most HAProxy incidents come from an operator applying the wrong mental model: treating HAProxy like a web server, like nginx, or like a black box that either works or does not. HAProxy is an event-driven state machine that multiplexes hundreds of thousands of connections across a small number of threads, and almost every confusing symptom traces back to one of a handful of internal mechanisms: the accept pipeline, the maxconn hierarchy, the buffer and connection pools, the health check engine, or a subsystem you forgot was running.

This article is the mental model. It does not fix a specific symptom. It explains what HAProxy is doing at all times, so that when a runbook tells you to check qcur or Idle_pct or show table, you know which internal mechanism that signal belongs to and why it matters.

What it is and why it matters

HAProxy is an event-driven Layer 4 (TCP) and Layer 7 (HTTP/HTTPS) proxy and load balancer. It accepts connections on frontends, evaluates ACL rules to route them to backends, and forwards traffic to individual servers within those backends. That three-level object model is the skeleton everything else hangs on. Almost every stat HAProxy exposes is scoped to one of these three.

Two properties of the engine shape everything operationally:

Event-driven, not process-per-connection. The core event loop uses the kernel’s most efficient poller (epoll on Linux) to multiplex connections across a configurable number of threads (nbthread). There is no thread per connection. One thread handles thousands of concurrent connections by waking only when a socket is ready. This is why CPU cost is driven by events (new connections, TLS handshakes, header parsing) rather than by connection count, and why Idle_pct (time spent waiting in poll()) is a more honest CPU saturation measure than system CPU. In current versions, threads are the only scaling model: nbproc was deprecated and scheduled for removal in the 2.4 manual, and multi-process directives no longer appear in current configuration documentation.

Two sides per proxied connection. For every proxied request, HAProxy holds a client-side connection (to the frontend) and a server-side connection (to the backend). These are independent. HAProxy can hold a client connection open while waiting for a server slot, and it can reuse a persistent server connection across many client requests. This asymmetry explains the resource math: roughly two file descriptors per proxied connection, two buffers per connection, and a connection pool on the backend side that behaves nothing like the frontend side.

How it works: the request pipeline

For an HTTP request, the internal pipeline looks like this:

flowchart LR
  A[Client] --> B[Accept on frontend socket]
  B --> C[TLS handshake]
  C --> D[Parse request, evaluate ACLs]
  D --> E[Backend selection and stick lookup]
  E --> F{Server slot free?}
  F -- no --> G[Queue]
  G --> F
  F -- yes --> H[Connect or reuse pooled connection]
  H --> I[Forward request and response]
  I --> J[Log at session end]

Step by step:

1. Accept. The connection arrives on a frontend listener socket and enters the OS accept queue (net.core.somaxconn). If that queue overflows, SYNs are dropped in the kernel and HAProxy never sees the connection at all. This is the classic “HAProxy looks fine but clients time out” case; it shows up in /proc/net/netstat as ListenOverflows and ListenDrops, not in any HAProxy counter.
2. TLS handshake. If the frontend terminates SSL, the handshake runs here. It is the most CPU-intensive operation HAProxy performs, and it consults the session cache or ticket. High new-connection rates with low session reuse are how frontends melt without any backend involvement.
3. Request parsing. Headers are read into a per-connection buffer (default 16KB, tune.bufsize). Request rules and ACLs are evaluated here. Requests that cannot be parsed become ereq and HAProxy-generated 400s.
4. Backend selection. The routing decision: ACLs, stick-table lookup, balancing algorithm. Session persistence and rate-limit state are consulted here.
5. Queue. If the chosen server or backend has reached its maxconn, the request enters a per-server or per-backend queue and waits. Queuing is backpressure, not an error. With maxqueue at its default of 0 the queue is unbounded; the request waits until a slot frees or a timeout fires.
6. Connect to server. Either a new TCP connection or reuse of an idle connection from the connection pool (http-reuse). Reuse is what keeps ctime near zero and backend TLS handshake cost down.
7. Forward request, receive response, forward response. Response rules are evaluated on the way back.
8. Logging. The log line is emitted at session close, or at request end in HTTP mode. If the log target cannot keep up, HAProxy drops log lines and only the DroppedLogs counter tells you.

The pipeline explains the latency decomposition HAProxy exposes: ttime (total) is approximately qtime (queue) + ctime (connect) + rtime (server response) + data transfer. When you know which pipeline stage each timer belongs to, a latency incident localizes itself.

The maxconn hierarchy

HAProxy enforces connection limits at four levels, independently:

• Global maxconn. The process-wide ceiling. HAProxy derives it from the file descriptor limit at startup if you do not set it explicitly.
• Frontend maxconn. Caps concurrent sessions per frontend. If unset, it inherits from the global limit.
• Backend maxconn. Caps connections into a backend as a whole.
• Per-server maxconn. Caps concurrent connections to one server. Default is no limit, which is often a misconfiguration: most application servers cannot absorb unbounded concurrency and degrade silently instead.

The operational consequence: any one of these can be the binding constraint while the other three look fine. A per-server limit can cause queuing (qcur rising) while the frontend has plenty of headroom. This is why saturation runbooks check scur/slim at every row type, not just the global CurrConns/Maxconn. Saturation against one of these limits, where the process is alive and healthy but new work queues or is rejected, is the single most common HAProxy operational issue.

The subsystems that matter

These run continuously, independent of any single request, and each owns a failure archetype.

Buffer pools. Pre-allocated memory for connection buffers, two per connection (request and response, each tune.bufsize). Under memory pressure, connections stall waiting for buffers, which produces latency spikes with no CPU or network explanation. The PoolFailed counter in show info is the confirmation signal; it should always be zero.

Connection pools. Idle backend connections kept alive for reuse via http-reuse. The connect vs reuse counters show the ratio of new to reused backend connections, and idle_conn_cur shows the pool depth. When reuse silently breaks (backend starts sending Connection: close, HTTP version change, config drift), you pay full TCP and possibly TLS cost per request, and ctime and backend handshake rates spike.

Stick tables. In-memory key-value stores for session persistence, rate limiting, and tracking. Fixed size. When full, the default is to purge expired entries; with nopurge, new entries are rejected outright. Either way, the functionality the table backs (rate limiting, persistence) silently stops working for affected clients. There is no eviction counter. show table is the only way to see utilization. Tables can be replicated between instances via the peers subsystem.

Health check engine. Runs independently of traffic, periodically probing servers over TCP, HTTP, agent-check, or external scripts. The resulting state (UP, DOWN, MAINT, DRAIN, plus transition states like UP 1/3) is the routing table. Two things to internalize: rise/fall thresholds mean there is a deliberate delay between a real failure and a status change, and a passing health check proves only that the check passes. A server can return 200 on /health while 500ing every real request. That gap is its own failure pattern.

DNS resolver. Used for dynamic server resolution (server-template, SRV records). Resolution failures do not immediately change routing: HAProxy keeps using cached results until TTL expiry, then it is stuck. In service-discovery deployments this means traffic can silently flow to stale IPs while every health-visible signal looks acceptable. show resolvers exposes sent queries, answers, timeouts, and errors.

Peers subsystem. Replicates stick-table state between HAProxy instances. Broken peer sync causes state divergence: rate limiting works on one node and not the other, session affinity breaks across failover. show peers shows connection state and replication activity per peer.

SSL engine. Handles TLS termination. Session cache, OCSP stapling, and certificate storage all consume memory, and handshake CPU cost dominates under high new-connection rates. The session cache is invalidated on every reload, which is why SslFrontendKeyRate spikes after each reload even with constant traffic.

Where it shows up in production

The failure archetypes below are not random; each maps to one mechanism above. Recognizing the mapping is most of diagnosis.

Deployment variants change which of these you can even see. TCP mode loses every HTTP-level signal (response codes, request rates). Multi-threaded mode (nbthread > 1) spreads load but makes some counters per-thread, so show activity per-thread counters become necessary to spot a single hot thread that the averaged Idle_pct hides. With busy-polling enabled, Idle_pct sits near zero by design and you must use show activity run-queue depth or system CPU instead. Active-passive pairs have a standby with no traffic signals at all, so failover detection replaces traffic monitoring there. Master-worker mode means the master is always alive while workers restart; monitor the worker, not the master. Reloads reset every cumulative counter, so rate math must handle resets and Uptime_sec is your reload detector.

Signals to watch in production

You do not need all of these on day one, but you should know which subsystem each one belongs to.

How Netdata helps

• Netdata collects HAProxy stats continuously, so scur/slim, qcur, hrsp_5xx, econ/eresp, and wretr/wredis are available as time series at every level (frontend, backend, server) without hand-rolled socat cron jobs.
• The latency decomposition (qtime, ctime, rtime, ttime) is charted together, which turns “latency is up” into “the queue stage is up” in one glance.
• Correlating server status transitions with per-server scur and rtime on the same dashboard makes the backend-collapse cascade visible as it develops, rather than after the last server goes DOWN.
• Idle_pct alongside SslFrontendKeyRate separates TLS-driven CPU exhaustion from general traffic load, which is the difference between “scale TLS” and “scale everything.”
• Because Netdata also collects host metrics, kernel-level signals like ListenOverflows/ListenDrops and file descriptor headroom sit next to HAProxy’s own counters, closing the “HAProxy saw nothing” blind spot.
• Counter resets on reload appear as such in the time series, so reload storms do not read as phantom rate spikes when you correlate with uptime.

Related guides

• HAProxy monitoring checklist: the signals every production proxy needs
• HAProxy monitoring maturity model: from survival to expert
• HAProxy maxconn reached: new connections queued and rejected
• HAProxy scur approaching slim: the concurrent-session saturation signal
• HAProxy maxconn hierarchy: global, frontend, backend, and per-server limits
• HAProxy backend queue building (qcur): requests waiting for a free server slot
• HAProxy Slowloris and idle-connection pile-up: high scur, low request rate
• HAProxy health check L4TOUT and L4CON: server marked DOWN and unreachable
• HAProxy health check L7STS: server DOWN on the wrong HTTP status
• HAProxy server flapping: rise, fall, and health checks oscillating UP and DOWN
• HAProxy health checks green but the application is broken: when UP does not mean healthy
• HAProxy backend losing servers: active server count and cascade risk