HAProxy peers sync broken: divergent stick tables across instances

You have two or more HAProxy instances fronting the same service, and their behavior no longer matches. Rate limiting triggers on one node but lets the same client through on another. A failover happens and every sticky session evaporates, even though you configured stick tables and a peers section specifically to survive that event. Or you compare show table output across instances and the entry counts are wildly different when they should be near-identical.

This is the signature of broken peer synchronization. The peers subsystem is the only mechanism HAProxy has for replicating stick-table state between instances. When it breaks, each instance builds its own private view of the world, and every feature that depends on shared state (session persistence, rate limiting, abuse tracking) silently degrades into per-instance behavior.

The failure is quiet. HAProxy does not log an obvious error when a peer connection drops, and nothing in the CSV stats tells you replication stopped. The only direct visibility is show peers on the runtime socket, which is why this problem often goes unnoticed until a failover exposes it.

What this means

HAProxy stick tables are in-memory key-value stores used for session persistence, rate limiting, and client tracking. Each table is local to the process that owns it. When you configure a peers section and reference it from a stick table, HAProxy pushes local table updates to the configured peer instances over dedicated TCP connections, so that every instance holds roughly the same state.

Replication is one-way per connection: each peer pushes its own local updates to the others. There is no authoritative copy and no consensus. Consistency is eventual and depends entirely on every peer connection being up, healthy, and processing updates.

When sync breaks, the tables diverge:

• Rate limiting becomes per-instance. A client limited to 100 requests per minute effectively gets 100 per minute per node. With N instances behind a load balancer, your real limit is N times what you configured.
• Session affinity breaks across failover. The backup instance never received the stick-table entries, so a failover reroutes users to random backends. Depending on the application, that means lost carts, re-authentication, or cache misses.
• Failover is silent. Nothing errors. Traffic flows. The divergence only surfaces as “users got logged out when we failed over” or “the rate limit didn’t hold during the attack.”

flowchart LR
  A[HAProxy instance A
table: 41200 entries] -. "peer connection down
or updates lost" .-> B[HAProxy instance B
table: 8900 entries]
  A --> R1[rate limit enforced on A]
  B --> R2[rate limit bypassed on B]

Common causes

Quick checks

All of these are read-only. Run them on every instance in the peer group, not just the one you suspect.

# 1. Peer connection and sync state (the primary signal)
echo "show peers" | socat unix-connect:/var/run/haproxy.sock stdio

# 2. Quick rollup of peer liveness from process info
echo "show info" | socat unix-connect:/var/run/haproxy.sock stdio | grep -i peer

# 3. Stick-table sizes and utilization, to quantify divergence
echo "show table" | socat unix-connect:/var/run/haproxy.sock stdio

# 4. What identity is this instance using?
# Peer name resolution order: -L command-line argument, then localpeer
# in the global section, then the system hostname.
ps -o args= -C haproxy | tr ' ' '\n' | grep -A1 '^-L$'
grep -E '^\s*localpeer' /etc/haproxy/haproxy.cfg
hostname

# 5. Peer port reachability from the other instance
# (run from peer B, targeting peer A's configured peer address/port)
nc -vz <peer-a-address> <peer-port>

# 6. Compare peers config across instances
grep -A20 '^peers ' /etc/haproxy/haproxy.cfg

A note on the stats socket: show peers returns nothing if the socket lacks sufficient privilege level or if the peers section name does not match. If you get empty output, verify the socket is configured with level admin before concluding there are no peers. There is no show info field that reports connected-peer liveness; show peers is the only direct view.

How to diagnose it

Work through these in order. Most peer sync failures are found in the first three steps.

1. Confirm the peers section is actually in use. Peers replication only exists if your stick tables reference a peers section. If no peers section is configured, divergence is expected behavior, not a bug. Check the stick-table lines for a peers <name> argument.

2. Read show peers on every instance. You are looking for each remote peer’s connection state. An established, working peer shows an established status (ESTA). Anything else, including a peer that never connects or a local peer marked (local,inactive), is your lead. Peer connections are initiated from one side, so a broken connection can look different on the two ends. Always check both.

3. Check local peer identity. The local peer’s name must exactly match one peer line in the peers section. The resolution order is the -L command-line argument, then localpeer in the global section, then the system hostname. The classic failure: the config was written expecting hostnames, someone later launched HAProxy with -L or in a container with a generated hostname, and the name no longer matches any peer line. The instance then cannot identify itself in the peer group and stays inactive. This is the single most common cause of “peers not syncing.”

4. Verify the network path on the peer port. The peers protocol runs over TCP on whatever port you configured on the peer lines. Confirm the port is open in host firewalls, security groups, and any network policy between instances. In container deployments, verify the address on the peer line is the address other instances can actually route to, not a host IP behind port forwarding. A forwarded port can pass a TCP SYN but break the protocol handshake if the advertised identity does not line up.

5. Quantify the divergence. Run show table on every instance and compare used counts per table. Small deltas are normal (replication is asynchronous). Large, persistent, or growing deltas confirm replication is not keeping up or not flowing at all. Also compare show peers output across instances: the per-peer connection state and per-table counters (update, localupdate, commitupdate, last_acked, last_pushed) are visible in the output, and healthy connections on both ends with diverging counters point to a replication-layer problem rather than a connectivity problem.

6. Check for a version-specific regression. If connections are established, counters advance, but entries still go missing under load, check your exact HAProxy version against the issue tracker and the peers/stick-table changelog entries in your branch. Peers replication has had version-specific regressions with healthy-looking show peers output but diverging counters; compare your version against the last known-good release in your deployment.

7. Check recent reloads and process churn. A peer that was reloaded rebuilds its tables from whatever the other peers teach it. During that relearning window, tables are legitimately divergent. If reloads are frequent (dynamic service discovery, ingress controllers), instances may never fully converge.

Metrics and signals to monitor

None of these appear in the CSV stats export. If your monitoring only scrapes show stat, you are blind to this entire failure class. Collection has to hit the runtime socket with show peers and show table on every instance.

Fixes

Fix the peer identity mismatch

Make the local peer name match a peer line exactly. The cleanest approach is to set it explicitly with the -L command-line argument per instance rather than relying on hostnames, which change under containers, autoscaling, and hostname regeneration. After correcting it, reload and confirm the local peer no longer shows as inactive. Tradeoff: -L per instance means per-instance process manager configuration (separate systemd units or templated unit arguments), which is more plumbing than a shared config but removes the ambiguity entirely.

Fix the network path

Open the configured peer port bidirectionally between all instances. In container deployments, put the container’s own routable address on the peer line, or run host networking, so the advertised identity matches what peers dial. Verify with a real TCP connection from each instance to every other instance’s peer address, not just a ping or a security group audit.

Align configurations across instances

The peers section and every replicated stick-table definition should be identical on all instances: same table names, same size, same expire, same store data types. If tables differ, peers may connect cleanly but never exchange entries for the mismatched tables. Generate the peers section and stick-table lines from the same template on every node rather than editing per-host.

Upgrade or downgrade away from a confirmed replication regression

If you have confirmed a version-specific replication bug (healthy connections, diverging counters, missing entries under load), the fix is a version change: pin to the last known-good release for your fleet until a patched release is available. Test replication under realistic load before rolling a new version across the peer group.

Size tables for the whole peer group

A table that fills up stops accepting new entries no matter how healthy sync is, and per-instance eviction makes divergence worse. Compare used against size during peak, and size tables with headroom for the full keyspace, not just one node’s share.

Prevention

• Monitor peer state explicitly. Collect show peers and show table from every instance on a regular interval. Alert on any peer leaving the established state and on stick-table used diverging beyond a tolerance between instances.
• Template the peers config. One source of truth for the peers section and replicated table definitions, rendered to all instances. Per-instance variance in this section is a divergence generator.
• Pin local identity. Use -L or localpeer deliberately rather than inheriting hostnames, especially in containers.
• Alert on reload frequency. If reloads outpace table relearning, your HA pair is permanently divergent during exactly the windows you built it for.
• Include peer state in failover testing. When you game-day a failover, verify the surviving instance’s tables contain the expected entries before declaring the test passed. “Traffic still flows” is not the same as “state survived.”
• Treat stick-table utilization as a shared metric. A full table on one instance usually means all instances are near full, and rate limiting is already degrading everywhere.

How Netdata helps

• Netdata collects HAProxy runtime-socket data per instance, so you can correlate stick-table utilization trends across all peer-group members on one dashboard instead of ssh-ing to each node during an incident.
• Per-instance table usage side by side makes divergence visible as it develops, not after a failover exposes it.
• Alerting on stick-table utilization approaching configured size catches the “rate limiting silently stopped” case that produces no error counter anywhere.
• Because peer sync failures often coincide with reloads and process churn, correlating HAProxy process events with table-metric discontinuities shortens the path to “this instance just relearned” versus “this instance is broken.”
• Combining table metrics with frontend signals (denied requests, session rates) on the same timeline shows the user-facing effect of divergence, which is what justifies the fix priority.

Related guides

• HAProxy 503 Service Unavailable: no server is available to handle this request
• HAProxy 504 Gateway Timeout: timeout server and the backend timeout cascade
• HAProxy backend losing servers: active server count and cascade risk
• HAProxy counter resets on reload: phantom drops and spikes in your metrics
• HAProxy scur approaching slim: the concurrent-session saturation signal
• HAProxy backend queue building (qcur): requests waiting for a free server slot