RabbitMQ Erlang run queue high: scheduler saturation and slow heartbeats

The RabbitMQ node is alive, queues are accepting messages, but everything feels slow: publishes lag, acks lag, the management API responds sluggishly, and the cluster log starts showing “missed heartbeats” from peers or clients. The node’s run_queue metric sits persistently above the CPU core count. OS-level CPU may even look unremarkable.

This is Erlang scheduler saturation. The BEAM VM has more runnable processes than its scheduler threads can execute, so work queues up and every operation on the node, including heartbeat processing, waits its turn. The secondary effect is worse than the primary one: delayed heartbeats get misread as dead peers, and the cluster starts reporting network partitions that do not exist.

This guide covers how to confirm scheduler saturation, separate it from ordinary CPU load, find the driver, and fix it without making the heartbeat problem worse.

What this means

RabbitMQ runs on the Erlang VM (BEAM). The VM starts a set of scheduler threads, typically one per CPU core, and every connection, channel, and queue is an Erlang process that those schedulers execute cooperatively. When a process is runnable but no scheduler is free, it waits in the run queue.

The run_queue metric (exposed in the Management API per node and via erlang:statistics(run_queue)) counts processes waiting to run, aggregated across all schedulers. Read it against the total core count, not against 1:

• run_queue at or near 0: schedulers keep up. Spikes during startup, queue purges, or policy application are normal and transient.
• run_queue sustained above 1 per core: work is queuing. Latency rises for everything.
• run_queue sustained above total cores: the node is saturated. This is the condition that delays heartbeats.

The heartbeat link is the dangerous part. RabbitMQ connections rely on heartbeat frames to detect dead peers: the default timeout is 60 seconds, frames are sent roughly every timeout/2, and about two missed frames mark the peer unreachable. Cluster nodes additionally use Erlang’s net_ticktime (default 60 seconds) to judge node health. When the schedulers are backlogged, the node may be perfectly healthy but too slow to send or process these frames on time. GC pauses make it worse: a long pause under memory pressure can exceed net_ticktime and produce a transient false partition that appears and heals on its own.

One more trap: OS CPU and Erlang scheduler utilization diverge. Schedulers may be pinned to a subset of cores, the machine may run other workloads, or the scheduler count may not match the actual cores available (a classic container misconfiguration). You can see 40% OS CPU while Erlang schedulers are at 95%. Trust the run queue and scheduler utilization, not top.

flowchart TD
  A[CPU driver: TLS, routing, churn, GC] --> B[Schedulers busy]
  B --> C[run_queue sustained above cores]
  C --> D[All ops slow: publish, ack, mgmt API]
  C --> E[Heartbeat frames delayed]
  E --> F[Missed heartbeat errors in logs]
  E --> G[net_ticktime exceeded: false partition]
  D --> H[Flow control on connections]
  G --> I[Quorum queues lose members, clients reconnect]

Common causes

Quick checks

All read-only and safe on a live node.

# 1. Run queue from the Management API (aggregate across schedulers)
curl -s -u guest:guest http://localhost:15672/api/nodes | jq '.[] | {name, run_queue}'

# 2. Same value directly from the VM
rabbitmqctl eval 'erlang:statistics(run_queue).'

# 3. Core count, for the comparison
nproc

# 4. Per-scheduler utilization (this call samples for 5 seconds and blocks)
rabbitmqctl eval 'scheduler:utilization(5000).'

# 5. Where scheduler time goes: emulator, gc, port, sleep
rabbitmq-diagnostics runtime_thread_stats

# 6. Churn: handshakes and channel setup eat scheduler time
curl -s -u guest:guest http://localhost:15672/api/overview | jq '.churn_rates'

# 7. Heartbeat and partition evidence in the log
grep -i "missed heartbeats\|partition" /var/log/rabbitmq/rabbit@$(hostname).log | tail -30

# 8. Cluster view: are partitions actually being reported?
rabbitmq-diagnostics cluster_status

Two notes on interpretation. First, scheduler:utilization(5000) blocks the caller for the sample window; that is expected, not a hang. The scheduler module (with scheduler:utilization/1) has shipped in Erlang/OTP since OTP 21, so it is available on every supported RabbitMQ (3.13 and 4.x mandate OTP 26 or newer). Second, if the management API itself is slow to answer check 1, that is consistent with saturation, not a separate API problem.

How to diagnose it

1. Confirm saturation, not a spike. Sample run_queue every few seconds for a couple of minutes. Startup, queue purges, and post-restart sync legitimately spike it. You care about sustained elevation above the core count.

2. Compare scheduler utilization to OS CPU. Run scheduler:utilization(5000) and watch OS CPU in parallel. Schedulers hot with OS CPU low points to a scheduler count mismatch or pinning problem. Both hot points to genuine CPU demand.

3. Break down where scheduler time goes. rabbitmq-diagnostics runtime_thread_stats splits time between emulator (executing code), gc, port (I/O), and sleep. High gc share points at memory pressure. High emulator with high publish rates points at routing and serialization. High port points at I/O-heavy work.

4. Identify the driver. Correlate the saturation window with message rates (message_stats in /api/overview), churn rates, and connection counts. Saturation that tracks publish rate is throughput-driven. Saturation with flat message rates but high churn is handshake-driven. Saturation with neither points at GC, stats collection, or a hot internal process.

5. Check GC and memory pressure. Take two samples of rabbitmqctl eval 'erlang:statistics(garbage_collection).' a few seconds apart and compare the deltas. Check rabbitmq-diagnostics memory_breakdown for a bloated binary heap. As a diagnostic only, rabbitmq-diagnostics force_gc forces GC across all processes: if run_queue and memory drop right after, lazy GC was contributing. force_gc itself is a CPU burst; do not run it repeatedly on a saturated node.

6. Quantify the heartbeat damage. Grep the log for “missed heartbeats” and check cluster_status for partitions. If partitions appear and self-heal, correlate their timing with run_queue peaks and GC activity before treating them as network events.

7. Rule out noisy neighbors. If the node shares a host or VM with other workloads, check host-level CPU. The Erlang runtime assumes it does not share CPU; time-slicing against other tenants inflates latency far beyond what the load suggests.

Metrics and signals to monitor

Fixes

Reduce CPU per message

If saturation tracks publish rate, the routing path is too expensive. Topic exchanges with thousands of bindings make every publish match routing keys against patterns. Large fan-outs turn one publish into work for hundreds of queue processes. Simplify routing topology, consolidate bindings, and split hot exchanges. Check message sizes too: serialization cost scales with payload.

Move TLS termination off the broker

TLS at the broker is one of the most common saturation drivers, and reconnect bursts multiply it because every handshake is CPU-heavy. Terminating TLS at a load balancer or proxy in front of the broker removes that cost, at the tradeoff of plaintext between the proxy and broker, which may be unacceptable in your threat model. If TLS must stay on the broker, reducing connection churn (below) matters even more.

Fix connection and channel churn

Connection setup involves the TLS handshake, AMQP negotiation, and Erlang process creation; channels are each a process too. Clients that open a connection per operation, reconnect without backoff, or use a channel-per-message pattern will saturate schedulers while doing almost no messaging. Enforce connection pooling, reuse channels, and add reconnect backoff. See RabbitMQ connection storm: reconnect loops, FD pressure, and CPU spent on handshakes and RabbitMQ channel leak and churn: the channel-per-message anti-pattern.

Fix scheduler and core mismatches

If scheduler utilization is high but OS CPU is low, check how many schedulers the VM started versus how many cores the node can actually use. In containers, a node started with the host core count under a 2-core CPU quota will thrash. Verify the container CPU limit and that the runtime sees it.

Relieve GC pressure

Saturation with a high gc share in runtime_thread_stats and a rising memory ratio means the VM is spending scheduler time collecting. Find the memory driver first: queue depth, unacked messages, or a bloated binary heap (rabbitmqctl eval 'erlang:memory(binary).'). Draining the memory pressure removes the GC load with it. Do not treat force_gc as a remediation loop; it is a diagnostic.

Rebalance and scale

Quorum queue leaders do all the publish and dispatch work for their queues, so a node holding most leaders works much harder than its peers. If one node saturates while others idle, check leader distribution and rebalance. If the workload genuinely exceeds one node’s cores, add capacity or move queues; no tuning makes a saturated scheduler faster.

Do not fix heartbeats by tightening timeouts

Lowering the heartbeat timeout on a saturated node makes false detections more frequent, not fewer. Values in the low single digits produce false positives even on a healthy node under burst load. Keep the default 60 seconds or a moderate value, and fix the saturation instead. The same applies to net_ticktime: raising it is sometimes justified on lossy links, but here it is a mask, not a fix.

Prevention

• Alert on the ratio, not the absolute. Ticket (do not page) when run_queue exceeds total cores sustained for several minutes, and when average scheduler utilization stays above 80%. Both are leading indicators well before heartbeats start failing.
• Trend churn separately from connection count. A stable connection count can hide heavy open/close churn that is quietly burning scheduler time.
• Keep CPU headroom. Plan capacity so peak load leaves schedulers below roughly 60% average. Past about 80%, small load increases produce disproportionate latency.
• Do not colocate. Run RabbitMQ nodes on hosts, or with CPU guarantees, that do not share cores with other heavy workloads.
• Correlate partitions with run_queue before declaring network incidents. A transient partition that coincides with a run_queue spike or GC pause is a symptom, not a network fault. Require partitions to persist before paging.
• Watch the write path after fixes. After reducing churn or topology cost, confirm run_queue drops and connections leave flow state; see RabbitMQ connection in flow state: credit-based backpressure explained for what that state means.

How Netdata helps

• Saturation over time: charting the node’s run_queue over minutes and hours shows sustained saturation instead of point samples during an incident.
• Scheduler vs OS CPU correlation: charting per-core OS CPU from Netdata next to the node’s scheduler utilization and run_queue makes the divergence pattern (saturated VM, idle host) visible immediately, which is the fastest route to a misconfiguration fix.
• Churn correlation: Netdata’s RabbitMQ connection and channel churn charts on the same view as CPU let you see whether scheduler time is going to handshakes or to messaging.
• Throughput overlay: publish, deliver, and ack rates against CPU and run_queue separate workload-driven saturation from overhead-driven saturation.
• Timeline correlation: partition events and resource alarms on the same timeline as run_queue spikes let you confirm false partitions instead of chasing phantom network faults.

Related guides

• RabbitMQ connection storm: reconnect loops, FD pressure, and CPU spent on handshakes
• RabbitMQ channel leak and churn: the channel-per-message anti-pattern
• RabbitMQ connection in flow state: credit-based backpressure explained
• RabbitMQ connection blocked / blocking: publishers frozen by a resource alarm
• RabbitMQ flow control vs resource alarms: the two throttling mechanisms operators confuse
• RabbitMQ Erlang distribution port saturation: the single link that carries all cluster traffic