$ guides / rabbitmq / rabbitmq-quorum-queue-wal-growth ▌

Operations Guides

RabbitMQ quorum queue WAL explosion: Raft log outpacing snapshots

Every operation on a quorum queue is appended to a Raft write-ahead log on disk. That log is supposed to be bounded: the queue periodically snapshots its state and discards the log segments the snapshot made redundant. When appends outpace snapshots, the WAL grows without bound, the data partition fills, and the node trips the disk alarm.

The disk alarm is not a quorum-queue-local event. When free disk on any node drops below disk_free_limit, RabbitMQ blocks all publishers on all nodes in the cluster. Consumers keep draining, but nothing new comes in. A single hot quorum queue can halt message ingestion for the entire cluster, including classic queues and streams that share the node.

This failure mode looks healthy from the application side for a long time. Queue depth stays stable, ack rates look fine, and memory is flat, because quorum queues stream messages to disk rather than holding them in RAM. The first symptom many teams see is the disk alarm itself, which is the cliff edge, not the warning.

What this means

Each quorum queue maintains its own WAL directory under {mnesia_dir}/quorum/{queue_name}/ on every node hosting a member of that queue. On a default Linux package install, mnesia_dir is typically /var/lib/rabbitmq/mnesia. The WAL is written by the Raft layer: publishes, acks, consumer registrations, and other state changes all become log entries.

Compaction is driven by snapshots. Once the queue’s state machine has been snapshotted up to a given Raft index, log segments below that index are eligible for deletion. Anything that delays snapshots or keeps old entries referenced keeps segments on disk:

Sustained high throughput: appends arrive faster than the snapshot cadence reclaims them.
A lagging or down follower: it cannot advance its snapshot position while catching up, and WAL growth accelerates on that member.
Slow or unacknowledged deliveries: entries that are still “live” cannot be discarded.
Consumer churn: operations on the queue state are themselves log entries, so churn grows the log even on queues with no backlog.

The cascade:

flowchart TD
  A[High publish rate or slow acks or consumer churn] --> B[Raft WAL appends outpace snapshots]
  C[Lagging or down follower] --> B
  B --> D[WAL segments accumulate in mnesia_dir/quorum/]
  D --> E[Data partition free space falls]
  E --> F[disk_free below disk_free_limit]
  F --> G[Disk alarm fires on one node]
  G --> H[All publishers blocked cluster-wide]

Common causes

Cause	What it looks like	First thing to check
Sustained throughput beyond snapshot cadence	WAL size tracks publish rate; shrinks in quiet periods, grows in bursts	Correlate WAL directory size with per-queue publish rate
Lagging or down follower member	One member node’s quorum directory grows much faster than peers; `online` list shorter than `members`	`rabbitmqctl list_queues name type online members`
Unacknowledged message accumulation	`messages_unacknowledged` high and flat; old log entries stay referenced	Per-queue unacked count and ack rate
Consumer or connection churn	WAL grows even with zero backlog; segment count climbs steadily	Connection and channel churn rates in `/api/overview`
Requeue or redelivery loops	High `redeliver` rate on the queue; each cycle appends log entries	Per-queue redeliver rate
Too many quorum queues on one node	Aggregate WAL across many queues exhausts the partition even though each queue looks moderate	Count quorum directories and total size under `quorum/`
Poison message without delivery limit	Stable depth, elevated redeliver, endless log appends for the same message	Redeliver rate plus head message age

Quick checks

These are all read-only.

# 1. Confirm whether the disk alarm is active on this node
rabbitmq-diagnostics check_alarms

# 2. Check free space on the data partition
df -h /var/lib/rabbitmq

# 3. Size the quorum WAL directories, largest first
du -sh /var/lib/rabbitmq/mnesia/quorum/*/ 2>/dev/null | sort -rh | head -20

# 4. Total WAL footprint on this node
du -sh /var/lib/rabbitmq/mnesia/quorum/

# 5. List quorum queues, their members, and which members are online
rabbitmqctl list_queues name type online members | grep quorum

# 6. RabbitMQ's own view of disk free vs the alarm limit
curl -s -u guest:guest http://localhost:15672/api/nodes | \
  jq '.[] | {name, disk_free, disk_free_limit, disk_free_alarm}'

# 7. Would stopping this node break quorum for any queue?
rabbitmq-diagnostics check_if_node_is_quorum_critical

On 4.x, rabbitmq-queues quorum_status <queue> shows per-member Raft detail including the Snapshot Index column; a follower whose snapshot index is stuck while the leader’s advances has stopped snapshotting.

How to diagnose it

Confirm the alarm state and the actual filesystem. Run check_alarms and df -h together. If RabbitMQ says alarm but the filesystem has plenty of space, your problem is a misconfigured disk_free_limit, not WAL growth. See the disk free limit alarm guide.
Attribute the disk consumption. Run the du commands above. If quorum/ dominates the partition, you are in the right article. If classic queue message stores, logs, or an erl_crash.dump dominate, this is a different incident.
Identify the top queues. The largest directories under quorum/ map to queue names. One queue growing means a workload or consumer problem. Many queues growing in parallel means aggregate overprovisioning of quorum queues on the node.
Check member health for the top queues. Compare online against members. A follower that is down or persistently catching up cannot advance its snapshot position, and its WAL balloons. Also check that leader distribution is not lopsided after a recent restart; the leader does all the publish and dispatch work for a queue.
Check consumer-side signals for the top queues. Pull messages_unacknowledged, message_stats (deliver, ack, redeliver), and consumers from /api/queues. High unacked with near-zero ack rate means old log entries stay live. High redeliver means requeue loops appending entries endlessly. Zero backlog but growing WAL points at churn rather than message data.
Estimate runway. Divide remaining free space (above disk_free_limit) by the measured WAL growth rate. That tells you whether you have hours to fix the root cause or minutes to relieve disk pressure first.

Metrics and signals to monitor

Signal	Why it matters	Warning sign
WAL directory size per queue (`du` on `quorum/<queue>`)	The direct measure of this failure; not exposed as a broker metric, so collect it from the OS	Sustained growth across snapshot cycles; size that never shrinks
`disk_free` vs `disk_free_limit`	The alarm is the cliff edge; the ratio is the warning	Ratio under 3.0, or falling fast
`disk_free_alarm`	Cluster-wide publishing halt when true	True for any sustained period
Per-queue `messages_unacknowledged`	Unacked entries keep log segments live	High and not draining
Per-queue redeliver rate	Requeue loops grow the log without progress	Elevated while ack rate is near zero
Connection and channel churn rates	Consumer churn appends log entries even on empty queues	Churn far above baseline
Quorum members `online` vs `members`	A lagging or down follower drives WAL growth on that member	Mismatch sustained beyond a restart window
Publish vs ack rate per queue	Sustained imbalance means the log grows faster than it can be trimmed	Publish exceeding ack for tens of minutes

Fixes

Relieve disk pressure first (if the alarm is close or active)

Free space on the data partition without touching the WAL: rotate or compress RabbitMQ logs, remove stale erl_crash.dump files, and clear anything else sharing the partition. Consumers are not blocked by the disk alarm, so letting them drain persistent backlogs also reclaims space. Do not delete WAL segment files by hand; that corrupts the queue.

If you need immediate relief and the queue is expendable, deleting and recreating the queue is the only way to force its WAL away. There is no CLI command to force compaction of a live queue. This destroys all messages in the queue, so treat it as a last resort and say so in the incident channel before doing it.

Recover or remove a lagging follower

If one member is down or stuck catching up, getting it healthy lets snapshots advance and the WAL shrink. If the node is not coming back, remove it from the queue’s membership so the remaining members stop carrying a dead position. The rabbitmq-queues CLI provides member management for this: rabbitmq-queues add_member <queue> <node> and rabbitmq-queues delete_member <queue> <node> (add -p <vhost> for non-default vhosts). Before restarting or removing any node, run check_if_node_is_quorum_critical so you do not tip another queue into minority.

Fix the consumer-side driver

Stuck consumers holding unacked messages: restart or fix the consumer application; the unacked entries are what keep old segments live. See consumers connected but not acknowledging.
Requeue loops: set a delivery limit on the queue so a poison message dead-letters after N attempts instead of cycling forever. RabbitMQ 4.0 introduced a default delivery limit of 20 for quorum queues; on 3.13 you must set x-delivery-limit explicitly. See the consumer timeout guide for the related consumer-side failure.
Consumer churn: fix clients that call basic.consume in tight reconnect loops without basic.cancel. Each registration is a log entry. See the connection storm guide.

Rebalance or reduce queue placement

If many quorum queues share one node, spread members across the cluster and rebalance leaders so no single node carries the write load for everything. Longer term, cap how many quorum queues land on one node, and question whether every queue needs to be quorum; queues that do not need the durability guarantees may belong on classic queues.

Raise `disk_free_limit` to something sane

The default is 50 MB, which a busy quorum workload can consume in seconds. Set an absolute value (2 GB or more) or use disk_free_limit.relative so the alarm reflects your actual partition size and gives you runway to react. This does not fix WAL growth; it stops a routine burst from becoming a cluster-wide halt. See the default 50MB footgun.

Upgrade if you are on an affected version

WAL and snapshot behavior has changed substantially across releases. RabbitMQ 4.0 added the default delivery limit (20) and checkpoint-based recovery, 4.0.x and 4.1.x shipped follower snapshot and checkpointing fixes, and 4.3 (via the Ra v3 upgrade) introduced a new log compaction mechanism in which a single unacknowledged message no longer pins all later segments. If you are fighting this on 3.13 or early 4.0, upgrading is a legitimate fix.

Prevention

Monitor the WAL directory explicitly. Overall disk-free monitoring does not distinguish WAL growth from everything else. Track du on quorum/ per node and per top queue, and alert on growth rate, not just absolute size.
Set delivery limits on every quorum queue so poison messages dead-letter instead of looping.
Size disk_free_limit for the partition you actually have, and keep free space at least 3x the limit.
Spread quorum membership and leadership across nodes, and rebalance after rolling restarts.
Capacity-plan disk for peak WAL, not steady state. WAL usage spikes during bursts and follower catch-up; provision for the worst case.
Keep consumers honest: monitor unacked counts, redeliver rates, and churn so the log never grows faster than snapshots can trim it.

How Netdata helps

Netdata collects RabbitMQ node metrics from the management API, including disk_free, alarm states, and per-queue message stats, so you can correlate WAL-driven disk decline with publish, ack, and redeliver rates on the same dashboard.
Per-queue messages_unacknowledged and redeliver trends surface the consumer-side drivers that keep log segments live, before the disk alarm fires.
Connection and channel churn charts expose the consumer-registration churn that grows the WAL on otherwise empty queues.
Node-level disk metrics from the OS collector let you watch the data partition alongside RabbitMQ’s own disk_free view, catching divergence between what the broker sees and what the filesystem reports.
Alarm state history (disk_free_alarm, mem_alarm) gives you the exact moment publishing halted, which anchors the incident timeline against WAL growth.