RabbitMQ disk free limit alarm: free disk space insufficient and publishing halted

Your RabbitMQ cluster just stopped accepting messages. Publishers are connected but frozen. Consumers are still draining queues normally. The broker log shows a disk alarm, and the management UI shows disk_free_alarm: true on at least one node.

This is the RabbitMQ disk free limit alarm. It fires when free space on the node’s data partition drops below disk_free_limit, and it blocks every publisher on every node in the cluster, not just the affected node. It is a hard circuit breaker, not a gradual degradation: one threshold crossing and ingestion stops instantly.

Recovery is usually fast once you know whether the disk is genuinely full or the limit is simply misconfigured. Consumers are not blocked, so the system can partially heal itself while you work. This guide walks through distinguishing those two cases and getting publishers flowing again.

What this means

RabbitMQ monitors free space on the partition holding its data directory (Mnesia database, message stores, quorum queue WAL segments), typically /var/lib/rabbitmq. When disk_free drops below disk_free_limit, the node raises the disk_free_alarm. The default limit is 50MB.

Two properties of this alarm matter operationally:

• It is cluster-wide. One node breaching the limit blocks publishers on all nodes. Your alert should say “cluster publishing halted”, not “node X disk low”.
• It is cliff-edge. There is no slowdown phase. Publish rate goes from normal to zero in one check interval. Disk is checked at least every 10 seconds, and up to 10 times per second as free space approaches the limit, so the alarm usually fires within seconds of the threshold crossing.

Consumers keep working during the alarm. Deliveries and acknowledgements continue, which means queue drain can free disk space if messages are persistent or paged out. Connections show state: blocked (tried to publish and got frozen) or state: blocking (alarm active, has not published yet).

flowchart TD
  A[Free disk on data partition drops below disk_free_limit] --> B[Node raises disk_free_alarm]
  B --> C[Alarm propagates cluster-wide]
  C --> D[All publishers blocked on all nodes]
  C --> E[Consumers keep draining queues]
  E --> F[Persistent and paged messages leave disk]
  F --> G[Free space recovers above limit]
  G --> H[Alarm clears, publishing resumes]
  D --> I[Publish rate drops to zero]

Common causes

The most common case in mature environments is the first one: the limit, not the disk, is the problem. On a server with a 500GB data partition, routine events like log rotation, a crash dump, or a burst of paged messages can consume 50MB in seconds and trip the alarm while the partition still has hundreds of gigabytes free. The default is a development-friendly value, not a production one.

Quick checks

These are read-only and safe to run during an incident.

# Confirm the alarm state from RabbitMQ itself
rabbitmq-diagnostics check_alarms

# See disk_free, disk_free_limit, and the alarm flag via the management API
curl -s -u guest:guest http://localhost:15672/api/nodes | \
  jq '.[] | {name, disk_free, disk_free_limit, disk_free_alarm}'

# Check actual free space on the data partition at the OS level
df -h /var/lib/rabbitmq

# Find what is consuming space (largest directories first)
du -h -d1 /var/lib/rabbitmq | sort -rh | head -20

# Check the log directory and look for crash dumps
du -sh /var/log/rabbitmq
ls -lh /var/log/rabbitmq/erl_crash.dump* 2>/dev/null

# See which queues hold the most messages
rabbitmqctl list_queues name messages_ready messages_unacknowledged messages_persistent

# Count blocked vs running connections to confirm publisher impact
curl -s -u guest:guest http://localhost:15672/api/connections | \
  jq 'group_by(.state) | map({state: .[0].state, count: length})'

The key comparison is disk_free from the API versus Avail from df. They should roughly agree. If df shows hundreds of gigabytes free but disk_free in the API shows a small number, check that RabbitMQ is looking at the partition you think it is: the data directory may have been moved or mounted differently.

How to diagnose it

1. Confirm the alarm is the cause of the publishing halt. Check disk_free_alarm: true on at least one node via /api/nodes, and verify that the cluster publish rate dropped to zero at the same time. Connections in blocked or blocking state corroborate it. Also check mem_alarm: if the memory alarm is also active, you have two problems and the memory wall behaves differently. See RabbitMQ memory resource limit alarm: publishers blocked across the whole cluster.

2. Identify which node raised the alarm. In a cluster, iterate /api/nodes and find the node with disk_free_alarm: true. That node’s data partition is the one to investigate. Other nodes blocking publishers are just reacting.

3. Decide: genuine full disk or mis-set limit. Run df -h on the data partition of the affected node. If free space is genuinely near zero (or near a sensibly configured limit), go to step 4. If free space is abundant and the limit is still at the 50MB default, the limit is the bug. Go to the fixes section.

4. Find the consumer of the space. Use du to rank directories under /var/lib/rabbitmq and /var/log/rabbitmq. A sudden multi-GB jump with a recent node crash points at an Erlang crash dump. Steady growth correlated with publish throughput on quorum queues points at WAL segments. Growth in the Mnesia or message store directories correlates with queue depth.

5. Check whether consumer drain is helping. Watch messages_ready and deliver/ack rates while the alarm is active. If persistent messages are being consumed and acked, disk may recover on its own. If consumers are absent or stuck, drain will not happen and you must free space manually.

6. Verify clearing behavior after freeing space. The disk check runs periodically, so the alarm can take up to about 10 seconds to clear after space is freed. If it does not clear, re-check df against disk_free_limit: you may have freed less than you think.

Metrics and signals to monitor

For the full signal taxonomy and severity model, see RabbitMQ monitoring checklist: the signals every production broker needs.

Fixes

Free space on the affected partition

If the disk is genuinely full:

• Delete or archive crash dumps. erl_crash.dump files can be many gigabytes and are only useful for postmortem analysis. Copy them off-box if you need them, then remove them.
• Rotate and compress RabbitMQ logs. If the log directory is on the data partition, reclaim it. Fix rotation afterwards so this does not recur.
• Let consumers drain. Do not block or restart consumers during the alarm; they are your recovery mechanism. If consumers are down, restoring them frees disk for persistent queues.
• Purge non-critical queues only as a last resort. Purging destroys messages permanently. Identify queues that are safe to lose before doing this, and prefer consumer drain.
• Do not restart the broker as a first move. A restart does not free disk, and a node that cannot write to its data directory may fail to come back cleanly. Free space first.

After freeing space, wait for the next disk check interval (up to ~10 seconds) and confirm disk_free_alarm returns to false and publishers resume.

Correct a mis-set disk_free_limit

If the disk is fine and the limit is the problem:

• Set the limit to a production-sane value. The playbook guidance is disk_free_limit.relative of 1.0-2.0 (1-2x RAM) or an absolute value of at least 2GB. The official production checklist recommends setting it to at least the memory high watermark, so the entire memory contents can be paged to disk with headroom. A common formulation is at least 1.5x the absolute memory watermark.
• You can change the limit at runtime with rabbitmqctl set_disk_free_limit; the change lasts until the next node restart. Persist it in rabbitmq.conf (disk_free_limit.absolute or disk_free_limit.relative) so it survives restarts.
• Version note on absolute vs relative: if both disk_free_limit.absolute and disk_free_limit.relative are set, RabbitMQ 3.11.5 and later give absolute precedence. On earlier versions, relative incorrectly won. If you run an older release and set both, the effective limit may not be what you configured.
• Version note on alarm clearing: RabbitMQ 4.2.0 fixed a bug where concurrent memory and disk alarms could clear the publisher-blocking state when only one resource recovered. On older 4.x, re-check both alarms if publishers stay blocked after the disk recovers.

Prevention

• Set a production disk_free_limit now. This is the single highest-value change. The 50MB default will eventually page you during a routine event. Alert when disk_free drops below max(3 * disk_free_limit, 1GB) so you get warning before the cliff.
• Alert on the trend, not just the alarm. The alarm is cliff-edge; the free-space trend is your early warning. Track disk consumption rate and compute runway: (disk_free - disk_free_limit) / consumption_rate.
• Isolate the data partition. Keep backups, snapshots, and application logs off the RabbitMQ data partition so external jobs cannot trip the broker’s circuit breaker.
• Rotate logs and clean crash dumps. Automate log rotation and alert on the presence of erl_crash.dump files.
• Watch quorum queue WAL growth separately. WAL segments grow fast under high throughput before compaction catches up. Overall free-disk monitoring can miss how quickly the quorum directories are expanding. See How RabbitMQ actually works in production: a mental model for operators for the storage internals behind this.
• Treat the alarm text correctly in paging. The page should say publishing is halted cluster-wide. Per-node “disk low” tickets understate a full ingestion stop.

How Netdata helps

• Alarm state and runway together: Netdata collects disk_free, disk_free_limit, and disk_free_alarm per node, so you can see the boolean alarm alongside the trend that caused it, instead of discovering the cliff with no history.
• Correlation with queue state: overlaying the alarm window with messages_ready, messages_persistent, and publish/deliver rates shows whether consumers were draining during the halt and which queues held the disk-consuming backlog.
• Connection state visibility: blocked and blocking connection counts confirm whether the alarm had real traffic impact or fired on an idle node, which changes the severity.
• Per-second disk metrics on the data partition: OS-level disk usage and I/O on /var/lib/rabbitmq next to broker signals distinguishes a fast WAL burst from slow log growth.
• Cross-node comparison: because the alarm is cluster-wide but the cause is node-local, seeing all nodes’ disk trends on one view immediately identifies which partition actually breached.

Related guides

• How RabbitMQ actually works in production: a mental model for operators
• RabbitMQ memory resource limit alarm: publishers blocked across the whole cluster
• RabbitMQ monitoring checklist: the signals every production broker needs
• RabbitMQ monitoring maturity model: from survival to expert