RabbitMQ disk_free_limit at the default 50MB: the production footgun

RabbitMQ ships with disk_free_limit set to 50MB. When free disk space on the node’s data partition drops below that limit, the broker raises a disk alarm and blocks every publisher on every node in the cluster. Consumers keep draining, but message ingestion stops until free space climbs back above the threshold.

The 50MB default exists so development installs on tiny machines do not silently eat a laptop’s disk. On a production server with terabytes of storage, it is not a safety margin; it is a tripwire one inch off the floor. A single log rotation, an Erlang crash dump, or a burst of persistent messages can consume 50MB in seconds, and the resulting incident looks exactly like a broker outage: publishers frozen, applications timing out, queue depth flat at zero growth while upstream systems back up.

What the disk alarm actually does

The disk alarm is one of RabbitMQ’s two cluster-wide resource alarms (the other is the memory watermark). The mechanics matter because they are more aggressive than most operators expect:

• It is cluster-wide. The alarm fires on one node, but the block propagates. All publishers on all nodes stop. Connections that attempted to publish show state: blocked; connections with an active alarm that have not yet published show state: blocking.
• Consumers are not blocked. They keep receiving and acknowledging messages, which is the intended recovery path: drain queues, free space, alarm clears.
• It is a cliff edge. There is no gradual degradation. Free space crosses the threshold and publishing halts instantly.
• Free space is checked periodically, not continuously. The broker checks disk space roughly every 10 seconds, increasing the frequency as the limit is approached. A fast-filling disk can overshoot the limit between checks, and after you free space it can take until the next check for the alarm to clear.

That last point has a sharp edge the official documentation calls out: if the limit is set too low and messages are being paged out rapidly, RabbitMQ can run out of disk and crash in the gap between checks. A low limit does not just fail to protect you; it can fail to protect you while also blocking your publishers.

Why 50MB is dangerous on a real server

The failure is almost never “the message store grew to fill a terabyte disk.” It is something mundane landing on the data partition:

• Log rotation or a burst of log volume. RabbitMQ logs, plus anything else writing to the same partition, can blow through 50MB during a single noisy event (a connection storm, an auth failure flood).
• An Erlang crash dump. When the BEAM VM crashes it writes erl_crash.dump, which on a large node can be gigabytes. One crash can both fill the partition and block the restart path.
• Quorum queue WAL growth. Quorum queues append every operation to a write-ahead log. Under sustained throughput, or when a follower falls behind and compaction stalls, WAL segments accumulate fast. This is exactly the workload where you least want a 50MB margin.
• A burst of persistent messages or aggressive paging to disk under memory pressure.
• Backups, snapshots, or co-located processes writing to the same partition.

There is also a nastier variant: the disk alarm death spiral. When publishing stops, some of the work that would eventually reclaim disk (compaction, cleanup) can stall or fall further behind, so the node does not recover on its own and you are doing manual cleanup at 3 a.m. A 50MB limit turns a non-event into that scenario.

flowchart TD
  A[Log rotation, crash dump, WAL burst, or backup writes to data partition] --> B[disk_free drops below 50MB default]
  B --> C[Disk alarm fires on one node]
  C --> D[All publishers blocked cluster-wide]
  D --> E[Publish rate drops to zero, upstream systems back up]
  C --> F[Consumers keep draining]
  F --> G{Enough space freed?}
  G -->|Yes| H[Alarm clears on next disk check]
  G -->|No, or crash between checks| I[Manual cleanup or node crash]
  I --> J[erl_crash.dump fills partition further]

Choosing a sane value

The goal is a limit large enough that routine, non-broker events can never trip it, and that gives you real runway to react before the disk is actually full.

Relative to RAM. Set disk_free_limit.relative to 1.0-2.0, meaning 1-2x system RAM. The reasoning: under memory pressure RabbitMQ pages queue contents to disk, so the disk should be able to absorb on the order of the broker’s full memory footprint plus headroom.

Absolute. Set a fixed value of at least 2GB, sized up for your workload and partition. Absolute values are predictable: you know exactly what will trip the alarm, regardless of what hardware the node lands on.

A word of caution on relative in containers. The relative limit is computed from the memory RabbitMQ believes it has. In containerized deployments, the memory visible via cgroups depends on the Kubernetes resource configuration, and there are cases where the cgroup reports the entire VM’s memory rather than the container’s limit. That means relative = 1.0 can silently become “1x the host’s RAM,” which may be far more disk than the partition even has, or a number you never intended. If you run RabbitMQ in containers, prefer an absolute limit unless you have verified what memory the broker actually detects.

Precedence when both are set. If disk_free_limit.absolute and disk_free_limit.relative are both present, the absolute value wins. This was not always true: older releases had a bug where the relative value overrode the absolute one, which bit operators whose base images or Helm charts shipped a relative default (some charts set relative = 1.0) that silently overrode their configured absolute value. The fix landed in 3.11.5. If you are on anything older, check for a stray relative setting in your config chain before trusting your absolute value. Relatedly, some vendor images load config files in an order where a later file’s relative setting overrides your absolute one; if your absolute limit appears to be ignored, inspect the effective config rather than assuming RabbitMQ is broken.

The Kubernetes Cluster Operator changed its default to 2GB years ago, so operator-managed clusters are less exposed. The 50MB default persists in the core broker itself, including 4.x.

Setting it

In rabbitmq.conf:

# Pick one. Absolute is preferred in containers.
disk_free_limit.absolute = 2GB

# Or relative to detected RAM (1.0 = 1x RAM):
# disk_free_limit.relative = 1.0

You can also change it at runtime without a restart:

# Runtime change: absolute
rabbitmqctl set_disk_free_limit 2GB

# Runtime change: relative to RAM
rabbitmqctl set_disk_free_limit mem_relative 1.0

The runtime form is useful for emergencies (you need publishers unblocked now and cannot free space immediately), but it does not survive a broker restart. Treat it as a bridge and put the real value in rabbitmq.conf before the next restart finds you unprotected again.

Verify what the broker is actually enforcing, not what you think you configured:

# Check effective limit, free space, and alarm state per node
curl -s -u guest:guest http://localhost:15672/api/nodes | \
  jq '.[] | {name, disk_free, disk_free_limit, disk_free_alarm}'

# Cross-check at the OS level against RabbitMQ's view
df -h /var/lib/rabbitmq

If disk_free_limit in the API does not match your config file, suspect the precedence issue above: something else in the config chain set a relative or absolute value after yours.

Alerting: do not inherit the footgun into your thresholds

Raising the limit is half the job. The other half is making sure your alerting floor is not anchored to the same broken default.

A common pattern is to alert when free space drops below some multiple of the limit, for example warning at 3x disk_free_limit. With the 50MB default, “3x the limit” is 150MB. On a modern server that alert is noise-adjacent and still far too late to give you meaningful runway.

Use a floor that combines the relative margin with an absolute minimum:

• Warn when disk_free < max(3 * disk_free_limit, 1GB)
• Page when disk_free_alarm is true, sustained for more than a minute, with evidence of traffic impact (publish rate was non-zero, or connections are in blocked/blocking state). The sustain and traffic conditions filter out cold starts and idle instances where the alarm fires but nobody is affected.

Two more things worth tracking:

• Rate of consumption, not just the level. Runway is (disk_free - disk_free_limit) / consumption_rate. A partition losing 100MB/minute is a very different incident from one losing 1MB/minute at the same absolute free space.
• The quorum WAL directory specifically. Aggregate free space hides which consumer is growing. Quorum queue WAL segments live in their own subdirectory and can grow much faster than classic queue storage.

Signals to watch in production

The publish-rate pattern is how you distinguish a disk halt from a publisher-side failure: in a disk alarm, connections stay up, publish rate goes to zero, and deliver/ack rates keep moving. The RabbitMQ mental model guide covers how this composite differs from the memory wall, which looks similar but blocks on mem_alarm instead.

Prevention checklist

• Set the limit explicitly. Absolute 2GB minimum, or relative 1.0-2.0 on bare metal/VMs where detected RAM is trustworthy. Never inherit the 50MB default into production.
• Verify the effective value via the API after every config change and after upgrades, especially on older versions or vendor images with layered config files.
• Prefer absolute in containers. Cgroup memory detection makes relative limits unpredictable under Kubernetes.
• Isolate the data partition. Keep RabbitMQ’s data directory on its own partition so co-located logs, backups, and crash dumps from other software cannot trip the broker’s alarm.
• Alert on the floor, not the limit. disk_free < max(3 * disk_free_limit, 1GB) as a warning, the alarm itself as a page with sustain and traffic conditions.
• Know your runtime override. rabbitmqctl set_disk_free_limit buys you breathing room mid-incident; pair it with a permanent config change.

How Netdata helps

• Netdata collects per-node disk_free, disk_free_limit, and disk_free_alarm from the management API, so you can see runway and alarm state per node in one view instead of querying each broker by hand.
• Correlating the alarm flag with publish, deliver, and ack rates makes the disk-halt signature obvious: publish flat at zero while consumer drain continues.
• Connection state tracking surfaces the blocked/blocking population, confirming whether an active alarm is actually impacting traffic.
• OS-level disk metrics for the data partition, alongside broker-reported free space, catch the divergence cases where RabbitMQ’s view and the filesystem’s view disagree, and let you compute burn rate and time-to-alarm.
• Historical retention lets you see whether a limit was silently reset, for example after a restart dropped a runtime-only set_disk_free_limit change.

For the full signal inventory this fits into, see the RabbitMQ monitoring checklist and the monitoring maturity model.

Related guides

• How RabbitMQ actually works in production: a mental model for operators
• RabbitMQ memory resource limit alarm: publishers blocked across the whole cluster
• RabbitMQ monitoring checklist: the signals every production broker needs
• RabbitMQ monitoring maturity model: from survival to expert