RabbitMQ messages lost on restart: durable queues, persistent messages, and transient traps

You restarted a RabbitMQ node, it came back healthy, and now downstream systems are reporting missing data. The queues are there. The exchanges and bindings are there. But messages that were sitting in queues before the restart are gone, or a queue itself vanished entirely.

This is almost never a RabbitMQ bug. It is a mismatch between what the operator thinks durability means and what RabbitMQ actually guarantees. The broker did exactly what it was configured to do; the configuration just guarantees less than you assumed.

What this means

RabbitMQ durability is not one property. It is a stack of three independent properties, and a message survives a restart only if every layer in its path opted in:

1. The queue must be durable. Durable queue metadata is written to disk and the queue is recovered on node boot. Transient (non-durable) queues are deleted on node boot, by design, along with everything in them.
2. The message must be published as persistent. In AMQP 0-9-1 this means delivery_mode=2. Messages published as transient are discarded during recovery, even if they sat in a durable queue. The queue shell survives; the contents do not.
3. For node loss (not just restart), the queue must be replicated. A durable classic queue lives on one node. If that node’s disk is gone, durability settings are irrelevant. Quorum queues replicate via Raft; in RabbitMQ 4.x, quorum queues and streams are the only replicated queue types.

The most common trap is layer 2: a durable queue full of transient messages. Operators see the queue reappear after a restart and assume its contents survived too. They did not. The broker kept the queue definition and discarded every transient message inside it.

There is a subtler trap even when you did everything right. On classic queues, RabbitMQ does not fsync before sending a publisher confirm. A confirmed persistent message on a durable classic queue can still be lost if the node crashes (power failure, kernel panic, OOM kill) in the window between the confirm and the fsync. Quorum queues fsync on a quorum of nodes before confirming, which closes that gap. This is one of the concrete reasons quorum queues are the recommended queue type in 3.13+. In 4.x, classic queues are non-replicated, leaving quorum queues and streams as the replicated options.

flowchart TD
  A[Message published] --> B{Queue durable?}
  B -- No --> C[Queue and messages deleted on restart]
  B -- Yes --> D{delivery_mode = 2?}
  D -- No --> E[Queue survives, messages discarded on recovery]
  D -- Yes --> F{Queue type}
  F -- Classic --> G[Survives clean restart; crash before fsync can still lose confirmed messages]
  F -- Quorum --> H[Persistent by default, fsync on quorum before confirm; survives restart and node loss]

Common causes

Quick checks

Run these read-only checks before the next maintenance window or restart. They tell you exactly what would survive.

# 1. List queues with durability, type, and message counts
rabbitmqctl list_queues name durable auto_delete type messages messages_persistent

The critical comparison on every queue is messages vs messages_persistent. Any gap between those two numbers is messages that will not survive a restart.

# 2. Same check via the management API, per queue
curl -s -u guest:guest http://localhost:15672/api/queues | \
  jq '.[] | {name, durable, auto_delete, type, messages, messages_persistent}'

# 3. Find durable queues holding only transient messages (the classic trap)
curl -s -u guest:guest http://localhost:15672/api/queues | \
  jq '.[] | select(.durable == true and .messages > 0 and .messages_persistent == 0) | {name, messages}'

# 4. Check the broker log for the restart type (clean stop vs crash recovery)
grep -i "starting message store recovery\|crash" /var/log/rabbitmq/rabbit@$(hostname).log | tail -20

If check 3 returns anything, you have queues that look safe and are not. That is the finding this article exists to surface.

How to diagnose it

If messages are already gone, you are doing forensics. If they are not gone yet, you are auditing. The steps are nearly the same.

1. Confirm the queue survived. If the queue itself is missing after restart, the queue was transient, auto-delete, or exclusive. Message durability is moot; the container was never going to survive. Check how the declaring application creates the queue.
2. Confirm the restart was clean. A planned systemctl restart rabbitmq-server or rabbitmqctl stop is a clean shutdown. A crash (OOM kill, power loss, erl_crash.dump in the data directory) is not. On classic queues, a crash widens the loss window because of the fsync gap described above.
3. Compare persistent vs total message counts from before the restart. If you have historical metrics, pull messages and messages_persistent for the affected queues from just before the shutdown. messages - messages_persistent is the upper bound of what restart loss explains. If the missing count matches that gap, you have your answer.
4. Check what the publishers actually send. Look at the publisher code or client library configuration for delivery_mode=2 (or the library’s equivalent, e.g. Persistent=true, MessageProperties.PERSISTENT_TEXT_PLAIN in the Java client). Many client libraries default to transient. This is the single most common root cause.
5. Check the queue type. rabbitmqctl list_queues name type shows classic, quorum, or stream. If it is classic and the missing messages were published in the seconds before a crash, the fsync gap is the likely explanation.
6. Rule out lookalikes. Before blaming restart loss, exclude the silent-loss patterns that produce the same symptom: unroutable messages dropped because mandatory=false, TTL expiry with no DLX, and max-length overflow with the default drop-head behavior. Check exchange bindings with rabbitmqctl list_bindings, queue policies with rabbitmqctl list_policies, and whether publish rate exceeds deliver rate while queue depth stays flat.

Metrics and signals to monitor

Fixes

Transient messages in a durable queue

Fix the publisher. Set delivery_mode=2 on publish for every message that must survive a restart. There is no broker-side setting that upgrades already-published transient messages to persistent, and no way to recover the ones already discarded.

Note the tradeoff: persistent messages go through the message store and disk path, which costs latency and I/O. Do not blanket-set persistence on high-volume telemetry or metrics fan-out where loss is acceptable. Decide per queue.

Non-durable queue that should survive

You cannot change durability on an existing queue. Re-declaring a queue with different properties fails with a channel error (PRECONDITION_FAILED, 406). The fix is:

1. Declare a new queue with durable=true and the correct type.
2. Rebind it to the same exchanges with the same binding keys.
3. Drain or move messages from the old queue (a shovel, or let consumers finish it).
4. Repoint consumers and publishers, then delete the old queue.

This is disruptive if done carelessly. Do it during a low-traffic window and confirm consumers are flowing on the new queue before deleting the old one.

Classic queue holding critical data

Migrate the queue to a quorum queue. Quorum queues make every message persistent by default, fsync on a quorum of nodes before confirming, and replicate across nodes, which also covers node loss rather than just restart. The migration path is the same shape as the durability fix above: declare a new quorum queue, rebind, drain, cut over. See the related guide on migrating to quorum queues for the full procedure.

Tradeoffs: quorum queues have Raft overhead, higher per-message memory cost, and require at least a three-node cluster to tolerate a node failure. For genuinely ephemeral work queues, a durable classic queue with persistent messages may still be the right answer.

The fsync gap on classic queues

If you must stay on classic queues, accept the residual risk: on a hard crash, messages confirmed in the last moments before the crash can be lost even with durable queue plus delivery_mode=2 plus publisher confirms. The mitigation is publisher confirms (so the application knows what was acknowledged and can republish what was not) plus idempotent consumers that tolerate redelivery. There is no broker-side configuration that makes classic queues fsync before confirm.

Prevention

• Declare everything durable by default in application code, and make transient queues an explicit, reviewed choice. From RabbitMQ 4.3.0, transient non-exclusive classic queues are deprecated and cannot be declared by default, so new clusters will push you this way anyway.
• Set delivery_mode=2 as the publisher default for any queue whose contents matter, and document per queue which class of traffic it carries.
• Enable publisher confirms and treat a missing confirm as a republish signal at the application level. RabbitMQ offers no guarantees for messages that were never confirmed.
• Monitor the messages vs messages_persistent gap per queue as a standing metric, not just during incidents. A growing gap on a “durable” queue is the transient trap forming in real time.
• Rehearse restarts in staging with realistic traffic and verify message counts before and after. A staged restart with a known message count is a five-minute test; most teams instead discover their durability gaps in production.
• Use quorum queues for anything that must survive node loss, and accept classic queues only where a single node’s disk is an acceptable blast radius.

How Netdata helps

• Netdata collects per-queue message counts including the persistent subset, so the messages vs messages_persistent gap is visible as a standing chart rather than a post-incident query.
• Node uptime and restart events are tracked, so you can line up exactly when a node went down against which queues lost messages.
• Publish, deliver, and ack rates per queue let you distinguish restart loss from the lookalikes: silent routing loss (publish healthy, deliver zero, depth flat) looks different from transient-message discard (depth present before restart, gone after).
• Correlating queue depth history across a restart window answers the forensic question directly: how much of the missing backlog was ever persistent in the first place.
• Resource alarms and connection blocked states around the restart window tell you whether the shutdown was clean or crash-adjacent, which determines whether the classic queue fsync gap is in scope.

Related guides

• RabbitMQ classic mirrored queues removed: migrating to quorum queues
• RabbitMQ connection blocked / blocking: publishers frozen by a resource alarm
• RabbitMQ connection in flow state: credit-based backpressure explained
• RabbitMQ consumers connected but not acknowledging: the consumer black hole
• RabbitMQ disk free limit alarm: free disk space insufficient and publishing halted
• RabbitMQ consumer timeout: delivery acknowledgement timed out and the channel is closed
• RabbitMQ connection storm: reconnect loops, FD pressure, and CPU spent on handshakes