RabbitMQ silent message loss: unroutable messages dropped when mandatory is false

Your publisher is healthy. The broker is healthy. Publish rate is non-zero, connections are up, no alarms, no errors in the log. And yet downstream systems are missing data. Queue depth is flat. Deliver rate is zero. Nothing looks broken, because nothing is broken in the usual sense: RabbitMQ is doing exactly what it was told to do.

When a message is published to an exchange and no queue binding matches its routing key, the broker has nowhere to route it. If the publisher did not set mandatory=true (and most client libraries default it to false), the broker silently discards the message. No return, no log line, no queue growth, no error surfaced to the publisher.

This is the most dangerous routing failure in RabbitMQ because it produces no symptom at the point of loss. Teams typically discover it hours or days later, when a downstream consumer reports missing data and someone has to reconstruct where it went. This guide covers how to confirm it is happening, how to find the routing fault, and how to make the loss visible and preventable.

What this means

In AMQP 0-9-1, routing is the exchange’s job. A publisher sends a message to an exchange with a routing key. The exchange evaluates its bindings and produces a list of target queues. If that list is empty, the broker’s behavior depends on two things:

• An alternate exchange (AE) on the target exchange. If one is configured, unroutable messages are republished to the AE instead of being dropped or returned.
• The mandatory flag on the publish. If no AE is configured and mandatory=true, the broker returns the message to the publisher with a basic.return. If false (the default in most client libraries), the message is dropped.

Neither is set by default. So the out-of-the-box behavior for a routing misconfiguration is silent, complete data loss with a fully green dashboard.

Two facts make this worse in production:

1. Publisher confirms do not save you. The broker confirms unroutable messages too. Once the exchange determines the message will not route to any queue, it issues the confirm. A publisher that enables confirms and treats every ack as “message safely queued” will see 100% confirm success while losing 100% of messages. Confirms guarantee the broker accepted the message, not that any queue did.
2. The return_unroutable counter only counts returns. Messages dropped because mandatory=false do not appear in it. If you alert only on return_unroutable, you catch the publishers that opted into returns and stay blind to everyone else.

flowchart TD
  P[Publisher sends message] --> E{Exchange: any matching binding?}
  E -->|yes| Q[Routed to queue]
  E -->|no| AE{Alternate exchange configured?}
  AE -->|yes| AEX[Republished to alternate exchange - counts as routed, no basic.return]
  AE -->|no| M{mandatory flag set?}
  M -->|mandatory=true| R[basic.return to publisher - counted in return_unroutable]
  M -->|mandatory=false| D[Silently dropped - no counter in message_stats]

Common causes

The unifying theme: this is almost always a topology or configuration fault, not an infrastructure fault. Nothing in RabbitMQ is overloaded, crashed, or alarmed. That is why it evades every resource-based alert.

Quick checks

These are read-only and safe to run during an incident.

# 1. The composite signature: publish vs deliver_get vs queue depth
curl -s -u guest:guest http://localhost:15672/api/overview | \
  jq '{publish: .message_stats.publish, deliver_get: .message_stats.deliver_get,
       return_unroutable: .message_stats.return_unroutable,
       ready: .queue_totals.messages_ready}'

# 2. List all bindings and eyeball the topology
rabbitmqctl list_bindings

# 3. Bindings for the suspect exchange specifically
curl -s -u guest:guest \
  'http://localhost:15672/api/exchanges/%2f/my.exchange/bindings/source' | jq .

# 4. Confirm the target queues exist and their state
curl -s -u guest:guest http://localhost:15672/api/queues | \
  jq '.[] | {name, state, messages_ready, consumers}'

# 5. Check policies for anything that would drop messages
#    (TTL without DLX, max-length with drop-head overflow)
rabbitmqctl list_policies

The first check is the detection signal in one command. The signature of silent routing loss is: publish > 0, deliver_get == 0 (or far below publish), messages_ready flat or zero, return_unroutable == 0. The last term being zero is the tell that the publishers are not using mandatory=true, because if they were, you would see returns.

If you run RabbitMQ 3.8 or later with the Prometheus plugin, there is a direct counter for the dropped case:

# Dropped vs returned unroutable messages (Prometheus endpoint, 3.8+)
curl -s http://localhost:15692/metrics | grep unroutable
# rabbitmq_channel_messages_unroutable_dropped_total   -> dropped (mandatory=false, no AE)
# rabbitmq_channel_messages_unroutable_returned_total  -> returned (mandatory=true)

rabbitmq_channel_messages_unroutable_dropped_total was introduced in 3.8.0. On older versions there is no broker-side counter for silent drops at all, and detection relies on the composite publish/deliver/depth pattern or firehose tracing. Note also that return_unroutable is a channel-level counter; it is not available per queue.

How to diagnose it

1. Confirm the composite signature. Pull /api/overview and verify publish is non-zero while deliver_get and queue depth are flat. If depth is growing, you have a consumer problem, not a routing problem. See RabbitMQ consumers connected but not acknowledging for that failure mode.

2. Check the returned counter. If return_unroutable is non-zero, at least some publishers use mandatory=true and the broker is telling you which channels are hitting unroutable messages. If it is zero, assume drops are happening silently and continue.

3. Isolate the exchange. Publishers log which exchange and routing key they use. Take the suspect exchange and list its bindings via the API. An empty or stale binding list is the fault. For topic exchanges, take a real routing key from publisher logs and evaluate it against the binding patterns by hand.

4. Check the target queues. A binding can be correct while the queue behind it was deleted and never recreated. Verify existence and state, not just the binding entry.

5. Correlate with deploys. Silent routing loss that starts cleanly at a point in time almost always correlates with a deployment: a service version that publishes to a new exchange, an infrastructure-as-code change that redeclared an exchange without its bindings, or a queue that was migrated. Check your deploy timeline before deep-diving the broker.

6. Rule out other silent-loss paths. Unroutable drops are not the only way messages vanish without errors. Messages can also expire via TTL with no dead-letter exchange configured, or be pushed out by a max-length policy with the default drop-head overflow. rabbitmqctl list_policies shows both. If bindings are fine, look here next.

Metrics and signals to monitor

A note on severity: because this is a configuration fault rather than an infrastructure fault, it will not self-escalate into an outage, and it is tempting to treat it as a ticket. Do not. If the lost messages are business-critical, confirmed silent loss is a page. The damage is proportional to how long it goes unnoticed, and detection lag is the whole problem.

Fixes

Restore the routing

The immediate fix is always to repair the topology: recreate the missing binding, recreate the deleted queue, or correct the routing key or topic pattern on the publisher. Messages already dropped are gone. RabbitMQ does not buffer unroutable messages anywhere, so there is nothing to recover broker-side. Recovery means republishing from the source system if it retains the data.

Set mandatory=true and handle returns

With mandatory=true, the broker sends a basic.return to the publisher for any unroutable message. Two operational requirements:

• The publisher must register a return handler. Without one, returned messages are silently discarded by the client library and you are back to invisible loss.
• The handler must do something observable: log, increment a metric, write to a fallback store. A handler that swallows returns is decorative.

This converts silent loss into an application-visible event and makes return_unroutable meaningful.

Add an alternate exchange

An AE catches every unroutable message on an exchange and republishes it, typically to a fanout exchange bound to a “dead letter for unroutable” queue you monitor and alert on. This works even for publishers you do not control and do not want to modify. Configure it via a policy rather than exchange arguments, so it can be managed centrally.

Caveats: the default exchange (the empty-string exchange) is special-cased and does not support alternate exchanges, so messages published there with no matching queue are uncatchable this way. And if a message routes via the AE, it counts as routed for the purposes of mandatory, so no basic.return is generated.

Keep publisher confirms, but for the right reason

Publisher confirms guarantee the broker accepted responsibility for the message. They do not guarantee it reached a queue. Keep confirms for durability against broker failure, but do not treat a confirm as end-to-end delivery. Confirm latency from the publisher side is also a useful early-warning signal for resource pressure, but it must be measured client-side; the broker does not expose it.

Prevention

• Alert on the composite, not just the counter. return_unroutable > 0 catches mandatory publishers. The composite “publish non-zero + deliver_get zero + flat depth” catches everyone else. You need both. With the Prometheus plugin on 3.8+, alert directly on any increase in rabbitmq_channel_messages_unroutable_dropped_total.
• Declare topology as code and reconcile it. Bindings created by hand in the management UI are the ones that disappear. Declaring exchanges, queues, and bindings from versioned configuration, applied on deploy, makes topology drift a diffable event.
• Standardize the publisher contract. Every publisher: confirms on, mandatory=true, a return handler that logs and counts, and a documented fallback for returned messages. Make the client library’s shared wrapper do this so individual teams cannot opt out by accident.
• Monitor the unroutable graveyard. If you use an alternate exchange, bind it to a queue and alert on depth > 0. An AE queue nobody watches is just a slower way to lose messages.
• Audit for the other silent-loss paths. TTL without a DLX and max-length with drop-head discard messages just as silently. Include both in the same periodic policy review.

How Netdata helps

• Netdata’s RabbitMQ collector pulls the management API message stats, so publish, deliver, ack, and return_unroutable rates are charted on the same timeline, making the publish-vs-deliver_get divergence visible without manual API polling.
• Per-queue charts for messages_ready and per-queue message rates let you spot the flat-depth-with-active-publishing signature on the specific queue that should be receiving.
• Because these are cumulative counters, Netdata computes the rates for you, so the composite pattern reads as line divergence rather than raw counter arithmetic.
• Topology-adjacent signals (queue count, consumer count, churn rates) on the same dashboard help you correlate the start of silent loss with the binding or queue change that caused it.
• The signal combination for this failure is exactly the “return unroutable rate + zero depth with zero deliver” composite that the RabbitMQ monitoring checklist lists as an expert-level signal.

Related guides

• How RabbitMQ actually works in production: a mental model for operators
• RabbitMQ monitoring checklist: the signals every production broker needs
• RabbitMQ consumers connected but not acknowledging: the consumer black hole
• RabbitMQ consumer_utilisation low: consumers attached but not keeping up
• RabbitMQ head message age: the queue latency that depth alone cannot show
• RabbitMQ connection blocked / blocking: publishers frozen by a resource alarm