$ guides / rabbitmq / rabbitmq-poison-message-blocking-queue ▌

Operations Guides

RabbitMQ poison message loop: a stuck queue head and endless redelivery

A queue with active consumers is making no progress. Queue depth is flat, messages are being delivered, but almost nothing is being acknowledged. The oldest message in the queue keeps getting older. This is a poison message loop: one message at the head of the queue crashes or is rejected by every consumer that picks it up, gets requeued, and is delivered again, forever.

The loop is expensive in three ways. First, the poison message itself never completes. Second, every valid message behind it is blocked, because the requeued message returns to the head. Third, each cycle burns consumer CPU, downstream calls, and log volume for zero business output. If prefetch is greater than 1, it gets worse: the poison message drags a batch of valid messages into the requeue cycle with it.

What this means

In normal operation, a consumer receives a message, processes it, and acks it. The ack removes the message from the broker. In a poison loop, the consumer either crashes while processing the message or explicitly rejects it with basic.nack or basic.reject and requeue=true. Requeue puts the message back at (or near) the head of the queue. The broker immediately redelivers it, with the redelivered flag set. The consumer fails again. Repeat indefinitely.

From the broker’s perspective, nothing is “broken”: delivery is happening, the consumer is connected, and no alarms fire. The composite signal is what gives it away:

Queue depth stable: messages are not accumulating, but they are not draining either.
Head message age climbing: the oldest message keeps getting older even though deliveries are occurring.
Redeliver rate elevated: the same message is being sent to consumers over and over.
Ack rate near zero: deliveries are not converting into completed work.
Unacked count stable at 1 or a small number: the in-flight set never grows because the same small set cycles.

flowchart TD
  A[Consumer fetches head message] --> B{Processing succeeds?}
  B -- ack --> C[Message removed, next message delivered]
  B -- crash or nack requeue=true --> D[Message returned to queue head]
  D --> A
  D -.-> E[Valid messages wait behind the poison message]

The prefetch interaction deserves emphasis. With prefetch greater than 1, the consumer holds the poison message plus several valid messages as unacknowledged. If the consumer crashes or its channel closes, the broker requeues the whole unacked batch, and the valid messages are trapped in the loop with the poisoned one. One bad message stalls not just itself but everything in the prefetch window behind it.

Common causes

Cause	What it looks like	First thing to check
Malformed or schema-incompatible payload	Consumer throws a deserialization or parse error on every delivery	Consumer application logs for a repeating exception on the same message
Consumer bug triggered by specific content	Crash or unhandled exception only for one message shape	Consumer error rate and stack traces correlated with this queue
Message exceeds consumer limits (size, complexity)	Consumer OOMs, times out, or exceeds a processing deadline on one message	Consumer memory/timeout logs; message size on the queue
Downstream dependency failure	Every message fails, not just one; looks like a poison loop but all messages cycle	Whether redeliver is elevated on multiple queues sharing the same dependency
No dead-letter exchange configured	Failed messages have nowhere to go except back to the head	Queue arguments and policies for a `dead-letter-exchange` setting
No delivery limit (classic queues, or quorum queues with the limit disabled)	Redelivery count unbounded	Queue type and effective `x-delivery-limit`

Distinguish “one bad message” from “everything fails” early. If the downstream database is down, every message looks poisoned, and the fix is the database, not the queue. The tell: poison is usually one message looping while others queue up behind it; a dependency failure shows a high redeliver rate across the whole queue with a failing downstream.

Quick checks

Substitute your vhost (the default / is URL-encoded as %2F) and queue name. The curl and list_* commands are read-only; the peek at the end requeues what it reads, so use it carefully (see note below).

# Per-queue depth, unacked count, consumers, and head message age
curl -s -u guest:guest http://localhost:15672/api/queues | \
  jq '.[] | {name, messages_ready, messages_unacknowledged, consumers, head_message_timestamp, state}'

# Per-queue message rates: look for redeliver >> ack
curl -s -u guest:guest http://localhost:15672/api/queues | \
  jq '.[] | {name, message_stats}'

# Same thing via rabbitmqctl
rabbitmqctl list_queues name messages_ready messages_unacknowledged consumers

# Detailed consumer info (prefetch, ack mode, channel)
rabbitmqctl list_consumers

# Peek at the head message without removing it (requeueing get)
curl -s -u guest:guest -X POST \
  http://localhost:15672/api/queues/%2F/my_queue/get \
  -H 'content-type: application/json' \
  -d '{"count":1,"ackmode":"ack_requeue_true","encoding":"auto"}'

Note on the peek: ack_requeue_true requeues the message after reading it, which sets the redelivered flag and, on quorum queues, may count toward the delivery limit depending on your broker version: releases before 4.3 counted every acquisition against the limit, while since 4.3 only genuine failures (basic.reject, client crash, AMQP 1.0 modified with delivery-failed=true) increment the delivery count. On a queue already in a redelivery loop this is usually harmless, but do not script it in a polling loop.

What to look for:

Depth flat, head_message_timestamp old and getting older: the head is stuck. This field depends on publishers setting the timestamp property; it can be absent or 0, so treat a missing value as “unknown”, not “healthy”.
redeliver rate non-trivial while ack rate is near zero on one specific queue: redelivery loop confirmed. (message_stats is only present for queues that have seen message activity since stats collection started.)
Unacked count equal to consumer count x prefetch, or pinned at a small constant: consumers are holding but not completing work.
The peeked message payload: malformed JSON, unexpected encoding, a schema version the consumer does not understand, or an abnormally large body are the usual findings.

How to diagnose it

Confirm the composite signal. For the suspect queue, verify all of: stable messages_ready, rising head age, redeliver rate well above zero, ack rate near zero, and consumers > 0. If depth is growing instead of flat, you are looking at a slow-consumer backlog, not a poison loop.
Rule out a downstream dependency failure. Check whether other queues consumed by the same application are also failing, and whether the consumer’s downstream (database, API) is healthy. If every message fails, fix the dependency first; the messages are not poisoned.
Identify the offending message. Use the requeueing get call above, or enable the firehose tracer (rabbitmq-plugins enable rabbitmq_tracing, then rabbitmqctl trace_on -p <vhost>) to capture deliveries on the affected queue. Look for the same message ID or payload appearing repeatedly with redelivered: true. Disable tracing afterwards (rabbitmqctl trace_off); leaving it on writes every traced message to disk.
Inspect the consumer failure mode. In consumer logs, find the exception thrown for that message. Determine whether the consumer is nacking with requeue=true explicitly, or crashing and letting the channel/connection close requeue the message implicitly. Both produce the same loop; the fix differs.
Check the queue’s defenses. Determine the queue type (GET /api/queues shows type in the arguments, or rabbitmqctl list_queues name type), then check whether a delivery limit and a dead-letter exchange are in effect via queue arguments and policies (rabbitmqctl list_policies). Classic queues have no native poison message handling; quorum queues track redeliveries and can enforce a limit.
Check prefetch. rabbitmqctl list_consumers shows prefetch per consumer. High prefetch plus a poison head means a whole batch of valid messages is trapped in the loop with it.

Metrics and signals to monitor

Signal	Why it matters	Warning sign
Per-queue `redeliver` rate	Direct measure of the loop	Elevated and sustained on one queue
Per-queue `ack` rate vs `deliver` rate	Deliveries that never complete	Deliver non-zero, ack near zero
`head_message_timestamp` (head age)	Latency of the oldest waiting message	Age climbs while depth is flat
`messages_ready` trend	Distinguishes poison (flat) from backlog (growing)	Flat depth with active consumers and no progress
`messages_unacknowledged`	Messages trapped in consumer buffers	Pinned at prefetch x consumers, or at a small constant
`consumer_utilisation`	Whether consumers can accept work	Low with a backlog present
Dead-letter queue depth	Where poison messages should be landing	Growing DLQ confirms failures are being captured; empty DLQ with high redeliver means no DLX path

Fixes

Stop the loop right now

Remove or quarantine the poison message manually. The fastest relief is getting the message out of the head position. Options, in order of preference:

Use the management API get with "ackmode":"ack_requeue_false" to pull the message and discard it. Copy the payload to a file first if you need it for debugging. This destroys that one message; that is the point.
Purge the queue only if every message in it is expendable. rabbitmqctl purge_queue destroys all messages, not just the poisoned one. Treat this as a last resort.

Do not restart the broker. A restart requeues everything and the poison message will simply resume its loop, now mixed in with everything else.

Fix the consumer’s failure handling

Nack with requeue=false and route to a DLX. The consumer should catch processing failures it cannot retry (parse errors, schema mismatches, validation failures) and reject with requeue=false. If a dead-letter exchange is configured on the queue, the message is dead-lettered instead of dropped, preserving it for inspection and replay. Reserve requeue=true for transient failures where a retry genuinely has a chance to succeed, and even then prefer bounded retries over infinite requeue.

Handle crash-based requeue. If the consumer process is dying mid-processing, the broker requeues all its unacked messages when the channel closes. That path bypasses your nack logic entirely, which is exactly why a broker-side delivery limit matters.

Enforce a delivery limit at the broker

Quorum queues: x-delivery-limit. Quorum queues track unsuccessful delivery attempts and, once a message exceeds the limit, drop it or dead-letter it if a DLX is configured. Set it as a queue argument (x-delivery-limit) or via a policy (delivery-limit). The default and the exact counting behavior have changed across releases: RabbitMQ 3.13 had no default limit (opt-in via x-delivery-limit), 4.0 introduced a default of 20, and since 4.3 the delivery count increments only for genuine failures (basic.reject, client crash, AMQP 1.0 modified with delivery-failed=true), so routine requeues such as basic.nack no longer count against the limit. Check the docs for your version before tuning.

A few operator notes:

The policy key is delivery-limit; the queue argument is x-delivery-limit. Using the wrong name in the wrong place silently does nothing.
Classic queues do not support poison message handling in any current version. If the loop is on a classic queue, your only broker-side defense is a DLX plus requeue=false discipline in the consumer, or migrating the queue to quorum.
Give the dead-letter queue its own delivery limit and a consumer. A DLQ with no limit and a failing consumer can loop the same message between the DLQ and the original queue.
Watch out for dead-letter routing cycles: if the DLX can route a message back to its original queue, the message can circulate. The broker detects a cycle and drops the message only if the cycle contains no rejections; a rejection in the cycle defeats that protection.

Tune prefetch. Keep prefetch modest so one poison message traps a small batch, not thousands. There is no universal value; size it from message processing time and consumer count, and revisit it whenever you see unacked pinned at the prefetch ceiling.

Prevention

Quorum queues with a delivery limit and a DLX on every workload queue. This makes the broker, not the consumer, responsible for terminating poison loops. Consumers crashing, throwing, or nacking incorrectly all funnel into the same bounded behavior.
Consumer-side poison handling as a library pattern. Wrap message processing in a handler that classifies exceptions: transient (retry with backoff), permanent (reject with requeue=false). Do not leave this to per-service reimplementation.
Monitor the composite, not one signal. Alert on the combination: redeliver rate elevated AND ack rate near zero AND head age climbing, sustained over several minutes. Any single one of these has benign explanations; together they are diagnostic. See the signal taxonomy in RabbitMQ monitoring checklist: the signals every production broker needs.
Treat the DLQ as a first-class queue. Alert on DLQ depth growth, give it a consumer or a replay procedure, and cap its own delivery limit. An unwatched DLQ becomes the incident later.
Schema discipline. Most poison messages are schema drift: a producer shipped a payload shape the consumer cannot parse. Contract testing between producers and consumers catches these before deploy.

How Netdata helps

Per-queue redeliver, deliver, and ack rates side by side: the deliver-vs-ack divergence plus elevated redeliver is the poison signature, and seeing all three on one queue in one view is the fastest confirmation.
Queue depth and unacked trends: flat ready depth with a pinned unacked count distinguishes a poison loop from a consumer backlog, which changes the entire response.
Head message age where available: tracking head_message_timestamp over time turns “the queue seems stuck” into “the head has not moved in 40 minutes”.
Consumer count and utilisation correlation: confirms consumers exist and are attached, ruling out the look-alike case of zero consumers with growing depth.
Cross-signal anomaly detection: a redeliver-rate anomaly on one queue while global broker rates look normal is exactly the kind of localized pattern that per-queue anomaly flags surface before depth-based alerts ever would.