$ guides / rabbitmq / rabbitmq-consumer-timeout ▌

Operations Guides

RabbitMQ consumer timeout: delivery acknowledgement timed out and the channel is closed

You found this in the broker log or your client logs:

delivery acknowledgement on channel 1 timed out

In 3.13+ the fuller broker-side message names the consumer, queue, vhost, delivery tag, and the timeout value used, and the channel is closed with a PRECONDITION_FAILED (406) channel exception. The client library then typically reopens the channel and resubscribes, until it happens again.

This is RabbitMQ’s consumer_timeout mechanism firing. When a consumer holds a delivery without acknowledging it for longer than the configured timeout, the broker concludes the consumer is stuck, force-closes the channel, and requeues the unacknowledged messages. It is a protective feature, not a bug: an unacknowledged message is pinned in broker memory for potential redelivery, and a consumer that never acks is a slow-motion memory leak.

The timeout is almost never the root cause. It is the terminal symptom of a consumer that takes too long per message: a slow database call, a deadlocked handler, a downstream API that stopped responding, or a timeout value set without knowing the actual processing-time distribution. This guide covers how the mechanism works, how to tell “genuinely stuck” from “legitimately slow”, and how to fix it without just pushing the number up.

For the broader model of how unacked messages, prefetch, and consumer health interact, see How RabbitMQ actually works in production.

What this means

The mechanism, in order:

A consumer subscribes with manual acknowledgements and receives deliveries, bounded by its prefetch count.
The broker starts a per-delivery timer when it sends the message.
The timeout is evaluated periodically, not continuously. Values below one minute are not supported, and values below five minutes are not recommended, so the effective timeout is approximate: it can fire somewhat later than configured.
If any delivery remains unacknowledged past the timeout, the broker logs the timeout, closes the channel with PRECONDITION_FAILED, and requeues all unacked deliveries on that channel.
Most client libraries react to the channel error by reopening the channel and resubscribing. The requeued messages are redelivered. If processing is still slow, the cycle repeats indefinitely.

The default is 30 minutes (1,800,000 ms). The version history matters because the default changed inside patch releases:

3.8.15: mechanism introduced with a default of 15 minutes. This was a breaking change in a patch release and took down applications with long-running consumers that had upgraded routinely.
3.8.17: default raised to 30 minutes after the fallout.
3.12: per-queue configuration added, via the x-consumer-timeout queue argument and the consumer-timeout policy key.
4.3: consumer timeout evaluation moved into quorum queues. Classic queues and streams no longer evaluate it, and for clients that advertise the consumer_cancel_notify capability the broker cancels only the timed-out consumer with a basic.cancel instead of closing the whole channel. If you upgrade to 4.3 and relied on this protection for classic queues, it silently stops applying.

Where the effective value comes from, highest precedence first:

Consumer argument x-consumer-timeout on basic.consume
Queue argument x-consumer-timeout at queue declaration
Queue policy key consumer-timeout
Global consumer_timeout in rabbitmq.conf

When both the queue argument and a policy are set, the lower of the two wins. Check all four levels before assuming you know which timeout is firing; the broker log line includes the timeout value used, which is the fastest way to identify the effective configuration.

Common causes

Cause	What it looks like	First thing to check
Consumer processing genuinely too slow	Timeouts cluster around the configured value; ack rate near zero between timeouts; unacked count sits at prefetch	Consumer-side processing latency per message
Stuck or deadlocked consumer	Unacked count frozen at exactly prefetch size, ack rate exactly zero, no progress ever	Consumer thread dumps, downstream dependency health
Poison message	Same queue times out repeatedly, redeliver rate elevated, head message age grows	Consumer logs for a repeated exception on one payload
Downstream dependency failure	Timeouts start suddenly across many consumers at once	Database, API, or service the consumer calls
Timeout set too low for the workload	Long batch jobs (reports, video, bulk imports) exceed 30 min	Actual p99 processing time vs effective timeout
Prefetch far too high	One consumer hoards hundreds of deliveries it cannot finish in time	`rabbitmqctl list_consumers` prefetch column

The single most common pattern: unacked count sitting at exactly the prefetch size with an ack rate of zero. That is a consumer holding its entire prefetch buffer and processing none of it. See RabbitMQ unacknowledged messages growing for that failure pattern in depth.

Quick checks

All read-only and safe to run during an incident.

# Find timeout events and the timeout value actually used
grep -i "timed out waiting for\|delivery acknowledgement" /var/log/rabbitmq/rabbit@$(hostname).log | tail -20

# Which consumers exist, with their prefetch and ack mode
rabbitmqctl list_consumers queue_name consumer_tag prefetch_count ack_required active

# Per-queue: who has unacked messages piling up
rabbitmqctl list_queues name messages_ready messages_unacknowledged consumers

# Policies that may set consumer-timeout
rabbitmqctl list_policies

# Global configured value
rabbitmqctl environment | grep consumer_timeout

# Per-queue message stats: ack rate vs deliver rate vs redeliver rate
curl -s -u guest:guest http://localhost:15672/api/queues | \
  jq '.[] | {name, messages_ready, messages_unacknowledged, consumers, message_stats}'

What to look for:

The log line names the queue and the timeout used. Start there. The queue tells you which consumer application to investigate; the timeout tells you which of the four configuration levels is in effect.
messages_unacknowledged at or near prefetch x consumers, ack rate zero: consumers are stuck, not slow.
Elevated redeliver on the affected queue: messages cycling through delivery, timeout, requeue, redelivery. Each cycle is wasted work and can mask a poison message.
A queue policy with consumer-timeout: per-queue overrides are a frequent surprise after upgrades, because the team that set the policy is often not the team reading the log at 3 a.m.

How to diagnose it

flowchart TD
  A[Channel closed:
delivery ack timed out] --> B{Ack rate before
the timeout?}
  B -->|Zero, unacked = prefetch| C[Stuck consumer:
deadlock, zombie process,
downstream down]
  B -->|Nonzero but too slow| D[Slow consumer:
processing time exceeds timeout]
  B -->|Timeouts repeat on one queue| E{Redeliver rate high?}
  E -->|Yes| F[Poison message loop:
same payload fails repeatedly]
  E -->|No| G[Workload exceeds timeout:
batch jobs, large payloads]
  C --> H[Fix consumer or dependency,
then reduce prefetch]
  D --> H
  F --> I[Nack with requeue=false + DLX,
x-delivery-limit on quorum queues]
  G --> J[Raise timeout deliberately
at queue or consumer level]

Identify the queue and effective timeout from the broker log line. This pins down which application is affected and which configuration level fired.
Check the ack rate for that queue over the window before the timeout. Zero acks with deliveries happening means stuck. Low but nonzero acks means slow.
Check the unacked pattern. Unacked frozen at exactly prefetch size is the signature of a consumer holding its whole buffer and doing nothing, with the timeout as its visible endpoint.
Look at redeliver rate on the queue. Elevated redeliver plus repeated timeouts on the same queue points at a poison message: one payload that crashes or hangs the handler every time it is delivered.
Correlate with downstream dependencies. A sudden onset across many consumers at once almost always means the database, API, or service the consumers call degraded. Consumer-side health is invisible to the broker; check it on the application side.
Measure actual processing time. Get the p50/p95/p99 per-message latency from the consumer application. If p99 is anywhere near the timeout, occasional timeouts are guaranteed under load, GC pauses, or dependency jitter.
Check prefetch. prefetch x processing_time is the time budget a consumer needs to drain its buffer. Prefetch of 500 with 10-second processing is 83 minutes of buffered work against a 30-minute timeout: a timeout is mathematically certain after any hiccup.

Metrics and signals to monitor

Signal	Why it matters	Warning sign
`messages_unacknowledged` per queue	Unacked messages are pinned in broker memory and cannot be paged out	Frozen at prefetch x consumers, or growing steadily
Ack rate per queue	Ground truth for “consumers are finishing work”	Zero while deliver rate is nonzero
Redeliver rate per queue	Detects timeout/requeue cycles and poison messages	Sustained elevation on one queue
Consumer utilisation per queue	Whether attached consumers can actually receive work	Below 0.5 with messages_ready > 0
Head message age	Latency of the oldest waiting message	Growing while depth stays flat (poison message)
Channel churn rate	Timeout events close channels; churn is the fingerprint	Spikes correlated with timeout log lines
Consumer-side processing latency	The real input to timeout sizing	p99 approaching the configured timeout

Protocol-level channel errors like PRECONDITION_FAILED are visible only in the logs, not in the Management API or Prometheus metrics. Log collection for the timeout message is what ties the metric symptoms to the mechanism.

Fixes

Fix the stuck consumer

If acks are zero, raising the timeout only delays the next closure. Find the stall: thread-dump the consumer, check its downstream dependencies, look for lock contention or an exhausted database connection pool. A consumer whose process died but whose TCP socket is still open (a zombie connection) will also hold deliveries until the timeout fires; closing the zombie connection via the management API requeues its messages immediately instead of waiting.

Fix the poison message

A payload that deterministically hangs or crashes the handler will time out on every delivery attempt. The fix is in the consumer: catch processing failures, nack with requeue=false, and route to a dead-letter exchange. On quorum queues, set x-delivery-limit so a repeatedly failing message dead-letters after N attempts instead of cycling forever.

Reduce prefetch

If consumers are honest but slow, prefetch is usually the amplifier. Each delivery in the prefetch buffer starts its own timer on receipt, and the consumer can only work one (or a few) at a time. Size prefetch so the whole buffer drains well within the timeout: prefetch x worst_case_processing_time should be a small fraction of the timeout, not a multiple of it.

Raise the timeout deliberately

Legitimate for long-running work: report generation, media processing, bulk imports. Set it as close to the workload as possible, not globally:

# Per-queue policy (quorum queues in 4.x; queue-scoped override in 3.12+)
rabbitmqctl set_policy queue_consumer_timeout "^batch_imports\." \
  '{"consumer-timeout":7200000}' --apply-to quorum_queues

Tradeoffs to be explicit about:

A higher timeout means a genuinely stuck consumer holds pinned memory longer. The timeout exists because unacked messages are the number-one precursor to memory alarms.
The mechanism detects stuck consumers; it does not enforce processing SLAs. Do not tune it down to chase latency.
Since RabbitMQ 4.3 moved timeout handling into quorum queues, setting consumer_timeout to undefined in advanced.config no longer disables the timeout and breaks consumer registration (the quorum-queue path requires an integer timeout). If you must effectively disable it, set a very high value instead.
In 4.3+, this only protects quorum queues. Classic queues no longer evaluate the timeout at all.

What not to do: restart the broker. The stuck deliveries are requeued on channel closure anyway; a restart loses in-memory state, forces every consumer to reconnect, and changes nothing about why processing is slow.

Prevention

Measure before you configure. Know the p99 processing time per queue before touching consumer_timeout. Size the timeout as a multiple of p99, not a round number.
Keep prefetch honest. The prefetch buffer must be drainable in a fraction of the timeout. This single constraint prevents most timeout incidents on healthy consumers.
Dead-letter by default. Every queue whose consumers can fail on a payload should have a DLX, and quorum queues should carry x-delivery-limit.
Alert on the leading indicators, not the log line. Unacked count at prefetch with zero ack rate, and per-queue redeliver rate, both fire minutes to hours before the first channel closure.
Watch consumer-side latency. The broker cannot see why a consumer is slow. Instrument processing time in the consumer application and alert when p99 trends toward the timeout.
Plan the 4.3 migration. If you rely on consumer timeouts on classic queues, that protection disappears on upgrade to 4.3. Migrate queues that need it to quorum queues first.

How Netdata helps

Per-queue messages_ready and messages_unacknowledged charts make the “unacked frozen at prefetch” signature visible at a glance, before the timeout ever fires.
Message rate charts (publish, deliver, ack, redeliver) let you confirm the diagnosis in one view: deliver nonzero, ack zero, redeliver climbing is the timeout loop in progress.
Channel churn correlation ties channel closures back to consumer timeout events in the broker log, distinguishing them from client bugs or protocol errors.
Consumer utilisation per queue separates “no consumers attached” from “consumers attached but ineffective”, which changes the fix.
Alerting on sustained unacked growth and elevated redeliver rate gives you the early warning the timeout mechanism itself cannot provide, since the log line only appears after the damage is done.
Correlating broker memory against unacked totals shows the cost of a raised timeout: how much pinned memory a stuck consumer accumulates while it waits to be detected.