RabbitMQ RESOURCE_LOCKED - cannot obtain exclusive access to queue

Your application logs show a channel-level exception, the channel closes, and the consumer never comes back:

reply-code=405, reply-text=RESOURCE_LOCKED - cannot obtain exclusive access to locked queue 'reply.xyz' in vhost '/'

This is AMQP error 405, a channel exception raised when a connection tries to declare, consume from, purge, or delete an exclusive queue that is still owned by a different connection. Exclusive queues are bound to the lifecycle of the connection that declared them: only that connection may use them, and they are deleted when that connection closes or is lost. The error is not a capacity problem, not an alarm, and not a permissions failure. It is an ownership conflict.

Two production scenarios generate almost every occurrence of this error: a reconnection race, where the client reconnects and re-declares the queue before the broker has reaped the old connection, and a deployment or scaling mistake, where two application instances use the same client-assigned name for what each believes is its own private queue.

What this means

An exclusive queue has exactly one owner: the connection that declared it. From the broker’s perspective:

• Any other connection that touches the queue gets 405 RESOURCE_LOCKED on the channel it used. The channel closes; the connection stays open.
• The queue is deleted automatically when the owning connection closes or its TCP session is lost.
• Exclusive queues are always classic queues. They are not replicated, do not survive a node restart, and cannot be declared on quorum queues or streams.

The key subtlety is timing. “Connection closed” from the broker’s point of view is not the same instant as “client called close()” or “client process died.” A client that vanishes without a clean AMQP close (network blip, process kill, LB idle timeout) is only detected as gone after the TCP layer reports the loss or the AMQP heartbeat times out. During that window, the exclusive queue still exists and is still locked. A fast-reconnecting client hits the lock with its own zombie.

Do not confuse exclusive queues with two related but distinct mechanisms:

• Exclusive consumer (x-exclusive on basic.consume): the queue is shared, but only one consumer may subscribe. A second consumer gets ACCESS_REFUSED, not RESOURCE_LOCKED.
• Single active consumer (x-single-active-consumer queue argument): many consumers may register, only one is active at a time, and the broker promotes the next consumer when the active one disconnects. This is the supported pattern for ordered, exclusive-style consumption on quorum queues.

Common causes

Quick checks

All commands are read-only. They use guest:guest, which the broker only accepts from localhost by default; substitute your admin credentials when querying remotely.

# Find the queue and its current state
curl -s -u guest:guest http://localhost:15672/api/queues | \
  jq '.[] | select(.name=="<queue_name>") | {name, node, state, auto_delete, arguments, consumers}'

# Find the connection that owns the queue (exclusive owner is listed on the queue object)
curl -s -u guest:guest 'http://localhost:15672/api/queues/%2F/<queue_name>' | \
  jq '{name, owner_pid, exclusive_consumer_tag, consumer_details}'

# List connections from the affected client host, with age and state
curl -s -u guest:guest http://localhost:15672/api/connections | \
  jq '.[] | select(.peer_host=="<client_ip>") | {name, state, connected_at, channels}'

# Check for recent channel exceptions and connection closes in the broker log
grep -i "RESOURCE_LOCKED\|connection_closed\|closing AMQP connection" \
  /var/log/rabbitmq/rabbit@$(hostname).log | tail -50

# Rule out a cluster partition if the lock will not clear
rabbitmq-diagnostics cluster_status

In the queue object, owner_pid identifies the connection that holds the exclusive lock. If the error is a reconnect race, you will often see the old connection still present at the moment the error fired, and gone a few seconds later.

How to diagnose it

Work through these in order. Most incidents resolve at step 2 or 3.

1. Confirm which queue and which operation failed. The client exception names the queue and the frame (queue.declare, basic.consume, queue.delete, queue.purge). The broker log records the same channel error with the connection and channel that raised it. Note the exact timestamp.

2. Identify the lock holder. Query the queue via the management API and read owner_pid. Map it back through /api/connections to a peer_host and user_provided_name if clients set one. If the owner is the same application instance that just failed, you are looking at a reconnect race. If the owner is a different instance, you have a name collision.

3. Check the reconnect race window. Compare the timestamp of the client’s reconnect attempt with the broker log’s close event for the old connection (connection_closed or a heartbeat timeout entry). If the close is logged after the reconnect attempt, the diagnosis is confirmed: the client outran the broker’s cleanup. Heartbeat-negotiated detection means this window is on the order of the heartbeat timeout, not milliseconds.

4. Check for a name collision across instances. If the queue has a client-assigned name, grep your codebase and deployment manifests for that string. A name like worker-reply-queue shared by every replica guarantees a 405 for every replica after the first. Server-generated names (declare with an empty name and read back the assigned one) are the intended pattern for exclusive queues.

5. If the lock never clears, suspect a partition artifact. After a network partition heals, an exclusive queue can survive on a node while its owning connection no longer exists anywhere. The queue then rejects every operation with 405, including deletion from the management UI. Confirm with rabbitmq-diagnostics cluster_status and by checking that the owner_pid connection is absent from every node’s connection list. The known workaround is restarting the node that hosts the stale queue, which is disruptive; schedule it.

6. If the caller uses basic.get, check queue arguments. basic.get is not supported on queues declared with x-single-active-consumer; the broker rejects the call. Switch the client to basic.consume.

flowchart TD
  A[405 RESOURCE_LOCKED on channel] --> B{Owner = same client instance?}
  B -- yes --> C[Reconnect race: old connection not yet reaped]
  C --> C1[Fix: retry with backoff, let heartbeat reap old conn]
  B -- no --> D{Fixed client-assigned queue name?}
  D -- yes --> E[Name collision across instances]
  E --> E1[Fix: server-generated names or unique per instance]
  D -- no --> F{Lock persists after partition?}
  F -- yes --> G[Stale queue on healed partition]
  G --> G1[Fix: restart node hosting the queue - disruptive]
  F -- no --> H{Client uses basic.get on SAC queue?}
  H -- yes --> I[Unsupported combination - use basic.consume]

Metrics and signals to monitor

RESOURCE_LOCKED itself is a log-only signal; the broker does not expose channel exceptions as metrics. What you can monitor is the context that produces it.

Correlate connection churn with the 405 log lines: if every error is bracketed by a connection close and a new connection, the race is your mechanism, and the churn alert is your early warning.

Fixes

Reconnect race

• Retry the declaration with backoff. The lock is transient; the old connection is reaped once TCP loss or heartbeat timeout is detected. A client that retries queue.declare after a short delay will succeed. Most client libraries’ automatic recovery already re-declares topology on reconnect, but they typically retry immediately, which is exactly when the lock is still held. Add bounded retry with jitter around the topology recovery step.
• Do not lower heartbeat timeouts to shrink the window without thinking it through. Shorter heartbeats reap dead connections faster but also increase false-positive disconnects on congested networks, which creates more reconnects, which creates more races. Fix the retry behavior first.
• Do not restart the broker for this. The lock clears on its own within seconds to a heartbeat interval.

Shared queue name across instances

• Use server-generated names. Declare the queue with an empty name and use the name the broker returns. Collisions become impossible.
• If the name must be client-assigned (some RPC frameworks require it), make it unique per instance: embed the instance ID, pod name, or a generated UUID.
• Audit frameworks that create hidden exclusive queues. JMS temporary destinations and several RPC reply-queue implementations declare exclusive queues with predictable names; a reconnect inside those frameworks hits the same race.

Stale queue after partition

• Confirm the partition has healed and the owner connection is truly gone from all nodes.
• The documented workaround is restarting the node hosting the stale queue. This is disruptive: it drops all connections to that node and, for quorum queues with leaders there, triggers leader elections. Schedule it and drain client traffic first.

basic.get on a single active consumer queue

• Replace polling with basic.consume. If you genuinely need pull semantics, remove x-single-active-consumer, but note that single active consumer cannot be toggled by policy; it is set at declaration time, so the queue must be deleted and re-declared, which drops any messages it holds. Drain or shovel the messages first.

Prevention

• Server-generated names for every exclusive queue. This single habit eliminates the collision class of 405 entirely and reduces the reconnect race to a self-healing retry.
• Bounded, jittered retry around topology recovery. Treat 405 on declaration as a transient error worth retrying, not a fatal one.
• Prefer single active consumer over exclusive queues for ordered single-consumer workloads that need to survive node restarts or run on quorum queues. SAC gives you failover: when the active consumer disconnects, another registered consumer is promoted, without a lock conflict.
• Alert on connection churn, not just connection count. A reconnect loop is the upstream event; RESOURCE_LOCKED is its symptom. Catching the churn early often catches the deploy or network fault causing it.
• Exercise partition handling deliberately. Know your cluster_partition_handling mode and rehearse what happens to exclusive queues during a partition before one happens to you.

How Netdata helps

Netdata does not see the 405 itself (it is a log event), but it surfaces the conditions around it:

• Connection churn rate from the management API exposes reconnect loops, the leading trigger for the reconnect race.
• Channel and queue churn make repeated failed re-declaration cycles visible as abnormal declare/create rates.
• Per-client connection counts and states help you spot a single peer_host accumulating connections or flapping.
• Queue and consumer counts confirm whether the exclusive queue’s consumer actually came back after recovery.
• Correlating churn spikes with deploy events and node restarts on one dashboard shortens the “race vs. collision” decision from minutes of log spelunking to one glance.

Related guides

• RabbitMQ connection storm: reconnect loops, FD pressure, and CPU spent on handshakes
• RabbitMQ consumer timeout: delivery acknowledgement timed out and the channel is closed
• RabbitMQ consumers connected but not acknowledging: the consumer black hole
• RabbitMQ consumer_utilisation low: consumers attached but not keeping up
• How RabbitMQ actually works in production: a mental model for operators