Ceph RGW GET/PUT latency: S3 request latency and queue length

When users report slow S3 GET or PUT responses, the RGW daemon is rarely the root cause. RADOS Gateway is a stateless HTTP frontend that translates REST calls into RADOS object operations. Its observed latency is dominated by the time those underlying operations take, plus whatever queuing happens when the gateway has more in-flight work than it can drain.

The RGW perf counters expose two distinct kinds of signal: per-operation latency accumulators (ceph_rgw_op_get_obj_lat_sum/_count and ceph_rgw_op_put_obj_lat_sum/_count) for the S3 operations themselves, plus queue gauges (ceph_rgw_qlen and ceph_rgw_qactive) that show whether requests are piling up inside the daemon. Treating those signals together is the difference between “S3 is slow” and “S3 is slow because one OSD hosting a bucket index shard is in OMAP collapse.”

What it means

The exact Prometheus-facing metric names depend on which Ceph version and exporter you run. Recent ceph-exporter builds expose the latency counters as ceph_rgw_op_global_get_obj_lat_sum/_count and ceph_rgw_op_global_put_obj_lat_sum/_count, while older builds and the internal perf counters drop the _global_ infix. The semantics are identical: use whichever name your collector exposes.

Average latency for an interval is the rate of the _sum counter divided by the rate of the _count counter:

# Average GET latency per object, by RGW instance (seconds)
rate(ceph_rgw_op_get_obj_lat_sum[5m])
  /
rate(ceph_rgw_op_get_obj_lat_count[5m])

The unit is seconds. The _sum accumulator covers the full request lifetime inside RGW: frontend parsing, bucket index lookup, RADOS read or write, and response streaming. PUT averages are dominated by write latency; GET averages by read latency and object size.

The two queue gauges tell you whether the gateway is the bottleneck or just the messenger.

A sustained mismatch between the two is the diagnostic. If qactive is high but qlen is low, the daemon is busy and keeping up. If qlen climbs while qactive plateaus, threads are blocked waiting on RADOS and the queue is draining slower than it is filling. That is the pattern that says “look below RGW.”

An example alerting condition for this pattern:

# Queue length disproportionate to request rate
ceph_rgw_qlen / (rate(ceph_rgw_req[5m]) + 1) > 10

Sustained for more than 300 seconds, this means the queue is growing out of proportion to incoming traffic. The +1 guards against divide-by-zero on idle gateways.

Common causes

Quick checks

These are safe, read-only commands. Run them in roughly this order. Note that ceph daemon commands require access to the admin socket (root or the ceph user).

# Top-level cluster state, including recovery and scrub lines
ceph -s

# Per-OSD commit and apply latency. Sort to find outliers.
ceph osd perf

# Per-pool read/write IOPS and bytes
ceph osd pool stats

# Per-OSD capacity, hierarchical view
ceph osd df tree

# Slow ops in flight on a specific OSD; "currently waiting for" tells you where they are stuck
ceph daemon osd.<id> dump_ops_in_flight

# Health detail, including LARGE_OMAP_OBJECTS, OSD_NEARFULL, SLOW_OPS
ceph health detail

# Per-bucket object count and shard count
radosgw-admin bucket stats --bucket=<bucket-name>

# RGW-side perf counters, including get_obj_lat / put_obj_lat internals
ceph daemon rgw.<id> perf dump | jq '.rgw'

How to diagnose it

RGW latency is downstream of RADOS latency. Diagnose top-down.

flowchart TD
    A[RGW GET/PUT latency rising] --> B{OSD apply/commit latency elevated?}
    B -->|No| C[Investigate RGW host: CPU, NIC, fd limit]
    B -->|Yes, cluster-wide| D{Recovery or deep scrub active?}
    B -->|Yes, isolated OSDs| E{LARGE_OMAP_OBJECTS active?}
    D -->|Yes| F[Throttle recovery and scrub]
    D -->|No| G[Check cluster-network saturation]
    E -->|Yes| H[Reshard bucket index]
    E -->|No| I[Check SMART and BlueStore DB spillover]

1. Confirm RGW is not the source. Check whether ceph_rgw_qlen and ceph_rgw_qactive are elevated across all RGW instances or just one. A single-instance spike points to the RGW host: CPU saturation, NIC saturation, file descriptor limits, or TLS termination overhead. A cluster-wide spike points below.

2. Cross-reference OSD latency. Plot ceph_osd_apply_latency_ms and ceph_osd_commit_latency_ms against the RGW latency series. A clean correlation confirms the bottleneck is in RADOS. If a small number of OSDs are outliers, those OSDs are blocking the PGs that RGW is reading from or writing to.

3. Check whether slow ops are present. ceph_healthcheck_slow_ops > 0 means operations have exceeded osd_op_complaint_time (default 30 seconds). Those operations are stuck, not just slow. Use ceph daemon osd.<id> dump_ops_in_flight and read the “currently waiting for” field of each slow op: waiting for subops from indicates a replication stall, waiting on pg indicates peering, waiting for rw locks indicates lock contention, and “reached pg but stalled” indicates a local device issue.

4. Rule out OMAP storms. If latency spikes coincide with LIST or PUT operations against one or two large buckets, check ceph health detail for LARGE_OMAP_OBJECTS. OSD logs in the worst cases show lines mentioning omap_iterator with latencies in tens of seconds. Identify affected buckets with radosgw-admin bucket stats and inspect shard count versus object count.

5. Rule out capacity pressure. Capacity and latency couple at nearfull (default 0.85). At backfillfull (default 0.90) recovery is blocked. At full (default 0.95) all writes fail. Even before those thresholds, individual OSDs that are disproportionately full become write hotspots.

6. Rule out BlueStore DB spillover. If a single OSD shows high commit latency but its data device is healthy, check whether RocksDB has spilled from the fast DB partition to the slow data partition: ceph daemon osd.<id> perf dump | jq '.bluefs | {slow_used_bytes, slow_total_bytes}'. Any nonzero slow_used_bytes means the OSD has hit the performance cliff.

7. Rule out recovery and scrub contention. Recovery and deep scrub compete with client I/O on the same disks and the same cluster network. ceph -s shows active recovery and scrub; correlation with the latency spike is usually obvious.

Metrics and signals to monitor

Fixes

Slow OSDs

Identify the outlier OSD with ceph osd perf. If a disk is failing, replace it; SMART data via ceph device get-health-metrics <devid> usually confirms. If the OSD is healthy but hot, reweight it lower with ceph osd reweight <osd-id> 0.9 (the OSD ID is a plain integer) to shed placement-group load. For BlueStore DB spillover, plan a ceph-bluestore-tool migration to a larger DB device, which requires OSD downtime.

OMAP storm on bucket index

Confirm with radosgw-admin bucket stats --bucket=<name>. Rule of thumb: each index shard should hold no more than roughly 100,000 entries, controlled by rgw_max_objs_per_shard (default 100,000). Check the reshard queue with radosgw-admin reshard list. Trigger a manual reshard if dynamic resharding has stalled. Resharding has been an online operation since Luminous: reads continue during the index rebuild and writes to the bucket are blocked only briefly. Scheduling resharding during low-traffic windows is still prudent for very large buckets.

Be aware that dynamic resharding has historically been able to reshard the same buckets repeatedly without dropping old index objects. If OMAP growth looks runaway, compare radosgw-admin bucket stats over time and check for stale index shards.

Capacity pressure

The fix is capacity. Delete data, add OSDs, or force RGW garbage collection with radosgw-admin gc process. Reweight overly full OSDs downward. Raising mon_osd_full_ratio is a last resort and only buys time.

RGW-side saturation

If only one gateway is slow, add capacity: more RGW instances behind the load balancer, more CPU on the host, or larger file descriptor limits. Confirm TLS termination is offloaded if RGW is handling it directly. For Beast frontend issues, ensure you are on a current patched Ceph release.

Recovery and scrub contention

Throttle recovery on the fly: ceph tell osd.* injectargs '--osd_max_backfills 1 --osd_recovery_max_active 1'. Set noscrub and nodeep-scrub temporarily during the incident, then unset them as soon as client latency recovers. Leaving those flags set indefinitely is one of the most common preventable Ceph failures.

Prevention

• Monitor per-OSD latency, not cluster averages. One OSD at 200ms while the cluster median is 5ms is the typical cause of intermittent RGW latency outliers.
• Track bucket growth against shard count. Alert when any bucket exceeds roughly 100,000 objects per shard. Reshard on your schedule, not in an incident.
• Watch for OMAP accumulation. LARGE_OMAP_OBJECTS is a late signal. Per-bucket object-count trends give earlier warning.
• Size BlueStore DB partitions for OMAP growth. RGW bucket indexes live in RocksDB. Default sizing guidance assumes moderate OMAP usage; RGW-heavy clusters need more DB space.
• Revisit recovery throttle settings as the cluster grows. Defaults that were safe with 20 OSDs can cause cascading failures with 200.
• Treat noscrub and nodeep-scrub as state, not actions. Alert if either is set for more than 24 hours without a matching maintenance window.
• Monitor RGW garbage collection rate. A near-zero ceph_rgw_gc_retire_object rate on a nearfull cluster means deleted data is not actually being reclaimed.

How Netdata helps

• The Netdata Ceph collector pulls ceph_rgw_op_*_obj_lat_sum/count, ceph_rgw_qlen, ceph_rgw_qactive, and ceph_rgw_req at per-second resolution, so you can see queue buildup in real time rather than as a five-minute average.
• Per-second OSD apply and commit latency lets you visually correlate an RGW latency spike with the specific OSD that caused it, without running ceph osd perf repeatedly by hand.
• ceph_healthcheck_slow_ops and ceph_health_detail labels including LARGE_OMAP_OBJECTS, OSD_NEARFULL, and SLOW_OPS render alongside the RGW latency chart, so the cause sits on the same dashboard as the symptom.
• The anomaly advisor can catch qactive flat-lining while qlen climbs before average latency crosses a static threshold. That pattern is the precursor to user-visible latency.
• BlueStore bluefs slow-device usage and RocksDB counters are collected per OSD, which is what you need to distinguish “slow disk” from “DB spillover” without dropping into the admin socket.

Related guides

• Ceph backfill_toofull: recovery blocked because target OSDs are full
• Ceph blocked ops: client I/O stuck behind a single slow OSD
• Ceph BlueStore RocksDB compaction stalls: periodic latency spikes
• Ceph BLUEFS_SPILLOVER: RocksDB metadata spilling onto the slow device
• Ceph BlueStore allocator fragmentation: rising latency at moderate fullness
• Ceph capacity death spiral: an OSD fails and recovery has nowhere to go
• Ceph client latency vs OSD latency: fast disks, slow clients
• Ceph deep scrub performance impact: I/O saturation that mimics an incident
• Ceph degraded objects: reduced redundancy and the race against a second failure
• Ceph FS_DEGRADED: standby MDS failed to take over a rank
• Ceph health detail: mapping ceph_health_detail checks to a cause
• Ceph HEALTH_ERR: reading the umbrella status and finding the real fault