Ceph LARGE_OMAP_OBJECTS: the RGW bucket-index OMAP storm and resharding

The LARGE_OMAP_OBJECTS health warning fires when a deep scrub finds a RADOS object carrying more OMAP keys than the configured threshold. On clusters running the RADOS Gateway (RGW), the most common trigger is the bucket-index shard: a single shard that has accumulated too many entries because a bucket holds millions of objects with too few index shards.

Once a shard crosses the threshold, the symptoms cluster around two areas. RGW clients see slow LIST responses and slow multipart coordination on the affected bucket. The OSD hosting the shard shows elevated commit latency, slow ops, and in the worst case BlueStore RocksDB pressure that resembles a compaction stall. The OSD usually remains up+in, so cluster-level health stays at WARN rather than ERR, which makes the problem easy to overlook until a user complains about a stuck ListObjects call.

The structural fix is resharding: increasing the number of index objects so per-shard OMAP stays within the threshold. Ceph has dynamic resharding (rgw_dynamic_resharding, on by default since Luminous) but it can stall, fail silently, or be unavailable in multisite configurations prior to Reef.

What this means

The warning is emitted by the OSD during deep scrub when it finds an OMAP object exceeding either osd_deep_scrub_large_omap_object_key_threshold (default 200,000 keys) or osd_deep_scrub_large_omap_object_value_sum_threshold (default 1 GB). The check is structural: it fires whenever the threshold is crossed during a scrub pass, regardless of whether the OMAP is being actively read or written.

The OMAP is RocksDB-backed in BlueStore. Bucket-index entries are OMAP keys on RADOS objects in the .rgw.buckets.index pool (or whatever the deployment names the index pool). Each object key in the bucket adds an OMAP key on the shard that owns it. With 11 default shards and a bucket holding tens of millions of objects, individual shards routinely cross the threshold.

Every operation that scans or modifies the oversized OMAP pays an increasing cost: bucket LIST, multipart upload coordination, GC metadata updates, and multisite sync. The OSD hosting the shard also feels the load because BlueStore must maintain the RocksDB structure backing the OMAP, which can push RocksDB past the DB device and into BlueStore DB spillover.

flowchart TD
    A[Bucket grows to millions of objects] --> B[Too few index shards]
    B --> C[Each shard accumulates OMAP keys]
    C --> D[Deep scrub finds OMAP greater than 200k keys]
    D --> E[LARGE_OMAP_OBJECTS health warning]
    C --> F[RGW LIST latency rises]
    C --> G[OSD commit latency rises on hosting shard]
    G --> H[RocksDB pressure and possible DB spillover]

Common causes

Quick checks

# Confirm health warning and read the message
ceph health detail | grep -A3 LARGE_OMAP_OBJECTS

# Per-OSD commit latency to find the OSD hosting the hot shard
ceph osd perf

# Show RGW reshard queue
radosgw-admin reshard list

# Bucket stats including shard count and per-shard object counts
radosgw-admin bucket stats --bucket=<bucket-name>

# Bucket limit check across all buckets
radosgw-admin bucket limit check

# Stale bucket instance entries (primarily pre-Mimic 13.2.5 clusters)
radosgw-admin reshard stale-instances list

# BlueFS slow device usage on suspected OSDs (DB spillover check)
ceph daemon osd.<id> bluefs stats

# Reshard status for a specific bucket
radosgw-admin reshard status --bucket=<bucket-name>

How to diagnose it

1. Confirm the warning is OMAP-driven and identify the affected RADOS object. ceph health detail lists the PG. The offending object is in the bucket-index pool.

2. Identify the bucket whose index lives on that PG. List index objects with rados -p <index-pool> ls, then map shard object names back to bucket names using radosgw-admin bucket stats --bucket=<name> until you find the matching shard ID.

3. Establish whether the bucket is over the reshard threshold. Rule of thumb: each shard should hold roughly 100,000 objects (recommended maximum around 102,400). A bucket with 10 million objects needs at least 50 to 100 shards.

4. Check resharding status. radosgw-admin reshard list shows pending entries. radosgw-admin reshard status --bucket=<name> shows the state of a specific bucket’s reshard process. A non-empty list that never drains indicates a stalled process.

5. Correlate with OSD performance. ceph osd perf shows commit and apply latency per OSD. The OSD hosting the hot shard typically shows a latency outlier versus peers on the same device class. A single outlier in a uniform tier is a strong signal.

6. Check BlueStore for spillover. If OMAP growth has pushed RocksDB past the DB device, you will see nonzero slow_used_bytes in ceph daemon osd.<id> bluefs stats. This is a separate but related failure mode covered in the Ceph BLUEFS_SPILLOVER guide.

Metrics and signals to monitor

Fixes

Allow dynamic resharding to run

If rgw_dynamic_resharding is false, set it to true and restart RGW daemons. By default the background scanner triggers a reshard when a bucket exceeds rgw_max_objs_per_shard (default 100,000 objects per shard). The scanner detects the condition and schedules a reshard entry.

# Process any pending reshard entries now
radosgw-admin reshard process

# Confirm state for a specific bucket
radosgw-admin reshard status --bucket=<bucket-name>

Tradeoff: dynamic resharding briefly locks the bucket during index rebuild. Schedule during low-traffic periods for high-traffic buckets. On multisite deployments, dynamic resharding is supported only on Reef and later. On earlier versions in multisite, manual resharding is the only path.

Manually trigger resharding

For buckets where dynamic resharding is unavailable (multisite pre-Reef) or stalled, run manual resharding.

# Destructive: rewrites the bucket index. Validate object counts first,
# snapshot the index pool if feasible, and test on a non-production
# bucket if you have never run this command. In-flight writes to the
# bucket during reshard can produce index inconsistency.
radosgw-admin bucket reshard --bucket=<bucket-name> --num-shards=<N>

Choose <N> based on expected steady-state object count. Plan for roughly 100,000 objects per shard. Note that rgw_max_dynamic_shards (default 1999) caps dynamic resharding but does not limit manual resharding. For very large buckets, manual resharding can take significant wall time and should be scheduled accordingly.

On multisite, manual resharding must be coordinated across zones and run on the master zone. The new bucket instance needs to propagate before old instances can be cleaned up.

Clean up stale bucket instances

On clusters prior to Luminous 12.2.11 or Mimic 13.2.5, resharding left stale bucket instance entries in the index pool. These accumulate OMAP keys independently and can re-trigger LARGE_OMAP_OBJECTS even after the live bucket has been resharded.

# List stale instances
radosgw-admin reshard stale-instances list

# Delete them
radosgw-admin reshard stale-instances delete

For multisite deployments where deleted buckets leave bilog entries behind, you may also need to trim the bilog before the warning clears. Run radosgw-admin bilog trim --bucket=<bucket-name> --bucket-id=<bucket-id> followed by a deep scrub on the affected PG.

Address BlueStore DB spillover separately

If slow_used_bytes is nonzero on the hosting OSD, resharding reduces future OMAP growth but does not reclaim already-spilled RocksDB data. Plan a DB partition migration using ceph-bluestore-tool, which requires OSD downtime. Resharding first prevents the spill from recurring immediately after the migration.

Restore lifecycle policies on older clusters

On clusters prior to Mimic 13.2.6 or Luminous 12.2.12, lifecycle policies could stop applying to a resharded bucket. If you reshard on an older cluster and lifecycle rules are in use, run the lifecycle reshard fix afterward.

radosgw-admin lc reshard fix --bucket=<bucket-name>

Prevention

• Pre-shard buckets at creation when object counts are known. Set rgw_override_bucket_index_max_shards in simple deployments or bucket_index_max_shards in zonegroup config. The default for new buckets is 11 shards, which is too few for any bucket expected to grow past roughly one million objects. Pre-sharded buckets show more deterministic performance than relying on dynamic resharding after the fact.
• Keep rgw_dynamic_resharding enabled unless you are on a multisite version that does not support it. Verify periodically that the reshard queue is draining.
• Monitor per-bucket object count versus shard count. Alert when a bucket approaches the per-shard threshold. The rule of thumb is roughly 100,000 objects per shard.
• Tune rgw_max_objs_per_shard if your workload regularly produces buckets that grow past the default. Lowering the threshold makes dynamic resharding trigger sooner, at the cost of more frequent reshard events.
• On multisite, upgrade to Reef or later to enable dynamic resharding support. Without that, every large bucket requires manual reshard coordination.
• Schedule periodic stale-instance sweeps on older clusters. radosgw-admin reshard stale-instances list should be part of routine maintenance on pre-Mimic 13.2.5 deployments.
• Size BlueStore DB partitions for OMAP workload. RGW-heavy clusters need larger DB partitions than the typical sizing guides suggest. Approximately 4 to 5 percent of the data partition is a starting point. More is required for heavy RGW deployments.
• Add AbortIncompleteMultipartUpload to lifecycle policies. Incomplete multipart uploads contribute to OMAP growth on the index shard, not just data pool consumption.

How Netdata helps

• The Ceph collector surfaces ceph_health_detail{name="LARGE_OMAP_OBJECTS"} as a labeled gauge, so you can alert on the specific check rather than the umbrella HEALTH_WARN that lumps it in with every other warning.
• Per-second ceph_osd_commit_latency_ms and ceph_osd_apply_latency_ms per OSD let you spot the hosting OSD as a latency outlier before users complain about slow LISTs.
• ceph_rgw_req, ceph_rgw_failed_req, ceph_rgw_qlen, and ceph_rgw_qactive give the RGW-side view to correlate client impact with backend OMAP pressure.
• ceph_healthcheck_slow_ops provides the bridge signal: when it spikes alongside the OMAP warning, you can confirm that the oversized shard is actively blocking I/O rather than just being structurally over threshold.
• ML-based anomaly detection on per-OSD commit latency catches the slow drift of RocksDB pressure building under a hot bucket-index shard, often before the next deep scrub runs and the health warning fires.
• Correlating LARGE_OMAP_OBJECTS onset with OSD latency spikes in the same window shortens diagnosis from “vague cluster slowdown” to “this specific bucket, this specific shard, this specific OSD”.

Related guides

• Ceph backfill_toofull: recovery blocked because target OSDs are full
• Ceph blocked ops: client I/O stuck behind a single slow OSD
• Ceph BlueStore RocksDB compaction stalls: periodic latency spikes
• Ceph BLUEFS_SPILLOVER: RocksDB metadata spilling onto the slow device
• Ceph BlueStore allocator fragmentation: rising latency at moderate fullness
• Ceph capacity death spiral: an OSD fails and recovery has nowhere to go
• Ceph client latency vs OSD latency: fast disks, slow clients
• Ceph deep scrub performance impact: I/O saturation that mimics an incident
• Ceph degraded objects: reduced redundancy and the race against a second failure
• Ceph FS_DEGRADED: standby MDS failed to take over a rank
• Ceph health detail: mapping ceph_health_detail checks to a cause
• Ceph HEALTH_ERR: reading the umbrella status and finding the real fault