Ceph RGW orphaned multipart uploads: space that disappears from view

Pool capacity climbs toward nearfull but bucket listings and radosgw-admin bucket stats cannot account for it. The obvious explanations (client growth, snapshots, OMAP bloat) do not fit. This pattern is frequently caused by orphaned multipart upload parts in the RADOS Gateway (RGW) data pool.

S3 multipart uploads create a manifest, upload parts into the bucket’s data pool, then call CompleteMultipartUpload to assemble the final object. When that final step never happens, when the client retries uploads of the same key, or when RGW aborts an upload incompletely, the individual parts remain in RADOS. They occupy raw space but no live bucket index entry references them.

The result is invisible capacity. ceph df detail shows the pool growing, radosgw-admin bucket stats attributes some bytes to buckets but most of the orphan mass is unaccounted for, and deleting visible data does not reclaim the space.

What this means

Each S3 multipart upload creates several classes of RADOS objects in the RGW data pool:

• A .meta object holding the upload manifest.
• One object per uploaded part, named with a multipart shadow prefix.
• After CompleteMultipartUpload, a final object keyed by the object name. The parts become tail segments referenced by that final object.

Space goes missing in three common ways:

1. The client abandons the upload (crash, timeout, application bug) without calling complete or abort, so the manifest and parts remain.
2. The client retries the upload of the same key, and on unfixed RGW versions the previous attempt’s parts are not cleaned up before the new parts are written.
3. RGW attempts to abort an upload but the abort partially succeeds, leaving either the manifest or some parts behind.

The orphans are invisible to normal bucket tooling because bucket indexes track live objects, not upload-in-progress state. radosgw-admin bucket list, radosgw-admin bucket stats, and per-bucket listings will not show them. They are visible only as raw RADOS objects in the data pool.

flowchart TD
    A[S3 client initiates multipart upload] --> B[.meta manifest created]
    B --> C[Parts written to data pool]
    C --> D{CompleteMultipartUpload called?}
    D -- yes, normal path --> E[Parts become tail of final object]
    D -- no: client crashed or timed out --> F[.meta and parts left behind]
    D -- retry on same key pre-fix --> G[old parts orphaned, new parts written]
    D -- abort partially fails --> H[.meta or parts left behind]
    F --> I[Orphaned RADOS objects]
    G --> I
    H --> I
    I --> J[Pool capacity consumed]
    J --> K[No bucket index entry references them]
    K --> L[Invisible to bucket stats and listings]

The leak is silent. No health check flags orphaned multipart parts. Capacity signals (see Ceph capacity death spiral) surface the symptom but do not attribute it.

Common causes

Quick checks

# Cluster-wide capacity position
ceph df detail

# Per-pool raw bytes versus stored bytes
ceph df detail | grep -A2 <rgw-data-pool>

# Per-bucket object count and size attribution
radosgw-admin bucket stats --bucket=<bucket-name>

# List all RADOS objects the RGW believes belong to a bucket
radosgw-admin bucket radoslist --bucket=<bucket-name>

# RGW GC queue depth (deleted-object backlog, not multipart orphans, but worth correlating)
radosgw-admin gc list --include-all

# RGW version (relevant for the re-upload leak fix)
ceph versions

All read-only and safe on production. The expensive step is radosgw-admin bucket radoslist on a large bucket; run it against one suspect bucket rather than cluster-wide.

How to diagnose it

1. Confirm the accounting gap. Compare ceph df detail raw bytes for the RGW data pool against the sum of radosgw-admin bucket stats across all buckets. A material, persistent gap that grows is the signature. Rule out snapshots and pool metadata with ceph df detail first.

2. Confirm there is no other major consumer. Check RBD and CephFS pool growth. Check for snapshot accumulation in the RGW pool if snapshots are in use.

3. Map bucket objects to RADOS. For the suspected bucket, run radosgw-admin bucket radoslist --bucket=<name> to get the set of RADOS object names RGW believes belong to that bucket.

4. List raw RADOS objects in the data pool. rados -p <pool> ls produces the set of objects actually present. Comparing this against the radoslist output identifies candidates that nothing references.

5. Run the supported orphan scan. The replacement for the deprecated radosgw-admin orphans find is the rgw-orphan-list tool. Point it at one data pool at a time:

   # rgw-orphan-list is experimental and writes intermediate files locally
   rgw-orphan-list <pool-name>
   

   The tool writes a local file of candidate orphan object names. Treat the output as a candidate list, not a deletion list, until you have verified entries. Unindexed buckets will cause every object in that bucket to appear as an orphan, so verify the bucket has an index before deleting anything.

6. Cross-check candidates. For each candidate, confirm it is a multipart shadow object by name pattern and confirm it is not a tail segment referenced by a live object via radosgw-admin bucket radoslist. Shared bucket markers from RGW’s shallow-copy optimization can make a tail object appear to belong to a different bucket, so do not delete based on marker alone.

7. Check bucket instance state for stuck multiparts. List the in-progress multipart uploads for the suspected bucket with the S3 API (aws s3api list-multipart-uploads --bucket <name>), then abort the stale ones with aws s3api abort-multipart-upload --bucket <name> --key <key> --upload-id <id>. radosgw-admin has no multipart list/abort subcommand.

Metrics and signals to monitor

Netdata surfaces per-pool capacity and RGW daemon counters via the Ceph collector. The diagnostic value comes from correlating raw pool growth against summed bucket stats, which is a manual comparison today.

Fixes

Force GC processing

Orphaned multipart parts are not always in the GC queue. For deleted objects that are still queued, processing GC first is cheap and safe:

# Process the GC queue now (does not affect multipart orphans directly)
radosgw-admin gc process

Abort in-progress uploads explicitly

If the S3 client lists multipart uploads still in progress for a bucket, abort them with the S3 API. This removes the manifest and any parts RGW still tracks. radosgw-admin has no multipart list or multipart abort subcommand.

# List in-progress multipart uploads for a bucket
aws s3api list-multipart-uploads --bucket <bucket-name>

# Abort a specific upload
aws s3api abort-multipart-upload --bucket <bucket-name> --key <object-key> --upload-id <upload-id>

Remove identified orphans manually

Once rgw-orphan-list has identified candidate orphans and you have cross-checked them, remove them directly from the data pool. This is destructive; there is no undo.

# Destructive: removes one RADOS object. Verify each name before deleting.
rados -p <rgw-data-pool> rm <object-name>

For bulk cleanup, scripts that iterate the orphan list file from rgw-orphan-list and call rados rm are common. Run them in batches and re-check ceph df detail between batches.

Bucket check fix (limited applicability)

radosgw-admin bucket check --fix --check-objects can reconcile some bucket index inconsistencies, but only for buckets with shard count zero. On dynamically sharded buckets this path does not work and may make indexing worse (this limitation is documented in Red Hat’s Ceph administrator guides and confirmed by operators on ceph-users). Do not rely on it as a primary cleanup mechanism.

Upgrade past the re-upload leak

If the cluster is running Quincy 17.2.7 or earlier and clients routinely overwrite large objects via multipart upload, the re-upload leak (tracker #16767, fixed in PR #51976) is a likely contributor. Upgrading to Quincy 17.2.8 or later stops new orphans from that code path. Existing orphans are not automatically cleaned by the upgrade; they still need manual identification and removal.

Prevention

• Apply an AbortIncompleteMultipartUpload lifecycle rule to every bucket. This is the primary preventive measure. A rule that aborts uploads after N days ensures abandoned uploads are cleaned up automatically. Configure via aws s3api put-bucket-lifecycle-configuration or your infrastructure-as-code tool.
• There is no cluster-wide default for this. No Ceph release exposes an rgw_abort_incomplete_multipart_upload_expiration config option; AbortIncompleteMultipartUpload lifecycle rules are the only built-in mechanism, so apply a rule to every bucket as described above.
• Make the rule report truthfully. Bug #43756 describes cases where AbortIncompleteMultipartUpload lifecycle processing reports COMPLETE while leaving the multipart uploads in place, typically when the .meta object was already removed by a previous failed attempt. Watch for that pattern if uploads persist despite an active rule.
• Avoid deleting buckets with uploads in progress. Bug #17164 documents that radosgw-admin bucket rm --purge-data does not abort in-progress multiparts first, so the parts are leaked. Abort uploads explicitly before purging a bucket.
• Watch for reshard interaction. Bug #43583 documents that multipart uploads in progress during a bucket reshard may become un-abortable because the multipart metadata is scattered across shards using a different hash than the resharding process expects. Avoid initiating large reshard operations while multipart uploads are active.
• Monitor the accounting gap. Track the difference between ceph df detail raw bytes for the RGW data pool and the sum of radosgw-admin bucket stats across buckets. A widening gap is the leading indicator that orphans are accumulating.
• Rule out the snapshot trap. ceph df detail also diagnoses snapshot space: data deleted in the live filesystem but retained by a snapshot continues to occupy raw space. The two failure modes look identical from a capacity perspective.

How Netdata helps

• The Ceph collector exposes ceph_pool_percent_used and ceph_cluster_total_used_raw_bytes, so you can see the leak rate of the RGW data pool, not just the absolute level.
• ceph_rgw_gc_retire_object is collected as a counter, so you can correlate GC processing rate against raw pool growth. Flat GC retire with rising pool usage rules out GC backlog and points toward multipart orphans.
• ceph_rgw_failed_req rate spikes can be correlated in time with capacity growth to identify workflows that abandon uploads.
• ceph_health_detail{name="OSD_NEARFULL"} and ceph_health_detail{name="OSD_FULL"} appear as labeled time series, so the trip into capacity-warning territory is visible as an event, not just a threshold breach.
• Per-pool capacity retained at per-second resolution over long history helps distinguish a slow orphan leak from a sudden client ingest spike.

Related guides

• Ceph backfill_toofull: recovery blocked because target OSDs are full
• Ceph blocked ops: client I/O stuck behind a single slow OSD
• Ceph BlueStore RocksDB compaction stalls: periodic latency spikes
• Ceph BLUEFS_SPILLOVER: RocksDB metadata spilling onto the slow device
• Ceph BlueStore allocator fragmentation: rising latency at moderate fullness
• Ceph capacity death spiral: an OSD fails and recovery has nowhere to go
• Ceph client latency vs OSD latency: fast disks, slow clients
• Ceph deep scrub performance impact: I/O saturation that mimics an incident
• Ceph degraded objects: reduced redundancy and the race against a second failure
• Ceph FS_DEGRADED: standby MDS failed to take over a rank
• Ceph health detail: mapping ceph_health_detail checks to a cause
• Ceph HEALTH_ERR: reading the umbrella status and finding the real fault