Ceph OSD commit and apply latency: reading per-OSD latency outliers

Two metrics tell you more about per-OSD health than almost anything else in Ceph: ceph_osd_commit_latency_ms and ceph_osd_apply_latency_ms. They are per-OSD gauges exported by the manager’s Prometheus module, labeled by ceph_daemon, and they measure the OSD’s internal I/O time, not the latency your clients experience end-to-end. A cluster can show healthy aggregate latency while one OSD quietly runs 5x slower than its peers of the same device class. The clients whose objects land on that OSD see tail latency spikes; everyone else sees normal performance. Cluster-wide averages hide this. Per-OSD comparison surfaces it.

This article covers what the two metrics actually measure, why they are equal on BlueStore, what normal looks like by device class, and how to read a single-OSD outlier versus a cluster-wide rise.

What these metrics measure

Both metrics are reported per OSD, labeled ceph_daemon, in milliseconds. They come from OSD perf counters surfaced through the manager’s Prometheus module.

ceph_osd_commit_latency_ms measures the time to persist a write to the WAL/journal path. On a BlueStore OSD with a dedicated WAL or DB device, this reflects the performance of that fast device. On FileStore, when commit latency spikes on one OSD while apply latency stays flat, the WAL/DB path is the bottleneck.

ceph_osd_apply_latency_ms measures the time to apply the write to the main data device: HDD, SATA SSD, or NVMe depending on your tier. High apply latency on a single OSD points at the data device itself. Causes include device saturation, heavy recovery or scrub load, or failing media.

Neither metric includes client-to-OSD network round-trip time. They are internal to the OSD’s processing pipeline. Client-observed latency includes network RTT, replication overhead to peer OSDs, and queuing in the messenger layer. A cluster can look fast at the OSD level while clients experience high latency from network congestion or replica slowness. Correlate OSD latency with slow ops and client-reported latency before concluding the OSD is the root cause.

BlueStore collapses commit and apply

If you are running BlueStore (the default since Luminous), ceph osd perf will show commit and apply latency as identical values on every OSD. This is expected behavior, not a bug: BlueStore feeds both values from the same internal commit-latency counter (BSPerfTracker::update_from_perfcounters assigns the same counter to both).

FileStore is not supported in Reef (18.2.0) and later: the backend was removed from the source tree, and its release notes state “FileStore is not supported in Reef”. On Reef and Squid every OSD runs BlueStore.

BlueStore writes directly to raw block devices and manages its own write-ahead log through BlueFS. Unlike FileStore, where the journal device and the data filesystem were distinct I/O paths with distinct performance characteristics, BlueStore’s commit and apply operations go through the same underlying stack. The two perf counters remain in the interface for compatibility with FileStore-era tooling, but they report the same number.

The practical consequence: on a BlueStore-only cluster, do not try to diagnose a WAL-versus-data-device split from these two metrics alone. You cannot. If you need to isolate WAL/DB performance from data device performance on BlueStore, you need deeper signals: BlueFS slow device usage, RocksDB compaction stats, and OS-level iostat on the individual block devices backing the OSD.

For FileStore clusters still in the wild: commit latency reflects the journal device and apply latency reflects the XFS data device. A meaningful gap between them is diagnosable. High commit with normal apply means the journal device is saturated or failing.

Normal ranges by device class

These ranges are operational baselines under sustained client load, not hard guarantees:

SATA SSD falls between these tiers. Exact numbers depend on drive model, firmware, queue depth, and workload mix. Small random writes are the worst case for all device classes.

What matters more than absolute values is comparison within a device class. An HDD OSD at 80ms apply latency is unremarkable if its peers on the same host and same device class sit at 60-90ms. The same OSD is a problem if every other HDD OSD is at 20ms. The signal is the outlier relative to peers, not the absolute number against a universal threshold.

Reading outliers: single OSD vs cluster-wide

The interpretation fork is the most important part of this article. There are two patterns, and they point at completely different root causes.

A single OSD at 5x the same-device-class median, sustained for more than 300 seconds, points at a local problem. Local means: this OSD’s hardware, this OSD’s DB/WAL device, this OSD’s configuration. The rest of the cluster is fine. Common causes include a failing disk (check SMART attributes), a saturated or failing DB/WAL NVMe, RocksDB compaction stalls, BlueStore DB spillover (metadata leaking from the fast DB partition to the slow data partition), or a CRUSH hot spot concentrating a heavy-write workload on one OSD.

A cluster-wide rise (most or all OSDs of a device class climbing together) points at a systemic problem. Systemic means: the network, the recovery load, the scrub schedule, or the cluster’s overall capacity pressure. Common causes include cluster-network congestion saturating the inter-OSD link, an active recovery storm competing with client I/O, deep-scrub running across many OSDs simultaneously, or capacity pressure forcing write amplification.

flowchart TD
    A["OSD latency elevated"] --> B{"One OSD or
cluster-wide?"}
    B -->|"Single OSD
5x median, >300s"| C["Local problem"]
    B -->|"Cluster-wide rise"| D["Systemic problem"]
    C --> C1["Failing disk
check SMART"]
    C --> C2["DB/WAL NVMe
saturation or failure"]
    C --> C3["RocksDB compaction
or DB spillover"]
    C --> C4["CRUSH hot spot"]
    D --> D1["Cluster network
congestion"]
    D --> D2["Recovery storm
competing with clients"]
    D --> D3["Deep scrub
across many OSDs"]
    D --> D4["Capacity pressure
or write amplification"]

The diagnostic action for each fork differs. For a single-OSD outlier, isolate that OSD: check its device health, its BlueFS stats, its RocksDB state, and its CRUSH weight. For a cluster-wide rise, step back: check recovery rate, scrub schedule, network utilization on the cluster interface, and capacity. Debugging a cluster-wide latency rise by investigating individual OSDs wastes time. Every OSD looks slow because the problem is above them.

The point-in-time snapshot trap

ceph osd perf and the Prometheus metrics are point-in-time snapshots, not histograms or rolling averages. The value at any given sample is the instantaneous latency at the moment of sampling.

This has two consequences. First, a single high reading may be a transient spike: a compaction event, a momentary write burst, a scrub pass. One sample is not an outlier. You need sustained elevation to call it a problem. The 5x device-class median sustained for more than 300 seconds threshold exists to filter out these transient spikes.

Second, because these are snapshots, you cannot derive percentiles from them. You cannot tell whether an OSD has a high P99 with a low mean, or a uniformly elevated latency. If you need distribution data, use the OSD perf histogram via the admin socket (ceph daemon osd.<id> perf histogram dump, available since Luminous), which provides bucketed latency. The Prometheus metrics give you the scalar. The histogram gives you the shape.

When sampling manually with ceph osd perf, run it several times over 10 or more seconds rather than reading a single output. Sustained elevation across samples is the signal. A single jump is noise.

Getting the data

The fastest CLI path:

# Check per-OSD commit and apply latency
ceph osd perf

This prints two columns, commit_latency(ms) and apply_latency(ms), for every OSD. Scan for the OSD that stands out from its same-device-class peers.

For a specific OSD’s deeper perf-counter state:

# Inspect a specific OSD's operation latency perf counters
ceph daemon osd.5 perf dump | jq '.osd | {
  op_latency: .op_latency,
  op_r_latency: .op_r_latency,
  op_w_latency: .op_w_latency,
  op_rw_latency: .op_rw_latency
}'

These are the OSD perf counters that exist; there are no op_commit_latency or op_apply_latency counters in the OSD perf dump. Each entry carries avgcount, sum, and avgtime sub-fields, so average latency between two samples is delta sum / delta count (or read avgtime directly).

ceph_osd_commit_latency_ms and ceph_osd_apply_latency_ms are not derived from these counters: they are the OSD’s objectstore-level commit/apply latency, reported in osd_stats and printed by ceph osd perf as point-in-time values (already in ms).

To check for BlueStore DB spillover, a common cause of latency cliffs that does not show up in capacity metrics:

# Check if RocksDB has spilled to the slow data device
ceph daemon osd.5 perf dump | jq '.bluefs | {
  slow_used_bytes: .slow_used_bytes,
  slow_total_bytes: .slow_total_bytes
}'

Any nonzero slow_used_bytes means metadata has spilled from the fast DB partition to the slow data partition. This is a performance cliff. RocksDB operations that ran at SSD or NVMe speed now run at HDD speed. The symptom is sustained elevated commit and apply latency on that OSD only, while its peers remain normal.

Signals to watch in production

How Netdata helps

• Netdata’s Ceph collector scrapes ceph_osd_commit_latency_ms and ceph_osd_apply_latency_ms per ceph_daemon at per-second resolution, so you see the outlier OSD the moment it diverges from its peers.
• Per-second collection makes the “sustained for more than 300 seconds” threshold practical. You can watch the outlier form in real time and confirm it is sustained, not a transient spike from a compaction event or scrub pass.
• Correlating per-OSD latency with ceph_healthcheck_slow_ops, recovery rate, and scrub activity in a single timeline view lets you distinguish a single-OSD hardware problem from a cluster-wide recovery or scrub impact without switching tools.
• ML anomaly detection flags OSDs whose latency deviates from their own historical baseline, which catches slow degradation (a disk failing over weeks) that a static 5x-median threshold would miss until the OSD is badly degraded.
• Device-class labeling lets you compare each OSD against peers of the same class automatically, which matters because HDD and NVMe have different normal ranges.

Related guides

• Ceph backfill_toofull: recovery blocked because target OSDs are full
• Ceph capacity death spiral: an OSD fails and recovery has nowhere to go
• Ceph health detail: mapping ceph_health_detail checks to a cause
• Ceph HEALTH_ERR: reading the umbrella status and finding the real fault
• Ceph HEALTH_WARN: which warnings are noise and which are structural
• How Ceph actually works in production: a mental model for operators
• Ceph MON_DOWN: a monitor out of quorum and reduced redundancy
• Ceph monitor quorum lost: the cluster can no longer update its maps
• Ceph monitoring checklist: the signals every production cluster needs
• Ceph monitoring maturity model: from survival to expert
• Ceph OSD down: telling a dead disk apart from a network blip
• Ceph OSD flapping: OSDs cycling up and down and the peering storm that follows