Ceph client latency vs OSD latency: fast disks, slow clients

You ran ceph osd perf, the commit_latency and apply_latency columns all look healthy, and you declared the cluster fast. Then a client team opens a ticket: writes are taking 50ms, 100ms, sometimes timing out. You re-check ceph osd perf. Still fast. The OSDs are not the bottleneck, but the clients are still slow.

ceph osd perf, the ceph_osd_*_latency_ms Prometheus metrics, and rbd perf image iostat all measure time inside the OSD data path. They do not measure the round trip the client experiences. Client-visible latency is the sum of OSD processing, public network RTT, replication round-trips to peer OSDs, CRUSH map recalculation during flapping, and queuing delay on the OSD’s public-facing messenger. A cluster can show 2ms commit latency on every OSD while clients see 50ms response times because the public network is saturated, or because public and cluster traffic share one NIC.

What this means

ceph osd perf reports two columns per OSD: commit_latency(ms) and apply_latency(ms). On BlueStore, which is every modern deployment, these are close but not identical: commit reflects the write-ahead log (BlueStore’s internal journal, equivalent to the RocksDB WAL) and apply reflects the data reaching the data device, so deferred writes and RocksDB throttling can separate them; with no separate FileStore-style journal, in practice they move together. On the legacy FileStore backend, commit measured the journal write and apply measured the data write. FileStore was deprecated in the Reef release, which no longer supports it (BlueStore has been the default backend since Luminous), so on any current cluster the two columns move together.

Both numbers measure OSD-internal processing: the time from when the OSD has the operation in hand to when it has committed the write locally. They do not include:

• network RTT between the client and the primary OSD (public network)
• network RTT between the primary and replica OSDs (cluster network)
• replication round-trips, since a 3x replicated write must complete on all three OSDs before the client receives an ACK
• CRUSH map recalculation on the client during OSD flapping
• queuing delay on the OSD’s public-facing messenger thread
• queuing delay from recovery and backfill competing on the same disks and NICs

The Prometheus metric ceph_osd_op_latency_sum with its _count pair is also OSD-side. It measures OSD processing time, not network transit in either direction: the OSD perf counters define op_latency as client-op latency including queue time, op_process_latency excluding queue time, and op_prepare_latency excluding queue time and the wait for the operation to finish.

rbd perf image iostat and rbd perf image iotop, which require the rbd_support MGR module, give per-image IOPS, throughput, and latency from the OSD data path. These latency measurements are not end-to-end. The assumption that the extra network segment is constant holds only with a quiet public network, and that assumption is the first thing that breaks.

flowchart TD
  Client["Client librados / librbd"] -->|"1. public network RTT"| Primary["Primary OSD"]
  Primary -->|"2. op queue, locks, peering"| Commit["BlueStore commit/apply
(what ceph osd perf measures)"]
  Commit -->|"3. cluster network RTT"| Replicas["Replica OSDs"]
  Replicas -->|"4. replica commit/apply"| Primary
  Primary -->|"5. public network RTT"| Client

The client experiences steps 1 through 5 as a single response time. ceph osd perf sees only step 2. Everything else, including the replication round-trip in steps 3 and 4, is invisible to that command.

Common causes

Quick checks

All read-only and safe to run during an incident.

# OSD-internal latency baseline
ceph osd perf

# Slow ops: client-perceived slowness past osd_op_complaint_time (default 30s)
ceph health detail | grep -A 10 SLOW_OPS

# Per-OSD op latency from the admin socket
ceph daemon osd.0 perf dump | jq '.osd | {op_latency, op_apply_latency, op_commit_latency}'

# In-flight and blocked ops on a specific OSD
ceph daemon osd.0 dump_ops_in_flight
ceph daemon osd.0 dump_blocked_ops

# Aggregate client I/O and recovery I/O
ceph -s

# Per-pool client IOPS
ceph osd pool stats

# Recovery throttle settings in effect
ceph daemon osd.0 config show | grep -E 'osd_recovery_max_active|osd_max_backfills|osd_recovery_sleep'

# RBD per-image latency (OSD-side only, requires mgr/rbd_support)
rbd perf image iostat

# Network interface saturation on OSD hosts
ip -s link show <public_iface>
ip -s link show <cluster_iface>

# Per-OSD messenger stats for queuing and connection churn
ceph daemon osd.0 perf dump | jq 'to_entries[] | select(.key | startswith("AsyncMessenger"))'

If you have shell access to a client host, query the librados or librbd admin socket directly. The socket path varies by deployment, but the pattern is:

# Client-side perf counters (path and name vary by deployment)
ceph daemon /var/run/ceph/ceph-client.<name>.asok perf dump

This is the only place you can see the round trip from the client’s perspective.

How to diagnose it

1. Confirm the gap is real. Pull ceph osd perf and note the worst OSD’s commit and apply latency. Then ask the client team for their measured latency, or check ceph health detail for SLOW_OPS. If OSD latency is single-digit milliseconds and clients are seeing tens or hundreds of milliseconds, the gap is between the OSD and the client, not inside the OSD.

2. Check slow ops first. ceph_healthcheck_slow_ops greater than zero means operations have been held longer than osd_op_complaint_time (default 30 seconds). That is not “slow,” that is stuck. Read the slow op descriptions with ceph daemon osd.<id> dump_blocked_ops: the “waiting for subops from” string points at replication, “waiting on pg” points at peering, “waiting for rw locks” points at contention. The description is more useful than the count.

3. Check whether public and cluster networks are separated. Look at ceph.conf or ceph config dump for public_network and cluster_network. If they are on the same subnet or the same physical NIC, recovery and replication traffic competes directly with client I/O. The Ceph network configuration reference is blunt about this: when OSDs replicate data more than once, the inter-OSD network load easily dwarfs the client-to-cluster load, and sharing the link introduces latency.

4. Check public network utilization. On each OSD host, run ip -s link show <public_iface> and watch the receive and transmit counters over a few seconds. Compare against link capacity. Also check /proc/net/snmp for TCPRetransSegs: high throughput plus rising retransmits is a link-layer problem that looks exactly like disk slowness to clients. If jumbo frames (MTU 9000) are configured on some nodes but not others, you get silent fragmentation and severe degradation.

5. Check recovery load. ceph -s shows a recovery line when data is moving. Recovery competes with client I/O on the same disks and, if networks are shared, the same NICs. If recovery is active and client latency is bad, the recovery throttle settings are the lever: osd_max_backfills (default 1), osd_recovery_max_active (default 0, which resolves to 3 concurrent recoveries on HDD OSDs and 10 on SSD OSDs via osd_recovery_max_active_hdd/_ssd), and osd_recovery_sleep (default 0).

6. Check for flapping. Ceph has no dedicated OSD_FLAPPING health check in the reviewed releases (Quincy/Reef/Squid); an OSD cycling up and down shows up as repeated up/down transitions and OSD map epoch churn. Each flap generates a new OSD map epoch, which forces clients to recalculate CRUSH and can briefly route requests to the wrong OSDs. This shows up as client latency spikes and retries, but ceph osd perf on the affected OSD can look completely normal between flaps. Check ceph osd tree repeatedly, or watch the OSD map epoch in ceph -s.

7. Check the client side if you can. If the client host is reachable, query its admin socket perf dump and look at the objecter or messenger counters. A gap between the client’s measured latency and the OSD’s op_latency is direct evidence of network or queuing overhead.

Metrics and signals to monitor

Fixes

Shared public and cluster NIC

This is the single most common cause of fast-OSD, slow-client clusters. If public_network and cluster_network route over the same physical NIC, replication and recovery traffic compete with client I/O at the link layer. For a 3-replica pool, every write triggers inter-OSD traffic that the Ceph docs note easily dwarfs the client-to-cluster traffic.

The structural fix is to separate the networks onto different physical links or VLANs with their own NICs. If that is not immediately possible, throttle recovery aggressively (see below) during incidents and treat the separation as a capacity-planning item. Jumbo frames (MTU 9000) end to end also help, but only if every switch port and NIC in the path agrees.

Recovery competing with client I/O

If ceph -s shows active recovery and client latency is bad, lower the recovery throttle temporarily:

# Throttle recovery to reduce client impact (runtime, non-persistent)
ceph tell 'osd.*' injectargs '--osd_max_backfills 1 --osd_recovery_max_active 1 --osd_recovery_sleep 0.5'

This extends the time the cluster spends degraded, so it is a tradeoff between client impact and data-loss exposure window. If client impact is critical, norecover and nobackfill will stop recovery entirely, but understand that this leaves the cluster at reduced redundancy for the duration. On Quincy and later, the mclock scheduler gives finer control over the balance between client, recovery, and scrub work; note that under mclock the sleep-based throttles such as osd_recovery_sleep are ignored (the option documentation states this), so the throttle commands only affect clusters on the classic scheduler.

CRUSH churn from flapping

If OSD_FLAPPING is active, the latency spikes are a symptom of the flapping, not a network problem. Stop the flapping first:

# Prevent the OSD from being marked out (cluster-wide; triggers no rebalance)
ceph osd set-noout

# If it keeps cycling, mark it down and stop the daemon
ceph osd down osd.<id>
systemctl stop ceph-osd@<id>

Per-OSD noout exists on current releases: ceph osd add-noout osd.<id> marks just that OSD, and ceph osd rm-noout osd.<id> clears it (the same add/rm forms exist for noup, nodown, and noin).

ceph osd down marks the OSD down. With noout set, it will not be marked out and no rebalancing starts, but PGs on that OSD become degraded. Without noout, marking it down triggers recovery on its PGs. Then investigate the root cause: SMART attributes on the disk, OSD logs, dmesg, and network stats on the host. Each flap generates an OSD map epoch and forces clients to recalculate placement, so a single flapping OSD can degrade latency for clients whose data has nothing to do with that OSD.

Public network saturation or errors

If the public interface is near link capacity, the options are traffic shaping, adding links (LACP with a sane hashing policy), or upgrading the link. Check the hashing algorithm on any LACP bond: a poor hash can leave one member link saturated while others are idle.

If TCPRetransSegs is climbing, the problem is physical or driver level: check for cable, switch port, NIC firmware, or MTU mismatch issues. MTU mismatch with jumbo frames is particularly nasty because it causes silent fragmentation.

Prevention

• Separate public and cluster networks at deployment time. Retrofitting is painful. Treat a shared NIC as a known technical debt with a deadline.
• Monitor per-OSD latency, not just cluster averages. A single outlier OSD causes tail latency for the clients that land on its PGs. Alert on the worst OSD relative to its device-class peers, not the mean.
• Alert on slow ops, not on OSD latency alone. ceph_healthcheck_slow_ops is the best signal that clients are actually stuck. OSD latency is necessary but not sufficient.
• Track OSD map epoch rate. A climbing epoch rate indicates flapping or excessive topology churn, both of which inflate client latency invisibly to ceph osd perf.
• Watch recovery flags. Forgotten norecover, nobackfill, and especially noout flags are common operational errors that distort latency and recovery behavior.
• Verify jumbo frame consistency end to end after any network change. A single non-jumbo port fragments traffic and degrades performance silently.

How Netdata helps

Netdata’s Ceph collector surfaces the signals that let you separate OSD-internal latency from client-visible latency without scraping admin sockets by hand:

• Per-second ceph_osd_commit_latency_ms and ceph_osd_apply_latency_ms per OSD let you spot the outlier OSD whose disk is slow while the cluster average looks fine.
• ceph_healthcheck_slow_ops as a gauge shows when operations are actually stuck past osd_op_complaint_time, which is the direct client-impact signal.
• ceph_health_detail labeled by check name surfaces OSD_FLAPPING, OSD_FULL, and recovery-related checks as individual series, so map churn and capacity problems are visible alongside latency.
• Host-level network metrics from Netdata’s generic collectors on the same OSD hosts let you correlate spikes in ceph_osd_commit_latency_ms with public or cluster interface saturation and TCP retransmits in the same view.
• Recovery rate metrics (ceph_pool_recovering_bytes_per_sec) alongside OSD latency let you confirm whether a latency spike coincides with recovery traffic competing for the same disks or NICs.

The value is correlation: seeing the OSD layer, the slow-ops signal, the host network, and the recovery rate on one per-second timeline makes the “fast disks, slow clients” gap obvious instead of mysterious.

Related guides

• How Ceph actually works in production: a mental model for operators
• Ceph monitoring checklist: the signals every production cluster needs
• Ceph monitoring maturity model: from survival to expert
• Ceph HEALTH_WARN: which warnings are noise and which are structural
• Ceph health detail: mapping ceph_health_detail checks to a cause
• Ceph backfill_toofull: recovery blocked because target OSDs are full
• Ceph capacity death spiral: an OSD fails and recovery has nowhere to go