$ guides / ceph / ceph-how-it-works-in-production ▌

Operations Guides

How Ceph actually works in production: a mental model for operators

Most Ceph incidents become legible the moment you stop reasoning about RBD, CephFS, and RGW as separate products and start reasoning about one system: RADOS placing objects on OSDs. The three client interfaces are thin translations. Underneath them, every write is an object, every object lives in a placement group, and every placement group is mapped to a set of OSDs by a deterministic algorithm.

This article covers the model, not the failure catalogue or the full signal list: the layering, how CRUSH places data without a lookup table, what the MONs actually do (and do not do), how BlueStore lays out an OSD on disk, and why the capacity thresholds behave like hard circuit breakers.

What it is and why it matters

Ceph is a distributed object store that exposes block (RBD), file (CephFS), and object (RGW, S3 and Swift compatible) interfaces on top of one shared substrate: RADOS, the Reliable Autonomic Distributed Object Store. The interfaces differ in protocol and in which daemon speaks them, but they all end as the same kind of operation: write this object, replicate it, acknowledge.

That collapse is the point. When an RBD volume stalls, a CephFS mount hangs, and an RGW PUT times out at the same time, you are not looking at three interface bugs. You are looking at one RADOS problem: a stuck PG, a saturated OSD, or a cluster approaching full.

The model also explains why Ceph fails the way it does. Placement is algorithmic, not table-driven, so topology changes ripple through the cluster. The MONs are authoritative for topology but sit off the data path, so a MON problem does not stop I/O immediately; it freezes the cluster’s ability to react. Capacity is enforced as hard thresholds, not gradual backpressure. These design choices produce characteristic failure shapes.

How it works

The diagram is the whole model in one picture: client interfaces collapse into RADOS, RADOS shards objects into placement groups, CRUSH maps each PG to an OSD acting set, BlueStore writes to raw devices, and the MONs distribute the maps that everyone agrees on.

flowchart TD
  RBD[RBD: block] --> RADOS
  CephFS[CephFS: file] --> RADOS
  RGW[RGW: S3/Swift] --> RADOS
  RADOS[RADOS object store] -->|object hashes into| PG[Placement Group]
  PG -->|CRUSH computes acting set| OSDs[OSD primary + replicas]
  OSDs -->|write| BS[BlueStore on raw device]
  MON[MONs: maps via Paxos] -.->|distribute maps| OSDs

Everything is an object in RADOS

RBD images are striped into objects (4 MB by default). CephFS stores file data as objects in a data pool and file metadata in a separate metadata pool. RGW stores object data in a data pool and bucket indexes as OMAP entries on RADOS objects. From RADOS’ perspective these are all the same thing: objects identified by pool and name, hashed into placement groups.

A pool is the unit of redundancy policy. Replicated pools keep N full copies. Erasure-coded pools split objects into k data chunks plus m coding chunks. The pool decides the redundancy model. CRUSH decides where the copies or chunks land.

Objects hash into PGs, PGs map to OSDs via CRUSH

A placement group (PG) is the sharding unit. An object’s pool and name are hashed, the hash selects a PG, and CRUSH deterministically maps that PG to an ordered set of OSDs called the acting set. The first OSD in the acting set is the primary, which coordinates client I/O for that PG. The rest are replicas (or, for erasure-coded pools, chunk holders).

The key insight, and the thing that makes Ceph unlike a hash ring or a lookup table, is that placement is computed, not stored. Every participant, clients included, can independently compute “PG X belongs on OSDs A, B, C” from the current CRUSH map. There is no central placement service to query and no table to keep in sync. When the topology changes, everyone recomputes.

This is why topology changes are expensive. Adding or removing an OSD changes the CRUSH input, which changes the mapping for many PGs, which triggers data movement (recovery or backfill) to bring the acting sets back in line with the new map.

There is one wrinkle worth knowing. The balancer module can store explicit PG-to-OSD overrides on top of CRUSH (the “upmap” mode) to improve distribution. The pure-computation model is still the right starting point, but overrides exist when you are debugging uneven distribution. Since Pacific (16.2.x), upmap has been the balancer module’s default mode, and in upmap mode automatic balancing is enabled by default (ceph balancer status shows the mode in effect).

MONs hold the maps and are off the data path

The monitors (MONs) run a Paxos cluster and maintain the authoritative cluster maps: OSD map, MON map, CRUSH map, PG map, MDS map. A majority quorum is required to commit any map update. Three MONs is the standard production deployment because it tolerates one failure while keeping majority.

The critical operational fact is that MONs are not in the data path. Client writes go directly from the client to the primary OSD to the replica OSDs. The MONs do not proxy data. What they do is authoritative topology: when an OSD goes down, the MONs commit that into the OSD map; when a PG’s acting set changes, that is reflected in the PG map; when you create a pool, the MONs record it.

This means a MON problem has a specific shape. Quorum loss does not immediately stop I/O. Existing clients keep talking to OSDs they already know about, using their cached maps. But no new map updates can be committed, so the cluster cannot react to any further topology change. If an OSD fails during a MON outage, PGs cannot remap to surviving replicas, and client I/O for those PGs eventually stalls as cached maps go stale.

MONs are also sensitive to clock skew. The default mon_clock_drift_allowed is 0.05 seconds (50ms); skew beyond that triggers warnings and can destabilize elections. MONs use a key-value store internally (RocksDB on modern releases). The store grows with OSD map epochs, so clusters with frequent topology churn, especially flapping OSDs, can bloat the MON store and slow elections.

BlueStore writes raw devices with a RocksDB DB and WAL

BlueStore is the OSD backend. It writes object data directly to raw block devices, bypassing a filesystem, and keeps metadata (object metadata, OMAP data, allocator state) in a RocksDB database. A BlueStore OSD can span up to three devices:

The main data device, the bulk storage, typically HDD or SATA SSD.
An optional DB device, holding the RocksDB database, ideally on fast media such as NVMe.
An optional WAL device, holding the write-ahead log, also ideally on fast media.

The DB device is the one that bites. RocksDB is sized for fast random access. If the DB device fills, RocksDB spills onto the main data device. Once metadata operations hit the slow device, latency jumps discontinuously. This is a cliff, not a slope. The OSD stays up, capacity metrics look fine, and the only signal is commit latency spiking during compaction. Many teams discover this only after months of unexplained latency on a subset of OSDs.

Since Squid (19.x), LZ4 is the default RocksDB compression in bluestore_rocksdb_options; Reef and earlier shipped kNoCompression.

The operational implication is unchanged regardless of compression defaults: keep the DB device off the slow data device, and size it for object count, not raw capacity. RGW bucket indexes, stored as OMAP, are a common driver of unexpected DB growth.

Capacity thresholds are cluster-wide circuit breakers

Ceph enforces three capacity thresholds, configured as ratios of OSD fullness and stored in the OSD map:

nearfull, default 0.85: HEALTH_WARN, recovery backfill may be throttled.
backfillfull, default 0.90: an OSD at this ratio refuses to accept backfill data, blocking recovery to that OSD.
full, default 0.95: the OSD refuses all writes. Client writes return ENOSPC.

These are not soft hints. They are hard stops, enforced per OSD, and the cluster-wide effect is dominated by the fullest OSD, not the average. A single full OSD blocks writes to every PG that has that OSD in its acting set, which is typically hundreds of PGs. A cluster at 80% average with one OSD at 95% is effectively full for a slice of its data.

This is why operators treat 85% as the real ceiling, not 95%. Recovery needs spare capacity. At nearfull you have already lost the headroom to absorb a host failure cleanly. Crossing backfillfull means you cannot heal at all without adding space. The thresholds are configurable, but raising them under pressure is a stopgap that buys time at the cost of risk.

Where this shows up in production

The mental model predicts the shapes you see in real incidents.

Topology changes cause cascading work. An OSD going down is not just “one disk offline”. It is a CRUSH map change that remaps every PG that OSD participated in, triggers peering on all of them, and starts recovery traffic across the cluster. A host failure with a dozen OSDs is a major event because of the PG remapping cascade, not because a dozen disks is a lot of raw capacity.

Recovery competes with clients. Recovery and backfill read from source OSDs and write to target OSDs on the same disks and the same network as client I/O. There is no separate recovery fabric. Without throttling, a single OSD failure can saturate the cluster network and spike client latency by an order of magnitude.

The cluster average lies. Capacity, latency, and PG distribution all have per-OSD variance that the aggregate hides. CRUSH does not guarantee even placement, and reweight adjustments are often needed. Monitoring the average instead of the worst OSD is the most common way teams miss a slow failure.

MONs freeze topology, not data. A MON outage during steady state is nearly invisible to clients. A MON outage during an OSD failure is catastrophic, because the cluster cannot remap PGs to recover. The danger window is the intersection of MON problems and OSD problems.

Common ways this model gets misapplied

Treating MONs as part of the data path. They are not. Adding MONs does not add write bandwidth, and MON latency does not directly add to client write latency under steady state. But MON health is load-bearing for recovery, so under-investing in MON hosts (slow disks, shared with other services, poor NTP) is a latent failure.
Assuming CRUSH gives even distribution. It gives statistically good distribution, not perfect distribution. On small clusters, or after adding a few OSDs, variance can be significant. Per-OSD capacity and per-OSD PG count both need monitoring.
Reading capacity as a soft signal. The thresholds are hard. The only way past full is to free space or add capacity. Raising the ratio is a gamble, not a fix.
Trusting aggregate latency. OSD commit and apply latency have long tails. A cluster with a 5ms median can have one OSD at 200ms, and the clients unlucky enough to land on that OSD’s PGs see the 200ms. Median is not the signal. The worst OSD is.
Forgetting the BlueStore DB device. It is the single most common source of unexplained latency cliffs. Check it whenever commit latency spikes on specific OSDs while the rest of the cluster looks fine.

Signals to watch in production

These signals map directly onto the mental model. Each corresponds to a piece of the architecture above.

Signal	Why it matters	Warning sign
MON quorum status	Without majority, no map updates commit and topology changes freeze.	Fewer than floor(N/2)+1 MONs in quorum.
Per-PG state counts	PGs not in active+clean are degraded, recovering, or stuck.	Sustained degraded or incomplete PGs, especially with zero recovery rate.
Per-OSD capacity utilization	The fullest OSD, not the average, drives effective capacity.	Any OSD approaching nearfull (0.85) or backfillfull (0.90).
OSD commit and apply latency	Commit reflects WAL/DB device health; apply reflects data device health.	Commit latency spiking on OSDs with a dedicated DB device signals DB spillover or DB device failure.
Slow ops count	Operations past `osd_op_complaint_time` (default 30s) are stuck, not slow.	Any sustained nonzero slow ops.
Recovery rate vs degraded count	Recovery should be making progress. Stalled recovery extends the data-loss exposure window.	Degraded PGs flat while recovery bytes/sec is near zero.
BlueFS slow device usage	Nonzero means RocksDB has spilled from the DB device to the data device.	Any nonzero slow device usage in bluefs stats.

How Netdata helps

Netdata’s per-second collection is most useful for Ceph precisely because the interesting events, PG peering cascades, OSD flaps, recovery stalls, happen on the scale of seconds and are easy to miss with 60-second scraping. The signals worth correlating:

MON quorum status against OSD flap events, to catch the dangerous intersection of MON instability and topology churn.
Per-PG state counts against per-OSD commit latency, to see whether latency spikes line up with recovery or peering bursts.
Per-OSD capacity against PG recovery states such as backfill_toofull, to predict capacity-driven recovery stalls before they happen.
Slow ops against OSD apply and commit latency, to distinguish a local device problem from a replication or network problem.
Recovery rate against degraded object count, to distinguish healthy healing from a stalled recovery that is extending the data-loss window.
Cluster flags (noout, norecover, nobackfill, noscrub, nodeep-scrub) against recovery activity, to catch the classic forgotten-flag outages.

Because Netdata collects host-level metrics alongside the Ceph exporter metrics, you can also put BlueStore commit latency next to the underlying NVMe device’s latency and utilization on the same timeline, which is usually the fastest path to confirming or ruling out DB spillover.