Ceph too many PGs per OSD: mon_max_pg_per_osd, peering cost, and sizing

PG count per OSD is one of the few Ceph sizing decisions with direct cost in both directions. Too many PGs per OSD means more memory and CPU spent on peering, recovery scans, and PG log maintenance across more logical units. Too few means CRUSH cannot spread data evenly, producing hotspots on specific OSDs while others sit idle.

mon_max_pg_per_osd (default 250; it was 200 during most of Luminous and was raised to 250 in the 12.2.10 timeframe) is the cluster’s failsafe against runaway PG counts. When an OSD approaches this number, Ceph raises TOO_MANY_PGS and blocks new pool creation, pg_num increases, and replication factor changes. This is a guard rail, not a performance target. The existing cluster keeps serving I/O; only topology changes that would add PGs are blocked.

Each PG carries fixed overhead per OSD that hosts it: a PG log, peering state, recovery tracking structures, and BlueStore metadata. This overhead is roughly linear in the number of PGs the OSD participates in. An OSD with 250 PGs does more peering work, holds more PG logs in memory, and takes longer to recover than an OSD with 100 PGs holding the same amount of data.

There is also a hard limit: osd_max_pg_per_osd_hard_ratio (default 3.0). At 3.0 times mon_max_pg_per_osd, an individual OSD refuses to instantiate new PGs entirely. With defaults, that is 750 PGs per OSD. Reaching this limit means PGs cannot peer on that OSD, which can leave PGs stuck in creating or peering states.

The operational target is 100 to 200 PGs per OSD. The PG autoscaler (enabled by default on Nautilus and later) works toward this range, but its conservative threshold and the dynamics of OSD loss mean you can still trip the warning.

How PG count per OSD changes

CRUSH maps each PG to a set of OSDs based on the pool’s pg_num, the replication factor, and the available OSDs. The per-OSD PG count is not static. It changes when:

• Pools are created or deleted. New pools add PGs distributed across OSDs by CRUSH.
• pg_num is increased. Existing PGs split, increasing the count on every OSD in the acting sets.
• OSDs are added or removed. CRUSH redistributes PGs. Adding OSDs lowers per-OSD counts; removing OSDs raises them.
• The autoscaler splits or merges PGs. When the autoscaler decides a pool’s PG count is wrong for its data volume, it splits (too few PGs) or merges (too many PGs) automatically.

The PG autoscaler runs in the manager daemon and evaluates each pool against mon_target_pg_per_osd (default 100). It only acts when the target differs from the current pg_num by more than a factor of 3 (its configurable threshold, default 3.0). This conservative threshold means a pool can be significantly over- or under-provisioned before the autoscaler intervenes. When it does act, splitting PGs causes brief I/O stalls as the new PGs peer and activate.

New pools created with the autoscaler enabled start with a single PG by default. The autoscaler then evaluates the pool’s actual data volume and splits as needed. Pools created with the --bulk flag start with a full complement of PGs based on the target calculation and scale down only if usage is uneven.

flowchart TD
    TRIGGER["OSD loss, pool creation, autoscaler split"] --> CHECK{"PG count per OSD"}
    CHECK -->|under 50| FEW["hot spots, uneven I/O"]
    CHECK -->|100 to 200| TARGET["target range, balanced load"]
    CHECK -->|near 250| WARN["TOO_MANY_PGS, new pools blocked"]
    CHECK -->|near 750| HARD["hard limit, new PGs refused"]
    AUTO["PG autoscaler"] -.->|merges or splits toward target| TARGET

Where this shows up in production

OSD loss is the most common trigger. A cluster running at 220 PGs per OSD across 10 OSDs can jump to roughly 244 PGs per OSD on survivors after one OSD fails. The total PG count has not changed, but the denominator (number of OSDs) has shrunk. Lose two OSDs on a small cluster and you can cross the 250 threshold without creating any new pools.

Small clusters are disproportionately affected. With only 3 OSDs and a replication factor of 3, every PG lands on every OSD. A pool with 256 PGs means 256 PGs per OSD on all three. Add a second pool with the same count and you are at 512 PGs per OSD, well past the warning threshold. The autoscaler helps here by starting pools at 1 PG, but clusters provisioned before autoscaling was default, or pools created with explicit pg_num, can carry legacy counts that are too high for the cluster size.

Pool proliferation in multi-tenant clusters. CephFS deployments with separate metadata and data pools, RGW deployments with separate index, data, and log pools, and erasure-coded pools alongside replicated pools can accumulate dozens of pools. Each pool adds its PG count to the per-OSD total. Even if each pool is modestly sized, the aggregate can cross the threshold.

Autoscaler splits during load. When the autoscaler splits PGs, the new PGs must peer and activate. During this window, affected PGs are temporarily unavailable for I/O. On a busy cluster, a large autoscaler-triggered split can cause visible latency spikes. The autoscaler steps pgp_num gradually to amortize the cost, but expect brief remapping and backfill during splits.

Sizing: targets, tradeoffs, and calculation

The target range is 100 to 200 PGs per OSD. This balances two competing costs:

• Below 100 PGs per OSD: CRUSH has fewer buckets to distribute objects across. With small pools, data concentrates on fewer OSDs. Individual OSDs become hotspots for specific workloads, and rebalancing after OSD loss is coarser because each PG move shifts more data.
• Above 200 PGs per OSD: Per-PG overhead starts to dominate. Peering after topology changes takes longer because more PGs need to negotiate state. Memory consumption from PG logs and metadata increases. Recovery after OSD failure involves more PG-level coordination. Above approximately 500 PGs per OSD, peering and RAM usage become excessive.

The standard sizing formula for a single pool:

PGs = (target_per_OSD * num_OSDs * data_fraction) / pool_size

Round up to the nearest power of 2. target_per_OSD is typically 200 for stable clusters. data_fraction is the proportion of cluster data this pool will hold (0 to 1). pool_size is the replication factor or (k+m) for erasure-coded pools.

For example, a 3x replicated pool expected to hold 50% of cluster data on a 30-OSD cluster:

PGs = (200 * 30 * 0.5) / 3 = 1000 -> round up to 1024

The official documentation recommends a target of 200 PGs per OSD for all but the very smallest deployments, and warns that values above 500 produce excessive peering traffic and RAM usage. The autoscaler’s own target (mon_target_pg_per_osd) defaults to 100, which is more conservative than that recommendation. This difference is intentional: the autoscaler starts small and splits upward as data arrives, while a fixed target assumes you want a static count sized for expected growth.

Checking and adjusting

Check the current per-OSD PG distribution and autoscaler state:

# Per-OSD PG count (look at the PGS column)
ceph osd df

# Autoscaler status and recommended PG counts for all pools
ceph osd pool autoscale-status

# Current mon_max_pg_per_osd value
ceph config get mon mon_max_pg_per_osd

If the warning fires because of OSD loss (not because you created too many pools), the fix is to restore the missing OSDs or add new ones. Do not raise mon_max_pg_per_osd to suppress the warning. If you must create a pool or increase pg_num while the warning is active, you can set mon_max_pg_per_osd to 0 temporarily to disable the check. This removes the guard rail entirely and can allow PG counts that cause pathological peering and memory consumption. Use it only as a deliberate, short-term override during an active incident.

If a pool has too many PGs for its actual data volume, ensure the autoscaler is enabled so it can reduce pg_num through merging:

# Enable autoscaler for a pool
ceph osd pool set <pool> pg_autoscale_mode on

For pools with overlapping CRUSH roots (for example, a pool spanning both SSD and HDD device classes), the autoscaler refuses to scale and issues a warning in the manager log. These pools require manual PG count management.

Signals to watch

How Netdata helps

• Per-second PG state metrics from the Ceph collector let you watch peering and creating counts change in real time. When the autoscaler splits PGs, you see ceph_pg_creating spike and then settle, correlating with any brief I/O stalls.
• Per-OSD metrics (up/down, latency, memory) let you detect when OSD loss is driving the per-OSD PG count upward. Correlating an OSD down event with a subsequent TOO_MANY_PGS health check clarifies cause and effect without manual ceph osd df polling.
• Health detail metrics with labels surface TOO_MANY_PGS and POOL_TOO_MANY_PGS as individual signals, so you can distinguish the cluster-wide guard rail from per-pool autoscaler advisories.
• Anomaly detection on PG state counts flags unexpected peering activity outside of known maintenance windows, which is often the first sign that the autoscaler is splitting PGs or that OSD loss has triggered redistribution.
• Recovery rate metrics (ceph_pool_recovering_bytes_per_sec) alongside PG state counts help you assess whether high PG counts are slowing recovery after an OSD failure.

Related guides

• Ceph backfill_toofull: recovery blocked because target OSDs are full
• Ceph capacity death spiral: an OSD fails and recovery has nowhere to go
• Ceph health detail: mapping ceph_health_detail checks to a cause
• Ceph HEALTH_ERR: reading the umbrella status and finding the real fault
• Ceph HEALTH_WARN: which warnings are noise and which are structural
• How Ceph actually works in production: a mental model for operators
• Ceph monitoring checklist: the signals every production cluster needs
• Ceph monitoring maturity model: from survival to expert
• Ceph OSD down: telling a dead disk apart from a network blip
• Ceph OSD flapping: OSDs cycling up and down and the peering storm that follows
• Ceph OSD_FULL: all writes stopped at the 95% full ratio
• Ceph OSD fullness imbalance: one OSD full while the cluster average looks fine