Disks

Guides

Cassandra Operations Guides 6 pages

Open the Cassandra Operations Guides hub

Cassandra Commit Log Disk Full: How To Fix It

Troubleshoot and recover when Cassandra commitlog segment exhaustion forces memtable flushes, overwhelms the write path, and blocks mutations.

Cassandra commitlog pending tasks: write-path I/O pressure

Diagnose and resolve sustained CommitLog PendingTasks in Apache Cassandra, where commitlog fsync bottlenecks delay write acknowledgments and cascade into dropped mutations.

Cassandra compaction death spiral: when writes outrun compaction throughput

Diagnose and recover from Cassandra compaction backlog when write rates exceed compaction throughput, causing SSTable accumulation, read amplification, and disk saturation.

Cassandra compaction strategies: STCS vs LCS vs TWCS vs UCS

Choose and operate the right Cassandra compaction strategy by understanding the I/O, space, and read amplification tradeoffs between STCS, LCS, TWCS, and UCS.

Cassandra disk space exhaustion: emergency recovery when the data volume fills

Emergency runbook for recovering a Cassandra node when disk space is exhausted, compaction stalls, and writes are rejected.

Cassandra hints accumulating: hinted handoff backlog and replay storms

Diagnose and resolve Cassandra hinted handoff backlog and replay storms before coordinator disk exhaustion and cascading replica failures.

Ceph Operations Guides 1 page

Open the Ceph Operations Guides hub

Ceph OSD down: telling a dead disk apart from a network blip

Triage a Ceph OSD marked down by separating disk failure from network loss, OOM-kill, and daemon crashes before recovery starts.

ClickHouse Operations Guides 3 pages

Open the ClickHouse Operations Guides hub

ClickHouse disk space collapse: why merges need free space and how the spiral starts

Why ClickHouse background merges need temporary disk headroom and how crossing the 85-90% threshold triggers a self-reinforcing spiral of stalled merges, accumulating parts, and accelerating consumption.

ClickHouse disk space monitoring: free_space, unreserved_space, and the 80% target

Understand ClickHouse disk space as an operational dependency, not just capacity. Interpret system.disks metrics, maintain merge headroom, and avoid the disk space death spiral.

ClickHouse No space left on device: emergency recovery when the data disk fills

Emergency recovery procedures and root-cause analysis for ClickHouse servers that hit disk-full conditions, including safe space reclamation and merge death-spiral prevention.

CockroachDB Operations Guides 2 pages

Open the CockroachDB Operations Guides hub

CockroachDB disk stall detected: storage_disk_stalled and node self-termination

Diagnose and resolve CockroachDB storage_disk_stalled alerts before the node self-terminates to prevent data inconsistency.

CockroachDB store is full: emergency disk recovery and why deletes don't free space

Emergency recovery procedures when CockroachDB stores run out of disk space, and why SQL DELETE operations do not immediately reclaim space.

Docker Operations Guides 6 pages

Open the Docker Operations Guides hub

Docker disk space full: how to troubleshoot /var/lib/docker

Diagnose and resolve Docker disk exhaustion on /var/lib/docker by identifying dominant space consumers across images, containers, volumes, build cache, and logs.

Docker image cleanup: safe pruning strategies for production hosts

Safe commands and filters to reclaim Docker disk space without stopping running containers or deleting data you still need.

Docker log rotation: preventing json-file logs from filling disk

Configure Docker json-file log rotation with daemon.json and per-container log-opt overrides to prevent disk exhaustion and daemon instability.

Docker logs taking too much disk space: how to fix log growth

Identify, safely truncate, and prevent unbounded Docker container log growth before it fills your disk and crashes the daemon.

Docker volume cleanup: finding and removing orphaned volumes

Find and safely remove orphaned Docker volumes that waste disk space and complicate operations.

Reading docker system df: where Docker disk usage actually lives

Interpret docker system df output accurately, find hidden disk consumers like BuildKit cache and container logs, and fix capacity issues before they cascade.

Elasticsearch Operations Guides 6 pages

Open the Elasticsearch Operations Guides hub

Elasticsearch cluster health yellow: unassigned replicas vs real allocation blocks

Distinguish benign yellow cluster states from real allocation blockers. Diagnose unassigned replicas, disk watermarks, and shard allocation failures in Elasticsearch.

Elasticsearch cluster_block_exception: blocked by, the read-only blocks explained

Map Elasticsearch cluster_block_exception codes to their root causes and clear read-only blocks on indices or the cluster.

Elasticsearch disk full: emergency recovery and freeing space safely

Emergency recovery steps when Elasticsearch data nodes exhaust disk space, including safe deletion order, replica reduction, and clearing flood-stage read-only blocks.

Elasticsearch disk watermark cascade: from low watermark to cluster-wide read-only

Operational guide to diagnosing and recovering from the Elasticsearch disk watermark cascade: low, high, and flood stage thresholds, read-only index blocks, and relocation storms.

Elasticsearch FORBIDDEN/12/index read-only / allow delete (api) — flood stage recovery

Recover from Elasticsearch cluster_block_exception when flood-stage disk watermark sets index.blocks.read_only_allow_delete, blocking writes.

Elasticsearch merge storms: segment explosion, I/O saturation, and refresh tuning

Diagnose and fix Elasticsearch merge storms caused by segment explosion, I/O saturation, and aggressive refresh intervals.

Kafka Operations Guides 5 pages

Open the Kafka Operations Guides hub

Kafka __consumer_offsets growing huge: compaction failure on the offsets topic

Diagnose and fix unbounded growth of Kafka's __consumer_offsets topic caused by a stalled or crashed log cleaner thread.

Kafka Broker Out Of Disk: How To Fix It

Diagnose and recover when a Kafka broker exhausts disk space on its log directories, including the cliff-edge shutdown behavior, uneven directory fill, and silent compaction failures.

Kafka disk I/O latency high: await, LocalTimeMs, and the slow-disk broker

Diagnose elevated disk I/O latency on Kafka brokers using iostat await and LocalTimeMs. Distinguish disk degradation from page cache misses and transient load.

Kafka log compaction falling behind: the dead cleaner thread and unbounded disk growth

Diagnose and fix silent Kafka log cleaner thread crashes that cause compacted topics like __consumer_offsets to grow without bound.

Kafka Log directory failed / OfflineLogDirectoryCount > 0: disk errors and JBOD recovery

Troubleshoot Kafka brokers with offline log directories due to disk I/O errors, JBOD failures, and recover partition availability without data loss.

Kubernetes Operations Guides 3 pages

Open the Kubernetes Operations Guides hub

Kubernetes API Server etcd Latency: How To Fix It

Detect elevated etcd latency in Kubernetes before it cascades into control plane saturation, 429 rejections, and cluster-wide write failures.

Kubernetes node DiskPressure: detection, eviction, and recovery

Detect disk pressure on Kubernetes nodes before evictions start. Understand nodefs, imagefs, inode exhaustion, and recovery steps for production clusters.

Kubernetes pod Evicted: detection, root cause, and prevention

Detect, diagnose, and prevent Kubernetes pod eviction caused by node memory, disk, inode, and PID pressure.

MongoDB Operations Guides 3 pages

Open the MongoDB Operations Guides hub

MongoDB disk full: emergency recovery when mongod can't write the journal

Emergency triage and recovery steps when MongoDB exhausts disk space and the journal cannot write, including WiredTiger space reclamation and recovery procedures.

MongoDB disk I/O saturation: correlating iostat with WiredTiger signals

Diagnose MongoDB storage bottlenecks by correlating OS disk saturation signals with WiredTiger checkpoint, journal, and cache metrics.

MongoDB journal sync latency high: the storage signal that warns 60 seconds early

Detect, diagnose, and fix high WiredTiger journal sync latency in MongoDB before it stalls j:true and w:majority writes.

MySQL Operations Guides 3 pages

Open the MySQL Operations Guides hub

MySQL Binary Log Disk Full: How To Fix It

Diagnose and fix MySQL binary log disk exhaustion caused by missing expiry, lagging replicas, or oversized transactions. Safe manual purge procedures and prevention.

MySQL Got error 28 from storage engine / No space left on device — recovery

Recover from MySQL errno 28 disk-full conditions by identifying whether binlogs, redo logs, tmpdir, or relay logs consumed the space, and apply the right fix without causing data loss.

MySQL slow commits with idle CPU: redo and binlog fsync pressure

Diagnose and fix MySQL commit latency when CPU is idle but transactions stall on redo log and binary log fsync pressure.

NGINX Operations Guides 1 page

Open the NGINX Operations Guides hub

NGINX proxy buffer spill to disk: proxy_buffers and temp file latency

Large upstream responses exceeding proxy_buffers spill to temporary files on disk, causing silent latency spikes with no error log entry.

PostgreSQL Operations Guides 3 pages

Open the PostgreSQL Operations Guides hub

PostgreSQL disk full: emergency recovery and root cause analysis

Emergency runbook for PostgreSQL disk-full incidents. Diagnose WAL accumulation, replication slots, temp files, and table bloat without guessing.

PostgreSQL pg_wal directory full: causes and emergency recovery

Diagnose unbounded WAL growth in PostgreSQL and recover safely when pg_wal fills the disk, including archive_command failures, replication slot retention, and emergency cleanup.

PostgreSQL replication slot bloat: when a stale slot fills the disk

Detect when a stale or orphaned PostgreSQL replication slot retains WAL and fills disk, diagnose the root cause, and recover safely.

RabbitMQ Operations Guides 2 pages

Open the RabbitMQ Operations Guides hub

RabbitMQ disk free limit alarm: free disk space insufficient and publishing halted

Diagnose and recover from the RabbitMQ disk_free_alarm that blocks all publishers cluster-wide when free disk drops below disk_free_limit.

RabbitMQ disk_free_limit at the default 50MB: the production footgun

Why RabbitMQ's default 50MB disk_free_limit halts publishing cluster-wide on any modern server, and how to set a safe absolute or relative limit with a sane alerting floor.

VMware vSphere Operations Guides 4 pages

Open the VMware vSphere Operations Guides hub

vCenter /storage/db full: vPostgres stops and the whole management plane dies

When the VCSA /storage/db partition fills, vPostgres cannot write WAL, crashes, and vpxd loses its database. Diagnose the cliff-edge failure and recover safely.

vCenter /storage/log full: the log-bomb disk death spiral

Diagnose and stop a vCenter /storage/log log bomb before it takes down vpxd and the management plane.

vSphere datastore full: 'No space left on device', paused VMs, and power-on failures

Recover from a full vSphere datastore: diagnose paused VMs, .vswp power-on failures, and 'No space left on device' errors, and find the snapshot or thin VMDK consuming the space.

vSphere datastore latency high: reading GAVG, DAVG, and KAVG

How to read vSphere's GAVG, DAVG, and KAVG counters to localize high datastore latency to the array, the VMkernel, or both.

Apache ZooKeeper Operations Guides 5 pages

Open the Apache ZooKeeper Operations Guides hub

ZooKeeper "fsync-ing the write ahead log took too long": the disk warning behind most write stalls

What the ZooKeeper fsync warning really means, why it is the most critical disk signal in any ensemble, and how to triage it before it costs you quorum.

ZooKeeper "Unable to load database on disk": corrupt snapshot on startup

Why a ZooKeeper node refuses to start with Unable to load database on disk, how to triage the corrupt file, and how to recover without losing the ensemble.

ZooKeeper autopurge not configured: snapshots and logs filling the disk over months

Diagnose and fix ZooKeeper disk exhaustion caused by disabled autopurge, with safe checks for snapshot and transaction log accumulation.

ZooKeeper dataLogDir sharing a disk with snapshots: the #1 fsync-latency footgun

Why ZooKeeper fsync latency spikes when the transaction log shares a disk with snapshots, how to confirm it, and how to move the log to a dedicated device safely.

ZooKeeper transaction log disk full: the crash with no graceful degradation

Diagnose and recover from a full ZooKeeper dataLogDir partition, including the crash-loop and silent data loss variants, plus the disk headroom rules that prevent recurrence.

Academy

Analyze & Reduce Disk I/O Bottlenecks: Top Techniques

Mastering Disk Performance for Optimal System Health

Key Observability Metrics | Infrastructure & APM Monitoring

Understanding & Implementing Essential Infrastructure Metrics For Effective Monitoring

Adaptec RAID Monitoring

Learn everything about monitoring & troubleshooting Adaptec RAID, what metrics are important to monitor and why, and how to monitor Adaptec RAID with Netdata.

DMCache Monitoring

Learn everything about monitoring & troubleshooting DMCache, what metrics are important to monitor and why, and how to monitor DMCache with Netdata.

Generic storage enclosure tool Monitoring

Learn everything about monitoring & troubleshooting Generic storage enclosure tool, what metrics are important to monitor and why, and how to monitor Generic storage enclosure tool with Netdata.

HDD Temperature Monitoring

Learn everything about monitoring & troubleshooting HDD temperature, what metrics are important to monitor and why, and how to monitor HDD temperature with Netdata.

MegaCLI MegaRAID Monitoring

Learn everything about monitoring & troubleshooting MegaCLI MegaRAID, what metrics are important to monitor and why, and how to monitor MegaCLI MegaRAID with Netdata.

NVMe Monitoring

Learn everything about monitoring & troubleshooting NVMe, what metrics are important to monitor and why, and how to monitor NVMe with Netdata.

S.M.A.R.T. attributes Monitoring

Learn about monitoring & troubleshooting S.M.A.R.T. attributes, what metrics are important and why, and how to monitor S.M.A.R.T. attributes with Netdata.

S.M.A.R.T. Monitoring

Learn everything about monitoring & troubleshooting S.M.A.R.T., what metrics are important to monitor and why, and how to monitor S.M.A.R.T. with Netdata.

StoreCLI RAID Monitoring

Learn everything about monitoring & troubleshooting StoreCLI RAID, what metrics are important to monitor and why, and how to monitor StoreCLI RAID with Netdata.