$ guides / clickhouse / clickhouse-no-space-left-on-device ▌

Operations Guides

ClickHouse No space left on device: emergency recovery when the data disk fills

When ClickHouse returns No space left on device to client inserts or the server log fills with write errors, the situation is past a simple capacity alert. ClickHouse does not degrade gradually on a full disk. Background merges halt immediately because they require temporary free space to write combined parts before removing source files. Once merges stop, small insert parts accumulate, metadata overhead grows, and disk usage accelerates. TTL-based expiration also stops because TTL cleanup is executed by merges. ClickHouse system tables such as system.query_log and system.part_log are MergeTree tables that can grow unbounded and consume the remaining space.

Recovery requires manual intervention. The goal is to reclaim enough space for merges to resume, identify what consumed the disk, and fix the root cause before the next cycle. Do not restart the server as a first step. A restart with a full disk can prevent ClickHouse from starting if it needs to write system tables or temporary files during initialization.

flowchart TD
    A[Disk > 85% full] --> B[Merges cannot allocate temp space]
    B --> C[Merges halt]
    C --> D[Small parts accumulate]
    D --> E[Metadata and indexes grow]
    E --> F[TTL cleanup stops]
    F --> G[Disk usage accelerates]
    G --> H[Inserts fail with No space left]

What this means

ClickHouse stores data in immutable parts. The merge engine continuously combines smaller parts into larger ones to maintain query performance and control file descriptor usage. A merge reads all source part files and writes a new merged part to the same disk before deleting the sources. A single merge temporarily requires space for both source and output parts.

When the disk crosses the threshold where the largest potential merge cannot be completed, ClickHouse stops scheduling new merges. Existing merges may stall or fail. Without merges, every insert creates a new part that persists indefinitely. The part count rises, increasing files, index entries, and memory structures. This metadata growth itself consumes additional disk space.

TTL relies on merges to remove expired rows, so old data stays on disk. If the volume also hosts ZooKeeper transaction logs, Keeper snapshots, or application logs, those can tip the system from “nearly full” to “hard stop.” Inserts fail, read queries may fail if they require temporary space, and the server can crash if it cannot write logs or system tables.

Common causes

Cause	What it looks like	First thing to check
Ingestion outpacing TTL or retention	Disk usage grows steadily day over day; old monthly partitions are still present and large.	`system.parts` for partition ages and sizes.
Merge death spiral from prior pressure	Disk is above 85%, active parts per partition are climbing, and `system.merges` is empty.	Merge activity and part count trends together.
Mutation backlog creating temporary copies	Disk usage is high despite flat ingestion; `system.mutations` shows long-running or stuck jobs.	Pending mutations with `is_done = 0`.
System tables growing unbounded	`system.query_log` or `system.part_log` consumes tens or hundreds of gigabytes.	`bytes_on_disk` grouped by database and table, including system tables.
Detached parts not cleaned up	Used disk space exceeds the sum of active parts; old detached directories linger.	`system.detached_parts` or the `detached/` directories on disk.
Non-ClickHouse files on the data volume	`df` shows less free space than `system.disks.free_space` suggests.	OS-level disk usage outside of ClickHouse data paths.

Quick checks

Run these read-only queries and commands in order.

# OS-level disk usage on the data mount
df -h /var/lib/clickhouse

-- ClickHouse's view of disk space and reservation
SELECT
    name,
    path,
    formatReadableSize(free_space) AS free,
    formatReadableSize(total_space) AS total,
    round(100 * (1 - free_space / total_space), 1) AS used_pct,
    formatReadableSize(unreserved_space) AS unreserved
FROM system.disks;

-- Largest partitions by on-disk size
SELECT
    database,
    table,
    partition_id,
    count() AS active_parts,
    sum(rows) AS total_rows,
    formatReadableSize(sum(bytes_on_disk)) AS size
FROM system.parts
WHERE active = 1
GROUP BY database, table, partition_id
ORDER BY size DESC
LIMIT 20;

-- Are merges running, or is the pipeline frozen?
SELECT
    count(*) AS active_merges,
    countIf(is_mutation = 1) AS mutations,
    countIf(is_mutation = 0) AS regular_merges
FROM system.merges;

-- Pending mutations that could be blocking merges and consuming space
SELECT
    database,
    table,
    mutation_id,
    command,
    create_time,
    is_done,
    parts_to_do
FROM system.mutations
WHERE is_done = 0
ORDER BY create_time;

-- Space consumed by system tables themselves
SELECT
    database,
    table,
    formatReadableSize(sum(bytes_on_disk)) AS disk_size,
    count() AS parts
FROM system.parts
WHERE active = 1
    AND database = 'system'
GROUP BY database, table
ORDER BY disk_size DESC
LIMIT 10;

-- Detached parts that still occupy disk but are invisible to queries
SELECT
    database,
    table,
    name,
    reason,
    formatReadableSize(bytes_on_disk) AS size,
    modification_time
FROM system.detached_parts
ORDER BY bytes_on_disk DESC
LIMIT 20;

-- Confirm whether inserts are being delayed or rejected
SELECT event, value
FROM system.events
WHERE event IN ('DelayedInserts', 'RejectedInserts');

How to diagnose it

Quantify the gap. Compare OS df output against system.disks. If the OS shows significantly less free space than ClickHouse reports, look for non-ClickHouse files and filesystem overhead; detached ClickHouse parts are on the same filesystem and are normally included in both views. Check for core dumps, package caches, or oversized logs.
Find the largest consumers. Rank partitions by bytes_on_disk using the query above. Focus on the top three. In most emergencies, one or two large tables or old partitions dominate.
Determine if merges are blocked. If system.merges returns zero rows while active parts are high, the merge pipeline is stalled. Check system.mutations next. Pending mutations can hold merge threads and temporarily duplicate parts.
Check system table bloat. If system.query_log, system.part_log, or system.text_log rank high in the size query, the monitoring instrumentation itself is contributing to the emergency. This is common on high-QPS clusters where TTL was never configured for log tables.
Assess detached parts. system.detached_parts shows parts that were removed from active service but not deleted. They continue to consume space. Large counts here usually follow failed ALTER operations, replication fetch failures, or manual DETACH commands that were never cleaned up.
Verify TTL health. If TTL is configured but expired data remains, confirm merges are running. TTL only drops data during merges. A full disk blocks merges, which blocks TTL, which prevents space reclamation.

Metrics and signals to monitor

Signal	Why it matters	Warning sign
Disk free space	Merges need temporary headroom equal to the largest source parts.	Free space below 20% or `unreserved_space` approaching zero.
Active part count per partition	Parts grow when merges halt, accelerating metadata overhead.	Count rising while merge activity is zero.
Merge activity	Confirms the background pipeline is actually making progress.	No entries in `system.merges` for more than 10 minutes during active inserts.
Mutation queue depth	Mutations block merges and temporarily double part sizes.	Any `is_done = 0` mutation with `parts_to_do` not decreasing over time.
System table sizes	`query_log` and `part_log` are MergeTree tables that can fill the disk.	Any system table larger than 10% of total data size.
Insert delays and rejections	Early indicators that the part count or disk pressure threshold is crossed.	`DelayedInserts` or `RejectedInserts` counters increasing.

Fixes

Immediate stabilization: stop the bleeding

Stop or throttle ingestors before deleting anything. Each insert creates a new part that adds metadata overhead. If replication is active, pause non-essential distributed sends and DDL operations. Do not restart ClickHouse yet; a restart with zero free space can fail during initialization.

Reclaim space from old partitions

Detach old partitions to take them out of the active set without deleting the files immediately. Detached partitions are no longer visible to queries, but they remain on disk in detached/; the space is reclaimed only when those files are deleted. Keep the quarantined directories if you may reattach them later.

-- Detach a specific old partition (safe, reversible)
ALTER TABLE db.table DETACH PARTITION ID '202401';

To permanently delete a partition and immediately reclaim space, use DROP PARTITION. This is destructive and irreversible.

-- Permanently drop a partition (destructive)
ALTER TABLE db.table DROP PARTITION ID '202401';

After deleting the detached directories, or dropping the partition, run the system.disks check again to confirm space was freed. Detaching alone does not free disk bytes.

Unblock system tables

If system.query_log, system.part_log, or similar tables are consuming significant space, truncate them for immediate relief. This destroys log history, so extract any needed forensic data first.

-- Truncate a system log table (destructive to history)
TRUNCATE TABLE system.query_log;

For long-term prevention, configure TTL on these tables in the server configuration so they do not grow unbounded. The change requires a server restart to apply.

Clean up detached parts

If detached parts from past operations are consuming space and are no longer needed, remove them from the detached/ directory. This is destructive; the data cannot be recovered without restoring from backup. Paths vary by database engine and configuration; verify the exact location before running destructive commands.

# Remove detached parts for a specific table (destructive)
rm -rf /var/lib/clickhouse/data/db/table/detached/*

Prefer querying system.detached_parts first to document what will be removed.

Cancel mutations that are doubling disk usage

If a long-running mutation is holding merge threads and creating temporary part copies, kill it. The mutation must be reissued later, but cancellation frees merge capacity and stops further temporary copying; already-applied work is not rolled back.

-- Kill a specific mutation
KILL MUTATION WHERE database = 'db' AND table = 'table' AND mutation_id = '0000000000';

Restore merge activity

Once free space is above the merge threshold, verify that merges resume:

-- Check for new merge activity
SELECT * FROM system.merges LIMIT 5;

Monitor system.parts over the next 10 to 30 minutes. Active part counts should begin trending downward. If parts do not decrease, investigate whether the background pool is saturated or whether I/O latency is preventing merge progress.

Structural fixes

If disk expansion is the only option, add space to the underlying volume or move older tables to tiered storage. Do not rely on TTL alone to save you in the immediate emergency; TTL requires merges, and merges require free space.

Prevention

Maintain merge headroom. Keep disk usage below 80% as an operational target. The safety margin must be at least 2x the size of the largest active part to allow a full merge to complete.
Configure TTL on system tables. Set retention policies for system.query_log, system.part_log, system.text_log, and system.trace_log. On high-traffic clusters, these can outgrow user data.
Monitor the merge-to-insert ratio. Track part creation rate against merge completion rate. If parts are created faster than they are merged, part count and disk usage will grow until they trigger a crisis.
Audit partitioning strategy. Overly granular partitioning (for example, by hour) multiplies part counts and metadata overhead. Daily or monthly partitioning is usually more space-efficient.
Clean detached parts proactively. Include system.detached_parts in routine checks. Detached parts left after incidents or failed alters silently consume space indefinitely.
Set disk utilization alerts below the danger zone. Page when disk usage exceeds 80%, not at 95%. The extra runway is needed for merge temp space.

How Netdata helps

Real-time correlation of disk utilization, merge activity, and active part count exposes the merge death spiral before inserts fail.
Per-partition part counts and growth rates predict disk pressure hours before the filesystem fills.
DelayedInserts and RejectedInserts event counters alert on write-pipeline backpressure.
System table size monitoring alongside user data catches cases where query_log or part_log become the primary disk consumer.
Background pool utilization and mutation queue depth distinguish disk-full causes from merge thread starvation.

The Netdata solution

ClickHouse monitoring with Netdata

Netdata monitors ClickHouse with per-second metrics and ML anomaly detection. Track merge debt, memory usage, replication lag, Keeper/ZooKeeper saturation, and disk headroom against the host signals that drive them.

See ClickHouse monitoring → Start monitoring free