Cassandra dropped mutations: silent write loss and load shedding

Your application logs successful writes, but reads return stale or missing data. An alert fires on DroppedMessage rate for MUTATION scope. The client never received an error, yet a replica discarded the write after it sat in the MutationStage queue past timeout. This is Cassandra load shedding. Silent write loss occurs whenever not enough other replicas succeed to meet the consistency level.

Dropped mutations are a lagging indicator. By the time they appear, the replica is already choking on commitlog I/O, CPU starvation, GC pauses, or thread pool exhaustion. The DroppedMessage counter is cumulative since JVM start; a single large value means nothing unless the rate is increasing. Any sustained non-zero rate is abnormal and demands immediate investigation.

What this means

When a coordinator forwards a write to a replica, the replica places the mutation into the MutationStage queue. A worker thread eventually picks it up, appends it to the commitlog, inserts it into the memtable, and sends an ACK. If the queue is backed up because the node cannot process mutations fast enough, the mutation ages past write_request_timeout_in_ms (default 2000 ms) and is silently discarded.

The coordinator may have already ACKed the write to the client if enough other replicas responded. The drop is invisible to the application. The missing data on that replica creates an inconsistency that will not self-heal unless the key is later touched by read repair or by an anti-entropy repair run within gc_grace_seconds. Hinted handoff does not protect against this: hints are generated only when a replica is marked DOWN by gossip, not when it is UP but overloaded and dropping mutations.

flowchart TD
    A[Coordinator forwards write] --> B[Replica MutationStage queue]
    B --> C[Thread processes mutation]
    C --> D[Commitlog append + memtable insert]
    D --> E[ACK to coordinator]
    F[Commitlog disk slow] --> G[Queue backlog]
    H[GC pause] --> G
    I[CPU saturation] --> G
    G --> J[Mutation exceeds timeout]
    J --> K[MUTATION silently dropped]
    K --> L[Data loss on replica]

Common causes

Quick checks

Run these safe, read-only commands to triage the scope of the drops and identify the saturated resource.

# Check dropped mutation counts and thread pool state
nodetool tpstats

# Check commitlog and data disk saturation
iostat -x 1

# Check compaction debt
nodetool compactionstats

# Check heap usage and pressure
nodetool info | grep "Heap Memory"

# Check for long GC pauses in recent logs
grep -iE "pause|stopped" /var/log/cassandra/gc.log | tail -20

# Check coordinator write latency
nodetool proxyhistograms

# Check node liveness and cluster view
nodetool status

# Check disk space on commitlog and data volumes
df -h /var/lib/cassandra/commitlog /var/lib/cassandra/data

How to diagnose it

1. Confirm a sustained drop rate. Run nodetool tpstats and note the Dropped section. The counters are cumulative. Sample the value, wait 60 seconds, and sample again. Any increase in the MUTATION row is a problem.
2. Inspect the MutationStage queue. In nodetool tpstats, look for Pending > 0 or Blocked > 0 under the request stages. Sustained pending means the write path cannot keep up; blocked means the queue is full and work is being rejected.
3. Check commitlog disk latency. Use iostat -x on the commitlog device. If await is elevated or %util is near 100%, fsync stalls are backing up the write pipeline. If commitlog and data share a device, separation is strongly recommended.
4. Check GC health. Parse GC logs for stop-the-world pauses. Pauses > 500 ms freeze all stage processing and cause queued mutations to expire. If heap after full GC > 75% of max, memory pressure is the root cause.
5. Check compaction status. Run nodetool compactionstats. If pending tasks are trending upward over hours, compaction is stealing I/O bandwidth from commitlog writes and read processing.
6. Check for asymmetric patterns. If only one rack or node shows drops, inspect that specific host for hardware degradation, uneven traffic, or a hot partition. Use nodetool tablehistograms to see if a single table dominates write latency.
7. Verify hinted handoff is not masking the issue. Hints are stored when a replica is marked DOWN by gossip. An overloaded node that remains UP will not receive hints for dropped mutations, so do not rely on hint replay to backfill lost data.

Metrics and signals to monitor

Fixes

Fix the root cause; do not restart Cassandra as a first response. Restarting clears the backlog temporarily but destroys the diagnostic state and does not prevent immediate recurrence.

Commitlog I/O saturation

If the commitlog device is saturated, move commitlog to a dedicated volume. This requires updating cassandra.yaml and restarting the node, so plan for maintenance. As a temporary relief, reduce write throughput from clients or batch jobs. If you are using commitlog_sync: batch, which fsyncs before ACK, switching to periodic trades durability (up to commitlog_sync_period_in_ms, default 10000 ms) for lower latency. Understand the data-loss implications before changing this.

MutationStage saturation from CPU pressure

Thread pool saturation is usually a symptom, not a root cause. Identify what is starving CPU. If compaction is the consumer, you can temporarily increase compaction_throughput_mb_per_sec to clear debt faster, or decrease it to throttle compaction and leave headroom for writes. If GC is consuming CPU, see the Cassandra GC death spiral guide. Identify large partitions, tombstone scans, or misconfigured caches rather than blindly resizing pools.

Disk I/O contention from compaction backlog

Increase compaction_throughput_mb_per_sec to let compaction catch up, but monitor read latency because this adds I/O load. Postpone repairs, bootstraps, and streaming that compete for disk. If you use SizeTieredCompactionStrategy and disk usage is above 50%, you are at risk of space exhaustion during a major compaction. Clear old snapshots and add capacity before compaction can recover.

Memory pressure and GC pauses

Reduce in-flight writes by throttling clients at the application layer. Check for large batch statements or unbounded partitions that allocate massive objects on heap. If heap after full GC > 85%, a rolling restart with increased heap may be needed. Do not exceed roughly 16 GB with G1GC, or pause times will worsen.

Prevention

• Alert on the rate of DroppedMessage, not the cumulative count. Counters reset on JVM restart, so rate is the only actionable signal.
• Keep commitlog on a dedicated disk, separate from data directories.
• Watch MutationStage pending tasks as a leading indicator. Any sustained pending predicts future drops.
• Monitor compaction pending trends, not just absolute values. A rising trend means I/O debt is accumulating.
• Keep JVM heap after full GC below 75% of max. Track this from GC logs, not just HeapMemoryUsage.
• Run anti-entropy repair within gc_grace_seconds so that any inconsistencies from transient drops are eventually reconciled.

How Netdata helps

Netdata collects DroppedMessage rates and MutationStage pending tasks from JMX and places them on the same timeline as disk I/O latency and GC pauses. This correlation shows whether drops lag a commitlog stall or a GC spike by seconds.

Per-disk await and utilization metrics for the commitlog volume let you distinguish disk saturation from CPU saturation without logging into the node.

Process RSS tracking surfaces off-heap memory pressure that JVM heap metrics miss, catching OOM-killer risk before it triggers.

Composite alerting on drops plus thread pool saturation plus GC pauses reduces false positives from single-metric blips.

Related guides

• Cassandra compaction strategies: STCS vs LCS vs TWCS vs UCS
• Cassandra compaction death spiral: when writes outrun compaction throughput
• Cassandra consistency levels explained: QUORUM, ONE, LOCAL_QUORUM, and EACH_QUORUM
• Cassandra zombie data resurrection: gc_grace_seconds and unrepaired tombstones
• Cassandra disk space exhaustion: emergency recovery when the data volume fills
• Cassandra GC death spiral: long pauses, gossip flapping, and recovery
• Cassandra GC pauses too long: diagnosing G1 stop-the-world pauses
• Cassandra heap pressure: sizing the JVM heap and tuning G1GC
• Cassandra monitoring checklist: the signals every production cluster needs
• Cassandra monitoring maturity model: from survival to expert
• Cassandra Not enough space for compaction: STCS space amplification and recovery
• Cassandra java.lang.OutOfMemoryError: Java heap space - causes and recovery