Cassandra monitoring checklist: the signals every production cluster needs

This checklist maps the signals a production Apache Cassandra cluster needs at four levels of monitoring maturity: survival, operational, mature, and expert. Each level is cumulative. A cluster at “mature” that does not alert on dropped mutations or node DOWN states has a survival-level gap, not a mature-level gap.

The levels correspond to how quickly you can detect and diagnose problems. Survival signals tell you something is broken. Operational signals tell you what is degraded. Mature signals tell you why. Expert signals let you predict failures before they happen. Most teams operate at Level 1 or Level 2, learn about Level 3 after their first major incident, and only reach Level 4 after repeated outages that Level 3 should have caught.

Use this as an audit tool. Walk through each table, confirm you are collecting the signal, confirm you are alerting on the threshold, and confirm the alert routes to the right severity. If any row is missing, that is your next monitoring task. Every signal here has a concrete JMX bean or CLI command behind it.

The monitoring maturity model

Each level roughly doubles the signal count, but the diagnostic value increases non-linearly. The composite patterns at the bottom of this checklist require signals from multiple levels to detect correctly. A single missing signal can render an entire alert class blind.

flowchart TD
    L4["Level 4: Expert
partition size, tombstone density, off-heap, gossip phi, capacity projections"]
    L3["Level 3: Mature
SSTable count, thread pools, hints, repair tracking, cache hits, FD usage, tombstones"]
    L2["Level 2: Operational
latency P99, timeouts, unavailables, dropped msgs, GC pauses, compaction pending"]
    L1["Level 1: Survival
node UN, native transport, disk space, dropped mutations, storage exceptions"]

    L1 --> L2 --> L3 --> L4

Level 1: survival

These signals answer one question: is the cluster accepting reads and writes right now? If any of these fire, you have an active outage or are minutes away from one.

Node liveness is the gossip-based phi accrual failure detector. Each node independently determines peer state, so run nodetool status on multiple nodes during a suspected partition. GC pauses longer than approximately 18 seconds (with default phi_convict_threshold=8) cause transient DOWN marking. If a node is flapping (more than 3 UP/DOWN transitions in 30 minutes), the root cause is almost always heap pressure, not network.

Dropped mutations mean the coordinator accepted a write but the replica silently discarded it because the message sat in its internal queue past the timeout. The counter is cumulative since process start. Alert on the rate of change, not the absolute value. Any non-zero rate in a healthy cluster is abnormal.

Storage exceptions are non-negotiable. Any non-zero value indicates disk failure, filesystem corruption, or SSTable corruption. Depending on disk_failure_policy, a storage exception can shut down gossip and native transport on the entire node.

Level 2: operational

These signals tell you the cluster is degraded before it breaks. A team operating at Level 2 catches the GC death spiral and compaction backlog before they become outages.

Unavailables versus timeouts: do not lump them into one “errors” metric. UnavailableException means the topology cannot satisfy the consistency level. Not enough replicas are alive. It fails immediately, no waiting. TimeoutException means replicas are alive but too slow to respond within the configured timeout window. Different root causes, different responses.

Heap usage: monitor the floor, not the oscillation. Heap swings between 40% and 80% constantly as objects are allocated and collected. Track heap used immediately after an old GC. If that floor trends upward over days, you have a memory leak or growing resident data structures (caches, bloom filter metadata, oversized partitions being read).

Pending compactions: alert on the trend, not the absolute. A stable count of 25 pending tasks may be normal for the workload. A count rising from 15 to 25 over a week means compaction is losing ground. Read latency will degrade as SSTables accumulate, but the lag is days, not minutes. By the time latency spikes, the backlog is severe.

Level 3: mature

These signals provide the “why” behind Level 2 symptoms. A team at Level 3 can distinguish a commitlog I/O bottleneck from a compaction throughput problem from a tombstone overread without guessing.

Repair tracking is the single most dangerous gap in Cassandra monitoring. If repair has not completed for a table within gc_grace_seconds (default 10 days), tombstones may be garbage-collected on some replicas while others still hold the original data. The deleted data reappears silently. There is no built-in alert for this. You must build it yourself. Target completing a full repair cycle within 50% of gc_grace_seconds (5 days at default) to leave a safety margin.

Thread pool saturation: the GOSSIP internal pool backing up is extremely serious. It means gossip is falling behind, which leads to false DOWN marking across the cluster. If you see pending tasks in the GOSSIP stage, investigate immediately. For request pools (MUTATION, READ), pending > 0 sustained means the node cannot accept work fast enough. Blocked tasks (the queue itself is full) means work is being rejected.

Disk I/O: keep commitlog and data on separate devices and monitor them independently. Commitlog device await > 10ms sustained on SSD is a PAGE-level problem because every write must wait for commitlog sync before acknowledgment.

Level 4: expert

These are the leading indicators and deep-dive signals that experienced operators add after their second or third major incident. They predict failures rather than react to them.

Off-heap memory: bloom filters (off-heap since 3.x), compression metadata, index summaries, chunk cache (4.0+), and Netty direct buffers all consume memory outside the JVM heap. A node can have heap at 60% while total RSS approaches system RAM. The Linux OOM killer strikes and operators cannot understand why. Track RSS minus heap as a first-class metric and alert when total RSS exceeds 80% of system RAM.

LWT metrics: lightweight transactions use Paxos (Paxos v2 is available in 4.1+ with paxos_variant: v2), adding 4 round-trips of latency. If LWT and normal operations share the same latency metrics, LWT tail latency is invisible. Monitor CASRead and CASWrite scopes separately.

Composite alerting patterns

The strongest alerting signals combine multiple metrics into composite conditions. Individual thresholds produce false positives during cold starts, repairs, and bulk loads. Composite patterns confirm active failures by requiring corroboration.

These patterns are why per-second, per-node correlation matters. The GC death spiral requires 4 signals to converge within a 10-minute window. A monitoring system that polls each metric independently every 60 seconds and evaluates each threshold in isolation will miss the pattern entirely.

Common monitoring gaps

1. Repair not monitored. The most dangerous and most common gap. Everything looks fine for months. Then gc_grace_seconds passes, tombstones are compacted away on some replicas, and deleted data resurrects. Alert when any table’s last successful repair exceeds 80% of its gc_grace_seconds.

2. Total heap instead of post-GC heap. Heap usage oscillates between 40% and 80% constantly. Track heap used immediately after an old GC. If that floor trends upward, you have a real problem. Most teams only notice when full GC pauses start.

3. Average latency instead of percentiles. A single large-partition read produces a 10-second outlier while 99% of reads complete in 2ms. The average looks mildly elevated. Always alert on P99 and investigate P999.

4. Compaction pending as a snapshot. A static “25 pending” is meaningless without trend context. Compaction pending increasing over 24 hours is a leading indicator of read degradation that will take days to become critical.

5. Off-heap memory ignored. Bloom filters, compression metadata, index summaries, chunk cache, and Netty buffers live off-heap. JVM heap looks healthy while total RSS approaches system RAM. The OOM killer strikes and nobody understands why.

6. Timeouts and unavailables conflated. Timeout means replicas are alive but slow. Unavailable means not enough replicas are alive. Different causes, different responses, different severity. Monitor and alert on them separately.

7. No per-node comparison. Cluster-aggregated metrics hide the one node with GC issues, disk degradation, or a hot partition. Every alert should fire per-node. Flag any node that deviates more than 2x from the cluster median on latency, dropped messages, GC pause, or compaction pending.

How Netdata helps

• Per-second granularity. Cassandra’s JMX metrics expose decaying reservoirs that smooth over rapid changes. Per-second collection catches the transient GC pause, the burst of dropped messages, and the gossip flap that minute-level polling misses entirely.
• Per-node decomposition. Correlating GC pause duration, heap usage, and dropped messages across individual nodes makes the outlier obvious within seconds, rather than buried in a cluster average.
• Composite pattern detection. The GC death spiral, compaction death spiral, and quorum loss patterns each require 3 to 5 signals to correlate. ML anomaly detection flags the co-occurrence of GC pauses, gossip state changes, and dropped mutations without requiring a custom multi-condition alert rule for each pattern.
• JVM and OS signals together. Off-heap memory (RSS minus heap), disk I/O per device, file descriptor counts, and JVM GC metrics all matter for Cassandra. Collecting them in one place lets you see that the OOM kill happened because off-heap grew, or that read latency spiked because compaction saturated the data device.
• Relationship-based alerting. Instead of fixed latency thresholds that produce false positives on cold starts and miss slow degradation, Netdata baselines each node’s normal behavior and alerts on sustained deviation from that baseline.

Netdata’s Cassandra monitoring with Netdata brings these signals together with per-second metrics and ML anomaly detection.

Related guides

• Cassandra adding and removing nodes safely: vnodes, tokens, and cleanup
• Cassandra Batch Too Large Warning: How To Fix It
• Cassandra node stuck in joining (UJ): bootstrap diagnosis
• Cassandra compaction strategies: STCS vs LCS vs TWCS vs UCS
• Cassandra clock skew: how NTP drift silently corrupts data
• Cassandra Commit Log Disk Full: How To Fix It
• Cassandra commitlog pending tasks: write-path I/O pressure
• Cassandra compaction death spiral: when writes outrun compaction throughput
• Cassandra consistency levels explained: QUORUM, ONE, LOCAL_QUORUM, and EACH_QUORUM
• Cassandra zombie data resurrection: gc_grace_seconds and unrepaired tombstones
• Cassandra disk space exhaustion: emergency recovery when the data volume fills
• Cassandra dropped mutations: silent write loss and load shedding