Cassandra java.lang.OutOfMemoryError: Java heap space - causes and recovery

The error java.lang.OutOfMemoryError: Java heap space in Cassandra system logs means the JVM exhausted its allocated heap and could not satisfy an allocation request. The process typically exits, or the Linux OOM killer terminates it. Gossip marks the node DOWN, clients experience timeouts, and the remaining replicas absorb the orphaned traffic.

Unlike gradual GC pressure that degrades latency over hours, a heap-space OOM is often a hard stop. The node was serving traffic, then it was not. The challenge is not recognizing the failure (the error string is unambiguous) but finding which of several competing heap consumers caused it.

Cassandra’s JVM heap holds memtables, key cache, prepared statement cache, index summaries, and intermediate query results during read merges. When any single consumer grows faster than GC can reclaim, or when a single operation allocates more than the entire free heap, the JVM throws OutOfMemoryError. Recovery requires identifying the specific consumer and either constraining it or increasing heap headroom.

What this means

When the JVM reports java.lang.OutOfMemoryError: Java heap space, the old generation has filled to its maximum (-Xmx) and GC could not free enough objects to satisfy the pending allocation. For Cassandra, this typically means one of two scenarios: the heap is undersized for the steady-state working set (caches, memtables, index summaries together exceed what the heap can hold), or a single operation allocated more than the entire free heap (a large partition read, an oversized batch, or a flood of in-flight requests).

The HeapMemoryUsage attribute on java.lang:type=Memory shows used == max at the moment of failure. In GC logs, you see back-to-back full GC cycles that reclaim almost nothing, then the OOM. The JVM exits or becomes unresponsive until the OS or a process supervisor terminates it.

After the node dies, gossip marks it DOWN after the phi accrual failure detector threshold is exceeded (default phi_convict_threshold=8, roughly 18 seconds of heartbeat absence). Hints accumulate on coordinators. If the OOM was caused by a workload pattern rather than a one-time anomaly, the traffic redistributes to other replicas, putting them at risk of the same failure.

flowchart TD
    A["Trigger: large partition read,
oversized batch, cache pressure"] --> B["Old generation fills to max heap"]
    B --> C["Full GC cannot reclaim enough memory"]
    C --> D["java.lang.OutOfMemoryError: Java heap space"]
    D --> E["JVM exits or OOM-killed"]
    E --> F["Gossip marks node DOWN"]
    F --> G["Clients see timeouts,
traffic redistributes"]
    G --> H["Surviving replicas absorb load
plus hint replay burst"]
    H --> I["Cascading heap pressure
on other nodes"]

Common causes

Quick checks

Run these on the affected node (or the closest surviving replica if the node is down). All are read-only unless noted.

# Check current heap usage and ratio
nodetool info | grep -i "Heap Memory"

# Check GC pause times and collection counts
nodetool gcstats

# Check for dropped messages (indicates overload before crash)
nodetool tpstats

# Check JVM heap configuration
ps aux | grep CassandraDaemon | tr ' ' '\n' | grep -E "Xm|UseG1GC|HeapDump"

# Check for OOM killer invocation in kernel log
dmesg | grep -i "killed process\|oom"

# Check total process RSS (compare against -Xmx to estimate off-heap footprint)
grep VmRSS /proc/$(pgrep -f CassandraDaemon)/status

# Parse GC logs for pause durations exceeding 1 second
grep -i "pause" /var/log/cassandra/gc.log* | awk '$NF > 1000'

# Check for batch warnings in system log
grep -i "Batch.*of size" /var/log/cassandra/system.log | tail -20

How to diagnose it

1. Confirm the OOM type. The error must be Java heap space, not Direct buffer memory or Metaspace. A Direct buffer memory OOM points to off-heap buffer exhaustion (see Cassandra commitlog pending tasks for related write-path pressure). A Linux OOM kill without a JVM OutOfMemoryError in the logs means off-heap memory (bloom filters, compression metadata, chunk cache, direct buffers) pushed total RSS past system limits, not heap exhaustion.

2. Check if a heap dump was captured. If -XX:+HeapDumpOnOutOfMemoryError was set in cassandra-env.sh, a .hprof file exists at the configured path. Open it with Eclipse MAT or a similar tool and sort retained size by class. The dominant consumer is usually obvious: org.apache.cassandra.db.rows objects indicate large partition reads; java.nio.HeapByteBuffer arrays indicate in-flight request buffers.

3. Correlate timing with workload events. Check whether the OOM coincided with a repair run, a bulk load, a new deployment, or a traffic spike. If sstableloader was running, it may have exhausted heap independently (it does not inherit the main Cassandra heap settings). If a repair was active, Merkle tree construction for wide partitions can consume significant heap.

4. Identify large partitions. Run nodetool tablehistograms <keyspace> <table> for suspect tables and check the partition size distribution. Partitions larger than ~100MB are problematic; partitions approaching the heap size will OOM the node when read. Also check nodetool toppartitions during normal operation to catch hot partitions early.

5. Check cache configuration. If row cache is enabled (row_cache_size > 0 in cassandra.yaml), it stores full rows on-heap and is a common OOM source. Row cache is disabled by default and should remain disabled for most workloads. Check the key cache size and prepared statement cache size against the total heap budget.

6. Analyze the heap-after-GC trend. If the node is still alive (or has been restarted), track heap usage immediately after full GC over time. A rising floor indicates either a memory leak or growing data structures. A stable floor with periodic spikes indicates transient allocation events (large reads, batches).

Metrics and signals to monitor

Fixes

Increase heap size (when the working set is genuinely too large)

Set -Xms and -Xmx to the same value in cassandra-env.sh to avoid resize cost. The recommended range for most Cassandra workloads is 8-16GB. G1GC (the default since Cassandra 5.0) handles heap well up to approximately 16GB. Beyond that, full GC pause duration becomes problematic. The practical ceiling with compressed oops is ~31GB; above that, the JVM disables compressed object references, effectively shrinking usable memory.

Tradeoff: increasing heap makes individual full GC pauses longer. A 16GB heap with G1GC can produce multi-second pauses during old-gen collection. If the OOM was caused by a transient spike rather than steady-state pressure, a larger heap buys time but does not fix the root cause.

Disable or shrink row cache

Row cache stores full partition data in memory and is invalidated on any write to the partition. It is disabled by default (row_cache_size: 0). If it was enabled, set it back to zero and restart. The default cache provider is org.apache.cassandra.cache.OHCProvider, which allocates off-heap; monitor total RSS to ensure off-heap allocation does not trigger the OS OOM killer.

Reduce batch sizes

Cassandra logs a warning when a batch exceeds the configured threshold. Logged batches that grow large enough to OOM the coordinator are a known pattern. Application changes are required: split large batches into individual statements or smaller chunks, use UNLOGGED batches only for partition-local writes, and never use batches for bulk loading (use sstableloader or bulk writes via individual mutations instead). See Cassandra Batch Too Large Warning: How To Fix It.

Fix large partition data model

If a partition is large enough to OOM the node when read, the data model needs restructuring. Cassandra merges entire partition contents in heap before returning results. A partition that has grown to hundreds of megabytes will consume that much heap on every read. Options include: bucketing the partition key (for example, adding a time component), switching to TWCS for TTL-dominated time-series data, or using paging to limit the result set per query.

Throttle in-flight requests

If the OOM was caused by a request flood (retry storm, connection pool misconfiguration, runaway analytics query), the fix is at the application or coordinator level. Reduce client-side retry aggressiveness, add rate limiting, and verify that speculative retry is not configured to ALWAYS. On the server side, ensure concurrent_reads and concurrent_writes are sized appropriately for the hardware, but do not increase them to mask an application-level problem.

Capture a heap dump for post-incident analysis

Set the following JVM flags in cassandra-env.sh to automatically capture a heap dump on the next OOM:

# JVM flags for automatic heap dump on OOM
JVM_OPTS="$JVM_OPTS -XX:+HeapDumpOnOutOfMemoryError"
JVM_OPTS="$JVM_OPTS -XX:HeapDumpPath=/var/lib/cassandra/heapdumps/"

Without these flags, once the JVM exits, the heap state is gone. You cannot retroactively capture a dump from a dead process.

Prevention

• Track heap-after-full-GC as a time series. This is the single most useful leading indicator. If the post-GC floor trends upward over days, investigate before it reaches 85%. Most monitoring setups track total heap usage, which oscillates constantly and tells you almost nothing.
• Keep row cache disabled. It is disabled by default. Do not enable it without a clear, measured reason and off-heap allocation.
• Monitor partition size growth. Use nodetool tablehistograms or nodetool toppartitions periodically. A growing max partition size is a ticking time bomb for heap.
• Page all range scans. Ensure client drivers use paging for any query that could return large result sets. Unpaged reads of large partitions are a direct path to OOM.
• Audit batch usage. Grep system logs for batch warnings. Every logged batch warning is a candidate for OOM if the batch grows further.
• Budget total memory. Ensure JVM heap plus off-heap allocations (bloom filters, compression metadata, index summaries, chunk cache, direct buffers) plus OS page cache fit within system RAM with at least 20% headroom. A heap that looks fine can still get the process OOM-killed if off-heap consumption is uncontrolled.
• Set HeapDumpOnOutOfMemoryError. Without it, you lose the evidence needed to diagnose intermittent OOMs that happen hours or days apart.

How Netdata helps

• Per-second JVM heap metrics from java.lang:type=Memory expose used, committed, and max heap with enough granularity to see the allocation spike that precedes an OOM, not just the steady-state average.
• GC pause duration per collector (G1 Young Generation, G1 Old Generation) lets you correlate long pauses with gossip DOWN events and dropped messages in the same time window.
• Dropped message rates by type (MUTATION, READ) surface the overload that often precedes heap exhaustion, giving you minutes of warning before the OOM.
• Thread pool pending and blocked tasks across MUTATION, READ, and Native-Transport-Requests stages show where requests are queuing, which helps distinguish a disk I/O bottleneck from a heap pressure bottleneck.
• Process RSS alongside heap metrics reveals the gap between JVM heap and total memory consumption, which is the gap that triggers OS OOM kills despite healthy heap readings.

Netdata’s Cassandra monitoring with Netdata brings these signals together with per-second metrics and ML anomaly detection.

Related guides

• Cassandra adding and removing nodes safely: vnodes, tokens, and cleanup
• Cassandra Batch Too Large Warning: How To Fix It
• Cassandra node stuck in joining (UJ): bootstrap diagnosis
• Cassandra compaction strategies: STCS vs LCS vs TWCS vs UCS
• Cassandra clock skew: how NTP drift silently corrupts data
• Cassandra Commit Log Disk Full: How To Fix It
• Cassandra commitlog pending tasks: write-path I/O pressure
• Cassandra compaction death spiral: when writes outrun compaction throughput
• Cassandra consistency levels explained: QUORUM, ONE, LOCAL_QUORUM, and EACH_QUORUM
• Cassandra zombie data resurrection: gc_grace_seconds and unrepaired tombstones
• Cassandra disk space exhaustion: emergency recovery when the data volume fills
• Cassandra dropped mutations: silent write loss and load shedding