Cassandra streaming failures: stalled bootstrap, decommission, and rebuild

Bootstrapping stuck in JOINING for six hours. Decommission streaming with zero byte progress after eight hours. Rebuild failed, leaving incomplete token ranges. These are streaming failures.

Streaming moves SSTables between nodes during topology changes and repair. When a session fails, the topology is left incomplete. When it stalls with no progress for more than thirty minutes, the node stays in a transitional state and may not handle traffic correctly. Root causes are usually network timeouts, source node bottlenecks, or configuration mismatches. Unlike client request timeouts, streaming failures do not always surface as explicit errors; a session can hang silently while the control channel stays open.

What this means

Cassandra uses streaming to transfer SSTable data during bootstrap, decommission, rebuild, and repair. In Cassandra 4.0+, Zero Copy Streaming (ZCS) transfers entire SSTable files directly off disk, bypassing object reification and reducing CPU overhead versus the partition-based path. A streaming session is a long-lived TCP connection with a control channel and one or more data channels.

A failed session is detected when the control channel closes or two consecutive keep-alive cycles fail. A stalled session is different: the connection is alive, but bytes stop moving. If no progress occurs for more than thirty minutes, the operation is frozen. The node remains in a transitional state (JOINING, LEAVING, or REBUILDING) and may not accept or redirect traffic correctly. There is no automatic abort timeout for a stalled bootstrap; the operator must choose between resume, restart, or a full wipe and retry.

Common causes

Quick checks

# Check active streaming sessions and progress
nodetool netstats

# Check cluster topology and node states
nodetool status

# List active SSTable tasks including streaming (Cassandra 4.0+)
cqlsh -e "SELECT keyspace_name, table_name, task_id, total, completed FROM system_views.sstable_tasks;"

# Search logs for streaming errors or bootstrap state
grep -iE "streaming|bootstrap|decommission|Some data streaming failed" /var/log/cassandra/system.log

# Check streaming throughput trend via JMX
# TotalIncomingBytes / TotalOutgoingBytes should change between samples

# Verify streaming tunables in configuration
grep -E "stream_throughput_outbound|internode_streaming_tcp_user_timeout|streaming_keep_alive_period" /etc/cassandra/cassandra.yaml

# Check if internode encryption is enabled (disables ZCS)
grep -A 5 "server_encryption_options" /etc/cassandra/cassandra.yaml

How to diagnose it

flowchart TD
    A[Streaming stalled >30 min] --> B{nodetool netstats active?}
    B -->|Yes| C{Bytes moving?}
    B -->|No| D[Check logs for session close]
    C -->|No| E[Check firewall vs keep-alive period]
    C -->|Yes slowly| F[Check throttle and source node IO]
    D --> G[Check for DOWN replicas]
    E --> H[Fix network timeout]
    F --> I[Raise throttle or relieve source]
    G --> J[Resume or override bootstrap]

1. Confirm the operation and state. Run nodetool status to check if the node is JOINING, LEAVING, or NORMAL. Run nodetool netstats to confirm active streaming sessions and whether bytes are moving.

2. Determine if the stream is failed or stalled. A failed session disappears from nodetool netstats and usually leaves an error in system.log. A stalled session remains listed but shows no byte progress for more than thirty minutes. On Cassandra 4.0+, query system_views.sstable_tasks to confirm whether active streaming work is registered.

3. Check for network layer drops. Compare the firewall or NAT idle timeout against the streaming keep-alive period. If the firewall timeout is shorter than 600 seconds (two keep-alive cycles), the firewall drops the connection even when the stream is healthy but slow. This is a common cause of mysterious streaming interruptions.

4. Verify throughput expectations. Calculate the dataset size divided by the configured stream_throughput_outbound (legacy stream_throughput_outbound_megabits_per_sec; default 24 MiB/s, about 200 Mbps). If the result is many hours, the operation may not be stalled; it may simply be throttled. However, if nodetool netstats shows no byte increase over a 15-minute window while the session is open, the stream is stalled regardless of throttle.

5. Inspect the source node. Streaming competes with compaction, flushes, and client traffic on the source. Check the source node for pending compactions, blocked thread pools, long GC pauses, and disk IO wait (iostat -x). If the source is in a GC death spiral or compaction backlog, it cannot feed the stream.

6. Check for Zero Copy Streaming fallback. In Cassandra 4.0+, ZCS is enabled by default, but it is automatically disabled when internode_encryption is enabled. There is no warning in nodetool netstats. If encryption is on, expect slower partition-based streaming and plan longer windows.

7. Evaluate CDC and materialized view overhead. For CDC-enabled tables, Cassandra 4.1+ has cdc_on_repair_enabled (default true): streamed CDC data is replayed through the commitlog during repair, while setting it false writes streamed CDC data as SSTables. There is no equivalent streaming-only parameter for materialized views; received data may still require view maintenance, so check view build/backlog separately.

8. Review timeout configuration. The legacy streaming_socket_timeout_in_ms property is not the effective control in modern releases. Use internode_streaming_tcp_user_timeout (legacy internode_streaming_tcp_user_timeout_in_ms), which defaults to 300 seconds, and streaming_keep_alive_period (legacy streaming_keep_alive_period_in_secs), which also defaults to 300 seconds.

9. Check for consistent bootstrap blocking. Since Cassandra 3.0, bootstrap requires all replicas for a token range to be available. If any replica is DOWN, the bootstrap will fail or hang. You can override this with -Dcassandra.consistent.rangemovement=false, but the joining node may miss data from the unavailable replica.

Metrics and signals to monitor

Fixes

Resume or restart a stalled bootstrap

On supported versions, a stalled bootstrap can sometimes be resumed from persisted progress; in Cassandra 5.0 resume is guarded by the bootstrap-progress setting:

# Attempt to resume from saved state.
# On Cassandra 5.0 this is guarded: use --force or explicitly set
# -Dcassandra.reset_bootstrap_progress=false (and restart) to permit resume.
nodetool bootstrap resume

If the node resumes but fails again, inspect the source node and network first. When resume is enabled, restarting the node can also resume bootstrap. If the bootstrap state is corrupted and you must start fresh, discard the progress and restart:

# Start Cassandra with the following JVM startup flag.
# It ignores persisted progress and re-streams the ranges; previously streamed
# files may remain on disk, so validate and reconcile local files as needed.
-Dcassandra.reset_bootstrap_progress=true

Related guides

• Cassandra node stuck in joining (UJ): bootstrap diagnosis
• Cassandra adding and removing nodes safely: vnodes, tokens, and cleanup
• Cassandra monitoring checklist: the signals every production cluster needs
• How Cassandra actually works in production: a mental model for operators
• Cassandra monitoring maturity model: from survival to expert
• Cassandra consistency levels explained: QUORUM, ONE, LOCAL_QUORUM, and EACH_QUORUM
• Cassandra GC death spiral: long pauses, gossip flapping, and recovery
• Cassandra GC pauses too long: diagnosing G1 stop-the-world pauses
• Cassandra java.lang.OutOfMemoryError: Java heap space — causes and recovery
• Cassandra heap pressure: sizing the JVM heap and tuning G1GC
• Cassandra pending compactions growing: the compaction backlog runbook
• Cassandra compaction death spiral: when writes outrun compaction throughput