Cassandra OperationTimedOutException: client-side timeouts vs server timeouts

OperationTimedOutException (driver 3.x) or DriverTimeoutException (driver 4.x) means the driver gave up before the coordinator responded. This is distinct from a server-side ReadTimeout or WriteTimeout, where the coordinator explicitly errors because replicas were too slow. Confusing the two leads to tuning the wrong timeout, masking a server capacity problem or adding unnecessary client latency. Use server-side signals and driver behavior to tell them apart.

When the driver timer fires first, the coordinator may still be processing the request. For writes, the mutation can be committed on some replicas even though the client sees an exception. When the server timer fires first, the coordinator returns a READ_TIMEOUT or WRITE_TIMEOUT to the driver. The diagnostic path and fix differ for each case.

flowchart LR
  subgraph ClientSide ["Client JVM"]
    App["Application"]
    Driver["Driver Request Timer"]
  end

  subgraph ServerSide ["Cassandra Cluster"]
    Coord["Coordinator"]
    R1["Replica"]
    R2["Replica"]
  end

  App -->|"1. execute()"| Driver
  Driver -->|"2. CQL request"| Coord
  Coord -->|"3a. replica requests"| R1
  Coord -->|"3b. replica requests"| R2

  Driver -->|"4a. timer expires
client-side timeout"| App
  Coord -->|"4b. READ_TIMEOUT /
WRITE_TIMEOUT response"| Driver

What this means

Client-side timeout: The driver starts a timer when sending a request. If it expires before the coordinator responds, the driver throws. In 3.x this is OperationTimedOutException; the retry policy invokes onRequestError. In 4.x it is DriverTimeoutException. Defaults shifted significantly between versions: 3.x defaults to roughly 12 seconds, 4.x defaults to 2 seconds.

Server-side timeout: The coordinator waits for enough replicas to satisfy the consistency level. If that does not happen within the configured timeout, the coordinator returns READ_TIMEOUT or WRITE_TIMEOUT. The driver surfaces this to onReadTimeout or onWriteTimeout. The coordinator responded; the driver is not guessing.

UnavailableException: If too few replicas are alive to attempt the request, the coordinator fails immediately. This is not a timeout; the fix is node recovery, not timeout tuning.

The driver timeout must exceed the relevant server-side timeout plus network round-trip time. If it is lower, the driver blames Cassandra while the server is still within its own limits.

Common causes

Quick checks

These read-only checks help determine which side fired first.

# Server-side timeout counters (JMX)
# org.apache.cassandra.metrics:type=ClientRequest,scope=Read,name=Timeouts
# org.apache.cassandra.metrics:type=ClientRequest,scope=Write,name=Timeouts

# Replica saturation and dropped messages
nodetool tpstats

# Coordinator latency percentiles (microseconds)
nodetool proxyhistograms

# Node liveness across the cluster
nodetool status

# Recent stop-the-world GC pauses
grep -i "pause" /var/log/cassandra/gc.log | tail -20

How to diagnose it

1. Check server-side timeout metrics. Query JMX for org.apache.cassandra.metrics:type=ClientRequest,scope=Read,name=Timeouts and the corresponding Write scope. If these counters are increasing, the coordinator fired first and the driver is surfacing a server-side error. If they are flat while the application sees timeouts, the driver timer fired first.

2. Compare coordinator latency to timeout walls. Use nodetool proxyhistograms to read coordinator-level latency percentiles. If p99 read latency is approaching half of read_request_timeout_in_ms (default 5000 ms) or spiking beyond it, the server is the bottleneck. If coordinator latency is well below the driver timeout but driver timeouts still occur, the network or the driver threshold is the problem.

3. Inspect replica saturation. On each replica, run nodetool tpstats and look at ReadStage and MutationStage pending tasks. Sustained pending greater than zero means the replica cannot keep up. Also check CompactionExecutor pending; if compaction is falling behind, reads slow down due to amplification.

4. Check GC pause duration. Long pauses freeze replicas. Inspect GC logs or JMX GarbageCollector MBeans for collection times. Pauses greater than 2 seconds directly drive coordinator timeouts because replicas stop responding.

5. Check for node DOWN states. Run nodetool status. Unavailable exceptions are distinct from timeouts, but a DOWN node shrinks the replica set and can overload the remaining nodes, causing them to time out.

6. Compare driver timeout configuration to server timeouts. The driver-side request timeout should be larger than the relevant server-side timeout (read_request_timeout_in_ms, write_request_timeout_in_ms, range_request_timeout_in_ms) plus the estimated network round-trip time between the application and the coordinator.

Metrics and signals to monitor

Fixes

Driver timeout is lower than server timeout

Increase the driver request timeout so it exceeds the relevant server timeout plus network RTT. In 4.x, adjust the global request timeout. In 3.x, adjust the socket read timeout. Tradeoff: clients wait longer. Ensure connection pools and circuit breakers accommodate the increase. Re-verify defaults after any major driver upgrade.

Server-side replica slowness

If server ClientRequest Timeouts are rising, do not increase the driver timeout without first reducing replica latency. Check for GC pressure, compaction debt, or disk I/O saturation. Increase compaction_throughput_mb_per_sec or add IOPS capacity if compaction is the bottleneck. Increase concurrent_compactors if CPU headroom exists. See Cassandra GC death spiral: long pauses, gossip flapping, and recovery and Cassandra compaction death spiral: when writes outrun compaction throughput.

Non-idempotent writes and driver retries

In driver 4.x, the default retry policy will not retry DriverTimeoutException for non-idempotent queries. This prevents duplicate writes. If your application sees write timeouts that are not retried, mark writes as idempotent when they are safe to replay, or implement explicit application-level handling. Tradeoff: automatic retries on non-idempotent writes risk duplicate data.

Netty timer issues in older driver releases

If you are running driver releases that include Netty 4.1.77 through 4.1.86, a HashedWheelTimer race condition can cause spurious client-side timeouts. Upgrade the Netty dependency or the driver to a fixed version.

Node failure and in-flight requests

When a node goes down, in-flight requests to that coordinator fail with client-side timeouts. The driver eventually marks the node DOWN via gossip, but the in-flight requests are lost. Ensure your application handles these explicitly rather than depending on automatic retry for non-idempotent operations.

Prevention

• Keep driver timeout greater than server timeout plus network RTT. Re-verify after driver upgrades; defaults shift dramatically between major versions.
• Monitor server-side ClientRequest Timeouts and Unavailables alongside driver exceptions. Alerting on only one side creates a blind spot.
• Keep coordinator read latency p99 below half of the server read timeout to leave headroom for tail latency and jitter.
• Watch ThreadPools pending tasks and GC pause duration as leading indicators. Server-side timeouts follow resource saturation; driver timeouts can appear without warning if the threshold is simply too aggressive.

How Netdata helps

Netdata correlates coordinator read/write latency with ClientRequest timeout and unavailable rates, showing whether the driver or the server fired first. It tracks JVM GC pauses and thread pool pending tasks per node to highlight replica slowness before it becomes client-visible. Disk I/O utilization per device separates commitlog contention from data directory saturation. Per-node timeout and unavailable metrics are shown alongside application error rates.

Related guides

• Cassandra compaction strategies: STCS vs LCS vs TWCS vs UCS
• Cassandra compaction death spiral: when writes outrun compaction throughput
• Cassandra consistency levels explained: QUORUM, ONE, LOCAL_QUORUM, and EACH_QUORUM
• Cassandra zombie data resurrection: gc_grace_seconds and unrepaired tombstones
• Cassandra disk space exhaustion: emergency recovery when the data volume fills
• Cassandra GC death spiral: long pauses, gossip flapping, and recovery
• Cassandra GC pauses too long: diagnosing G1 stop-the-world pauses
• Cassandra heap pressure: sizing the JVM heap and tuning G1GC
• Cassandra monitoring checklist: the signals every production cluster needs
• Cassandra monitoring maturity model: from survival to expert
• Cassandra Not enough space for compaction: STCS space amplification and recovery
• Cassandra java.lang.OutOfMemoryError: Java heap space - causes and recovery