CockroachDB /health?ready=1: load balancer checks, draining, and impaired nodes

CockroachDB exposes a readiness endpoint at GET /health?ready=1 on its HTTP port (default 8080). Load balancers and Kubernetes readiness probes use it to decide whether to route SQL traffic to a node. The node returns HTTP 200 when ready and HTTP 503 when not. This signal separates “process is alive” from “this node should receive client connections.”

The most common operator mistake is using a plain TCP check against the SQL port (26257) or the plain /health endpoint instead of /health?ready=1. A TCP check succeeds as long as the port is listening, telling you nothing about whether the node is draining, write-stalled, or GC-thrashing. The plain /health endpoint returns 200 whenever the process is running, regardless of draining state. Neither is safe for routing decisions.

What the endpoint checks

The ?ready=1 query parameter transforms /health from a liveness check into a readiness check. The endpoint returns:

• 200 OK (empty body) when the node is operational and ready to accept SQL connections.
• 503 Service Unavailable (JSON error body) when the node is not ready to serve traffic.

Per the official documentation, 503 is returned when:

1. The node is in the wait phase of the shutdown sequence (draining).
2. The node is decommissioning or decommissioned.
3. The node cannot communicate with a majority of other nodes (cluster unavailability).

The “cannot communicate with a majority” condition is implemented through node liveness: since v23.2 the readiness check fails when the node’s own liveness record is expired, which happens when the node can no longer write heartbeats to the liveness range, i.e. it lost quorum connectivity. Issue #116194 (still open) asks for a stronger check that verifies whether the SQL gateway can reach a majority quorum of ranges beyond its own liveness, because a node with marginal liveness connectivity can still report 200 while some of its ranges are unreachable.

How it works

The readiness check is a lightweight HTTP request served by the node’s HTTP server. It does not test the full SQL path (TCP connect, TLS handshake, pgwire protocol, query execution). It checks whether the node’s internal state machine has reached a point where it believes it can serve clients.

flowchart TD
    A["LB / probe checks /health?ready=1"] --> B{Node draining?}
    B -->|yes| F["503 Service Unavailable"]
    B -->|no| C{Decommissioning?}
    C -->|yes| F
    C -->|no| D{SQL server started?}
    D -->|no| F
    D -->|yes| E["200 OK"]
    F --> G["LB removes node after N failed checks"]
    E --> H["LB routes SQL traffic"]

What the endpoint does not check:

• Whether ranges on this node have quorum connectivity.
• Storage engine health (write stalls, L0 sublevel overflow, compaction backlog).
• Inter-node network latency or partition status.

A node can return 200 while experiencing write stalls, admission control throttling, or elevated GC pauses. The endpoint tells you the node believes it is ready, not that it is performing well. A synthetic SELECT 1 probe exercises the full client path and catches failures the readiness endpoint misses.

Where it shows up in production

Load balancer configuration

When using /health?ready=1, the check interval and failure threshold must be coordinated with CockroachDB’s shutdown timing.

During graceful shutdown, the node enters a drain phase. It stops accepting new client connections and transfers range leases. The server.shutdown.initial_wait cluster setting (previously server.shutdown.drain_wait, kept as an alias) controls how long the node waits before draining begins. The default is 0s, meaning the node starts draining immediately without giving the load balancer time to detect the 503 and stop routing new connections.

For HAProxy with default inter 2000 fall 3, the load balancer checks every 2 seconds and removes a backend after 3 consecutive failures. The node must return 503 for at least 6 seconds before the load balancer stops routing. With server.shutdown.initial_wait at 0s, new connections arrive at a draining node.

Set server.shutdown.initial_wait to at least your LB’s check interval multiplied by its fall threshold, plus margin. For the HAProxy example above, 8 seconds covers the 6-second detection window with a safety buffer.

Kubernetes readiness probes

The official CockroachDB StatefulSet manifest uses /health?ready=1 as the readiness probe path with failureThreshold: 2 and periodSeconds: 5. With these defaults, a pod is marked not-ready after roughly 10 seconds of 503 responses (2 failed checks at 5-second intervals). Ensure server.shutdown.initial_wait covers that window. The liveness probe is explicitly commented out in the manifest with the note: “We recommend that you do not configure a liveness probe on a production environment, as this can impact the availability of production databases.”

Set terminationGracePeriodSeconds to the sum of all server.shutdown.* timeouts, per the official recommendation; in most cases a value above 300 seconds is not required. If the grace period is too short, Kubernetes SIGKILLs the pod before CockroachDB finishes draining, causing connection resets and brief range unavailability.

CPU starvation and health check timeouts

Under severe CPU load, requests to the health endpoint can hang or timeout. This was reported as early as 2020 (issue #44832): a node near 100% CPU could take 20+ seconds to respond to /health?ready=1, causing probes to fail and mark the pod not-ready. It was partially addressed in v20.1 by making /health require no authentication and perform no KV operations (PR #45119), and further mitigated by the admission control work introduced experimentally in v21.2, which keeps an overloaded node serving critical work such as the health endpoint.

Follow the manifest’s guidance and leave the liveness probe disabled in production. A liveness probe failure triggers a pod restart, which causes lease transfers, connection resets, and brief unavailability. Restarting a node that is slow but not dead often makes the situation worse.

Tradeoffs and common misuses

TCP checks versus the readiness endpoint

A plain TCP check against port 26257 succeeds whenever the process is listening. It cannot detect:

• A draining node (accepting no new connections).
• A node with active write stalls (storage engine refusing writes).
• A node in GC-thrashing oscillation (alternating between alive and unresponsive).
• A node that has lost liveness but whose process is still running.

If your load balancer uses TCP checks, it will route traffic to impaired nodes during incidents.

Plain /health versus /health?ready=1

The /health endpoint without ?ready=1 returns 200 whenever the process is running. It does not reflect draining state. Using it as a readiness check means traffic continues to route to a draining node during graceful shutdown.

Readiness endpoint versus synthetic SQL probe

A SELECT 1 probe via cockroach sql -e "SELECT 1" exercises TCP connect, TLS handshake, pgwire protocol, authentication, and SQL execution. It catches TLS certificate issues, authentication problems, and pgwire-level errors that the HTTP readiness check misses.

The tradeoff is cost: a SQL probe opens a real connection and is more expensive to run frequently. A reasonable pattern is /health?ready=1 for the load balancer’s fast check (every 2-5 seconds) and a SELECT 1 probe for deeper monitoring (every 10-30 seconds).

The partitioned-node gap

/health?ready=1 does not detect network partitions. A node isolated from quorum can still report 200. For partition detection, correlate the readiness check with:

• ranges_unavailable metric (nonzero means ranges have lost quorum).
• Node liveness status changes.
• Raft leader-not-found errors in logs.
• Inter-node RPC latency spikes.

If you rely solely on the readiness endpoint for health routing, a partitioned node will keep receiving traffic it cannot serve.

Signals to watch in production

How Netdata helps

• Per-second metrics collection captures liveness transitions, write stalls, and RPC latency spikes at the resolution they occur, not 15 or 30 seconds after the fact.
• Correlate /health?ready=1 probe results with ranges_unavailable, storage_write_stalls, and node liveness on the same timeline. If the readiness endpoint returns 200 but ranges are unavailable or writes are stalling, the node is impaired despite its own self-assessment.
• Per-node CPU utilization and Go GC pause metrics distinguish “health check failed because the node is unhealthy” from “health check timed out because the HTTP server was CPU-starved.”
• Admission control queue depth signals when the system is at capacity, which often precedes degradation that eventually surfaces as health check failures.

Netdata’s CockroachDB monitoring brings these signals together with per-second metrics and anomaly detection.

Related guides

• CockroachDB admission control throttling: queue depth, store-write, and capacity headroom
• CockroachDB clock_offset_meannanos high: catching clock drift before self-termination
• CockroachDB clock skew cascade: how shared NTP drift causes quorum loss
• CockroachDB clock synchronization error: this node is more than 500ms away from at least half of the known nodes
• CockroachDB compaction backlog growing: when Pebble can’t keep pace with writes
• CockroachDB context deadline exceeded: timeouts, slow ranges, and overload
• CockroachDB CPU saturation: Raft ticking, SQL execution, and the per-node ceiling
• CockroachDB detecting hot ranges: per-range QPS, CPU asymmetry, and the Hot Ranges page
• CockroachDB disk space running out: capacity_available trends and the 20% rule
• CockroachDB disk stall detected: storage_disk_stalled and node self-termination
• CockroachDB file descriptor exhaustion: SSTables, connections, and ulimit
• CockroachDB Go GC pauses high: when garbage collection threatens Raft heartbeats