CockroachDB /health?ready=1: load balancer checks, draining, and impaired nodes

CockroachDB exposes a readiness endpoint at GET /health?ready=1 on its HTTP port (default 8080). Load balancers and Kubernetes readiness probes use it to decide whether to route SQL traffic to a node. The node returns HTTP 200 when ready and HTTP 503 when not. This signal separates “process is alive” from “this node should receive client connections.”

The most common operator mistake is using a plain TCP check against the SQL port (26257) or the plain /health endpoint instead of /health?ready=1. A TCP check succeeds as long as the port is listening, telling you nothing about whether the node is draining, write-stalled, or GC-thrashing. The plain /health endpoint returns 200 whenever the process is running, regardless of draining state. Neither is safe for routing decisions.

What the endpoint checks

The ?ready=1 query parameter transforms /health from a liveness check into a readiness check. The endpoint returns:

• 200 OK (empty body) when the node is operational and ready to accept SQL connections.
• 503 Service Unavailable (JSON error body) when the node is not ready to serve traffic.

Per the official documentation, 503 is returned when:

1. The node is in the wait phase of the shutdown sequence (draining).
2. The node is decommissioning or decommissioned.
3. The node cannot communicate with a majority of other nodes (cluster unavailability).

However, issue #116194 documents that the readiness endpoint only checks local node state: whether the node is initializing or draining, and whether the SQL server has started. It does not verify whether the SQL gateway can reach a majority quorum of ranges. A node partitioned away from quorum may still report 200 via /health?ready=1. This is an open enhancement request, not resolved as of this writing.

How it works

The readiness check is a lightweight HTTP request served by the node’s HTTP server. It does not test the full SQL path (TCP connect, TLS handshake, pgwire protocol, query execution). It checks whether the node’s internal state machine has reached a point where it believes it can serve clients.

flowchart TD
    A["LB / probe checks /health?ready=1"] --> B{Node draining?}
    B -->|yes| F["503 Service Unavailable"]
    B -->|no| C{Decommissioning?}
    C -->|yes| F
    C -->|no| D{SQL server started?}
    D -->|no| F
    D -->|yes| E["200 OK"]
    F --> G["LB removes node after N failed checks"]
    E --> H["LB routes SQL traffic"]

What the endpoint does not check:

• Whether ranges on this node have quorum connectivity.
• Storage engine health (write stalls, L0 sublevel overflow, compaction backlog).
• Inter-node network latency or partition status.

A node can return 200 while experiencing write stalls, admission control throttling, or elevated GC pauses. The endpoint tells you the node believes it is ready, not that it is performing well. A synthetic SELECT 1 probe exercises the full client path and catches failures the readiness endpoint misses.

Where it shows up in production

Load balancer configuration

When using /health?ready=1, the check interval and failure threshold must be coordinated with CockroachDB’s shutdown timing.

During graceful shutdown, the node enters a drain phase. It stops accepting new client connections and transfers range leases. The server.shutdown.initial_wait cluster setting (previously server.shutdown.drain_wait, kept as an alias) controls how long the node waits before draining begins. The default is 0s, meaning the node starts draining immediately without giving the load balancer time to detect the 503 and stop routing new connections.

For HAProxy with default inter 2000 fall 3, the load balancer checks every 2 seconds and removes a backend after 3 consecutive failures. The node must return 503 for at least 6 seconds before the load balancer stops routing. With server.shutdown.initial_wait at 0s, new connections arrive at a draining node.

Set server.shutdown.initial_wait to at least your LB’s check interval multiplied by its fall threshold, plus margin. For the HAProxy example above, 8 seconds covers the 6-second detection window with a safety buffer.

Kubernetes readiness probes

The official CockroachDB StatefulSet manifest uses /health?ready=1 as the readiness probe path with failureThreshold: 2 and periodSeconds: 5. With these defaults, a pod is marked not-ready after roughly 10 seconds of 503 responses (2 failed checks at 5-second intervals). Ensure server.shutdown.initial_wait covers that window. The liveness probe is explicitly commented out in the manifest with the note: “We recommend that you do not configure a liveness probe on a production environment, as this can impact the availability of production databases.”

terminationGracePeriodSeconds should be at least 300 seconds and at least 5 seconds longer than the sum of all server.shutdown.* timeouts. If the grace period is too short, Kubernetes SIGKILLs the pod before CockroachDB finishes draining, causing connection resets and brief range unavailability.

CPU starvation and health check timeouts

Under severe CPU load, requests to the health endpoint can hang or timeout. Before v21.2, a node near 100% CPU could take 20+ seconds to respond to /health?ready=1, causing readiness probes to fail and mark the pod not-ready. This was partially addressed by removing authentication overhead on the health endpoint, and further mitigated by admission control in v21.2+, which prevents CPU starvation from reaching the point where the HTTP server becomes unresponsive.

CockroachDB Cloud’s own production deployments do not use a Kubernetes liveness probe. A liveness probe failure triggers a pod restart, which causes lease transfers, connection resets, and brief unavailability. Restarting a node that is slow but not dead often makes the situation worse.

Tradeoffs and common misuses

TCP checks versus the readiness endpoint

A plain TCP check against port 26257 succeeds whenever the process is listening. It cannot detect:

• A draining node (accepting no new connections).
• A node with active write stalls (storage engine refusing writes).
• A node in GC-thrashing oscillation (alternating between alive and unresponsive).
• A node that has lost liveness but whose process is still running.

If your load balancer uses TCP checks, it will route traffic to impaired nodes during incidents.

Plain /health versus /health?ready=1

The /health endpoint without ?ready=1 returns 200 whenever the process is running. It does not reflect draining state. Using it as a readiness check means traffic continues to route to a draining node during graceful shutdown.

Readiness endpoint versus synthetic SQL probe

A SELECT 1 probe via cockroach sql -e "SELECT 1" exercises TCP connect, TLS handshake, pgwire protocol, authentication, and SQL execution. It catches TLS certificate issues, authentication problems, and pgwire-level errors that the HTTP readiness check misses.

The tradeoff is cost: a SQL probe opens a real connection and is more expensive to run frequently. A reasonable pattern is /health?ready=1 for the load balancer’s fast check (every 2-5 seconds) and a SELECT 1 probe for deeper monitoring (every 10-30 seconds).

The partitioned-node gap

/health?ready=1 does not detect network partitions. A node isolated from quorum can still report 200. For partition detection, correlate the readiness check with:

• ranges_unavailable metric (nonzero means ranges have lost quorum).
• Node liveness status changes.
• Raft leader-not-found errors in logs.
• Inter-node RPC latency spikes.

If you rely solely on the readiness endpoint for health routing, a partitioned node will keep receiving traffic it cannot serve.

Signals to watch in production

How Netdata helps

• Per-second metrics collection captures liveness transitions, write stalls, and RPC latency spikes at the resolution they occur, not 15 or 30 seconds after the fact.
• Correlate /health?ready=1 probe results with ranges_unavailable, storage_write_stalls, and node liveness on the same timeline. If the readiness endpoint returns 200 but ranges are unavailable or writes are stalling, the node is impaired despite its own self-assessment.
• Per-node CPU utilization and Go GC pause metrics distinguish “health check failed because the node is unhealthy” from “health check timed out because the HTTP server was CPU-starved.”
• Admission control queue depth signals when the system is at capacity, which often precedes degradation that eventually surfaces as health check failures.

Netdata’s CockroachDB monitoring brings these signals together with per-second metrics and anomaly detection.

Related guides

• CockroachDB admission control throttling: queue depth, store-write, and capacity headroom
• CockroachDB clock_offset_meannanos high: catching clock drift before self-termination
• CockroachDB clock skew cascade: how shared NTP drift causes quorum loss
• CockroachDB clock synchronization error: this node is more than 500ms away from at least half of the known nodes
• CockroachDB compaction backlog growing: when Pebble can’t keep pace with writes
• CockroachDB context deadline exceeded: timeouts, slow ranges, and overload
• CockroachDB CPU saturation: Raft ticking, SQL execution, and the per-node ceiling
• CockroachDB detecting hot ranges: per-range QPS, CPU asymmetry, and the Hot Ranges page
• CockroachDB disk space running out: capacity_available trends and the 20% rule
• CockroachDB disk stall detected: storage_disk_stalled and node self-termination
• CockroachDB file descriptor exhaustion: SSTables, connections, and ulimit
• CockroachDB Go GC pauses high: when garbage collection threatens Raft heartbeats