CockroachDB certificate expired: TLS handshake failures and online rotation

CockroachDB uses mutual TLS for every connection: node-to-node, client-to-node, and admin UI. There is no plaintext fallback and no grace period. When a certificate expires, affected connections fail immediately.

Node certificate expiry prevents nodes from completing TLS handshakes with each other, which can look like a network partition or quorum loss. Client certificate expiry prevents applications from connecting. CA certificate expiry invalidates the entire trust chain at once, breaking every connection simultaneously.

CockroachDB defaults certificate lifetimes to 87840h (10 years), so expiry often surfaces years after initial setup, when the original operator has moved on and nobody configured monitoring for it. CA certificate expiry is the most commonly forgotten item in multi-year deployments.

Three certificate types exist and expire independently:

• Node certificates are presented by each node to other nodes and to SQL clients. Expired node certs cause inter-node TLS handshake failures, ranges_unavailable spikes, and potential quorum loss.
• Client certificates are presented by SQL clients (applications, CLI tools). Expired client certs cause client connection failures with no inter-node errors.
• CA certificate signs all node and client certificates. An expired CA breaks the trust chain for every certificate it signed, causing complete cluster isolation.

flowchart TD
    A["Certificate expired"] --> B{"Which cert type?"}
    B -->|Node cert| C["Inter-node TLS failures"]
    B -->|Client cert| D["Client connections fail"]
    B -->|CA cert| E["All connections fail"]
    C --> F["Ranges unavailable"]
    C --> G["Lease transfers spike"]
    C --> H["Quorum loss possible"]
    D --> I["sql_conn_failures rises"]
    D --> J["New connections rejected"]
    E --> K["Complete cluster isolation"]
    F --> L["Fix: SIGHUP rotation"]
    I --> M["Fix: rotate cert, restart clients"]
    K --> N["Fix: CA rotation, combined cert"]

The characteristic log signature:

http: TLS handshake error from <addr>: remote error: tls: bad certificate
serving SQL client conn: remote error: tls: bad certificate

Common causes

Quick checks

# List all certificates with file paths, usage, expiry dates, and errors
cockroach cert list --certs-dir=/path/to/certs

# Check expiry of specific certificate files
openssl x509 -in /path/to/certs/node.crt -enddate -noout
openssl x509 -in /path/to/certs/ca.crt -enddate -noout
openssl x509 -in /path/to/certs/client.root.crt -enddate -noout

# Check certificate expiration metrics from the Prometheus endpoint
curl -s http://localhost:8080/_status/vars | grep -E 'security_certificate_expiration_(ca|node)'

# Search for TLS handshake errors in CockroachDB logs
grep "TLS handshake error\|bad certificate" /path/to/logs/cockroach.log

# Check key file permissions (CockroachDB enforces strict ownership)
ls -la /path/to/certs/*.key

Diagnosis

1. Identify the failing connection type. Inter-node errors point to node or CA cert problems. Client-only errors (with healthy inter-node communication) point to client cert problems.

2. List all certificates with expiry dates. Run cockroach cert list --certs-dir=<dir> on any node. Compare each expiry date against the current date.

3. Check the CA certificate specifically. CA expiry is easy to miss in long cockroach cert list output. Run openssl x509 -in ca.crt -enddate -noout directly.

4. Check Prometheus metrics. Current versions expose the expiration timestamps as separate metrics: security_certificate_expiration_ca for the CA certificate and security_certificate_expiration_node for the node certificate (older releases used a single security_certificate_expiration metric with a type label). There is no built-in metric for an individual client certificate’s expiration; track client cert expiry externally.

5. Verify key file permissions. CockroachDB enforces strict key file ownership: the key must be owned by the process user or root, with group ownership matching the process group. Wrong permissions cause rotation to fail even with valid certificates.

6. Check all nodes, not just one. If certificates were provisioned at different times or partially rotated, different nodes may have different expiry dates.

Metrics and signals

Fixes

Online rotation via SIGHUP (node and CA certificates)

CockroachDB reloads certificate files from the certs directory on SIGHUP, without restarting. This works for both node certificates and CA certificates.

1. Generate new certificates using cockroach cert create-node or your PKI tooling.
2. Replace the certificate files in the certs directory (same filenames).
3. Send SIGHUP to the cockroach process:

# Reload certificates without restarting the node
kill -HUP $(pgrep -x cockroach)

4. Verify the new certificate is loaded: run cockroach cert list --certs-dir=<dir> and confirm the updated expiry date. Test with a SQL query.

Rotate one node at a time. Confirm each node has reloaded successfully before moving to the next to avoid simultaneous reloads across the cluster.

CA certificate rotation

CA rotation requires a multi-step process because nodes and clients with old certificates must still be verified during the transition.

1. Create a combined CA certificate file with the new CA cert first, followed by the old CA cert. This allows CockroachDB to verify both old and new certificates during the transition.
2. Replace ca.crt with the combined file on all nodes.
3. Send SIGHUP to each node sequentially to reload the combined CA.
4. After all nodes have the combined CA, rotate node certificates (signed by the new CA) via SIGHUP on each node.
5. Rotate client certificates (signed by the new CA). Client certs require the client process to restart, not just SIGHUP on the server.
6. After all nodes and clients have certificates signed by the new CA, remove the old CA cert from the combined ca.crt file and SIGHUP again.

CA rotation must complete before node and client cert rotation. Clients only pick up new CA certificates after restart, so plan the transition with adequate lead time.

Client certificate rotation

SIGHUP only reloads certificates on the CockroachDB server process, not on client applications. Each application instance must be restarted after the new client certificate is placed on disk.

There is no built-in metric for client certificate expiration. Track client cert expiry externally using openssl x509 -enddate -noout or your PKI tooling.

Operator-managed deployments (Kubernetes)

In operator-managed deployments, SIGHUP-based online rotation is not available. Pods must be restarted to pick up new certificates, whether those certificates are generated automatically (by the self-signer utility or cert-manager) or manually.

Plan for rolling pod restarts during certificate rotation. Ensure the rolling restart does not cause quorum loss by waiting for each pod to rejoin and stabilize before restarting the next.

For cert-manager-managed certificates, ensure fsGroup is set correctly on the volume mount so CockroachDB can read the key files with the expected ownership.

Key file permission issues

CockroachDB enforces strict key file ownership. The key file must be owned by the process user or root, with group ownership matching the process group.

To temporarily bypass permission checks during debugging:

export COCKROACH_SKIP_KEY_PERMISSION_CHECK=true

Do not use this in production. Fix the actual file ownership and mode on the key files.

Prevention

• Monitor security_certificate_expiration_ca and security_certificate_expiration_node. Alert at 30 days remaining (plan rotation) and at less than 24 hours remaining with no replacement staged (urgent).
• Track client certificate expiry externally. Run a scheduled job that checks expiry dates with openssl x509 -enddate -noout and alerts before expiry.
• Alert before expiry, not at expiry. CockroachDB has no grace period. Alert days or weeks in advance.
• Document the CA certificate. Record its creation date, expiry date, and rotation procedure in your runbook. The CA cert is the most commonly forgotten item in multi-year deployments.
• Test rotation before expiry. Rotate certificates well before they expire so you have time to fix issues. Discovering that operator-managed deployments do not support SIGHUP at 3 AM under incident pressure is avoidable.

How Netdata helps

• Per-second collection of security_certificate_expiration_ca and security_certificate_expiration_node keeps the expiry countdown visible rather than only surfacing during a scheduled check.
• ML anomaly detection on sql_conns and sql_conn_failures can flag unexpected connection drops that correlate with certificate problems, even when the certificate metric itself was not being watched.
• Cross-signal correlation between TLS handshake errors in logs, ranges_unavailable spikes, and node liveness changes confirms the failure is certificate-related rather than a network partition or disk stall.
• Node-level visibility shows per-node certificate states, which matters when certificates were provisioned at different times or partially rotated.

Netdata’s database monitoring brings these signals together with per-second metrics and ML anomaly detection.

Related guides

• CockroachDB clock offset high: catching clock drift before self-termination
• CockroachDB clock skew cascade: how shared NTP drift causes quorum loss
• CockroachDB clock synchronization error: this node is more than 500ms away from at least half of the known nodes
• CockroachDB connection storm after failover: reconnect stampedes and surviving-node overload
• CockroachDB context deadline exceeded: timeouts, slow ranges, and overload