CockroachDB backup job failures: RPO breaches, duration trends, and stuck jobs

CockroachDB scheduled backups run through the internal jobs system, visible via crdb_internal.jobs (backed by system.jobs). When a scheduled backup fails silently, stalls indefinitely, or grows so slowly that it cannot complete within its interval, your recovery point objective (RPO) is at risk. The failure is insidious: backups often appear healthy until you realize the last successful completion was 36 hours ago.

The severity distinction is sharp. A single backup failure with a recent prior success is a TICKET. The retry will probably succeed. But when no successful backup exists within your RPO window (for example, last success over 24 hours ago with a 24-hour RPO), you are in PAGE territory.

Two related failure modes are harder to catch. Backup duration creep: a backup that took 20 minutes six months ago now takes 3 hours, approaching or exceeding the schedule interval. Overlapping backups compete for I/O and eventually fail. And stuck jobs: a backup appears running but never completes. The schedules.BACKUP.failed counter does not increment for stuck jobs, so standard failure alerts stay quiet while the RPO window silently expires.

What this means

Each backup job has a lifecycle: running, paused, reverting, succeeded, failed, canceled. Jobs that fail with retryable errors are automatically retried with exponential backoff. Jobs that fail with a non-retryable error move to failed and increment jobs.backup.resume_failed. But a job can also get stuck in running indefinitely if it cannot make progress, and that stuck job does not increment any failure counter.

The critical metrics for RPO detection:

If you only alert on schedules.BACKUP.failed, you will miss stuck jobs entirely. The most reliable RPO breach signal is schedules.BACKUP.last-completed-time not advancing past your RPO window.

flowchart TD
    A["Scheduled backup fires"] --> B{"Completes?"}
    B -->|Yes| C["last-completed-time updates"]
    C --> D{"Within RPO window?"}
    D -->|Yes| E["Healthy"]
    D -->|No| F["PAGE: RPO breach"]
    B -->|Fails with error| G["resume_failed or resume_retry_error"]
    G --> H{"Prior success within RPO?"}
    H -->|Yes| I["TICKET: retry may succeed"]
    H -->|No| F
    B -->|Stuck running| J["currently_running nonzero"]
    J --> K["No failure counter increments"]
    K --> L["last-completed-time frozen"]
    L --> F

Common causes

Quick checks

# Check all non-terminal backup jobs
cockroach sql -e "SELECT job_id, job_type, status, running_status, fraction_completed, created FROM crdb_internal.jobs WHERE job_type = 'BACKUP' AND status NOT IN ('succeeded', 'canceled') ORDER BY created DESC;"

# Check last completed backup timestamp (per node - scrape every node)
curl -s http://localhost:8080/_status/vars | grep 'schedules.BACKUP.last-completed-time'

# Check failed backup count
curl -s http://localhost:8080/_status/vars | grep 'schedules.BACKUP.failed'

# Check currently running backup jobs
curl -s http://localhost:8080/_status/vars | grep 'jobs.backup.currently_running'

# Check resume failure counters
curl -s http://localhost:8080/_status/vars | grep -E 'jobs.backup.(resume_failed|resume_retry_error)'

# Check disk space per store
curl -s http://localhost:8080/_status/vars | grep -E 'capacity\.(available|used)'

# Check L0 sublevels (backup I/O contention indicator)
curl -s http://localhost:8080/_status/vars | grep 'storage_l0_sublevels'

Backup metrics are reported per node. You must scrape every node’s Prometheus endpoint for a complete picture. Missing one node can hide a stuck backup job running on that node.

How to diagnose it

1. Determine if this is an RPO breach or a single failure. Compare schedules.BACKUP.last-completed-time against your RPO window. If the timestamp is older than your RPO (for example, over 24 hours with a 24-hour RPO), treat this as a PAGE. If the timestamp is recent and only one backup failed, it is a TICKET.

2. Identify the job state. Query crdb_internal.jobs for all non-terminal backup jobs. Look for jobs in running status with low fraction_completed, jobs in reverting status, or jobs in failed status.

3. Distinguish stuck from slow. A backup job in running state with fraction_completed that has not advanced in 30 or more minutes is stuck. A job where fraction_completed is advancing slowly is slow but progressing. Both eventually breach RPO if the trend continues.

4. Check for retryable vs non-retryable errors. If jobs.backup.resume_retry_error is incrementing, the job is retrying after transient failures (likely destination storage). If jobs.backup.resume_failed is incrementing, the error is non-retryable and the job will not recover without intervention.

5. Correlate with storage health. Backup jobs read data from every range. If the storage layer is under pressure (L0 sublevels elevated, write stalls, disk I/O saturated), backup throughput drops. Check storage_l0_sublevels, write stall metrics, and disk I/O utilization.

6. Check protected timestamp records. A hung backup job can hold a protected timestamp record that prevents MVCC garbage collection, causing silent disk space growth. Check protected timestamp metrics if disk usage is climbing without explanation.

7. Review the backup destination. High-latency destinations (S3 in a different region, network-constrained storage) extend backup duration. Verify network throughput between cluster nodes and the backup destination.

Metrics and signals to monitor

Fixes

Stuck backup job

A stuck job (running state, no progress) requires intervention. Use CANCEL JOB to force the job into reverting state:

-- Cancel the stuck job (use the job_id from crdb_internal.jobs)
CANCEL JOB <job_id>;

The job will attempt cleanup, which itself takes time and I/O. After cancellation, monitor jobs.backup.currently_running to confirm the job leaves running state. Do not assume cancellation is instant. Reverting a large backup can take significant time and I/O.

Non-retryable backup error

Query the error details:

SELECT job_id, status, error FROM crdb_internal.jobs
WHERE job_type = 'BACKUP' AND status = 'failed'
ORDER BY created DESC;

Common non-retryable causes include permission errors on the destination, invalid backup syntax, or schema mismatches in incremental chains. Fix the underlying issue, then re-run the backup or resume the schedule.

Destination storage failure

If resume_retry_error is incrementing, the job is retrying after transient destination failures. Check connectivity from cluster nodes to the backup destination. For S3/GCS, verify IAM permissions, endpoint reachability, and bucket region. CockroachDB retries with exponential backoff, so transient failures may resolve on their own. Persistent failures require manual intervention.

I/O contention degrading backup throughput

Backup I/O competes with foreground traffic and compaction. If backups are slow because the storage layer is saturated, consider scheduling backups during lower-traffic windows or adjusting the backup rate. If L0 sublevels are climbing and write stalls are occurring during backups, pause the backup schedule until storage health recovers. See CockroachDB compaction backlog growing for compaction diagnosis.

Duration exceeding interval

If backup duration is approaching or exceeding the schedule interval, backups are at risk of overlapping or missing their window. This usually indicates data growth outpacing backup infrastructure. Options: increase backup destination throughput (higher-IOPS endpoint, closer region), split the backup into smaller targeted backups, or run more frequent incremental backups to reduce per-run volume.

Prevention

Alert on last-completed-time, not just failure count. The schedules.BACKUP.failed metric misses stuck jobs. Build your RPO alert around schedules.BACKUP.last-completed-time not advancing past your RPO window. This catches both failed and stuck jobs with a single condition.

Enable the last-backup-time metric on your schedules. The schedules.BACKUP.last-completed-time metric only updates if the schedule was created with the updates_cluster_last_backup_time_metric option. Without this option, the metric does not exist and RPO detection is significantly harder.

Track duration trends, not just pass/fail. A backup that succeeds every time but takes progressively longer is heading toward failure. Plot backup duration over weeks. Alert when duration exceeds 50% of the backup interval.

Scrape every node. Backup metrics are per-node. A monitoring setup that scrapes only one node will miss backup jobs running on other nodes.

Verify metrics are actually emitted. Scheduled backups may not emit all backup-specific metrics in some configurations. Verify that your scheduled backups produce the metrics you expect by running a test backup and checking the Prometheus endpoint.

How Netdata helps

• Per-second metric collection from every CockroachDB node means stuck jobs on any node are immediately visible through jobs.backup.currently_running without advancing last-completed-time.
• Anomaly detection on backup duration trends catches the slow drift from 20 minutes to 3 hours before it exceeds the interval.
• Correlation between backup jobs and storage health shortens diagnosis: L0 sublevels, disk I/O, and admission control queue depth appear in the same view when a backup stalls.
• Protected timestamp and disk space monitoring together catch the silent cascade where a hung backup blocks MVCC garbage collection and disk usage climbs.
• Composite alerting can combine last-completed-time staleness with currently_running persistence to distinguish a normally running long backup from a stuck one, reducing false-positive pages during legitimate large backups.

Related guides

• CockroachDB admission control throttling: queue depth, store-write, and capacity headroom
• CockroachDB certificate expired: TLS handshake failures and online rotation
• CockroachDB clock_offset_meannanos high: catching clock drift before self-termination
• CockroachDB clock skew cascade: how shared NTP drift causes quorum loss
• CockroachDB clock synchronization error: this node is more than 500ms away from at least half of the known nodes
• CockroachDB compaction backlog growing: when Pebble can’t keep pace with writes
• CockroachDB connection storm after failover: reconnect stampedes and surviving-node overload
• CockroachDB context deadline exceeded: timeouts, slow ranges, and overload
• CockroachDB CPU saturation: Raft ticking, SQL execution, and the per-node ceiling
• CockroachDB detecting hot ranges: per-range QPS, CPU asymmetry, and the Hot Ranges page
• CockroachDB disk space running out: capacity_available trends and the 20% rule
• CockroachDB disk stall detected: storage_disk_stalled and node self-termination