Docker monitoring checklist: the signals every production host needs

Production Docker incidents rarely look like Docker problems at first. They show up as application latency, deployment failures, or hosts that suddenly refuse to schedule containers. By the time you notice, the daemon may be hung, a log file has filled the disk, or a container has been silently throttled into unusable latency. This checklist groups the essential production signals into three priority tiers: must-have alerts that keep the host alive, should-have metrics that expose resource pressure before it becomes an outage, and nice-to-have security and internal signals for mature environments. Every signal includes where to read it from the raw cgroup filesystem or the Docker API so you can instrument hosts without guessing paths.

What This Means

Docker is not a single process to monitor. It is a stack of interdependent components: dockerd, containerd, runc, a storage driver, and a network bridge. Each layer emits its own failure signals. A storage driver deadlock can hang the daemon even though containers keep running. A container can report moderate CPU usage while the CFS bandwidth controller throttles it into p99 latency spikes. A missing log rotation configuration can fill the disk while all container-level metrics look normal. This checklist maps each signal to its authoritative source so you can distinguish between a healthy host, a degraded one, and a host about to fail.

Common Causes

Quick Checks

Run these commands to get a snapshot of host health before you set up continuous monitoring.

# Check daemon responsiveness (should return OK in under 1 second)
time curl -s --max-time 5 --unix-socket /var/run/docker.sock http://localhost/_ping

# Check container states for dead or restarting containers
docker ps -a --format '{{.State}}' | sort | uniq -c

# Check restart counts for all containers
docker inspect --format '{{.Name}} {{.RestartCount}}' $(docker ps -aq)

# Check for OOM-killed containers
docker ps -aq | xargs -I{} docker inspect --format '{{.Name}} OOMKilled={{.State.OOMKilled}}' {}

# Check Docker disk usage and reclaimable space
docker system df

# Check CPU throttling across containers (cgroup v2, systemd driver)
for cg in /sys/fs/cgroup/system.slice/docker-*.scope; do
  echo "$(basename $cg): $(grep -E 'nr_throttled|throttled_usec' $cg/cpu.stat)"
done

# Check memory OOM events across containers (cgroup v2, systemd driver)
for cg in /sys/fs/cgroup/system.slice/docker-*.scope; do
  echo "$(basename $cg): $(grep oom_kill $cg/memory.events)"
done

# Check daemon file descriptor usage against its limit
ls /proc/$(pgrep dockerd)/fd | wc -l
cat /proc/$(pgrep dockerd)/limits | grep "open files"

# Check for privileged containers
docker ps -q | xargs -I{} docker inspect --format '{{if .HostConfig.Privileged}}PRIVILEGED: {{.Name}}{{end}}' {} | grep PRIVILEGED

cgroup paths vary by driver and distribution. For systemd, which is the default on cgroup v2, the paths above use system.slice/docker-<id>.scope. If your host uses the cgroupfs driver, look under /sys/fs/cgroup/docker/<id>/ instead. Verify with systemd-cgls or ls /sys/fs/cgroup/system.slice/.

How To Diagnose It

Use this flow during an incident or when onboarding a new host.

1. Verify the daemon is responsive. Run curl --unix-socket /var/run/docker.sock http://localhost/_ping with a 5-second timeout. If it hangs, you have a daemon deadlock or storage driver hang. Check if container processes are still alive with ps aux | grep containerd-shim.
2. Check for disk exhaustion. Run docker system df and df -h /var/lib/docker/. If the filesystem is over 80% full, disk pressure is likely the root cause of secondary failures.
3. Inspect container states. Run docker ps -a. Any container in dead state indicates storage corruption. Containers in restarting indicate a crash loop.
4. Classify crashes. For any stopped container, check docker inspect for ExitCode and OOMKilled. Exit code 137 with OOMKilled: true means memory exhaustion. Exit code 1 means an application error. Exit code 139 means a segfault.
5. Check for throttling before blaming the application. If latency is high but CPU usage looks moderate, read cpu.stat in the container cgroup. Increasing nr_throttled means the CFS quota is too low.
6. Validate network from inside the container. Run docker exec <id> cat /etc/resolv.conf to confirm the embedded DNS at 127.0.0.11, then test resolution with nslookup. Network errors in docker stats indicate veth or bridge issues.
7. Audit security exposure. Scan for privileged containers, docker socket mounts, and added capabilities. Unexpected changes here can indicate compromise or misconfiguration.

Metrics & Signals To Monitor

Must-Have: Availability & Survival

These five signals tell you whether the daemon is working, whether disk is filling, and whether containers are running without crashing.

Must-Have: Resource Utilization

These signals catch resource pressure inside containers before it becomes an outage.

Should-Have: Storage, I/O & Daemon Internals

Monitor these to catch disk growth, I/O contention, and daemon stress before they cascade.

Nice-To-Have: Security & Deep Internals

Add these after you have coverage of the tiers above.

Fixes

Apply fixes based on the signal category.

If Disk Is The Bottleneck

As an emergency stopgap while containers run, truncate the largest unrotated log files: truncate -s 0 /var/lib/docker/containers/<id>/<id>-json.log; external truncation can break docker logs -f. Reclaim space with docker image prune -a for unused images and docker volume prune for unused volumes. Set log-opts in /etc/docker/daemon.json with max-size and max-file to prevent recurrence.

If Containers Are Crash-Looping

Check docker inspect for exit code and OOMKilled. If OOM, raise the memory limit or fix the leak. For exit code 1, read docker logs. Break the restart loop temporarily with docker update --restart=no <id> while you debug.

If CPU Throttling Is Causing Latency

Calculate the throttle percentage from nr_throttled / nr_periods in cpu.stat. Raise the CPU limit or switch to cpuset pinning for latency-sensitive workloads instead of CFS quotas.

If The Daemon Is Hung

Confirm container processes are still alive via ps or ctr -n moby containers list. If live-restore is enabled, restart dockerd with systemctl restart docker; running containers will survive. Without live-restore, a restart kills all containers.

Prevention

• Configure log rotation in /etc/docker/daemon.json with max-size and max-file defaults.
• Set memory limits that leave headroom for native allocations. For JVM workloads, set -Xmx to roughly 75 percent of the container limit.
• Enable meaningful health checks in every production image.
• Automate cleanup of exited containers and dangling images with a scheduled docker system prune or equivalent.
• Avoid --privileged and docker socket mounts in production workloads. Drop capabilities and run as non-root.
• Set net.netfilter.nf_conntrack_max to at least 262144 on busy hosts and monitor utilization.

How Netdata Helps

• Correlates container CPU usage with throttling metrics on the same chart, exposing the silent cause of latency spikes.
• Reads cgroup v2 memory.events to alert immediately on OOM kills without waiting for docker inspect.
• Tracks per-container disk usage and log growth to catch storage pressure before the host filesystem fills.
• Monitors dockerd health, API latency, and file descriptor usage alongside container metrics to detect daemon stress.

Related Guides

• Docker container high CPU usage: causes and fixes
• Docker container high memory usage: how to diagnose it
• Docker container keeps restarting: causes, checks, and fixes
• Docker container memory leak: how to find one and prove it
• Docker container running but unhealthy: how to diagnose health check failures
• Docker CPU throttling: the hidden cause of container latency
• Docker daemon not responding: how to troubleshoot a hung dockerd
• Docker disk space full: how to troubleshoot /var/lib/docker
• Docker DNS not working inside containers
• Docker exit code 137: OOMKilled or SIGKILL?
• Docker log rotation: preventing json-file logs from filling disk
• Docker logs taking too much disk space: how to fix log growth