Docker log rotation: preventing json-file logs from filling disk

Docker’s default json-file log driver appends every line of container stdout and stderr to a JSON file on the host under /var/lib/docker/containers/<id>/. Without size limits, that file grows monotonically. A single verbose container can consume tens of gigabytes, and because the driver is the default, this often happens silently until /var/lib/docker fills. At that point image pulls fail, container creates are rejected, and the daemon may hang on storage operations. This guide covers how to cap log files with daemon.json and per-container log-opt overrides, verify the caps are working, and choose a different driver when json-file is not appropriate.

What This Means

In production, json-file is convenient because docker logs works out of the box and the output is machine-readable JSON. The tradeoff is that the daemon keeps every log line on disk unless you tell it otherwise. The logs are stored in the container directory, but docker system df reports the writable layer, not json-file logs, so it can understate their disk cost. Exited containers also retain their logs until the container is removed. When disk exhaustion hits, the failure mode is a cascade: writes slow, daemon latency spikes, image pulls and container creates fail, and in extreme cases dockerd becomes unresponsive while containers continue running. Rotation is not a performance tweak. It is an availability requirement for any node using json-file.

Common Causes

Quick Checks

# Check the active default logging driver
docker info | grep -i "logging driver"

# Inspect daemon configuration for log options
cat /etc/docker/daemon.json

# Check a container's specific log configuration
docker inspect --format '{{json .HostConfig.LogConfig}}' <container_id>

# See container directory sizes, which include json-file logs and container metadata
du -sh /var/lib/docker/containers/*/ 2>/dev/null | sort -rh | head -10

# List stopped containers that still retain logs and layers
docker ps -a --filter status=exited --format '{{.ID}} {{.Names}}'

# Check Docker-wide disk usage and reclaimable space
docker system df -v

If daemon.json is missing or has no log-opts block, the host has no default rotation. If a container’s LogConfig shows no max-size, that container is unprotected regardless of the daemon default.

How To Diagnose It

1. Confirm the driver and locate the largest consumers. Run docker info | grep -i "logging driver" and du -sh /var/lib/docker/containers/*/. If the driver is json-file and individual container directories are hundreds of megabytes or larger, logs are the likely culprit. This tells you whether the fix is rotation or a driver migration. Next, inspect the daemon configuration.

2. Inspect daemon.json for log-opts. Look for a log-opts key containing max-size and max-file. Why: the daemon default applies to every container created after the daemon reads the policy. Result: if these keys are missing, nothing limits log growth. Next: add the policy and plan a daemon restart.

3. Inspect existing containers for overrides. Run docker inspect --format '{{json .HostConfig.LogConfig}}' on large containers. Why: per-container --log-opt flags override the daemon default. Result: if a container specifies its own driver or omits max-size, it will grow without the daemon safeguard. Next: recreate the container with explicit limits.

4. Verify container creation time relative to policy changes. Check when the container was created versus when daemon.json was last modified and the daemon restarted. Why: Docker applies logging configuration at container creation time; changes are not retroactive. Result: containers started before the policy change continue under the old rules. Next: recreate stale containers after the daemon restart.

5. Audit exited containers. Run docker ps -a --filter status=exited. Why: stopped containers retain log files until removed. Result: a large population of exited containers can explain disk pressure even when running containers look fine. Next: prune exited containers after confirming you do not need their logs.

6. Check application log velocity. Run docker logs --tail 100 <container_id>. Why: if an application emits megabytes per minute, a small max-size may rotate too frequently, while a large one may still exhaust disk if max-file is high. Result: you may need to reduce application verbosity or switch to a streaming driver such as fluentd or syslog.

flowchart TD
    A[Check json-file log sizes] --> B{Rotation configured in daemon.json?}
    B -->|No| C[Add max-size and max-file to daemon.json]
    B -->|Yes| D{Container created after config?}
    D -->|No| E[Recreate container to pick up policy]
    D -->|Yes| F[Verify log file count and sizes]
    F -->|Still growing| G[Check application log volume]
    F -->|Within limits| H[Monitor disk growth trends]
    C --> I[Restart dockerd]
    G --> J[Reduce verbosity or switch log driver]

Metrics & Signals To Monitor

Fixes

If The Cause Is Missing Rotation Configuration

Create or edit /etc/docker/daemon.json:

{
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "10m",
    "max-file": "3"
  }
}

Apply the change with a daemon restart:

# Disruptive: restarts dockerd and briefly affects all containers
systemctl restart docker

After the restart, only new containers pick up the policy. Existing containers continue with their previous configuration until recreated.

If The Cause Is Per-Container Overrides Without Limits

Recreate the container with explicit log options:

docker run --log-driver json-file \
  --log-opt max-size=10m \
  --log-opt max-file=3 \
  ...

Docker applies log configuration at creation time. You cannot update log driver options on a running container.

If The Cause Is An Unsuitable Log Driver

If you are shipping logs to a central system, writing everything to json-file on disk may be redundant. Consider an alternative:

Switching the daemon default requires editing daemon.json and restarting dockerd. Existing containers remain on their original driver until recreated.

If The Cause Is Application Log Volume

Reduce verbosity at the application level if possible. If the workload legitimately emits high volumes, avoid json-file entirely and stream directly to fluentd or syslog so the host disk is not the bottleneck.

If The Cause Is Accumulated Exited Containers

Remove stopped containers and reclaim their logs and writable layers:

# Destructive: removes all stopped containers
docker container prune -f

Warning: This deletes stopped containers and their logs. Confirm you do not need the data. For targeted cleanup, remove specific containers by ID or name.

Prevention

• Set log-opts in daemon.json as part of host provisioning. Do not rely on operators to remember --log-opt flags.
• Monitor Docker disk usage by containers and alert at 70% of the /var/lib/docker filesystem. Waiting until 95% leaves no room for cleanup operations.
• Schedule automated pruning of exited containers.
• Evaluate log driver choice during service design. json-file is convenient for local debugging but becomes a liability on high-traffic nodes.
• Test rotation by starting a verbose container and verifying that total log size stays within max-size multiplied by max-file.

How Netdata Helps

• Netdata tracks Docker disk usage by category, so you can see when container layers and logs dominate the total.
• Container state counts surface exited containers that are quietly consuming disk.
• Host disk space monitoring on the /var/lib/docker filesystem gives early warning before rotation policies are stressed.
• Correlating disk usage spikes with container start events helps distinguish log growth from image pulls or build cache.

Related Guides

• Docker logs taking too much disk space: how to fix log growth
• Docker disk space full: how to troubleshoot /var/lib/docker
• Docker daemon not responding: how to troubleshoot a hung dockerd
• Docker container keeps restarting: causes, checks, and fixes
• Docker monitoring checklist: the signals every production host needs