$ guides / kubernetes / kubernetes-volume-mount-failures ▌

Operations Guides

Kubernetes pod stuck on volume mount: CSI, permissions, and timeouts

You scale a StatefulSet and the new pods sit in ContainerCreating for ten minutes. The node is Ready. The CSI driver pods are running. kubectl describe shows no FailedMount events, yet the containers never start. The absence of volume events often misleads operators into checking image registries or resource quotas instead of the storage path. The kubelet volume manager is blocked somewhere between attach and mount, and Kubernetes will not retry fast enough to hide the problem.

Unlike image pull errors or scheduling failures, stuck volume mounts often leave the node looking healthy. Existing pods keep running. Only workloads that need storage fail to start, which makes the symptom easy to blame on the application instead of the storage path.

What this means

The control-plane attach/detach controller reconciles volume attachment, while the kubelet volume manager reconciles node-local volume state and mounts. For CSI volumes, the kubelet calls node RPCs to stage and publish the volume. Each operation holds a goroutine. If an operation hangs, that goroutine blocks. When enough hang, the volume manager saturates, and subsequent pods with volumes stall while the node stays Ready.

One hung NFS mount or a cloud volume still finishing force-detach can queue up and delay every new stateful pod on the node. Additionally, the kubelet applies fsGroup ownership changes recursively during mount; on large volumes this can stall pod startup without producing FailedMount events.

Common causes

Cause	What it looks like	First thing to check
CSI driver unavailable on the node	Events mention RPC errors or driver name; mount setup fails	CSI node plugin pod on the target node
Stale VolumeAttachment or multi-attach	Events mention multi-attach errors or attach waits indefinitely	VolumeAttachment objects and which node owns them
Permission or fsGroup mismatch	Container exits with permission denied on the volume path, or mount fails with access errors	Pod securityContext fsGroup and storage backend ownership
NFS or network storage hang	No FailedMount events; pod ContainerCreating for more than 10 min; node Ready	Manual mount test from the node to the storage endpoint
Cloud provider detach delay	Attach operation exceeds 2 minutes after previous node loss	Cloud provider console or volume status for pending detach

Quick checks

# Confirm the pod phase is ContainerCreating
kubectl get pod $POD_NAME -o jsonpath='{.status.phase}'

# Inspect events for MountVolume, AttachVolume, or RPC errors
kubectl describe pod $POD_NAME | grep -A 20 Events

# Identify if the volume is attached to a different node
kubectl get volumeattachment -o wide

# Verify the CSI node plugin is present and running
kubectl get pods -n $CSI_NAMESPACE -l app=$CSI_DRIVER --field-selector spec.nodeName=$NODE_NAME

# Scan cluster-wide volume errors
kubectl get events --all-namespaces | grep -E "FailedMount|AttachVolume|MountVolume"

# Read kubelet logs for low-level volume and CSI errors
journalctl -u kubelet --since "30 minutes ago" | grep -i "mount\|volume\|csi"

# Check storage operation latency from kubelet metrics
kubectl get --raw /api/v1/nodes/$NODE_NAME/proxy/metrics | grep storage_operation_duration_seconds

# Verify fsGroup is configured when containers run as non-root
kubectl get pod $POD_NAME -o jsonpath='{.spec.securityContext.fsGroup}'

# Inspect currently mounted volumes under the kubelet directory
mount | grep kubelet

# Test NFS reachability directly from the node with a timeout to avoid indefinite hangs
timeout 10 mount -t nfs $SERVER:$PATH /tmp/testmnt
# Unmount after testing: umount /tmp/testmnt

# Check CSI node plugin logs for RPC timeouts or driver errors
kubectl logs -n $CSI_NAMESPACE -l app=$CSI_DRIVER --field-selector spec.nodeName=$NODE_NAME --tail=200 | grep -iE "error|timeout|rpc"

How to diagnose it

Determine whether the delay is in attach or mount. Look at kubectl describe pod events and scroll past image pull and scheduling messages to find volume entries. If you see AttachVolume.Attach with a timestamp older than two minutes, the problem is attach. If you see MountVolume.SetUp without completion, the problem is mount.
Inspect VolumeAttachment objects. Compare each VolumeAttachment’s .spec.nodeName against the pod’s .spec.nodeName. If the volume is attached to a different node, flag it as stale for cleanup after you verify the previous workload has stopped.
Verify the CSI node driver pod on the target node. If the pod is not running, volume operations cannot proceed.
Check kubelet storage operation metrics. As rough thresholds, local volumes should mount in under one second, block devices under ten seconds, network volumes under sixty seconds, and cloud attach operations under two minutes. Operations exceeding these indicate a backend hang or slowness.
Examine permission errors. If the container starts but cannot write to the mount path, compare the pod’s securityContext.fsGroup and runAsUser against the filesystem ownership on the storage backend. Correct mismatches in the pod spec.
For network storage, test reachability from the node directly. If a manual mount command hangs, the issue is outside Kubernetes. If it succeeds but the pod still stalls, the problem is likely in the kubelet volume manager or CSI driver.
Review kubelet logs for RPC errors from CSI drivers or mount syscall failures. RPC errors with codes like Internal or DeadlineExceeded point to driver-specific bugs or resource exhaustion.
Check for volume manager saturation. If multiple pods on the same node are stuck in ContainerCreating and the node is Ready, look for the first hung operation. Resolving that operation often unblocks the rest.

flowchart TD
    A[Pod stuck in ContainerCreating] --> B{Events show attach or mount?}
    B -->|Attach timeout / Multi-Attach| C[Check VolumeAttachment objects]
    B -->|MountVolume.SetUp failed / RPC error| D[Check CSI driver pod health and logs]
    B -->|No events / stale for >10 min| E[Test storage reachability from node]
    C -->|Attached to wrong node| F[Delete stale VolumeAttachment]
    C -->|Cloud attach >2 min| G[Check cloud provider detach status]
    D -->|CSI pod unhealthy| H[Restart CSI node plugin]
    D -->|RPC permission error| I[Check pod fsGroup and SELinux context]
    E -->|Mount hangs| J[Review NFS mount options or network path]
    E -->|Mount succeeds| K[Check kubelet storage operation metrics]
    F --> L[Wait for controller to reattach and verify pod startup]
    G --> L
    H --> L
    I --> L
    J --> L
    K --> L

Metrics and signals to monitor

Signal	Why it matters	Warning sign
`storage_operation_duration_seconds`	Reveals attach and mount latency before pods visibly stall	p99 exceeds 30 s for network volumes or 2 min for cloud attach
`storage_operation_duration_seconds{status="fail-unknown"}`	Tracks volume operations that completed with an unknown failure	Sustained rate increase over a 5-minute window
`volume_manager_total_volumes`	Count of volumes the kubelet is managing	Sudden spike correlating with ContainerCreating pods
Pod phase ContainerCreating duration	Direct symptom of stuck mounts	Duration > 5 minutes
Node Ready condition	Nodes can remain Ready while the volume manager is blocked	Ready=True but volume-mounted pods fail to start
`kubelet_pod_worker_duration_seconds`	Slow reconciliation can indicate blocked volume operations	Sustained > 30 s
`kubelet_runtime_operations_duration_seconds`	Distinguishes runtime slowness from storage issues	`list_containers` p99 > 5 s
VolumeAttachment object age	Stale attachments block reattachment	Older than the cloud provider’s normal detach window

Fixes

If the cause is CSI driver failure

Restart the CSI node plugin pod on the affected node. Check the CSI controller and node plugin logs for RPC errors such as Internal, DeadlineExceeded, or Unavailable. These indicate the driver cannot reach the storage backend, the node plugin socket is unreachable, or the driver is exhausting its memory limits. If the driver is crashing due to resource pressure, increase its memory limit or move it to a node pool with headroom. Do not restart kubelet until the driver is healthy.

If the cause is stale VolumeAttachment or multi-attach

Delete the stale VolumeAttachment object:

kubectl delete volumeattachment <name>

Warning: Only delete a VolumeAttachment after you verify the volume is no longer in use on the previous node. Deleting it while the volume is still mounted can cause data corruption or unclean filesystem state; with a matching pod or node intent, the attach/detach controller will recreate the required attachment. For cloud-backed volumes after a node failure, allow the cloud provider’s force-detach window to complete before manually detaching the volume, or detach it through the provider’s console if the cluster is stuck.

If the cause is permission or ownership mismatch

Add or correct securityContext.fsGroup in the pod spec to align with the group owning the files on the backing storage. For SELinux enforcing environments, ensure the volume context is compatible or configure seLinuxOptions in the securityContext. Recreate the pod to apply changes.

If the cause is network storage hang

For NFS, prefer mount options that prevent indefinite hard hangs, such as soft and a reasonable timeo value, while accepting that soft can return I/O errors to the application if the server becomes unreachable. If a mount is already hung, identify the blocking process before acting. Do not kill kubelet or the mount helper without first assessing workload impact; an unclean termination can leave the volume in an inconsistent state and prevent clean unmount. Restarting the kubelet without fixing the storage endpoint rarely helps and can extend recovery time.

If the cause is cloud API throttling or detach delay

Reduce the rate of new volume attachments to stay within the cloud provider’s API limits and volume queue depths. Verify that the previous node has fully released the volume before rescheduling stateful pods. If a node was preempted or terminated, expect a forced-detach delay.

Prevention

Monitor storage_operation_duration_seconds and alert on p99 thresholds before pods stall.
Set fsGroup and expected runAsUser explicitly in pod specs for volumes written by non-root containers.
Configure NFS and network storage mounts with soft and timeo to avoid hard hangs that saturate the volume manager.
Keep CSI driver and sidecar versions current, and monitor their pod health independently from kubelet health.
Maintain node disk and inode headroom; disk pressure slows image pulls and can indirectly delay volume reconciliation.
Track the gap between running and desired pod counts on each node to detect volume manager saturation early.
Document your storage backend’s attach limits and enforce them through CSI driver flags or scheduler constraints.
Test storage endpoint fail-over and recovery procedures in staging so that hang behavior is understood before it happens in production.

How Netdata helps

Correlates storage_operation_duration_seconds with node disk I/O wait and network latency to distinguish a slow storage backend from a saturated kubelet volume manager.
Tracks kubelet sync loop and PLEG relist latency alongside pod start duration to identify whether volume mounts are the bottleneck during startup storms.
Alerts on node filesystem usage and inode consumption before disk pressure triggers evictions that complicate volume recovery.

The Netdata solution

Kubernetes monitoring with Netdata

Netdata monitors Kubernetes with per-second metrics across the control plane, nodes, and every pod, with ML anomaly detection and zero per-pod configuration. Correlate API-server and etcd latency, kubelet PLEG stalls, scheduling pressure, and OOMKills in one place.

See Kubernetes monitoring → Start monitoring free