Kubernetes pod stuck Terminating: finalizers, grace periods, and force delete

A pod stuck in Terminating stays visible in the API server after kubectl delete, sometimes for minutes or hours. Usually a finalizer blocks removal, a CSI volume is still attached, or the container is ignoring SIGTERM. Force deleting without diagnosis orphans containers and can violate StatefulSet guarantees. Check the signals first, then decide whether to wait, patch a finalizer, or force delete.

What This Means

When you delete a pod, the API server sets metadata.deletionTimestamp and starts a graceful deletion window controlled by terminationGracePeriodSeconds (default 30). The kubelet executes any preStop hooks, sends SIGTERM to all containers, and waits up to the grace period before sending SIGKILL.

The pod object stays in the API server until metadata.finalizers is empty. Controllers, CSI drivers, and operators add finalizers to enforce cleanup ordering. If a controller is down, a volume cannot unmount, or a node fails, the finalizer persists and the pod stays Terminating. Since Kubernetes 1.27, kubelet transitions deleted pods to a terminal phase (Failed or Succeeded) before the API server removes them, with two exceptions: static pods and force-deleted pods that have no finalizer.

If the application ignores SIGTERM or a preStop hook hangs, the effective termination time can exceed the configured grace period. In Kubernetes 1.26, upstream issue #115817 documents behavior where the kubelet may grant a second full grace period after a hung preStop hook times out, doubling the wait. For pods with CSI volumes, the kubelet depends on the volume manager to signal the CSI driver when it is safe to unmount. On node shutdown, the CSI driver may terminate before workload volumes fully unmount, leaving the Attach/Detach controller to wait several minutes before issuing a force detach.

Common Causes

Quick Checks

# Check deletion timestamp and configured grace period
kubectl get pod <pod-name> -o jsonpath='deletionTimestamp={.metadata.deletionTimestamp}{"\n"}gracePeriod={.spec.terminationGracePeriodSeconds}{"\n"}'

# Inspect finalizers blocking deletion
kubectl get pod <pod-name> -o jsonpath='{.metadata.finalizers}'

# Check if containers are still running on the node
crictl ps -a | grep <pod-name>

# List volume attachments for the node
kubectl get volumeattachment -o json | jq '.items[] | select(.spec.nodeName=="<node>") | {name: .metadata.name, attached: .status.attached}'

# Check kubelet logs for unmount or volume errors
journalctl -u kubelet --since "10 minutes ago" | grep -i "unmount\|volume.*failed\|pleg"

# Check node readiness
kubectl get node <node-name> -o jsonpath='{range .status.conditions[?(@.type=="Ready")]}{.type}={.status}{"\n"}{end}'

# Review recent pod events for lifecycle hook failures
kubectl get events --field-selector involvedObject.name=<pod-name> --sort-by='.lastTimestamp'

# Check API server DELETE latency during mass termination events
kubectl get --raw /metrics | grep 'apiserver_request_duration_seconds_bucket{.*verb="DELETE".*resource="pods"'

How To Diagnose It

1. Confirm the delete reached the API server. Verify deletionTimestamp is set. If it is missing, the delete request may not have reached the API server.

2. Inspect finalizers. If metadata.finalizers is not empty, identify which controller owns the finalizer. A missing or crashed controller never removes the finalizer. Patching it manually allows the API server to complete deletion, but you must accept that whatever cleanup the finalizer was protecting is skipped.

3. Check if containers are still running. If crictl ps -a on the node shows the container as Running, the kubelet has not yet sent SIGKILL. The grace period has not expired, or a preStop hook is still executing. If the grace period has expired and the container is still running, the application is likely ignoring SIGTERM.

4. Look for volume attachment hangs. If the pod used a CSI PersistentVolume, check kubectl get volumeattachment. A stuck VolumeAttachment on a NotReady node often means the CSI driver or attach-detach controller has not completed cleanup. The controller-manager may wait roughly six minutes before force-detaching. If the node is shut down, the CSI driver on the node may have been terminated before unmount completed.

5. Evaluate node health. A node with an unhealthy PLEG, runtime disconnection, or kubelet OOM can lose the ability to report container status or complete unmounts. If the node is NotReady, eviction may be initiated but the kubelet cannot complete it.

6. Choose the intervention. If a finalizer is stuck and the owning controller is permanently gone, patch the finalizers. If a volume is stuck and the node is dead, wait for the attach-detach controller timeout or manually detach if the storage backend allows. If the application is ignoring SIGTERM, reduce terminationGracePeriodSeconds after fixing the application. Only use force delete when you accept that the container may continue running on the node and that StatefulSet at-most-one semantics may be violated.

flowchart TD
    A[Pod stuck Terminating] --> B{Finalizers present?}
    B -->|Yes| C[Check owning controller]
    B -->|No| D{Containers still running?}
    C --> E[Patch finalizers null or fix controller]
    D -->|Yes| F{preStop hook running?}
    D -->|No| G{VolumeAttachment exists?}
    F -->|Yes| H[Wait or reduce grace period]
    F -->|No| I[Check SIGTERM handler in app]
    G -->|Yes| J[Verify CSI driver and node status]
    G -->|No| K[Force delete as last resort]
    J --> L[Detach stuck volume or remove finalizer]
    I --> M[Fix app signal handling]

Metrics & Signals To Monitor

Fixes

If The Cause Is Finalizers

Removing a finalizer tells the API server to delete the object immediately. It does not stop running processes or clean up external resources. Use this only when the owning controller is gone and you accept the leak.

# Destructive: removes all finalizers and skips external cleanup
kubectl patch pod <pod-name> -p '{"metadata":{"finalizers":null}}' --type=merge

If the finalizer belongs to a CSI driver and the volume is already unmounted at the storage layer, manual removal is usually safe. Verify the backend state first.

If The Cause Is A Volume Detach Hang

For CSI volumes on a failed node, wait for the attach-detach controller to force-detach after its timeout. If you cannot wait, verify the volume is no longer mounted on the node. You can then remove the VolumeAttachment manually if your storage backend supports it.

Warning: Manual VolumeAttachment removal can corrupt data if the volume is still mounted or writing somewhere.

If you see repeated GetDeviceMountRefs errors in kubelet logs for a shared SMB or NFS mount, another pod on the same node may hold the mount reference. Terminate co-located pods that share the mount to break the reference.

If The Cause Is Ignored SIGTERM Or A Hung preStop Hook

Fix the application to handle SIGTERM by draining connections and exiting. If you use a preStop hook, give it a strict timeout shorter than the grace period. Note that a preStop hook timeout does not reduce the remaining grace period in some Kubernetes versions, which can double the total wait time.

If The Cause Requires Force Deletion

Force deletion is the escape hatch when graceful deletion is impossible.

# Destructive: skips graceful shutdown and volume unmount
kubectl delete pod <pod-name> --grace-period=0 --force

Tradeoffs: the pod name is freed immediately, but the container may keep running on the node. For StatefulSet pods, this can violate at-most-one semantics because the controller may create a replacement before the original container stops. Only force delete StatefulSet pods when you can verify the original is no longer running or communicating with peers.

Prevention

• Set terminationGracePeriodSeconds to the application’s actual drain time. Do not leave the default 30 seconds if the app exits in 5.
• Ensure custom controllers remove their finalizers promptly during deletion handling.
• Monitor CSI driver health. A driver that crashes during unmount leaves finalizers and VolumeAttachments behind.
• Avoid shared mounts where multiple pods on the same node reference the same underlying SMB or NFS path.
• Test application SIGTERM handling in CI. A container that exits on SIGTERM avoids most Terminating delays.
• Keep node density and kubelet resource reservations within tested limits. Kubelet memory pressure or PLEG stalls directly delay termination.

How Netdata Helps

• Correlate kubelet_pleg_relist_duration_seconds spikes with pod termination delays to detect kubelet-side bottlenecks.
• Monitor container runtime operation errors and kubelet sync loop duration to catch unmount failures before pods stall.
• Track API server DELETE latency to identify control plane saturation during mass terminations.
• Alert on node NotReady transitions and kubelet OOM events that prevent graceful shutdown.

Related Guides

• Kubernetes kubelet not responding: PLEG, runtime, and certificate issues
• Kubernetes kubelet memory leak: detection and OOM cycle
• Kubernetes kubelet certificate expired: detection, rotation, and recovery
• Kubernetes eviction cascade: when one node failure takes down the cluster
• Kubernetes API server slow or unresponsive: causes and fixes