Kubernetes Pod Security Standards violations: detection and remediation

You apply a Deployment and the ReplicaSet creates zero pods. You roll out a DaemonSet update and new pods vanish without events. You add a namespace label to improve security posture, and CI pipelines fail with opaque admission errors. Pod Security Admission (PSA) blocks non-compliant workloads silently when enforce mode is active. To fix these violations, correlate namespace labels with enforcement outcomes, parse audit log annotations into exact spec changes, distinguish admission-time rejections from runtime failures, and prevent enforcement from silently blocking future rollouts.

What this means

Pod Security Admission (PSA) enforces three Pod Security Standards (PSS) levels per namespace: privileged (unrestricted), baseline (blocks known privilege escalations), and restricted (hardening best practice). Enforcement is controlled through labels:

  • pod-security.kubernetes.io/<MODE>: <LEVEL>
  • pod-security.kubernetes.io/<MODE>-version: <VERSION> (optional)

MODE is enforce, audit, or warn. LEVEL is privileged, baseline, or restricted.

When a namespace carries an enforce label, the API server rejects pods that violate the specified level. audit mode writes a violation annotation to the audit log but allows the pod. warn returns a warning to the user. These modes operate independently; a namespace can enforce restricted while warning on baseline violations.

PSA evaluates existing pods retroactively when an enforce label is added or changed. Running non-compliant pods are not evicted, but any future creation or update is blocked. The workload appears healthy until the next rolling update, when new pods are rejected and the rollout stalls.

PSA also evaluates workload resources (Deployment, Job, DaemonSet, and so on) in audit and warn modes, catching violations before the controller produces pods. Exemptions are configured statically in the AdmissionConfiguration passed to the API server, scoped to usernames, runtimeClasses, or namespaces. Exempted requests skip all PSA checks entirely.

flowchart TD
    A[Namespace labeled enforce restricted] --> B{Pod creation attempt}
    B -->|Violates policy| C[API server rejects pod]
    B -->|Audit mode| D[Audit log writes audit-violations annotation]
    B -->|Warn mode| E[kubectl returns warning to caller]
    C --> F[Deployment shows zero ready replicas]
    D --> G[Grep audit logs for pod-security.kubernetes.io/audit-violations]
    E --> H[Check namespace labels for warn mode]
    G --> I[Map message to securityContext field]
    H --> I
    I --> J[Patch workload and recreate]

Common causes

CauseWhat it looks likeFirst thing to check
Missing restricted securityContext fieldsDeployment ReplicaSet creates pods that are immediately rejectedContainer securityContext for runAsNonRoot, capabilities.drop, and seccompProfile
Namespace moved to enforce without prior auditExisting pods run, but rollouts or scale-ups failNamespace labels and the age of the enforce label
runAsNonRoot without explicit runAsUserPod is admitted but container exits with a uid errorWhether runAsUser is set in the pod or container spec
Mutating webhook injecting privileged fieldsWorkload resource passes PSA but the running pod violates policyMutatingWebhookConfiguration and diff between template and live pod
Disallowed volume type under restrictedStatefulSet or Job pods rejected on creationVolume types against the restricted allowlist
Retroactive enforcement on old workloadsNew pods blocked in a namespace that previously allowed everythingExisting pod compliance in the labeled namespace

Quick checks

# Check namespace PSS labels
kubectl get namespace <namespace> -o jsonpath='{.metadata.labels}'

# Search audit logs for recorded violations
grep pod-security.kubernetes.io/audit-violations /var/log/kubernetes/audit.log

# Check API server logs for PodSecurity prefix messages
kubectl logs -n kube-system kube-apiserver-<node> | grep -i PodSecurity

# Find privileged containers cluster-wide
# Caution: on large clusters, scope to a namespace to avoid client-side memory pressure
kubectl get pods -A -o json | jq '.items[] | select(.spec.containers[]?.securityContext.privileged == true) | {name: .metadata.name, namespace: .metadata.namespace}'

# Find pods using hostPath volumes
# Caution: on large clusters, scope to a namespace to avoid client-side memory pressure
kubectl get pods -A -o json | jq '.items[] | select(.spec.volumes[]?.hostPath != null) | {name: .metadata.name, namespace: .metadata.namespace, path: .spec.volumes[].hostPath.path}'

# Inspect container securityContext for missing runAsUser
kubectl get pod <pod> -o jsonpath='{.spec.containers[*].securityContext}'

How to diagnose it

  1. Identify the enforcement scope. Read the namespace labels. Look for pod-security.kubernetes.io/enforce, audit, and warn. If enforce is set, violations block pod creation. If only audit or warn are present, the workload is admitted while leaving a trail.

  2. Collect the exact violation message. In audit mode, filter the API server audit log for the annotation pod-security.kubernetes.io/audit-violations. The value states the policy version and the exact field path that failed, for example: would violate PodSecurity "restricted:latest": allowPrivilegeEscalation != false (container "example" must set securityContext.allowPrivilegeEscalation=false).

  3. Distinguish admission rejection from runtime failure. If the API server rejected the pod, the controller shows failed replicas or a stalled rollout. If the pod was admitted but uses runAsNonRoot: true without an explicit runAsUser, the container runtime will fail to start the container. Check the container status for uid errors.

  4. Check for exemptions. Look for the annotation pod-security.kubernetes.io/exempt in audit logs. If present, the request matched an exemption dimension and skipped all checks. Verify that the exemption is intentional and narrowly scoped.

  5. Inspect workload templates, not just pods. A submitted Deployment may appear compliant, but a mutating webhook can inject violating fields before the pod is created. Diff the live pod spec against the workload template.

  6. Verify existing pods in retroactively labeled namespaces. PSA evaluates existing pods when an enforce label is added but does not evict them. Patch controller templates before the next rollout to avoid creation failures.

  7. Check for mutating webhook interference. If the live pod contains privileged: true, hostPath volumes, or other denied fields absent from the workload template, a mutating webhook injected them. Review the MutatingWebhookConfiguration and check the API server metric apiserver_admission_webhook_admission_duration_seconds to confirm the webhook is active in the path.

  8. Validate the policy version pin. If the namespace uses latest or an unpinned version, a cluster upgrade can introduce new restrictions. Check pod-security.kubernetes.io/enforce-version and compare it to the cluster version to rule out version drift.

Metrics and signals to monitor

SignalWhy it mattersWarning sign
Audit log events with pod-security.kubernetes.io/audit-violationsThis is the direct evidence of policy breachesSustained increase or new violation types appearing
Pod creation failure rate for pod resourcesEnforce mode blocks non-compliant pods before they startElevated apiserver_request_total with non-2xx code on pod creation in labeled namespaces
Deployment rollout statusRestricted enforcement stalls rollouts when replicas cannot be createdreadyReplicas below spec.replicas with ProgressDeadlineExceeded
ReplicaSet failures in enforce-labeled namespacesBlocked creations never reach schedulingDesired replicas exceed ready replicas with no new pod objects created
API server admission webhook latencyMutating webhooks can inject violating fields after PSA validationapiserver_admission_webhook_admission_duration_seconds elevated for mutating webhooks
CrashLoopBackOff in restricted namespacesRuntime failures from runAsNonRoot without runAsUserContainers exiting with uid or permission errors
Namespace label changesRetroactive enforcement can suddenly block future workloadsAddition of enforce labels without preceding audit mode

Fixes

If the cause is a restricted profile field violation

Add the required fields to the container securityContext. Set runAsNonRoot: true. PSA restricted accepts this as satisfying the non-root user requirement, but if the image lacks a non-root USER directive, pair it with an explicit runAsUser integer to avoid runtime uid errors. Add capabilities.drop: ["ALL"]. Set seccompProfile.type: RuntimeDefault.

If the cause is a baseline profile field violation

Remove privileged: true, hostNetwork, hostPID, hostIPC, and hostPath volumes from the pod spec. Under baseline, seLinuxOptions.role and seLinuxOptions.user must be empty or undefined.

If the cause is a disallowed volume type

Replace the volume with a type permitted under restricted: configMap, csi, downwardAPI, emptyDir, ephemeral, persistentVolumeClaim, projected, or secret. Remove hostPath, nfs, gcePersistentDisk, and other forbidden volume types.

If the cause is runtime failure from incomplete securityContext

If the pod is admitted but the container fails with a runAsNonRoot uid error, add an explicit runAsUser value to the container or pod securityContext. Verify that the image does not default to root.

If the cause is retroactive enforcement on existing workloads

Patch the workload template to comply, then trigger a rolling update:

kubectl rollout restart deployment/<name> -n <namespace>

Existing pods continue running until replaced. PSA does not evict them.

If the cause is mutating webhook interference

Review the MutatingWebhookConfiguration. If the webhook injects privileged sidecars, hostPath volumes, or removes securityContext fields, fix the webhook or narrow its scope. If the webhook is required and the injected behavior must be exempted, add a targeted exemption in the AdmissionConfiguration for that webhook’s namespace or runtime class, but avoid broad exemption lists.

If the cause is exemption misconfiguration

Audit the AdmissionConfiguration exemptions. Exempted namespaces, usernames, or runtime classes skip enforce, audit, and warn entirely. Remove overly broad entries and document any remaining exemptions.

Prevention

Apply audit labels to namespaces for at least one full release cycle before switching to enforce. Pin the policy version explicitly, for example pod-security.kubernetes.io/enforce-version: v1.25, instead of using latest. Test workload templates with server-side dry-run before deploying to production:

kubectl apply --dry-run=server -f manifest.yaml -n <namespace>

Review exemption configurations quarterly. Monitor audit log annotations proactively; a violation in audit mode today becomes a blocked deployment tomorrow. Document allowed volume types and securityContext patterns for teams deploying into restricted namespaces.

How Netdata helps

  • Correlate API server latency spikes with admission webhook delays that mutate pods after PSA validation.
  • Track CrashLoopBackOff counts to surface runtime failures from incomplete securityContext.
  • Monitor controller workqueue depth and deployment rollout status to catch stalled rollouts caused by enforce rejections.
  • Alert on elevated API server error rates for pod resources, providing an early signal that PSS enforcement is blocking creations.
  • Track API server request rates and response codes to detect admission rejections in labeled namespaces.