Kubernetes bound service account tokens: rotation, audience, and expiry

Pods fail with 401 Unauthorized, CSI volume mounts hang with token errors, or security audits surface long-lived credentials that never rotate. Bound tokens are projected, audience-scoped, and short-lived. Legacy tokens are long-lived static Secrets. After Kubernetes 1.24, both coexist in most upgraded clusters. This guide covers the TokenRequest API lifecycle, kubelet rotation behavior, audience binding, and version-specific changes from 1.24 through 1.33 to help you diagnose auth failures and remove stale credentials safely.

What this means

Kubernetes bound service account tokens are JWTs issued by the TokenRequest API. Each token is bound to a specific Pod, carries an aud claim array, and expires. The default TTL is 1 hour (3600 seconds), with a minimum of 10 minutes (600 seconds). The kubelet requests the token from the API server using the Pod’s ServiceAccount and mounts it via the serviceaccount volume plugin. It rotates the file when the token age exceeds 80 percent of its TTL or when it is older than 24 hours. At the default TTL, rotation begins roughly 48 minutes after issuance.

If no audience is specified in the TokenRequest, the token audience defaults to the API server identifier (the value of --service-account-issuer). Recipients must verify that they identify as one of the aud values. A token projected with audience: vault cannot be validated by the Kubernetes TokenReview API unless api or the matching issuer audience is included in the audience list.

Projected token files are written with mode 0600, or 0640 if the Pod sets securityContext.fsGroup. Legacy secret-mounted tokens default to world-readable 0644.

Legacy tokens are Secret objects of type kubernetes.io/service-account-token. They are static, unbound, and carry no expiration. Since v1.24, Kubernetes no longer auto-creates these Secrets for every ServiceAccount, but clusters upgraded from older versions often retain them. From v1.29, unused auto-generated legacy tokens are labeled kubernetes.io/legacy-token-invalid-since and tracked by the serviceaccount_invalid_legacy_auto_token_uses_total metric. The LegacyServiceAccountTokenCleanUp controller, GA in v1.30, automatically deletes unused auto-generated legacy tokens after the cleaner’s one-year cleanup period. A token must also be referenced by a ServiceAccount and not mounted by a pod to be eligible for marking and deletion; check the controller’s settings and kubernetes.io/legacy-token-invalid-since and kubernetes.io/legacy-token-last-used labels for the actual age on a cluster.

flowchart LR
    A[TokenRequest API] -->|binds to Pod, audience, TTL| B[Projected Volume]
    B --> C[Kubelet Volume Plugin]
    C -->|rotate at >80% TTL or >24h| D[Token File on Disk]
    D --> E[Application or CSI Driver]
    E -->|verify aud| F[K8s API or External IDP]

Common causes

Quick checks

# Decode a projected token to inspect claims
kubectl exec -n <ns> <pod> -- cat /var/run/secrets/kubernetes.io/serviceaccount/token | \
  cut -d. -f2 | base64 -d 2>/dev/null | jq '{aud, exp, iat}'

# Check token file age in seconds to verify rotation cadence
kubectl exec -n <ns> <pod> -- sh -c 'echo $(( $(date +%s) - $(stat -c %Y /var/run/secrets/kubernetes.io/serviceaccount/token) ))'

# List legacy service account token Secrets across the cluster
kubectl get secrets -A --field-selector type=kubernetes.io/service-account-token \
  -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers

# Check a CSIDriver's token request audience
kubectl get csidriver <name> -o jsonpath='{.spec.tokenRequests}'

# Check API server token issuer setting on a control plane node
ps aux | grep kube-apiserver | grep -o '\-\-service-account-issuer=[^ ]*'

# Find stale legacy tokens marked invalid (v1.29+)
kubectl get secrets -A -o json | \
  jq -r '.items[] | select(.metadata.labels."kubernetes.io/legacy-token-invalid-since") | "\(.metadata.namespace)/\(.metadata.name)"'

How to diagnose it

1. Identify the token type. If the Pod relies on a Secret reference created before 1.24, it is likely legacy. If the Pod spec uses a projected volume or automount on a 1.24+ cluster, it is bound.
2. Inspect the JWT claims. Decode aud, exp, and iat. Confirm the aud array contains every audience the recipient expects. For Kubernetes TokenReview, the audience must include api or the matching issuer identifier.
3. Verify rotation. Check the token file age on the Pod filesystem. At the default 1-hour TTL, kubelet begins rotation after roughly 48 minutes. If the file age exceeds 80 percent of expirationSeconds, investigate kubelet volume health and node conditions on that node.
4. Check kubelet logs. Look for errors related to projected volume updates or TokenRequest calls. On systemd-based nodes, use journalctl -u kubelet. PLEG delays, disk pressure, or volume manager stalls can block the kubelet from writing the rotated token.
5. Audit legacy secrets. In clusters upgraded past 1.24, list Secrets by type kubernetes.io/service-account-token. Look for the kubernetes.io/legacy-token-invalid-since label introduced in v1.29. The cleanup controller removes these automatically in 1.30+ if they remain unused; verify controller activity before manual deletion.
6. Examine CSI drivers. If the failure occurs during volume mount, inspect CSIDriver.spec.tokenRequests. ServiceAccountNodeAudienceRestriction was beta and disabled by default in v1.32; it became beta and enabled by default in v1.33, so test each affected CSI driver and audience on a staging node before upgrading. Ensure the requested audience is present in the Pod spec or explicitly granted via RBAC with verb request-serviceaccounts-token-audience.
7. Correlate 401 errors. Match the failing Pod’s service account, the token’s exp claim, and the API server or external service logs. Clock skew between node and control plane can make a valid token appear expired.

Metrics and signals to monitor

Fixes

If the cause is audience mismatch

Add the required audience to the Pod’s projected volume spec. If a token is used for both an external vault (audience: vault) and the Kubernetes API (audience: api), include both in the audiences list. Do not assume the default issuer audience satisfies external verifiers.

If the cause is rotation stall

Evict the Pod to force a fresh projected volume mount on a healthy kubelet. Warning: Eviction is disruptive and will terminate running containers.

Check the node for kubelet volume manager errors, PLEG delays, or disk pressure that prevent the kubelet from updating the token. Restarting the kubelet remounts all projected volumes on the node, but this is disruptive to every Pod on the node.

If the cause is legacy token accumulation

Delete manually created legacy Secrets only after confirming no workload references them. Warning: Deleting an in-use Secret will cause immediate 401 failures.

For auto-generated legacy Secrets, rely on the LegacyServiceAccountTokenCleanUp controller (GA in v1.30) to remove unused tokens after the one-year cleanup period; verify the controller’s activity and labels before assuming deletion timing. You may delete Secrets carrying the kubernetes.io/legacy-token-invalid-since label if you have verified they are unused.

If the cause is CSI driver token request failure

If you hit a v1.32 service-account-audience regression with a CSI driver, test with Kubernetes 1.33 or later, where ServiceAccountNodeAudienceRestriction is enabled by default; verify the driver’s release notes before upgrading. Ensure the CSI driver token request audience is declared in the Pod’s projected volume or that the node’s kubelet has RBAC to request that audience via the request-serviceaccounts-token-audience verb.

If the cause is provider breakage post-1.24

Update Terraform providers and Helm charts that rely on kubernetes_service_account.default_secret_name. Create an explicit Secret of type kubernetes.io/service-account-token if a static reference is required, or switch to kubectl create token or the TokenRequest API in your pipeline.

Prevention

• Use projected volumes for all new workloads. Set automountServiceAccountToken: false on ServiceAccounts that do not need API access, and mount projected tokens explicitly where needed.
• Set expirationSeconds to a value your workload and kubelet can sustain. The default 3600 seconds is appropriate for most cases; avoid values below 600 seconds unless you have verified kubelet can rotate under load.
• Audit Secrets quarterly. Remove legacy tokens before they accumulate. Enable LegacyServiceAccountTokenCleanUp if not already active.
• Validate CSI driver audiences in staging before upgrading to versions that enforce ServiceAccountNodeAudienceRestriction.
• Monitor TokenRequest rates and 401 errors per service account. Treat a rising 401 rate from a known service account as a token hygiene alert.

How Netdata helps

• Surfaces API server request rates for the serviceaccounts/token subresource to detect TokenRequest storms.
• Tracks kubelet volume operation latency and PLEG health to catch rotation delays before tokens expire.
• Monitors etcd object counts and control plane storage growth to surface legacy token accumulation.
• Alerts on node disk pressure that can block kubelet projected volume updates.
• Correlates API server 401/403 rates with node-level signals to narrow the failure domain.

Related guides

See How the Kubernetes control plane works for the mental model behind TokenRequest and kubelet volume management.

See Kubernetes API server certificate rotation for handling control-plane credential expiry.

See Kubernetes API server slow or unresponsive if TokenRequest latency is part of a broader API degradation.

See Kubernetes anonymous API access to distinguish token failures from anonymous auth probes.