$ guides / kubernetes / kubernetes-anonymous-access-detection ▌

Operations Guides

Kubernetes anonymous API access: detection, audit, and lockdown

Anonymous requests to the Kubernetes API server are authenticated as system:anonymous and evaluated as part of the system:unauthenticated group. On many self-managed clusters this behavior is enabled by default, which means requests that arrive without a valid client certificate, bearer token, or other credential are passed to the authorization layer instead of being rejected immediately. Public discovery endpoints and health checks are common legitimate uses, but anonymous access to namespaced resources, secrets, or RBAC objects is a direct security exposure.

The audit log is the authoritative record of this activity. If you have not reviewed it specifically for system:anonymous, you may already have undetected probing, misconfigured automation, or overly permissive bindings. This guide walks through a focused audit procedure: detecting anonymous requests in audit logs, classifying them into legitimate and illegitimate patterns, locking the API server down via --anonymous-auth=false where possible, and pruning RBAC bindings that reference anonymous identities.

flowchart TD
    A[Audit log: grep for system:anonymous] --> B{Endpoint pattern?}
    B -->|/healthz /version| C[Legitimate baseline]
    B -->|/api /apis| D[Discovery only if intended]
    B -->|Secrets Pods RBAC| E[Critical: immediate lockdown]
    C --> F[Continue monitoring]
    D --> G[Review RBAC bindings]
    E --> G
    G --> H[Disable --anonymous-auth or prune RBAC]
    H --> I[Verify 401 responses and log drop]

What anonymous API access means and why it matters

When the API server receives a request with no authentication material, it does not immediately return 401. Instead, if --anonymous-auth is true, it assigns the identity system:anonymous and the group system:unauthenticated and proceeds to authorization. RBAC, webhook, or other authorizers then decide whether to allow or deny the request based on rules that may explicitly or implicitly cover that group. Many distributions grant anonymous users read access to discovery and health endpoints by default, which is why unauthenticated curls against /version or /healthz often succeed.

The problem arises when custom or vendor bindings accidentally include system:unauthenticated in broader roles, or when the API server is exposed to networks where scanners probe for accessible resources. Because anonymous requests look like normal API traffic in many aggregate metrics, they are easy to miss unless you query audit logs specifically for the username.

Prerequisites

Access to Kubernetes audit logs. The commands below assume JSON audit logs written to /var/log/kubernetes/audit.log. If you use a webhook backend, a managed control plane logging stream, or a different file path, adapt the queries accordingly.
Permissions to read RBAC resources across all namespaces.
For self-managed control planes, shell access to control plane nodes or the ability to edit static pod manifests if you plan to disable anonymous auth at the API server level.

Procedure

1. Confirm whether anonymous authentication is enabled

# Check the active API server flags
ps aux | grep kube-apiserver | grep -o '\-\-anonymous-auth=[^ ]*'

If the flag is missing, the default is typically true on self-managed clusters. If it returns --anonymous-auth=true, anonymous access is explicitly enabled. If it returns --anonymous-auth=false, the API server already rejects unauthenticated requests with 401 and you can focus purely on RBAC hygiene.

2. Establish a baseline from audit logs

# Count anonymous requests over the retention window
grep '"username":"system:anonymous"' /var/log/kubernetes/audit.log | wc -l

# See what endpoints are being hit
grep '"username":"system:anonymous"' /var/log/kubernetes/audit.log | \
  jq -r '.requestURI' | sort | uniq -c | sort -rn | head -20

Expect to see /healthz, /livez, /readyz, /version, /api, and /apis. Any request to namespaced resources, /api/v1/secrets, /api/v1/pods, or subresources such as exec or portforward is abnormal.

3. Identify successful anonymous access

# List anonymous requests with HTTP 200 responses
grep '"username":"system:anonymous"' /var/log/kubernetes/audit.log | \
  jq 'select(.responseStatus.code == 200) | {verb: .verb, resource: .objectRef.resource, requestURI: .requestURI, sourceIPs: .sourceIPs}'

Focus on LIST, GET, CREATE, UPDATE, PATCH, or DELETE verbs against sensitive resources. Even a single 200 from an anonymous user to secrets, configmaps, or RBAC resources is a critical finding.

4. Classify legitimate versus illegitimate sources

Legitimate: repeated GET /healthz, GET /livez, GET /readyz, or GET /version from known load balancer IPs or monitoring agents.
Illegitimate: LIST secrets, GET /pods, RBAC changes, or sweeps across multiple resource types from unknown source IPs. Sustained elevation above baseline also warrants investigation, even if the endpoints appear harmless.

5. Audit RBAC for anonymous or unauthenticated bindings

Anonymous access is only possible if RBAC allows it. Search for bindings that reference the anonymous user or unauthenticated group.

kubectl get clusterrolebindings,rolebindings --all-namespaces -o json | \
  jq '.items[] | select(.subjects[]?.name == "system:anonymous" or .subjects[]?.name == "system:unauthenticated") | {kind: .kind, name: .metadata.name, namespace: .metadata.namespace, role: .roleRef.name}'

Review each result. If a binding is not explicitly documented and scoped to public discovery endpoints, remove it.

6. Lock down the API server (self-managed control planes)

Edit the kube-apiserver static pod manifest, typically located at /etc/kubernetes/manifests/kube-apiserver.yaml on control plane nodes, and add:

- --anonymous-auth=false

The kubelet will restart the static pod automatically. In HA clusters, update one instance at a time and verify /readyz returns 200 before proceeding to the next.

If you are running a managed control plane where the flag cannot be changed directly, rely on step 5 and network policies to block anonymous access to resources.

7. Clean up discovered bindings

Delete any ClusterRoleBindings or RoleBindings that reference system:anonymous or system:unauthenticated unless they are strictly required for a documented use case.

kubectl delete clusterrolebinding <name>
kubectl delete rolebinding <name> -n <namespace>

Verifying lockdown

After changing the API server flag, test from a host that has no Kubernetes credentials:

curl -k https://<apiserver>:6443/api/v1/pods

You should receive an HTTP 401 Unauthorized response. Re-run the audit log query from step 2. Anonymous request volume should drop to zero or reflect only denied requests. If you see continued 200 responses, there may be another API server instance or a managed endpoint that still permits anonymous access.

Verify that legitimate health checks still function. If your load balancer performs HTTP GET probes against /livez or /readyz, confirm it receives 200 or 401 as expected depending on probe configuration. A TCP connect probe alone may succeed even when anonymous auth is disabled because the TLS handshake and port are still open, but an HTTP probe will reveal the 401.

Common pitfalls

Load balancer health checks that use anonymous HTTP 200 from /healthz will break when --anonymous-auth=false is set. Update probes to use authenticated health checks, or configure the load balancer to accept HTTP 401 as a healthy response for the API server port.
Some cluster bootstrap workflows may require anonymous access to discovery endpoints during node registration. If you disable anonymous auth completely, test node joins in a non-production environment before applying the change broadly.

Signals to monitor

Signal	Why it matters	Warning sign
Anonymous request rate in audit logs	Detects scanning or misconfigured clients	Sustained requests beyond `/healthz` or `/version`
RBAC changes referencing `system:anonymous`	Catches privilege escalation or accidental grants	New bindings referencing `system:anonymous` or `system:unauthenticated`
API server `401` response rate	Unauthenticated clients are being rejected	Spike after lockdown that does not settle indicates a dependent workload is broken
Audit log gaps	Ensures attempts are not being hidden	Missing events during periods of known activity

How Netdata helps

Correlate apiserver_request_total{code="401"} spikes with API server latency to identify clients that are retrying aggressively after losing anonymous access.
Monitor audit log delivery rates and alert on gaps that could hide unauthorized access attempts.
Track RBAC-related audit events to surface unexpected privilege grants to system:anonymous or system:unauthenticated.
Visualize API server error code distributions to confirm that anonymous requests transition from 200 to 401 after lockdown, giving you immediate feedback that the procedure worked.

The Netdata solution

Kubernetes monitoring with Netdata

Netdata monitors Kubernetes with per-second metrics across the control plane, nodes, and every pod, with ML anomaly detection and zero per-pod configuration. Correlate API-server and etcd latency, kubelet PLEG stalls, scheduling pressure, and OOMKills in one place.

See Kubernetes monitoring → Start monitoring free