$ guides / zookeeper / zookeeper-noauth-error ▌

Operations Guides

ZooKeeper KeeperErrorCode = NoAuth: ACL denials on protected znodes

KeeperErrorCode = NoAuth for /path appears in client logs when a ZooKeeper operation is rejected because the calling session lacks the ACL permission required for that operation on that znode. ZooKeeper does not log the denial server-side by default; to record it, enable audit logging (audit.enable=true, ZooKeeper 3.6+). NoAuth is not a transient connectivity issue. The request reached a server, the server evaluated the znode’s ACL, and the session did not match.

ZooKeeper always enforces ACLs. A fresh ensemble uses OPEN_ACL_UNSAFE (world:anyone with CREATE, READ, WRITE, DELETE, ADMIN). NoAuth only appears after someone has explicitly set a restrictive ACL on a znode using one of the schemes digest, sasl, ip, auth, or x509. The error implies a protected znode exists and a client reached it without the matching credential.

The usual causes are a misconfigured or missing SASL/Digest credential, a client that never called addAuthInfo before issuing operations, a credential rotation that left one client behind, or an unauthorized access attempt. Repeated NoAuth from a single source warrants investigation as either a misconfiguration or a probe.

What this means

ZooKeeper evaluates ACLs per operation. Each znode stores an ACL: a list of (scheme:expression, permissions) entries. When a client performs an operation, the server walks the ACL and allows the operation only if some entry matches the client’s authenticated identity and grants the required permission. The permission bits are CREATE, READ, WRITE, DELETE, and ADMIN. getACL requires READ or ADMIN (fixed in 3.4.14 and 3.5.5, per ZOOKEEPER-1392; present in 3.6.0 and later). setACL requires ADMIN.

Facts that bite operators:

ACLs are not recursive. A parent’s ACL does not protect its children. Each znode carries its own ACL, set at creation time. A common cause of NoAuth is assuming an ACL set on a parent propagates to children.
The auth scheme is a wildcard for “identities this session has authenticated as.” Creating a node with an auth-scheme ACL before the creating session has called addAuthInfo fails with InvalidACL. If a node ends up with an empty or misapplied ACL, no client will match.
The world scheme with anyone is the permissive default. Once a node is created with a restrictive ACL, only matching identities or the super user can change it.
exists() now checks READ in versions 3.9.2, 3.8.4, and 3.7.3 and later (ZOOKEEPER-2590). Clients that previously probed for node existence without READ permission now receive a permission error where they previously received a silent ok.

The error surface differs by client. The Java client throws org.apache.zookeeper.KeeperException$NoAuthException: KeeperErrorCode = NoAuth for /path. The CLI (zkCli.sh) reports KeeperError = NoAuth, or, prior to ZOOKEEPER-3891 (fixed in 3.7.0), a misleading Authentication is not valid message for insufficient permissions. The server does not log the denial by default; with audit logging enabled (3.6+), the audit log records it. All are the same event.

flowchart TD
  A["NoAuth on /path"] --> B{"Client called addAuthInfo?"}
  B -- No --> C["Add addAuthInfo before
protected operations"]
  B -- Yes --> D{"Identity matches ACL?"}
  D -- No --> E["Fix digest password
or SASL principal"]
  D -- Yes --> F{"Upgraded to
3.9.2 / 3.8.4 / 3.7.3?"}
  F -- Yes --> G["exists now needs READ.
Grant it or stop probing."]
  F -- No --> H["Source IP expected?
If not, treat as probe."]

Common causes

Cause	What it looks like	First thing to check
Client never called `addAuthInfo`	First operation on a protected znode fails immediately with NoAuth; same client works on world-readable nodes	Inspect the connection path in the client. Confirm `addAuthInfo` is called before any create, getData, or setData on protected paths
Digest credential mismatch	NoAuth appears after a password rotation; only some clients affected; `zk_auth_failed_count` stays 0 if `addAuthInfo` was omitted, but increments if an invalid digest was sent	Diff the password on the client against the password used to set the ACL; recompute the SHA-1 hash
SASL/Kerberos mismatch	NoAuth tied to a specific principal; ticket renewal failures in client logs; possibly the ZOOKEEPER-4885 pattern of a non-SASL client used after Kerberos recovery	Check JAAS config and keytab principal; compare against the ACL expression; run `klist` on the client host
ACL set with `auth` before authentication	Node created with a missing or wrong ACL; intended owner cannot operate on it	`getACL /path` (as super user if needed) and inspect the actual entries
Version-driven `exists()` change	NoAuth or `Insufficient permission` on `exists()` probes after upgrade to 3.9.2/3.8.4/3.7.3+; previously worked	Confirm server version via `srvr`; identify clients probing without READ
Unauthorized access attempt	NoAuth from source IPs outside the client inventory; bursts from a single source	Cross-reference source IP against known clients; check connection logs

Quick checks

These are read-only. In ZooKeeper 3.5.3+, four-letter commands must be whitelisted via 4lw.commands.whitelist in zoo.cfg. If mntr returns nothing, whitelisting is the first thing to fix.

# The server does not log ACL denials by default (client-side NoAuth is the signal).
# With audit logging enabled (3.6+, audit.enable=true), read the audit log instead:
grep -hE "result=failure" /var/log/zookeeper/zookeeper_audit.log | tail -30

# Failed authentication handshakes (SASL/digest). Distinct from ACL denials.
echo mntr | nc localhost 2181 | grep zk_auth_failed_count

# Confirm the server is healthy enough to be the source of truth
echo ruok | nc localhost 2181
echo isro  | nc localhost 2181

# Server version. Determines whether the ZOOKEEPER-2590 exists() change applies.
echo srvr | nc localhost 2181 | grep -E "Zookeeper version|Mode"

# Connected clients and source IPs. Expensive on busy servers; use sparingly.
echo cons | nc localhost 2181 | head -40

The log path differs by distribution. /var/log/zookeeper/zookeeper.log is the common default; check your service definition if that file is absent.

How to diagnose it

Confirm the error is NoAuth. Confirm the client log shows KeeperErrorCode = NoAuth for /path. The server does not log the denial by default; if audit logging is enabled (3.6+), the audit log records the failed operation with the user, path, and session. If the client shows ConnectionLoss or SessionExpired, this is a different failure.
Read the ACL on the znode. Run getACL /path from a session that has READ or ADMIN on the node. If no live client has access, use the super user (see Fixes) or read the ACL from a snapshot off-line. The ACL tells you which scheme and expression the client must match.
Identify the client’s authenticated identity. For digest, the identity is user:<base64(SHA1(user:password))>. For sasl, it is the Kerberos principal. For x509, it is the client certificate subject. Compare this identity against the ACL expression exactly.
Check whether addAuthInfo was called before the failing operation. In the Java client this is zoo.addAuthInfo("digest", "user:password".getBytes()) or the SASL equivalent at connection. A common bug is creating the ZooKeeper handle and immediately issuing operations before authentication completes.
Check the server version. If you recently upgraded to 3.9.2, 3.8.4, or 3.7.3, exists() now checks READ. Clients that worked before by probing without READ will now fail. This is a behavior change, not a misconfiguration.
Check for a Kerberos event correlation. With SASL/Kerberos, look for ticket renewal failures in client logs. ZOOKEEPER-4885 (open as of the 3.9.3 timeframe) describes a case where a non-SASL client is created after a Kerberos failure and never recovers, producing persistent NoAuth even after Kerberos is healthy again. The signal is NoAuth from a single client that began at the same timestamp as a renewal failure.
Check the source distribution. If NoAuth comes from one host or one client identity repeatedly, it is almost certainly a misconfiguration or a probe. If it affects many clients at once, suspect a credential rotation or an ACL change.

Metrics and signals to monitor

Signal	Why it matters	Warning sign
`zk_auth_failed_count` (mntr)	Counts failed authentication handshakes. Distinct from ACL denials, but the two co-occur when the root cause is a wrong digest password	Non-zero rate after a credential rotation, or sustained rate from a single source
Audit log `result=failure` entries (3.6+)	Server-side record of failed operations when audit logging is enabled	Sustained or bursty rate from one source IP
`cons` four-letter output	Per-connection source IP and session ID. Maps a log entry back to a client	Unexpected source IPs appearing in the connection list
Server version (from `srvr`)	Determines whether the ZOOKEEPER-2590 `exists()` ACL change applies	Recently upgraded to 3.9.2, 3.8.4, or 3.7.3, or later
`zk_ensemble_auth_fail` (mntr)	Server-to-server authentication failures. A different problem (quorum auth), but easy to confuse with client NoAuth	Any non-zero increment
`zk_non_mtls_remote_conn_count` (mntr)	In x509-based deployments, counts non-mTLS remote connections	Non-zero in an mTLS-required environment

Fixes

Client never called `addAuthInfo`

Call addAuthInfo on the ZooKeeper handle before issuing any operation against a protected znode:

ZooKeeper zk = new ZooKeeper(connectString, sessionTimeout, watcher);
zk.addAuthInfo("digest", "user:password".getBytes(StandardCharsets.UTF_8));
// now safe to operate on ACL-protected nodes

addAuthInfo is asynchronous in the Java client. The credential is sent to the server and the server responds. Operations issued before the response may still fail with NoAuth. If you need a synchronous guarantee, issue a no-op such as exists on a world-readable node and wait for it to succeed before issuing protected operations.

Digest credential mismatch

If the ACL was set with digest:user:<sha1> and the client presents user:wrongpassword, correct the password on the client. The hash is base64(SHA1(user:password)). Compute it locally to verify:

# Compute the digest auth hash for verification
echo -n "user:password" | openssl dgst -sha1 -binary | base64

Compare against the ACL expression visible via getACL /path. If you cannot recover the original password, your options narrow:

Reset the ACL using the super user (see below). Requires zookeeper.DigestAuthenticationProvider.superDigest to have been configured on the server.
Reconstruct the node: delete and recreate with the correct ACL. This is destructive. Ephemeral children vanish, watches fire, and any data on the node is gone. Do this only outside peak traffic and only after checking what depends on the node.

SASL/Kerberos mismatch

The ACL expression for SASL is the Kerberos principal (for example, zkclient@EXAMPLE.COM). Ensure the client’s JAAS config presents the matching principal. Common pitfalls:

The JAAS config file is not loaded (java.security.auth.login.config system property missing or pointing at the wrong file).
The principal in the keytab does not match the principal in the ACL.
The ticket has expired and renewal failed. ZOOKEEPER-4477 (fixed in 3.8.1) addressed a case where a single renewal failure prevented all future renewals on Java 9 and later.
ZOOKEEPER-4885 (open as of 3.9.3): after a Kerberos failure, the client falls back to a non-SASL client and never recovers, producing persistent NoAuth even after Kerberos recovers. The workaround is to recreate the ZooKeeper handle on the client side.

Rebuilding the ZooKeeper handle drops the session, which clears ephemeral nodes and watches. Do this only after confirming the client is stuck in the ZOOKEEPER-4885 pattern.

ACL set with `auth` before authentication

The auth scheme expands to “all identities this session has authenticated as.” If the node was created with an auth-scheme ACL before the creating session called addAuthInfo, the server rejects it with InvalidACL. If an empty or unmatchable ACL bypassed validation, the fix is to recreate the node with the correct ACL, or to use the super user to run setACL.

Version-driven `exists()` change

This is not a misconfiguration. If you upgraded to 3.9.2, 3.8.4, or 3.7.3 and clients now fail exists() where they previously succeeded, you have two options:

Grant READ on the probed nodes to the probing identity.
Change the client to not depend on exists() returning ok without READ.

Do not work around this with zookeeper.skipACL=yes. That disables all ACL checking on the server and removes a security control. The exists() change closes a probe that should never have been permitted.

Unauthorized access attempt

If the source IP is not in your client inventory, the NoAuth is doing its job. The fix is to prevent the source from reaching the ZooKeeper port: network policy, firewall rules, or removal of the pod or job. In parallel, audit your ACL posture. Confirm that sensitive subtrees (Kafka assignments, HBase region state, distributed lock nodes) carry restrictive ACLs, because the default world:anyone makes them readable and writable by any client that can reach the port.

Super user recovery

For any cause that requires changing an ACL you cannot reach, configure the super user escape hatch on the server:

-Dzookeeper.DigestAuthenticationProvider.superDigest=super:base64(SHA1(super:password))

Then connect as addauth digest super:password and bypass all ACL checks. This is the only recovery path for a node whose ACL has locked out every legitimate client. Configure it before you need it, and store the password in your secrets manager.

Prevention

Wire addAuthInfo into the connection path. Every client should call it as part of establishing the handle, before any application-level operation.
Set restrictive ACLs at creation time. Create znodes with explicit (scheme:expression, permissions) rather than relying on defaults.
Remember ACLs are not recursive. Children do not inherit the parent’s ACL. Protect each child explicitly, or set the creating client’s default ACL.
Configure superDigest before you need it. Document the password in your secrets store. You will need it during a lockout.
Track server version in monitoring. The exists() change in 3.9.2/3.8.4/3.7.3 is breaking. Know when you cross that boundary.
Keep audit logging on for sensitive subtrees. audit.enable=true in zoo.cfg (3.6+) logs mutations with the session and identity. NoAuth denials themselves are already in the server log.
Do not use skipACL in production. zookeeper.skipACL=yes disables all ACL checking. It is a debug escape hatch, not a configuration.

How Netdata helps

Correlate zk_auth_failed_count with server log spikes. Authentication handshake failures often precede or accompany NoAuth storms. Seeing both move together in one view narrows the cause from “client never authenticated” to “client authenticated as the wrong identity.”
Map connection source IPs to NoAuth bursts. Per-second collection on zk_num_alive_connections and the connection rejected counter lets you spot a single source IP producing a burst, the signature of a misconfigured rollout or a probe.
Track server version across the ensemble. Knowing which nodes are on 3.9.2 or later tells you immediately whether the exists() ACL change is in play, without grepping release notes mid-incident.
Surface ensemble-wide auth failures. zk_ensemble_auth_fail increments on server-to-server authentication failures, a different problem (quorum auth) but easy to confuse with client NoAuth. Having both on one dashboard prevents misdiagnosis.
Catch the Kerberos recovery trap. Sustained NoAuth from a single client that began at the same timestamp as a Kerberos ticket renewal failure matches the ZOOKEEPER-4885 pattern. High-resolution collection makes that timestamp correlation visible.