ZooKeeper authentication failures: SASL/Digest auth_failed_count climbing

The zk_auth_failed_count counter exposed by ZooKeeper’s mntr four-letter command (available since the 3.6.0 metrics framework) counts failed connection handshakes and X509 client-certificate authentication failures on the Netty transport. It does not move on Digest or SASL (GSSAPI/Kerberos) authentication failures: those return AUTHFAILED to the client and show up in the server log (Authentication failed for scheme: ..., No password found for user: ...), not in this counter. In a stable, locked-down production ensemble this counter is effectively flat between restarts. When it moves, connections are being rejected during the handshake.

The metric is per-server and cumulative since process start. It does not break out by auth scheme, source IP, or principal, so the counter alone tells you something is wrong but not who or why. You resolve the “who and why” by reading the ZooKeeper log, the surrounding metrics (zk_ensemble_auth_fail, zk_connection_rejected), and the deployment timeline.

A short burst right after a credential rotation is expected and self-resolving. A sustained rate above roughly 10 failures per second suggests widespread misconfiguration (a fleet of clients on stale credentials) or an unauthorized access attempt. Either way it deserves a ticket, not silence. When the failures are server-to-server rather than client-to-server, you are looking at a quorum threat, not a client nuisance.

What this means

zk_auth_failed_count covers two authentication paths:

• Digest authentication: username and password verified locally by the server’s DigestAuthenticationProvider. A failure means the credentials the client sent do not match what the server has on file.
• SASL authentication: typically GSSAPI/Kerberos in production, sometimes DIGEST-MD5. A failure means the SASL handshake did not complete. Causes include an expired ticket, a missing server-side JAAS entry, a hostname mismatch against the service principal, or a client library that is not configured to do SASL at all.

Failures on either path return AUTHFAILED to the client and close the connection; the counter itself only moves for Netty TLS/X509 handshake failures. To know which path is failing, read the server log line that accompanies the failure.

Operational notes that shape the diagnosis:

• Since ZooKeeper 3.5.3, four-letter commands must be whitelisted via 4lw.commands.whitelist. If mntr returns nothing, your monitoring is blind to this counter and everything else it exposes. The AdminServer on port 8080 (3.5+) is an HTTP alternative that can be authenticated separately.
• zk_auth_failed_count is a connection-establishment signal (Netty transport only). The companion metric zk_ensemble_auth_fail tracks server-to-server (quorum peer) authentication failures. Any non-zero value there is more serious because it threatens quorum, not just a single client.
• zk_connection_rejected is a different failure: the per-source-IP maxClientCnxns limit (default 60) has been hit. Reconnection storms from auth-failing clients can drive both counters together.
• The ZooKeeper security page is explicit that Digest transmits the password during authentication and stores an unsalted SHA-1 hash. Treat Digest as coarse access control, not as a confidentiality mechanism. Prefer SASL/Kerberos or mTLS where credential secrecy matters.

Common causes

Quick checks

All of the following are read-only and safe to run on a live server.

# Check the auth_failed counter twice, seconds apart, to confirm it is moving
echo mntr | nc localhost 2181 | grep zk_auth_failed_count

# Server-to-server auth failures (higher-stakes, threatens quorum)
echo mntr | nc localhost 2181 | grep -E 'zk_ensemble_auth'

# Confirm mntr is even whitelisted (empty reply means monitoring is blind)
echo ruok | nc localhost 2181     # imok = process alive

# Confirm the node is functional, not in read-only mode
echo isro | nc localhost 2181     # rw = healthy; ro = quorum lost

# Correlate with connection rejection (per-IP maxClientCnxns saturation)
echo mntr | nc localhost 2181 | grep -E 'zk_(connection_rejected|num_alive_connections)'

# Recent auth-related log lines (log path varies by distribution)
grep -iE 'auth|sasl|denied|noauth' /var/log/zookeeper/zookeeper.log | tail -50

# Per-connection snapshot to identify noisy source IPs (expensive on loaded servers, sample sparingly)
echo cons | nc localhost 2181 | head -30

How to diagnose it

The counter tells you only that something is failing. The diagnostic job is to classify the failure, identify the source, and decide whether the threat is to a single client or to the ensemble.

flowchart TD
    A["zk_auth_failed_count moving"] --> B{"Sustained non-zero rate?"}
    B -- No, isolated burst --> C["Recent credential rotation?"]
    C -- Yes --> D["Expected. Watch for self-heal."]
    C -- No --> E["Single misconfigured client. Identify via logs."]
    B -- Yes --> F{"zk_ensemble_auth_fail also moving?"}
    F -- Yes --> G["Quorum threat. Check CVE-2023-44981 patch level and quorum JAAS."]
    F -- No --> H{"Which auth scheme is failing?"}
    H -- Digest --> I["Verify client credentials and server JAAS digest entries"]
    H -- SASL / GSSAPI --> J["Check Kerberos: klist, KDC reachability, hostnames vs principals"]
    H -- Unknown --> K["Grep ZK log for source IPs and exact error strings"]

1. Confirm rate, not absolute value. zk_auth_failed_count is cumulative since restart. Run mntr twice, seconds apart, and compute the delta. A flat counter that has been static for weeks is not the problem you are chasing.
2. Classify the failure window. A short burst that begins within minutes of a known deploy and self-resolves is almost certainly credential propagation. A sustained rate is either fleet-wide misconfiguration or an unauthorized attempt.
3. Check zk_ensemble_auth_fail first. Server-to-server auth failures are higher-stakes than client auth failures because they threaten quorum. If this counter is moving, prioritize it over zk_auth_failed_count. Verify patch level against CVE-2023-44981, which allowed a SASL Quorum Peer auth bypass on unpatched 3.7.0 through 3.7.1, 3.8.0 through 3.8.2, 3.9.0, and earlier 3.x releases.
4. Read the log for the scheme. The error string disambiguates Digest from SASL. “No password found for user: null” indicates the server’s JAAS config is missing the digest entry for the connecting user. “quorum member’s saslToken is null” indicates a missing or malformed server section in the JAAS config for quorum auth. “GSS initiate failed [Caused by GSSException: … Checksum failed]” points to a Kerberos hostname alias mismatch (ZOOKEEPER-4334) or a stale keytab.
5. Identify the source. cons is expensive on a loaded server, but a single sampled snapshot will show which IPs are connected. Cross-reference against your client inventory. Unknown IPs in a locked-down environment warrant immediate network-ACL review.
6. Correlate with zk_connection_rejected. If maxClientCnxns (default 60 per source IP) is being hit in the same window, the failures may be compounding a reconnection storm. Containerized deployments where many pods share a host IP hit this limit quickly.
7. Cross-reference the deployment timeline. Most sustained spikes in production correlate with a config push, a keytab rotation, a KDC maintenance window, or a network change. The counter rarely lies about when; it just does not say what.

Metrics and signals to monitor

Fixes

Credential rotation not propagated

Refresh client configs or keytabs on the affected fleet. For Kerberos, ensure the keytab is readable by the client process and that the principal in the JAAS config matches the keytab contents. For Digest, update the server’s DigestAuthenticationProvider entries and verify the super-digest configuration if you use one. Verify against a single known-good client before rolling fleet-wide.

Expired or renewal-failed Kerberos ticket

ZooKeeper’s client library does not renew Kerberos tickets itself: a renewable TGT is refreshed either by the JVM’s Krb5LoginModule when configured with renewTGT=true, or by OS-side kinit renewal, and renewal fails if the KDC is unreachable at the renewal window. Investigate KDC reachability from the ZooKeeper host, then either restart the affected clients with a fresh ticket or fix the KDC. If this is recurring, review how the client fleet renews TGTs and the KDC’s ticket lifetime and renewal policy.

Server-side JAAS misconfigured or missing

The JAAS config (-Djava.security.auth.login.config) must be explicitly passed in the ZooKeeper server start script. Missing this is a common source of “ZooKeeper server cannot authenticate itself properly” errors. Verify the config file has a Server section for client-facing SASL and, if you have enabled quorum SASL, the corresponding QuorumServer and QuorumLearner sections.

Hostname alias mismatch (ZOOKEEPER-4334)

If the ZooKeeper server connects using a hostname alias that does not match the service principal in the keytab, GSSAPI checksum failures occur. Align the hostname the client uses with the principal, or regenerate the keytab against the canonical FQDN.

maxClientCnxns saturation

If auth-failing clients are reconnecting in a tight loop and saturating the per-IP connection limit, raise maxClientCnxns with care. The default of 60 is per source IP, and containerized deployments with many pods behind one host IP can exceed it trivially. The real fix is usually to stop the reconnect storm at the source, not to raise the limit indefinitely.

Unauthorized access attempts

If the source IPs are not in your client inventory, this is a security event, not an operational nuisance. Lock down the ZooKeeper client port with network ACLs, enable audit.enable=true (3.6+) for forensic capture of auth and mutation events, and require SASL or mTLS at the server. Do not rely on Digest alone for confidentiality given its cleartext password transmission and unsalted SHA-1 hash storage.

SASL Quorum Peer bypass (CVE-2023-44981)

If zk_ensemble_auth_fail is incrementing, prioritize patching. CVE-2023-44981 allowed a SASL ID without the instance part (for example eve@EXAMPLE.COM instead of eve/host@EXAMPLE.COM) to skip the authorization check entirely when quorum.auth.enableSasl=true. Affected versions include 3.7.0 through 3.7.1, 3.8.0 through 3.8.2, 3.9.0, and earlier 3.x releases. Fixed in 3.7.2, 3.8.3, and 3.9.1. Verify patch level on every ensemble member.

Prevention

• Alert on rate, not absolute value. The counter is cumulative since restart. Alert on increase(zk_auth_failed_count[5m]) > 0 outside maintenance windows, and page on zk_ensemble_auth_fail incrementing at all.
• Gate credential rotations in maintenance windows. A short burst after a rotation is expected; treat any sustained rate as an incident.
• Monitor zk_ensemble_auth_fail as a separate, higher-severity signal. Client auth failures are a nuisance. Quorum peer auth failures are an availability and integrity threat.
• Enable audit logging on production ensembles. audit.enable=true (3.6+) gives you per-operation forensics for sensitive znode mutations and authentication events.
• Require SASL or mTLS at the server. sessionRequireClientSASLAuth (3.6.0+) lets the server reject clients that do not authenticate via SASL.
• Lock down the client port with network ACLs. Defense in depth matters. An unknown source hitting auth failures is worth investigating even when auth is enforced at the application layer.
• Track patch level against the Apache ZooKeeper security advisories. SASL-related CVEs have historically been high-severity and easy to miss on slow-moving ensembles.

How Netdata helps

• Per-second collection of zk_auth_failed_count and zk_ensemble_auth_fail exposes rate changes long before a slower scrape interval would surface them.
• Correlation views let you put auth failures next to zk_connection_rejected, zk_num_alive_connections, and zk_stale_sessions_expired to distinguish a misconfigured client from a reconnection storm from a downstream cascade.
• ML anomaly detection flags unusual increments even when the absolute rate is below a static threshold, useful for catching slow credential drift or low-and-slow unauthorized probing.
• Per-node dashboards localize which ensemble member is seeing the failures, which matters when only one node is reachable from a problematic network path.
• Long-term retention makes it cheap to answer “when did this start?” by overlaying the counter on deployment events weeks after the fact.

Related guides

• ZooKeeper data size growing: using ZooKeeper as a database is an anti-pattern
• ZooKeeper autopurge not configured: snapshots and logs filling the disk over months
• ZooKeeper avg_latency hides write stalls: why the headline number lies
• ZooKeeper “Cannot open channel to N at election address”: the blocked election port
• ZooKeeper “Client session timed out, have not heard from server”: the heartbeat miss
• ZooKeeper connection drops spiking: sessions dying in bursts
• ZooKeeper KeeperErrorCode = ConnectionLoss: the transient disconnect every client hits
• ZooKeeper dataLogDir sharing a disk with snapshots: the #1 fsync-latency footgun
• ZooKeeper “Detected pause in JVM or host machine (eg GC)”: the pause-monitor warning
• ZooKeeper data tree digest mismatch: detecting corruption before it spreads
• ZooKeeper transaction log disk full: the crash with no graceful degradation
• ZooKeeper follower doing a SNAP sync: full snapshot transfer and its blast radius