$ guides / zookeeper / zookeeper-nonode-error ▌

Operations Guides

ZooKeeper KeeperErrorCode = NoNode: operating on a path that doesn't exist

A client logs KeeperErrorCode = NoNode for /some/path. The server returned Code.NONODE (integer -101), which the Java client surfaces as KeeperException.NoNodeException. The failed operation was a getData, getChildren, setData, delete, or create against a znode that is not currently in the data tree.

NoNode is not a server fault. It is the API contract enforced correctly: ZooKeeper refuses to operate on a missing path. The operator’s job is to find out why the path is missing. Three cases cover almost every incident: the path was never created (usually a missing parent), the path was deleted by the server (an ephemeral tied to an expired session, or a container/TTL node auto-cleaned), or a deploy changed the znode layout clients expect.

The case that surprises operators is a sudden burst of NoNode across many ephemeral paths at once. That is rarely an application bug. It is usually the downstream signature of a session-expiration storm: a leader election, GC pause, or network event expired many sessions simultaneously, the server deleted their ephemerals, and other clients are now racing to read or update paths that just vanished.

What this means

ZooKeeper’s API contract is unforgiving about path existence:

create("/a/b", ...) throws NoNodeException when parent /a does not exist. The standard create() signatures do not auto-create parents.
getData, getChildren, setData, setACL, and delete throw NoNodeException when the target is absent. exists is the documented exception: it returns null instead of throwing.
Ephemeral znodes are deleted by the server when the creating session expires. Any later operation by any client returns NoNode.
Container znodes (3.5.3+) are deleted by the server once their last child is removed. The Programmer’s Guide explicitly warns clients to handle NoNodeException when creating children under container parents and to recreate the container.
TTL znodes (3.5.3+, requires zookeeper.extendedTypesEnabled=true) are deleted after their TTL elapses.

One subtlety: ZooKeeper logs Got user-level KeeperException at INFO for NoNode during create and delete. This is normal for many coordination patterns. Kafka’s ISR-change cleanup, for example, routinely produces INFO-level NoNode. Do not alarm on INFO-level NoNode in isolation; alarm on rate, on burst, or on correlation with session-expiration signals.

flowchart TD
  A[NoNode error logged] --> B{Single path or burst?}
  B -- Single path --> C[Check parent exists]
  B -- Burst across many paths --> D[Session expiration storm]
  C --> E{Parent present?}
  E -- No --> F[App bug: missing parent]
  E -- Yes --> G[Check container or TTL flags]
  D --> H[Correlate connections, ephemerals, GC]

The branching point is the second box. Single path failing repeatedly points to an application or coordination bug. Many paths failing in a narrow window points to a session-expiration cascade.

Common causes

Cause	What it looks like	First thing to check
Missing parent path	`NoNode` on `create("/a/b")`; parent `/a` never created	`ls /a` in zkCli; Curator `creatingParentContainersIfNeeded()`
Ephemeral vanished after session expiry	Burst of `NoNode` across many ephemeral paths; correlates with `zk_stale_sessions_expired` increase	`mntr` rate of `zk_stale_sessions_expired`
Container parent auto-deleted	`NoNode` creating children under a container parent; succeeds after parent recreated	Whether parent znode is a container node
TTL node expired	`NoNode` reading a TTL node after its TTL elapsed	Whether `zookeeper.extendedTypesEnabled=true` and node was created with `-t`
Deploy changed layout	`NoNode` begins immediately after a release; clients still use old paths	Recent deploy diff, client config
Race: `exists()` then `create()` without watch	Sporadic `NoNode` under contention	Application concurrency model; whether watches serialize
Kafka broker not registered	`NoNodeException for /brokers/ids/<id>`	Broker start order; ZooKeeper connect string in client
Wrong ensemble in client config	`NoNode` for paths that exist on a different ZooKeeper cluster	Client’s connect string vs. intended cluster

Quick checks

These are read-only. None mutate the data tree.

Note: 4lw commands (ruok, mntr, isro, etc.) require explicit whitelisting via 4lw.commands.whitelist in ZooKeeper 3.5.3+. A refused connection is a config issue, not an ensemble problem.

# Confirm the process is alive and serving reads
echo ruok | nc localhost 2181
echo isro | nc localhost 2181

# Ensemble role and stability
echo mntr | nc localhost 2181 | grep -E 'zk_server_state|zk_uptime|zk_looking_count'

# Session-expiration signals
echo mntr | nc localhost 2181 | grep -E 'zk_stale_sessions_expired|zk_connection_drop_count|zk_ephemerals_count'

# JVM pause pressure (a common root cause of session storms)
echo mntr | nc localhost 2181 | grep -E 'zk_.*jvm_pause'

# Confirm the path actually exists (read-only commands inside zkCli)
zkCli.sh -server localhost:2181
#   ls /suspected/parent
#   get /suspected/target
#   stat /suspected/target

For Kafka-specific NoNode for /brokers/ids/<id>:

zkCli.sh -server localhost:2181 ls /brokers/ids
echo mntr | nc localhost 2181 | grep -E 'zk_ephemerals_count|zk_num_alive_connections'

For correlating a NoNode burst with a possible session storm:

# Compare zxids across the ensemble; divergence indicates replication lag (mntr has no zk_zxid; use srvr)
for s in zk1 zk2 zk3; do
  echo "$s:"; echo srvr | nc $s 2181 | grep Zxid
done

# Check leader-unavailable history
echo mntr | nc localhost 2181 | grep -E 'zk_.*leader_unavailable'

How to diagnose it

Triage by error volume. Single path failing repeatedly is almost always an application or coordination bug. Many paths failing in a narrow window is almost always a session-expiration cascade. The remaining steps branch on this.
If burst, rule out a session storm first. Pull mntr for zk_stale_sessions_expired, zk_connection_drop_count, zk_num_alive_connections, and zk_ephemerals_count. A simultaneous drop in connections and ephemerals with a spike in expired sessions confirms mass session expiration.
If burst, identify the trigger. Cross-correlate with JVM pause metrics (GC pauses are a common trigger), zk_looking_count (leader election), and any leader-unavailable counter (write outage). A network partition or storage-induced fsync stall will show up in fsync-time metrics.
If single path, identify which operation failed. The exception message includes the path. The operation type comes from the application stack trace, not the server response. Knowing whether it was create vs. setData vs. delete narrows the cause immediately.
Check parent path existence. For create("/a/b") failures, verify /a exists. For Curator-based clients, confirm creatingParentContainersIfNeeded() is on the builder.
Check znode flags. A parent created as a container node will be auto-deleted when its last child is removed. Subsequent create calls under it will throw NoNode. The fix is to handle the exception and recreate the container.
Map the path to the owning application. /brokers/ids, /hbase/rs, /kafka/consumers, and similar paths belong to specific frameworks. The framework’s documentation usually describes the expected NoNode patterns. Kafka’s ISR-change cleanup routinely logs INFO-level NoNode.
Verify client session health. If the failing client uses ephemeral paths, check its session-expiry behavior. A client that briefly disconnected and reconnected with a new session will find its previous ephemerals deleted.
Check for layout drift. If errors began immediately after a deploy, diff the deploy. Clients upgraded to expect new paths while the server still has old paths (or vice versa) is a common deploy-day incident.

Metrics and signals to monitor

Signal	Why it matters	Warning sign
`zk_stale_sessions_expired`	Direct counter of expired sessions; each expiration deletes that session’s ephemerals	Any non-zero rate outside maintenance
`zk_connection_drop_count`	Connections dropping typically precede session expirations	Spike preceding a `NoNode` burst
`zk_ephemerals_count`	Mass deletion of ephemerals is the proximate cause of related `NoNode`	Sharp drop
`zk_num_alive_connections`	Mass disconnect is the upstream cause	>50% drop in a 1-minute window
`zk_looking_count`	Elections cause session migrations and brief write unavailability	Increment outside planned maintenance
JVM pause metrics	GC pauses are a common root cause of session storms	Approaching `minSessionTimeout` (default 4000ms)
Leader-unavailable counter	Cumulative write outage duration	Any non-zero delta
Fsync-time metrics	Fsync stalls are another common session-storm trigger	p99 > 10ms sustained
`zk_digest_mismatches_count`	Rules out data corruption as a separate cause	Any increment (separate incident)

Fixes

The fix is cause-specific. Restarting ZooKeeper is almost never the right move: it will not fix an application bug and will likely make a session storm worse by triggering another election.

Missing parent path (application bug)

The application must create parent paths before children:

Raw ZooKeeper clients: issue a create for each parent first, handling NodeExistsException as a no-op success.
Apache Curator: call creatingParentContainersIfNeeded() on the create builder. This is the most common fix when migrating from raw client code.
Atomic multi-znode operations: use the multi API to create parent and child in a single transactional op.

Burst of NoNode after a session-expiration storm

The NoNode errors are a symptom. Fix the upstream cause:

If JVM pause time is elevated, address heap pressure. Inspect zk_znode_count and zk_watch_count for growth trends. Consider G1GC or ZGC tuning.
If zk_looking_count is incrementing, investigate the leader-election trigger: disk, GC, or network.
If fsync time is elevated, move dataLogDir onto dedicated storage.
Client-side: ensure clients re-register ephemerals on session reconnect. Many frameworks do this automatically; custom coordinators often do not.

Container parent auto-deleted

This is expected behavior. The application must:

Catch NoNodeException when creating children under a container parent.
Recreate the container, then retry the child creation.
Optionally treat the empty container as a signal to clean up related state.

Deploy changed the layout

Roll back the deploy or push corrected client path configuration.
If the layout change is intentional, coordinate the migration: pre-create new paths before deploying clients that expect them, and keep old paths alive until clients migrate.

Kafka broker not registered

Confirm the broker process is running and its ZooKeeper connect string is correct.
Wait for broker registration to complete before issuing metadata-dependent client commands.
Verify the broker can reach the ZooKeeper ensemble (firewall, security groups).

Race condition (exists-then-create without watch)

Use a watch on the parent path instead of polling exists.
Treat NoNode as a recoverable signal: re-set the watch and retry.
For coordination patterns (leader election, distributed locks), prefer a battle-tested framework (Curator, Kafka’s own utilities) over hand-rolled watches.

Prevention

Treat NoNode as a recoverable application-level signal, not an unrecoverable error. Most coordination frameworks already do this; custom clients often do not.
Use container nodes for parent paths that should auto-exist with their children. Requires ZooKeeper 3.5.3+.
For Curator clients, default to creatingParentContainersIfNeeded() unless you have a specific reason not to.
Monitor zk_stale_sessions_expired and JVM pause metrics so you catch a session storm before clients report cascading NoNode.
Do not alarm on INFO-level NoNode server logs in isolation. Alarm on rate changes correlated with session-expiration signals.
Pin the znode layout in a deploy-time contract. Document which paths each service owns, and verify the contract during CI for client code that depends on those paths.
Runbooks should distinguish application NoNode from session-storm NoNode. They have different incident commanders and different fixes.

How Netdata helps

Per-second collection of zk_stale_sessions_expired, zk_connection_drop_count, and zk_ephemerals_count shows the exact second a session storm begins, before downstream services log NoNode.
Anomaly detection on JVM pause and fsync metrics flags the upstream GC or disk stall that typically triggers the cascade.
Correlating zk_num_alive_connections drops with zk_looking_count increments distinguishes a client-side network event from a leader election.
The ZooKeeper dashboard surfaces zk_server_state per node, so a leader gap is visible alongside the downstream NoNode reports.
Delta-based views of leader-unavailable counters confirm whether a NoNode burst coincides with ensemble-level write unavailability.