$ guides / zookeeper / zookeeper-command-not-in-whitelist ▌

Operations Guides

ZooKeeper "X is not executed because it is not in the whitelist": four-letter-word commands blocked

The error string is exact. When you run echo mntr | nc localhost 2181 against a ZooKeeper 3.5.3+ server that has not been configured for it, the server replies:

mntr is not executed because it is not in the whitelist.

Same shape for ruok, isro, stat, conf, envi, cons, wchs, and the rest of the four-letter-word (4lw) command set. Only srvr works out of the box, because the bundled zkServer.sh status check depends on it.

The change is intentional. On the 3.5.x line, 3.5.3 introduced 4lw.commands.whitelist with a default of srvr only (isro is also enabled when read-only mode is on); 3.5.0 through 3.5.2 had no whitelist at all. The motivation is documented: wchp and wchc are CPU-intensive enough to be usable in denial-of-service attacks. On the 3.4.x line, where the property arrived in 3.4.10, the default disabled exactly those two commands and kept everything else enabled. From 3.5.3 on, operators must open additional commands explicitly.

The failure mode that hurts is silent. Health probes and metric scrapers that previously consumed mntr, ruok, or isro start receiving either the whitelist error string or nothing. Many scrapers do not treat that as a fetch failure; they parse the body as an empty payload and emit zero-valued metrics. From the dashboard’s point of view the cluster looks healthy at exactly the moment it has stopped being observed.

What this means

Four-letter-word commands are short text probes sent over the client port (default 2181). They predate the JMX layer and the HTTP AdminServer. The ones most monitoring stacks depend on are:

mntr - the monitoring dump. Emits all zk_* key/value pairs: zk_server_state, zk_avg_latency, zk_outstanding_requests, zk_zxid, zk_znode_count, and dozens of others.
ruok - shallow liveness. Returns imok if the JVM thread that handles the command is responsive. Does not confirm quorum, leadership, or that the data tree is loaded.
isro - read/write state. Returns rw if the node is in a functional ensemble, ro if it has lost quorum but is still serving stale reads.
stat and srvr - server summary (role, version, counters, latency). srvr is the default whitelist entry because the bundled scripts rely on it.
conf - parsed zoo.cfg.

When a 4lw command is not on the whitelist, the server writes the literal string “<cmd> is not executed because it is not in the whitelist.” back over the socket and closes the response. There is no log entry by default. There is no metric increment. The server itself is fine. Only your observability has stopped.

Three things make this painful after an upgrade:

The error does not look like an error to a scraper. Telegraf’s ZooKeeper input logs unexpected line in mntr response: 'mntr is not executed because it is not in the whitelist.' and then stops emitting metrics. A pipeline that treats absent metrics as zero now shows a cluster sitting at zk_avg_latency = 0, zk_outstanding_requests = 0, zk_znode_count = 0. It looks healthy. It is unobserved.
ruok returning nothing reads as “process down” to a naive probe, while isro returning nothing removes the only reliable signal that distinguishes a live read-write node from one that has lost quorum. The ro state - a node alive but unable to serve writes - is exactly the failure isro was built to expose.
The change is version-gated and bites on upgrade. Pre-3.4.10 had no whitelist at all. 3.4.10 and later (3.4.x line) had 4lw.commands.whitelist, defaulting to every command except wchc and wchp; 3.5.0 through 3.5.2 had no whitelist at all. 3.5.3+ introduces the restrictive srvr-only default. Ensembles that ran for years without touching the property lose monitoring the day they cross that boundary.

Common causes

Cause	What it looks like	First thing to check
Default whitelist on 3.5.3+	Every 4lw except `srvr` returns the whitelist error string	`grep 4lw.commands.whitelist zoo.cfg` returns nothing
Upgrade from 3.4.x or 3.5.0-3.5.2	Monitoring was fine, broke immediately after upgrade	`echo srvr \| nc localhost 2181 \| grep Zookeeper` to confirm new version
Distribution image ships closed whitelist	Containerised ZK (Confluent images historically, some Helm charts) hits this on first deploy	Inspect the active `zoo.cfg` mounted into the container
Restrictive list missing commands	`mntr` works, `ruok` does not (or vice versa)	Diff the whitelist string against the set your scrapers call
Whitelist set on the wrong property name	`4lw.commands.whitelist` is in `zoo.cfg` but the JVM was started with `-Dzookeeper.4lw.commands.whitelist` overriding it	`ps -ef \| grep QuorumPeerMain` for `-D` overrides

Quick checks

# Confirm the symptom: which commands return the whitelist error?
for c in ruok mntr isro stat srvr conf cons envi wchs; do
  printf '%-6s -> ' "$c"
  echo "$c" | nc -w 2 localhost 2181 | head -1
done

# Confirm ZooKeeper version. srvr is the only 4lw on by default in 3.5.3+.
echo srvr | nc localhost 2181 | grep -E 'Zookeeper version|Mode'

# Read the effective whitelist from disk.
grep -E '4lw\.commands\.whitelist|zookeeper\.4lw\.commands\.whitelist' \
  /etc/zookeeper/conf/zoo.cfg /opt/zookeeper/conf/zoo.cfg 2>/dev/null

# Check whether the JVM overrides the file property.
ps -ef | grep '[Q]uorumPeerMain' | grep -o -- '-Dzookeeper.4lw.commands.whitelist=[^ ]*'

# Confirm the AdminServer is up (3.5+). Listens on 8080 by default.
curl -s --max-time 2 http://localhost:8080/commands/server_stats | head -c 200 ; echo

# Reproduce exactly what the scraper sees.
echo mntr | nc localhost 2181 | head -5

All of the above are read-only. None change ZK state.

How to diagnose it

Reproduce the error from the host. echo mntr | nc localhost 2181 should produce the literal whitelist string. If it produces metrics, the problem is elsewhere (network ACL, wrong host, scraper auth).
Confirm the ZooKeeper version. The whitelist default only changed in 3.5.3; if you are on 3.4.x or 3.5.0-3.5.2 and seeing the error, someone set an explicit restrictive whitelist.
Read zoo.cfg for 4lw.commands.whitelist. The property uses dots (4lw.commands.whitelist) in the file and as a Java system property (zookeeper.4lw.commands.whitelist). The Java system property wins if both are set.
Inspect the running process command line for -D overrides. A container image or a wrapper script can inject an override that bypasses zoo.cfg.
Cross-check what each scraper actually calls. Common pattern: liveness probe uses ruok, readiness probe uses isro, Telegraf or a Prometheus exporter uses mntr, and operators run stat ad hoc. Each must be on the whitelist.
Verify whether anything already consumes the AdminServer on port 8080. If a scraper already speaks HTTP, migrating it is cheaper than maintaining the whitelist.

flowchart TD
  A[4lw returns "not in the whitelist"] --> B{ZK version}
  B -->|3.4.x or 3.5.0-3.5.2| C[Explicit whitelist set - find it]
  B -->|3.5.3+| D[Default is srvr only]
  D --> E{Scraper needs}
  E -->|mntr, ruok, isro, stat| F[Add to 4lw.commands.whitelist]
  E -->|HTTP already supported| G[Move scraper to AdminServer :8080]
  F --> H[Rolling restart]
  G --> H

Metrics and signals to monitor

Signal	Why it matters	Warning sign
`zk_server_state` (from `mntr`)	Confirms the node’s role. Without it you cannot tell leader from follower or detect split-brain	Metric disappears from the scraper, or stays frozen at its last value
`isro` (`rw` / `ro`)	The only 4lw that distinguishes a functional node from one serving stale reads with quorum lost	Readiness probe silently green while the cluster is read-only
`ruok` (`imok`)	Shallow liveness; not sufficient alone but consumed by load balancers and orchestrators	Probe reports the node down even though `QuorumPeerMain` is running
`zk_outstanding_requests`	Leading indicator of pipeline saturation	Stuck at zero because the scraper never sees the real value
`zk_avg_latency`, `zk_max_latency`	Headline latency metrics	Flatline at zero is the signature of a broken scrape, not a healthy cluster
`zk_synced_followers`, `zk_pending_syncs` (leader only)	Replication health on the leader	Absent from the leader’s scrape output
`zk_looking_count`	Election frequency. Any non-zero outside maintenance is worth a ticket	Stale value hides repeated elections
`zk_uptime`	Process uptime, useful for suppressing cold-start false positives	A frozen uptime hides restarts

The pattern to alert on is not “metric crossed a threshold”. It is “metric source went away”. Treat a missing mntr scrape as a page, not a data gap.

Fixes

Two paths. Pick based on what your scrapers can speak.

Whitelist the commands your monitoring uses

Edit zoo.cfg:

4lw.commands.whitelist=srvr,ruok,isro,mntr,stat

Use a comma-separated list. Avoid * in production - it re-enables wchp and wchc, the CPU-intensive commands the whitelist exists to gate. If you need conf, envi, or cons for debugging, add them deliberately and remove them once the session ends; cons is expensive on ensembles with many client connections.

Apply with a rolling restart. There is no reload signal for 4lw.commands.whitelist; the property is read at startup. On a 3-node ensemble, restart one follower at a time, confirm it rejoins and mntr works, then proceed.

After the restart, verify each command individually:

for c in ruok mntr isro stat srvr; do
  printf '%-6s -> ' "$c"
  echo "$c" | nc -w 2 localhost 2181 | head -1
done

If you are running in a container, check where the config is mounted. Confluent images and some Helm charts historically did not set a whitelist and required the 4lw.commands.whitelist property to be added to zoo.cfg or zookeeper.properties through the image’s configuration mechanism.

Move scrapers to the AdminServer

ZooKeeper 3.5.0+ ships an embedded Jetty AdminServer on port 8080. The endpoint shape is /commands/<command> and the response is JSON. To discover the exact command routes available on your version:

curl -s http://localhost:8080/commands/ | python3 -m json.tool | head -40

The monitoring dump (the mntr equivalent) is typically:

curl -s http://localhost:8080/commands/monitor | python3 -m json.tool | head -30

Other useful endpoints include the server statistics route (the srvr/stat equivalent) and the environment route (the envi equivalent). The exact path names may differ across releases; query the /commands/ root listing to confirm what your server exposes.

The official documentation notes that 4lw commands are deprecated in favour of the AdminServer. No removal date has been announced and 4lw is still functional in current 3.8.x and 3.9.x releases, but new deployments should prefer HTTP where the scraper supports it.

The AdminServer has its own security surface. If you expose it beyond localhost, place it behind a firewall and keep current on patch releases. Recent AdminServer issues include CVE-2024-51504, an IP-authentication bypass fixed in 3.9.3, and CVE-2025-58457, a snapshot/restore permission check fixed in 3.9.4. Do not assume that closing the 4lw whitelist is the only thing that matters.

Do not silence the error

Pointing the health probe at srvr (which is always whitelisted) and calling it done is the same trap as using ruok as your only check: srvr confirms the JVM is up but says nothing about quorum, replication, or whether the node can serve writes. The fix is to whitelist the commands you need, not to downgrade your checks to the ones that already work.

Prevention

Set the whitelist explicitly in your base zoo.cfg template. Do not rely on the default. Treat it as part of cluster bootstrap, not a runtime tweak.
Pin the whitelist in version control alongside zoo.cfg. When someone upgrades the cluster, the whitelist moves with the config.
Include a 4lw sanity check in post-deploy smoke tests. Run ruok, mntr, and isro against every node and assert non-empty responses.
Alert on missing scrapes, not just on threshold crossings. A pipeline that silently emits zeros when the source disappears is a monitoring failure dressed up as a metric.
For new deployments, prefer the AdminServer over 4lw. The deprecation notice has been in the docs since 3.5.3. Even if 4lw is never removed, the HTTP endpoint is easier to secure and scrape.
When upgrading across the 3.4.x-to-3.5.3+ boundary, treat the whitelist change as a breaking change in your release notes. It is the most common silent monitoring break after a ZooKeeper upgrade.

How Netdata helps

Netdata’s ZooKeeper collector issues the 4lw commands you configure. If mntr is not whitelisted, the per-second scrape fails visibly in the collector logs and the affected charts go to “no data” rather than to zero. Flatline-at-zero is the failure mode that hides outages; explicit “no data” is the one you want.
Per-second resolution means the gap between “whitelist changed” and “alert fires” is short. A daily scraper can lose a day of visibility before anyone notices; a per-second scraper cannot.
Correlating the missing mntr scrape against an ensemble-wide deployment event (rolling restart, image bump) makes the upgrade-induced regression obvious. The signature is “all zk_* metrics disappeared from this node at the same instant”, which is visually distinct from “one metric crossed a threshold”.
Netdata’s ML anomaly detection flags the joint pattern of “metric source went away while process uptime reset” as anomalous, even when no static threshold has been set on the affected charts.