ZooKeeper Watcher Principle and Command Line Practice Guide

This is article 29 in the Big Data series. Deep analysis of ZooKeeper Watcher mechanism internal working principle, and zkCli command line practice demonstrating complete flow of node operations and event monitoring.

Complete illustrated version: CSDN Original | Juejin

Watcher Mechanism Overview

ZooKeeper’s Watcher is a lightweight event-driven distributed Pub/Sub mechanism, allowing clients to subscribe to ZNode state changes and receive notifications asynchronously.

Core description: “One-time trigger, asynchronous notification, lightweight transmission”.

Three Components of Watcher

Complete Watcher work involves three roles:

Client Thread  ←→  ClientWatchManager  ←→  ZooKeeper Server
(发起注册/处理通知)  （本地管理 Watcher 列表）  （保存 Watcher，触发事件）

Client Thread: When calling getData/getChildren/exists with watch=true, initiates registration request
ClientWatchManager: Watcher registry maintained locally on client, indexed by path
ZooKeeper Server: Records client Session ID in ZNode’s WatcherList

Complete Workflow

Registration Phase

客户端调用 getData("/config", watch=true)
    → 请求发送到 ZooKeeper Server
    → Server 在 /config 节点的 WatcherList 中记录 Session ID
    → 同时返回 /config 的当前数据

Trigger Phase

其他客户端调用 setData("/config", "new-value")
    → Server 修改数据，ZXID 递增
    → Server 检查 /config 的 WatcherList
    → 向所有注册了 Watcher 的客户端发送 WatchedEvent
    → WatcherList 中该条目**自动清除**（一次性）

Notification Phase

客户端收到 WatchedEvent (type=NodeDataChanged, path=/config)
    → 执行注册的回调逻辑
    → 如需持续监听，必须重新调用 getData("/config", watch=true)

Four Features Detailed

Feature	Detailed Description
One-time	Automatically expires after trigger, avoid server accumulating many Watchers causing memory leak
Asynchronous	Notification via independent thread callback, doesn’t block main business logic
Lightweight	WatchedEvent only contains event type + node path, not full data content
Ordered	Same client notifications strictly delivered in trigger order

Supported Event Types

Event Type	Trigger Operation	Registration API
`NodeCreated`	Node created	`exists(path, watch)`
`NodeDeleted`	Node deleted	`getData` / `exists`
`NodeDataChanged`	Node data changed	`getData(path, watch)`
`NodeChildrenChanged`	Child node list changed	`getChildren(path, watch)`

zkCli Command Line Practice

Connect to Cluster

# Connect to any node (load balanced to Leader/Follower)
/opt/servers/apache-zookeeper-3.8.4-bin/bin/zkCli.sh -server h121.wzk.icu:2181

Node Creation

# Create persistent node
create /wzk 123456

# Create persistent sequential node (auto appends sequence number at path end)
create -s /wzk-order 654321
# Actually created: /wzk-order0000000001

# Create ephemeral node (auto deleted after Session disconnects)
create -e /wzk-temp 123123

# Create ephemeral sequential node
create -e -s /lock/node "client-1"

Query Operations

# List root node's child nodes
ls /

# View node data
get /wzk

# View node metadata (version number, timestamps, etc.)
stat /wzk

# View child node list
ls /wzk

Modification and Deletion

# Modify node data (dataVersion auto increments)
set /wzk-temp 111222

# Delete leaf node (error if has child nodes)
delete /wzk-temp

# Recursively delete node and all child nodes
deleteall /wzk

Watcher Registration and Trigger Demo

Terminal A (Register monitoring):

zkCli.sh -server h121.wzk.icu:2181

# Read data and register Watcher
get -w /wzk

# Register child node monitoring
ls -w /wzk

Terminal B (Trigger event):

zkCli.sh -server h122.wzk.icu:2181

# Modify data, trigger NodeDataChanged
set /wzk "new-value"

# Create child node, trigger NodeChildrenChanged
create /wzk/child "child-data"

Terminal A receives notification:

WATCHER::
WatchedEvent state:SyncConnected type:NodeDataChanged path:/wzk

WATCHER::
WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/wzk

Note: After receiving notification, original Watcher is expired. To continue monitoring, must re-execute get -w /wzk.

Typical Watcher Application Scenarios

1. Configuration Center

Configuration node changes → All services subscribed to that node receive notification → Re-pull configuration and hot reload, no restart needed.

2. Service Discovery

Service instance creates ephemeral node on startup → Client monitors service list node → Instance goes offline, node automatically disappears → Client receives NodeChildrenChanged notification, updates local service list.

3. Distributed Lock

All competitors create ephemeral sequential nodes → Smallest sequence number holds lock → Lock holder only needs to monitor previous node → When predecessor node deleted, receives notification and tries to acquire lock (fair lock, avoids thundering herd).

4. Leader Election

Each node creates ephemeral sequential node → Node with smallest sequence becomes Leader → Other nodes monitor predecessor node → When current Leader goes down, automatically triggers new round of election.

Summary

Watcher’s “one-time trigger” design minimizes server resource consumption, at the cost of client needing to actively re-register in business logic. Understanding “register → trigger → notification → re-register” cycle is key to correctly using ZooKeeper monitoring mechanism. Next article will implement same monitoring logic via Java API (ZkClient library).