ZooKeeper for Beginners (4): Inside ZooKeeper

Tutorial index: Big Data for Beginners: tutorial series

The first three ZooKeeper tutorials left you able to use it. This post goes a step further and gives you a general idea of how ZooKeeper actually works.

Two Types of Znodes

1. Ephemeral: once the client disconnects from the server, the node it created deletes itself.
2. Persistent: once the client disconnects, the node it created remains.

Four Kinds of Directory Nodes (default is persistent)

1. PERSISTENT: the node survives after the client disconnects.
2. PERSISTENT_SEQUENTIAL: the node survives after disconnection, and ZooKeeper appends a sequential number to its name.
3. EPHEMERAL: the node is deleted once the client disconnects.
4. EPHEMERAL_SEQUENTIAL: the node is deleted once the client disconnects, with ZooKeeper appending a sequential number to its name.

A word on those sequentially numbered nodes: in distributed systems, sequence numbers can impose a global ordering on all events, letting clients infer event order from the numbers.

ZooKeeper’s Stat Structure

Every znode in the ZooKeeper namespace has an associated stat structure, similar to the stat structure of a file in a Unix/Linux filesystem. Here are the fields, and what they mean:

• cZxid: the transaction ID of the change that created this znode.
• mZxid: the transaction ID of the change that last modified this znode.
• pZxid: the transaction ID of the change that last added or removed a child.
• ctime: znode creation time in milliseconds since 1970-01-01T00:00:00Z.
• mtime: znode last modification time in milliseconds since 1970-01-01T00:00:00Z.
• dataVersion: the number of changes made to this znode’s data.
• cversion: the number of changes made to this znode’s children.
• aclVersion: the number of changes made to this znode’s ACL.
• ephemeralOwner: for an ephemeral znode, the session ID of its owner; for a non-ephemeral znode, this field is zero.
• dataLength: the length of the znode’s data field.
• numChildren: the number of children of this znode.

ZooKeeper Watchers

Earlier we tried ZooKeeper watchers and found them pretty powerful, so let’s look at how they work internally. new ZooKeeper calls the org.apache.zookeeper.ZooKeeper#ZooKeeper(java.lang.String, int, org.apache.zookeeper.Watcher, boolean, org.apache.zookeeper.client.HostProvider, org.apache.zookeeper.client.ZKClientConfig) constructor, which contains:

cnxn = createConnection(
    connectStringParser.getChrootPath(),
    hostProvider,
    sessionTimeout,
    this,
    watchManager,
    getClientCnxnSocket(),
    canBeReadOnly);
cnxn.start();

So what is ClientCnxn? Constructing it calls org.apache.zookeeper.ClientCnxn#ClientCnxn(java.lang.String, org.apache.zookeeper.client.HostProvider, int, org.apache.zookeeper.ZooKeeper, org.apache.zookeeper.ClientWatchManager, org.apache.zookeeper.ClientCnxnSocket, long, byte[], boolean), which contains these lines:

sendThread = new SendThread(clientCnxnSocket);
eventThread = new EventThread();

And in org.apache.zookeeper.ClientCnxn#start:

sendThread.start();
eventThread.start();

SendThread and EventThread both extend org.apache.zookeeper.server.ZooKeeperThread, which extends java.lang.Thread — so it spins up two child threads, one for receiving and one for sending.

In other words, each ZooKeeper node keeps a list of watchers; when we register a watch we get added to that list. When the data changes, ZooKeeper notifies the EventThread, which then invokes callbacks into our program.

The Zab Protocol

Zab stands for ZooKeeper Atomic Broadcast.

ZooKeeper uses Zab to guarantee eventual consistency for distributed transactions. This is a beginner tutorial so we’ll only cover it briefly; plenty of experts have written it up online. To go deeper, read up on Paxos — Zab borrowed heavily from Paxos, though it isn’t a copy; it modified it.

Zab does essentially two things: electing a leader (crash recovery mode), and getting work done (message broadcast mode).

Leader Election (Crash Recovery Mode)

When the cluster first boots, or when the leader crashes leaving no leader, an election starts. Each node votes for itself first, then enters the looking state to observe what everyone else voted for. If everyone voted for themselves, they compare who has more influence. Every event gets a Zxid assigned to preserve ordering in the distributed system, so whoever holds the newest Zxid data version wins. If two nodes tie on the newest Zxid, they compare their IDs and the larger one wins. Once more than half the votes have picked a leader, nodes starting later don’t bother comparing — they just follow the leader. That’s why clusters are sized odd: to guarantee a result and avoid a hung 50/50 split.

Getting Work Done (Message Broadcast Mode)

When a client request enters the cluster, a receiving node that isn’t the leader forwards it to the leader. The leader turns the request into a Proposal and broadcasts it to all nodes. Each node keeps its own FIFO pending-write queue and compares the event’s Zxid against its own data: if a node’s recorded Zxid is 10 and the write request’s Zxid is 9, it votes no; if the request is 11, it votes yes. The votes go back to the leader, and if the leader receives ACKs agreeing from more than half the nodes, it broadcasts a commit so every node writes successfully.

If everyone agrees but one node dissents, that node’s data is out of sync — it kills itself and re-syncs from the leader, preserving cluster-wide consistency.

Imagining Very Large Clusters

Voting among a handful of machines is fast, but what about a cluster of tens or hundreds of thousands? Voting every time is obviously unrealistic; write throughput would collapse as the network burned entirely on voting. That’s where the Observer role comes in.

An Observer does exactly what its name says: observe. It watches what the leader and followers do, making it easy to scale a ZooKeeper cluster dynamically without hurting performance. Observers accept client connections and execute the leader’s state-update commands, but they don’t participate in voting — which is also why an observer going down has no effect on the cluster’s operation.