What is Apache Zookeeper Used for?

Apache ZooKeeper is a service used by a cluster (group of nodes) to coordinate between themselves and maintain shared data with robust synchronization techniques. ZooKeeper is itself a distributed application providing services for writing a distributed application.

Below are some of the common services provided by zookeeper:

• Naming service − Identifying the nodes in a cluster by name. It is similar to DNS, but for nodes.
• Configuration management − Latest and up-to-date configuration information of the system for a joining node.
• Cluster management − Joining / leaving of a node in a cluster and node status at real time.
• Leader election − Electing a node as leader for coordination purpose.
• Locking and synchronization service − Locking the data while modifying it. This mechanism helps you in automatic fail recovery while connecting other distributed applications like Apache HBase.
• Highly reliable data registry − Availability of data even when one or a few nodes are down.

Distributed applications offer a lot of benefits, but they throw a few complex and hard-to-crack challenges as well. ZooKeeper framework provides a complete mechanism to overcome all the challenges. Race condition and deadlock are handled using fail-safe synchronization approach.
Another main drawback is inconsistency of data, which ZooKeeper resolves with atomicity.

ZooKeeper node is referred as znode.

There are following two types of nodes in Zookeeper:

• Leader Node – Leader Node is the only node responsible for processing the write requests. All other nodes called followers simply delegate the client write calls to the Leader node. (We currently have two leader election algorithms in ZooKeeper: LeaderElection and FastLeaderElection) We don’t mark any node as a leader while setting up the Apache ZooKeeper cluster. It instead is elected internally among all the nodes of the cluster.
• Apache ZooKeeper uses the concept of majority for the same i.e. Node that gets the highest number of votes is elected as Leader.
• This serves as the basis of a recommendation that suggests having an odd number of nodes in a cluster for best failover and availability. E.g. if we create a cluster of four nodes and two nodes go offline for some reason. Apache ZooKeeper will be down as half of the nodes have gone offline it is not possible to gain a majority for the Leader node election. However, if we create the cluster of five nodes, even if two nodes go offline, Apache ZooKeeper will still be functional as we still have the majority of nodes in service.
• Follower Nodes – All nodes other than Leader are called Follower Nodes. A follower node is capable of servicing read requests on its own. For write requests, it gets done through Leader Node.
• Followers also play an important role in electing a new leader if the existing leader node goes down.

Below are some of instances where Apache ZooKeeper is being utilized :

• Apache Storm, being a real-time stateless processing/computing framework, manages its state in ZooKeeper Service.
• Apache Kafka uses it for choosing the leader node for the topic partitions.
• Apache YARN relies on it for the automatic failover of the resource manager (master node).
• Yahoo! utilities it as the coordination and failure recovery service for Yahoo!.
• Message Broker, which is a highly scalable publish-subscribe system managing thousands of topics for replication and data delivery.
• It is used by the Fetching Service for Yahoo! crawler, where it also manages failure recovery.