← Back to Blog

Kafka Doesn't Need Zookeeper Anymore? Kafka KRaft Mode

kafka message queue zookeeper

If you've worked with Kafka before, you're probably used to the model of using Zookeeper alongside Kafka. Have you ever wondered, like me, why we need Zookeeper at all instead of just using Kafka directly?

For those who don't know, Zookeeper is responsible for:

  • Managing Kafka's metadata: ZooKeeper stores the Kafka cluster's metadata, including broker locations, topic config, and partitions.
  • Leader election: ZooKeeper is used to elect a leader for each partition of a topic. This ensures only one broker is responsible for handling writes for a specific partition at any given time.
  • Managing consumer group offsets: ZooKeeper is used to store the current read position (current offset) of each consumer group. This lets a consumer continue reading from its previous position after an error or a restart.
  • Health check: ZooKeeper monitors the Kafka cluster's health and can notify admins if any issue crops up.
Kafka Zookeeper overview

Zookeeper brings certain benefits to a system, but in some cases it adds complexity to the system and increases the chance of errors due to having more components, etc. With the new Kafka model I'm introducing in this post, we no longer need Zookeeper at all! This model operates under Kafka's KRaft mode.

What is Kafka KRaft?

Kafka KRaft is a new consensus protocol developed with the goal of removing Zookeeper's presence from the system entirely. Instead of storing data across 2 separate system components — Zookeeper and Kafka — with this protocol, Kafka stores its metadata on itself. Using this protocol requires enabling KRaft mode on the Kafka cluster; KRaft uses a new Quorum Controller with the Raft consensus protocol to elect a leader.

Kafka KRaft architecture

What sins has Zookeeper committed to deserve being removed?

If you've been using Kafka with Zookeeper just fine, there's no reason to change the model! But if you've run into issues, or your system has grown with more topics and more partitions, it's worth weighing things again. Here's why we should consider removing Zookeeper:

  • Due to Zookeeper's limits, Kafka can only support 200,000 partitions
  • When a broker joins the cluster, the syncing and re-electing of a leader can overload Zookeeper and slow the system down.
  • Setting up a Kafka cluster with Zookeeper is fairly complex and hard to build.
  • Sometimes syncing metadata from Kafka to Zookeeper fails
  • Guaranteeing security is harder (protecting one thing is easier than protecting two)
  • ....

With a few of the factors above, and with some experience battling Kafka for a while, I think switching to KRaft Mode is well worth considering!

How does KRaft Mode work?

This mode uses the new Quorum Controller in Kafka. The Quorum Controller uses an event-sourced storage model and the Raft consensus mechanism to sync metadata between Quorums (brokers).

Kafka Quorum Controller

One of the Quorum Controllers acts as the leader, generating events in the metadata topic — the remaining Quorum Controllers sync from the leader by acknowledging these events. So when a broker fails and needs to rejoin, it can catch up on missed events from the event log.

Unlike the Zookeeper-based model, when there's a Leader change, the new Quorum Controller already has the synced metadata available in memory, instead of having to reload state from Zookeeper.

Benefits of using KRaft Mode

  • Increases Kafka's scalability, easily scaling to millions of partitions
  • Easier to install and maintain
  • Increases the system's stability, easier to monitor and manage (since the model is simpler)
  • Faster Kafka startup and restart
  • Increased recovery ability when an incident occurs

Here's a comparison table of Shutdown and Recovery time between the 2 models, With Zookeeper and With Quorum Controller — you can see a fairly large difference.

Zookeeper vs Quorum Controller shutdown and recovery time comparison

Quick start with Kafka KRaft mode

To quickly try out this KRaft mode, you can use docker to spin it up:

To try it quickly, you can use the bitnami Kafka image bitnami/kafka:latest and enable KRAFT mode via the environment variable KAFKA_ENABLE_KRAFT=yes

docker run -p9092:9092 --name kafka -e ALLOW_PLAINTEXT_LISTENER=yes -e KAFKA_CFG_AUTO_CREATE_TOPICS_ENABLE=true -e KAFKA_ENABLE_KRAFT=yes bitnami/kafka:latest

Note: KRaft mode currently isn't recommended for a production environment!!! You should only use it for test, staging, dev environments, etc.

If you found this post useful, please UpVote and Follow me to catch more posts

References

Here are the sources I referenced and used images from:

Have thoughts on this?

I'd love to hear your perspective. Send me a message.

Get in touch