Updating a Kafka environment to use KRaft (Kafka Raft) mode instead of ZooKeeper involves several steps. KRaft mode is a new feature in Apache Kafka that allows Kafka brokers to manage metadata directly, eliminating the need for ZooKeeper. This transition involves setting up a new Kafka cluster in KRaft mode, migrating data, and updating configurations.

Prerequisites

1. Apache Kafka 2.8.0 or later: KRaft mode is available starting from Kafka 2.8.0, but it's stable for production in Kafka 3.0.0 and later.
2. Backup Data: Ensure you have a backup of your existing Kafka data and configurations.

Steps to Update Kafka Environment to Use KRaft

Step 1: Prepare Your Environment

1. Download and Install Kafka:

   • Download the latest version of Kafka from the official Apache Kafka website.

   wget <https://downloads.apache.org/kafka/3.2.0/kafka_2.13-3.2.0.tgz>
   tar -xzf kafka_2.13-3.2.0.tgz
   cd kafka_2.13-3.2.0
   
   

Step 2: Configure KRaft Mode

1. Create a New Configuration File:

   • Create a new Kafka configuration file for KRaft mode, e.g., server.properties.

   # Basic Kafka settings
   process.roles=broker,controller
   node.id=1
   controller.quorum.voters=1@localhost:9093
   listeners=PLAINTEXT://localhost:9092,CONTROLLER://localhost:9093
   inter.broker.listener.name=PLAINTEXT
   log.dirs=/tmp/kraft-combined-logs
   
   

2. Additional Configuration:

   • Set other necessary configurations like log retention, replication factor, etc., as per your existing setup.

   log.retention.hours=168
   num.partitions=1
   default.replication.factor=1
   min.insync.replicas=1
   
   

Step 3: Format the Storage for KRaft

1. Format the Kafka Storage:

   • Before starting the Kafka broker in KRaft mode, format the storage to initialize the metadata.

   bin/kafka-storage.sh format -t <uuid> -c config/server.properties
   
   

   • Replace <uuid> with a unique identifier. You can generate one using uuidgen or similar.

   uuidgen
   
   

Step 4: Start Kafka in KRaft Mode

1. Start the Kafka Broker:

   • Start the Kafka broker using the new configuration file.

   bin/kafka-server-start.sh config/server.properties
   
   

Step 5: Migrate Data and Clients

1. Reconfigure Clients:
   • Update your Kafka clients to connect to the new Kafka brokers in KRaft mode. Update the bootstrap servers to point to the new KRaft brokers.
2. Migrate Topics and Data:
   • Use tools like kafka-reassign-partitions.sh and kafka-mirror-maker.sh to migrate topics and data from the old ZooKeeper-based cluster to the new KRaft-based cluster.
3. Verify Data Migration:
   • Ensure that all topics, data, and consumer groups have been successfully migrated to the new cluster.

Step 6: Decommission the Old Cluster

1. Shutdown Old Brokers:
   • Once you've verified that the new KRaft-based cluster is working correctly and all data has been migrated, you can decommission the old ZooKeeper-based Kafka brokers.