Kafka Monthly Digest: July 2022

This 54th edition of the Kafka Monthly Digest covers what happened in the Apache Kafka community in July 2022.

For last month’s digest, see Kafka Monthly Digest: June 2022.

Releases

There is currently one release in progress, 3.3.0. A new bugfix release, 3.2.1, is available.

3.3.0

The release process for 3.3.0 continued. Feature freeze happened on July 6, followed by code freeze on July 20. We are now in the stabilization period, which typically lasts a couple of weeks. The first release candidate is expected in early August. You can find the release plan on the Kafka wiki.

3.2.1

On July 13, David Arthur volunteered to run this bugfix release. He published the first release candidate on July 22, and since no issues were found, he released Kafka 3.2.1 on August 2. In particular, this release fixes a few major issues including regressions in the consumer group protocol (1, 2 and 3), a memory leak in Kafka Connect, and a bug in the OAuth re-authentication logic. You can find the complete list of changes in the release notes, or in the release plan on the Kafka wiki.

Kafka Improvement Proposals

Last month, the community submitted 12 KIPs (KIP-851 through KIP-862). I'll highlight a few of them:

• KIP-853: KRaft voter changes: In KRaft mode, consensus is achieved by a configured group of voters. Currently, in order to change this list of voters, you need to shut down all controller nodes. This KIP proposes a mechanism to dynamically add or remove voters at runtime. This will make KRaft clusters much easier to scale and more resilient when voters experience disk failures or other issues.

• KIP-856: KRaft disk failure recovery: Following on from the previous proposal, this KIP aims at improving the recovery of a voter in case of a disk failure. It will rely on the ability to add and remove voters to effectively replace a voter with a failed disk while keeping the quorum active.

• KIP-858: Handle JBOD broker disk failure in KRaft: This KIP aims to add JBOD support to KRaft. KIP-589, which was voted a while ago, should have covered this use case; however, it was never implemented due to scalability issues found during the implementation process, as it required making a number of calls proportional to the number of partitions. This new KIP addresses that concern by explicitly assigning partitions to log directories in the metadata log, so if a log directory fails, it's easy to identify the affected partitions.

• KIP-860: Add client-provided option to guard against replication factor change during partition reassignments: While a partition is being reassigned, its metadata may contain a mix of the old and new replicas, and hence it could momentarily appear to have a larger replication factor. It's possible to list active reassignments but there are race conditions that can make identifying the replication factor of partitions confusing. This is especially an issue when submitting new partition reassignments that should preserve the replication factor. This KIP proposes adding a new field in the AlterPartitionReassignments request to indicate whether the reassignment is expected to change the replication factor or not.

Community releases

• Sarama 1.35: Sarama is a pure Go language Kafka client. This new version adds support for Kafka 3.2.0 and contains a few fixes in the consumer group protocol and the admin client.

• strimzi-kafka-operator 0.30: Strimzi is a Kubernetes Operator for running Kafka. The main change in this release is that StrimziPodSet is now used by default instead of StatefulSets for deploying Kafka and ZooKeeper. It also brings a few Cruise Control improvements; for instance, network and CPU broker capacity overrides are now supported.

Blogs

I selected some interesting blog posts and articles that were published last month:

• Using Apache Kafka to process 1 trillion inter-service messages
• On the availability of data pipelines

To learn more about Kafka, visit Red Hat Developer's Apache Kafka topic page.