Kafka Monthly Digest: January 2023

This 60th edition of the Kafka Monthly Digest covers what happened in the Apache Kafka community in January 2023. This is the 5th anniversary of this series! If you are interested in a bit of history about this series, 2 years ago I explained on Twitter how I got started.

For last month’s digest, see Kafka Monthly Digest: December 2022.

Releases

The 3.3.2 bugfix release is available. There is currently one release in progress, 3.4.0.

3.4.0

The release process for 3.4.0 continued. David Arthur published the first release candidate, RC0, on January 11. Unfortunately the community found a blocker issue, KAFKA-14618. On January 20, Sophie Blee-Goldman published RC1 but another issue was found, so on January 31 David Arthur published RC2. The vote is currently on-going.

3.3.2

Chris Egerton released 3.3.2 on January 23. This bugfix release fixes 20 issues. You can find more details in the release notes.

KIPs

Last month, the community submitted 6 KIPs (KIP-896 to KIP-901). I'll highlight a few of them:

• KIP-896: Remove old client protocol API versions in Kafka 4.0: The Kafka protocol defines a number of messages that clients and brokers use to exchange data. Each message is versioned and whenever the project changes one of these messages, a new version is created. For example Produce (the message used by producers to send records to brokers) has currently 9 versions. At the moment, clients and brokers still support all versions. This is great for compatibility as, for example, the latest clients are still able to connect to 0.9 brokers but this has a significant cost in code complexity and the testing matrix is huge. This KIP proposes removing support for API messages that are older than Kafka 2.1. That version released in November 2018 and the most popular third party clients already support newer APIs.
• KIP-898: Modernize Connect plugin discovery: When a Connect instance starts up, it scans its plugin paths to discover all the available plugins. This is an expensive operation as it effectively loads all classes found in the class and plugin paths to find plugins. This KIP aims at improving this mechanism by using Service Loaders and relying on plugins to provide manifests. The KIP also introduces a tool to generate manifests for migrating existing plugins.
• KIP-900: KRaft kafka-storage.sh API additions to support SCRAM for Kafka Brokers: KRaft mode is still missing a few features, including SCRAM support. If brokers themselves use SCRAM, you need a mechanism to bootstrap the setup before brokers are running (in ZooKeeper mode this is done using the kafka-configs.sh tool with the --zookeeper flag). This KIP proposes updating the kafka-storage.sh script to allow creating credentials before starting brokers and setting up SCRAM authentication.

Community Releases

• strimzi-kafka-operator 0.33: Strimzi is a Kubernetes Operator for running Kafka. This new version adds support for Kafka 3.3.2 and requires Kubernetes 1.19 or above. It includes support for automatically restarting failed connectors, better FIPS support, and improves significantly the scalability of the User operator.
• Sarama 1.38: Sarama is a pure Golang Kafka client. In addition of a handful of fixes, this release improves the memory usage when using zstd, adds support for Metadata v7 and adds a few new producer metrics.
• Librdkafka 2.0. Librdkafka is a Kafka client in C/C++. This version adds support for listing, describing and deleting consumer groups via the Admin API. It now supports OpenSSL 3.0 and fixes a few issues with the idempotent and transactional producers.

Blogs

I selected some interesting blog articles that were published last month:

• Intelligent, automatic restarts for unhealthy Kafka consumers
• Horizontally scaling Kafka consumers with rendezvous hashing

To learn more about Kafka, visit Red Hat Developer's Apache Kafka topic page.