Kafka Monthly Digest: November 2021

December 7, 2021
Mickael Maison
Related topics:
Event-drivenKafkaMicroservices
Related products:
Red Hat OpenShift

This 46th edition of the Kafka Monthly Digest covers what happened in the Apache Kafka community in November 2021, including the imminent release of Apache Kafka 3.1.0, notable Kafka Improvement Proposals (KIPs), community project releases for Jikkou 0.7 and AKHQ 0.19.0, and more.

For last month's digest, see Kafka Monthly Digest: October 2021.

Releases

Two new Kafka bugfix versions were released in November, and Kafka 3.1.0 is in progress.

Bugfix releases for Kafka 2.7.2 and 2.6.3

I released Kafka 2.6.3 and 2.7.2 on November 22. These two bugfix releases include a fix for CVE-2021-38153.

Kafka 3.1.0 is nearly here

The next minor Kafka version, 3.1.0, is also nearing release. Code freeze happened on November 12 and we are now in the stabilization phase. There are still a few open blockers%20ORDER%20BY%20priority%20DESC%2C%20status%20DESC%2C%20updated%20DESC%20%20%20%20%20%20) but it should be ready in the next couple of weeks.

Kafka Improvement Proposals

Last month, the community submitted 16 KIPs (KIP-789 to KIP-805; note that 790 was skipped). I'll highlight just a few of them:

  • KIP-796: Interactive Query v2: Interactive Query is a functionality in Kafka Streams that allows you to query the internal state of Streams processors. The API dates back to the early days of Streams and has a few problems and limitations. This KIP proposes a brand new API, which is designed to be easier to use and maintain and will also offer more flexibility for users.

  • KIP-800: Add reason to JoinGroupRequest and LeaveGroupRequest: It can be hard to troubleshoot rebalancing issues because relevant logs are spread between brokers and clients. This KIP would make it simpler to troubleshoot such issues by requiring that consumers provide a reason when they join or leave a group.

  • KIP-801: Implement an Authorizer that stores metadata in __cluster_metadata: By default, Kafka has a built-in authorizer implementation, AclAuthorizer, that stores access control lists (ACLs) in ZooKeeper. As Kafka is moving away from ZooKeeper (see KIP-500 for more details), this KIP proposes replacing the built-in authorizer with a new fully compatible implementation that stores ACLs directly in Kafka.

  • KIP-802: Validation support for Kafka Connect SMT options: Kafka Connect has a validation endpoint that you can use to check a connector's configuration before starting it. However, this endpoint does not currently validate Single Message Transformations (SMT), so the only way to know if an SMT configuration is correct is to try starting a connector. This KIP aims to address this gap by updating the validate endpoint to check the full configuration.

Community releases

This section covers a few notable open source community project releases:

  • Jikkou 0.7: Jikkou is a command-line tool for automating Kafka topic management. The latest version supports adding partitions to topics, and the topic validation options have been improved.

  • AKHQ 0.19.0: AKHQ is a GUI for Apache Kafka. Version 0.19 adds support for JSON and Protobuf schemas. There are also various improvements to the authentication options and to the topic data explorer, which now handles the __consumer_offset and __transaction_state internal topics. A number of fixes are included in this version, as well.

  • node-rdkafka 2.12: This new release of node-rdkafka is now based on librdkafka 1.7.

Blogs

I recommend the following blogs and articles that were published last month:

To learn more about Kafka, visit Red Hat Developer's Apache Kafka topic page or the Kafka product homepage.

Last updated: September 20, 2023