This 46th edition of the Kafka Monthly Digest covers what happened in the Apache Kafka community in November 2021, including the imminent release of Apache Kafka 3.1.0, notable Kafka Improvement Proposals (KIPs), community project releases for Jikkou 0.7 and AKHQ 0.19.0, and more.
For last month's digest, see Kafka Monthly Digest: October 2021.
Two new Kafka bugfix versions were released in November, and Kafka 3.1.0 is in progress.
Bugfix releases for Kafka 2.7.2 and 2.6.3
2.6.3 addresses 11 JIRA tickets. For more details, see the release notes.
2.7.2 addresses 26 JIRA tickets. For more details, see the release notes.
Kafka 3.1.0 is nearly here
The next minor Kafka version, 3.1.0, is also nearing release. Code freeze happened on November 12 and we are now in the stabilization phase. There are still a few open blockers%20ORDER%20BY%20priority%20DESC%2C%20status%20DESC%2C%20updated%20DESC%20%20%20%20%20%20) but it should be ready in the next couple of weeks.
Kafka Improvement Proposals
Last month, the community submitted 16 KIPs (KIP-789 to KIP-805; note that 790 was skipped). I'll highlight just a few of them:
KIP-796: Interactive Query v2: Interactive Query is a functionality in Kafka Streams that allows you to query the internal state of Streams processors. The API dates back to the early days of Streams and has a few problems and limitations. This KIP proposes a brand new API, which is designed to be easier to use and maintain and will also offer more flexibility for users.
KIP-800: Add reason to JoinGroupRequest and LeaveGroupRequest: It can be hard to troubleshoot rebalancing issues because relevant logs are spread between brokers and clients. This KIP would make it simpler to troubleshoot such issues by requiring that consumers provide a reason when they join or leave a group.
KIP-801: Implement an Authorizer that stores metadata in __cluster_metadata: By default, Kafka has a built-in authorizer implementation,
AclAuthorizer, that stores access control lists (ACLs) in ZooKeeper. As Kafka is moving away from ZooKeeper (see KIP-500 for more details), this KIP proposes replacing the built-in authorizer with a new fully compatible implementation that stores ACLs directly in Kafka.
KIP-802: Validation support for Kafka Connect SMT options: Kafka Connect has a validation endpoint that you can use to check a connector's configuration before starting it. However, this endpoint does not currently validate Single Message Transformations (SMT), so the only way to know if an SMT configuration is correct is to try starting a connector. This KIP aims to address this gap by updating the
validateendpoint to check the full configuration.
This section covers a few notable open source community project releases:
Jikkou 0.7: Jikkou is a command-line tool for automating Kafka topic management. The latest version supports adding partitions to topics, and the topic validation options have been improved.
AKHQ 0.19.0: AKHQ is a GUI for Apache Kafka. Version 0.19 adds support for JSON and Protobuf schemas. There are also various improvements to the authentication options and to the topic data explorer, which now handles the
__transaction_stateinternal topics. A number of fixes are included in this version, as well.
I recommend the following blogs and articles that were published last month:
- Sleeping good at night: Kafka configurations tweaks
- Debezium UI support for Single Message Transformations
- Scaling Kafka Consumer for billions of events
- Kafka Streams with Quarkus