This 48th edition of the Kafka Monthly Digest covers what happened in the Apache Kafka community in January 2022. The release of Apache Kafka 3.1.0 is the big news, including updates to Kafka Connect and Kafka Streams. I'll also discuss new KIPs and open source releases in January 2022.
This is the fourth anniversary of this series! If you are interested in the history behind the Kafka Monthly Digest, last year I explained on Twitter how I got started.
For last month's digest, see Kafka Monthly Digest: December 2021.
Kafka 3.1.0 release
David Jacot released Kafka 3.1.0 on January 24 and published an announcement on the Apache blog. As always, you can find the complete list of changes in the release notes or the release plan on the Kafka wiki.
Kafka 3.1 adds support for Java 17. Note that running Kafka in KRaft mode (without ZooKeeper) is still not ready for production.
This new minor release brings a few interesting features, which I'll highlight in the next sections.
Kafka brokers and clients
Updates to the Kafka broker and clients include the following:
- A production-ready OAuth implementation with OpenID Connect (OIDC) support (KIP-768).
- Metrics improvements include new metrics for active and fenced brokers (KIP-748) and consistent names for latency metrics (KIP-773).
- Topic IDs that were introduced in Kafka 2.8 are now used in
Updates to Kafka Streams include the following:
- Foreign-key joins can now be performed using custom partitioners (KIP-775).
- Querying ReadOnlySessionStore, ReadOnlyWindowStore, and ReadOnlyKeyValueStore with unbounded ranges is now supported (KIP-763 and KIP-766).
- The eager rebalance protocol is now deprecated. The new protocol, cooperative rebalancing, has been the default since Kafka 2.2. Eager will be removed in the next major release (KAFKA-13439).
- All uncaught exceptions are now wrapped into StreamsException (KIP-783).
Kafka Improvement Proposals
Last month, the community submitted seven KIPs (KIP-812 to KIP-818). I'll highlight just a few of them:
KIP-813: Shareable state stores: This KIP proposes a mechanism for reusing existing Kafka Streams state stores in other Streams applications. In scenarios where multiple applications perform similar processing steps, this could be useful to avoid doing the same processing twice.
KIP-814: Static membership protocol should let the leader skip assignment: This KIP's goal is to improve support for static consumer group membership (introduced in Kafka 2.4). Today, in some cases, consumer groups with static membership can end up without a leader and miss metadata updates.
KIP-817: Fix inconsistency in dynamic application log levels: There are two methods for changing Kafka's log at runtime: Java Management Extensions (JMX) and the Admin API. However, there are some inconsistencies in the levels they support. For example, it's not possible to set a logger to
OFFvia the Admin API. This KIP aims at clearing up those inconsistencies.
Community releases for Apache Kafka
This section covers a few notable open source community project releases:
A Great Day Out With... Apache Kafka: This is a curated list of resources, projects, tools, and products for people getting started with Kafka. The most recent update added the latest key Kafka resources people should be aware of when working with the platform.
Sarama 1.31: Sarama is a pure Go language Kafka client. This new release adds support for
IncrementalAlterConfigsin the Admin client, improves request pipelining for the producer, and as usual fixes a few bugs.
Kafka blogs and articles
Here are a few of the most noteworthy Kafka-related blogs and articles published in January 2022:
- 5 common pitfalls when using Apache Kafka
- Kafka Streams: Iterative development and blue-green deployment
- Towards a MongoDB-backed Apache Kafka Streams state store
To learn more about Kafka, visit Red Hat Developer's Apache Kafka topic page.