This 61st edition of the Kafka Monthly Digest covers what happened in the Apache Kafka community in February 2023.
For last month’s digest, see Kafka Monthly Digest: January 2023.
Releases
There is a new release, 3.4.0, and the planning for 3.5.0 has already started:
3.4.0
David Arthur released Kafka 3.4.0 on February 7 and published an announcement on the Apache blog. As always, you can find the complete list of changes in the release notes or the release plan on the Kafka wiki.
This new minor release brings a lot of bug fixes and a few new interesting features, which I'll highlight in the next sections.
Kafka brokers and clients
Updates to the Kafka broker and clients include the following:
- Migration from ZooKeeper to KRaft is available in early access. You can find the migration steps in the documentation. Note that due to KAFKA-14698, spurious errors are currently printed to the broker and controller logs while the migration is on-going. (KIP-866)
- Consumer partition assignments can now be rack aware. Note that none of the built-in assignors have been updated yet to support this feature.(KIP-881)
- The JMX metrics reporter now works the same way as other reporters. This means it can be disabled using
auto.include.jmx.reporterand from Kafka 4.0 it will be disabled if not present inmetric.reporters. (KIP-830)
Kafka Connect
Updates to Kafka Connect include the following:
- MirrorMaker can use a custom Admin interface to interact with clusters by setting
forwarding.admin.class. This allows better integration in environments that use custom tools to manage Kafka resources such as topics and consumer groups. (KIP-787)
Kafka Streams
Updates to Kafka Streams include the following:
- Result records can now be sent to multiple partitions using the new StreamPartitioner.partitions() method. (KIP-837)
3.5.0
On February 8, I volunteered to be the release manager for the next minor release, Kafka 3.5.0. The next milestone is KIP freeze currently targeted for March 22. The release date is expected in May, you can find the release plan with all the details in the wiki.
KIPs
Last month, the community submitted 8 KIPs (KIP-902 to KIP-909). I'll highlight a few of them:
- KIP-903: Replicas with stale broker epoch should not be allowed to join the ISR: This KIP aims at addressing a replication issue that could lead to data loss in very specific cases. At the moment when the leader of a partition wants to change the in-sync replicas (ISR), it only provides the broker ids in the AlterPartition request it sends to the controller. This KIP will add the broker epoch to ensure brokers are not added to the ISR if their current epoch does not match the epoch specified by leader.
- KIP-905: Broker interceptors: Over the years there has been a few attempts (KIP-686, KIP-729) at providing broker side interceptors to inspect or validate records. Due to multiple issues, so far none of these proposals were accepted by the community. This KIP is a new proposal that focuses solely on intercepting Produce requests by introducing a
ProduceRequestInterceptorinterface. - KIP-909: DNS Resolution Failure Should Not Fail the Clients: This KIP's goal is to improve how DNS resolution failures are handled by clients. Currently bootstrap servers are resolved in the constructors of clients, and if that fails a ConfigException is thrown. The proposal is to move DNS resolution out of the constructors and in case of failures throw a NetworkException that users can catch if they want to retry.
Community Releases
None of the projects I follow have made a release in February. If you know a popular open source community project that I don't cover, please let me know on Twitter or Mastodon.
Blogs
I selected some interesting blog articles that were published last month:
- DDD Aggregates via CDC-CQRS Pipeline using Kafka & Debezium
- Mastodon usage — counting toots with Kafka, DuckDB & Seaborn 🐘🦆📊
- Tracing in Apache Kafka: from OpenTracing to OpenTelemetry
To learn more about Kafka, visit Red Hat Developer's Apache Kafka topic page.
