Kafka Monthly Digest: June 2024

This 77th edition of the Kafka Monthly Digest covers what happened in the Apache Kafka community in June 2024.

For last month’s digest, see Kafka Monthly Digest: May 2024.

Releases

There is one new release, 3.7.1:

3.7.1
Igor Soarez published the first release candidate for 3.7.1 on June 10. A couple of small issues were found, so RC2 was published on June 19. and it passed the vote. Kafka 3.7.1 released on July 1 and contains over 60 fixes including several blocker issues. For more details, you can check the release announcement on the blog and the release notes.

There are two releases in progress, 3.8.0 and 3.9.0:

3.8.0
The work towards 3.8.0 is still ongoing. Instead of waiting for KIP-853 and unclean leader election support in KRaft mode to be implemented, the community decided to release 3.8.0 and complete these 2 features in 3.9.0. So code freeze happened on June 12 and we are now in the stabilization phase. There is currently one blocker issue remaining, once it's resolved we'll get the first release candidate. You can find the release plan in the wiki.
3.9.0
On June 14, Colin McCabe volunteered to be the release manager for the 3.9.0 release. As 3.8.0 was initially planned to be the last 3.X release before Kafka 4.0, the community decided to have a shorter release cycle for 3.9.0. The release date is targeted for August. You can find the release plan in the wiki.

Last month, the community submitted 16 KIPs (KIP-1050 to KIP-1065). I'll highlight a few of them:

KIP-1054: Support external schemas in JSONConverter: When using JsonConverter in a Connect sink pipeline, the converter expects the JSON schema to be included in the record. In cases JSON records are produced to Kafka without a schema, you first need to transform them. Also in most cases including the schema in the record significantly increases the record size. A common workaround is to use a Schema Registry but this still requires transforming the records to inject the schema id. This KIP proposes adding a new configuration to JsonConverter to provide the JSON schema so it does not need to be included in records when exporting data from Kafka to an external system with Connect.
KIP-1059: Enable Producer to resolve send() method errors: When using transactions, if a single record fails to be produced, the whole batch has to be aborted. This KIP aims at allowing users to ignore failed records by providing a new send() method in the Producer API.
KIP-1061: Allow exporting SCRAM credentials: When describing SCRAM credentials via the Admin API, for security reasons only the mechanism (SHA-256 or SHA-512) and the iteration count are returned. With ZooKeeper, administrators wanting to copy SCRAM credentials between clusters could retrieve the salt and secrets directly from ZooKeeper. With KRaft this is more complicated as by default these are stored in the metadata log. The proposal is to provide an encryption key to brokers so they can export encrypted SCRAM credentials.

Debezium 2.7: Debezium is a Change Data Capture platform. There is now a standalone connector for MariaDB (previously it was part of the MySQL connector). This release also includes significant improvements to most of the connectors. There are a few breaking changes too that are worth checking before you upgrade.
kroxylicious 0.6.0: Kroxylicious is an open source pluggable framework for writing network proxies that understand the Apache Kafka protocol. This release introduces a new filter for validating OAuth bearer tokens before forwarding them to brokers. It also adds an integration with AWS KMS.

I selected some interesting blog articles that were published last month: