Skip to main content
Redhat Developers  Logo
  • Products

    Featured

    • Red Hat Enterprise Linux
      Red Hat Enterprise Linux Icon
    • Red Hat OpenShift AI
      Red Hat OpenShift AI
    • Red Hat Enterprise Linux AI
      Linux icon inside of a brain
    • Image mode for Red Hat Enterprise Linux
      RHEL image mode
    • Red Hat OpenShift
      Openshift icon
    • Red Hat Ansible Automation Platform
      Ansible icon
    • Red Hat Developer Hub
      Developer Hub
    • View All Red Hat Products
    • Linux

      • Red Hat Enterprise Linux
      • Image mode for Red Hat Enterprise Linux
      • Red Hat Universal Base Images (UBI)
    • Java runtimes & frameworks

      • JBoss Enterprise Application Platform
      • Red Hat build of OpenJDK
    • Kubernetes

      • Red Hat OpenShift
      • Microsoft Azure Red Hat OpenShift
      • Red Hat OpenShift Virtualization
      • Red Hat OpenShift Lightspeed
    • Integration & App Connectivity

      • Red Hat Build of Apache Camel
      • Red Hat Service Interconnect
      • Red Hat Connectivity Link
    • AI/ML

      • Red Hat OpenShift AI
      • Red Hat Enterprise Linux AI
    • Automation

      • Red Hat Ansible Automation Platform
      • Red Hat Ansible Lightspeed
    • Developer tools

      • Red Hat Trusted Software Supply Chain
      • Podman Desktop
      • Red Hat OpenShift Dev Spaces
    • Developer Sandbox

      Developer Sandbox
      Try Red Hat products and technologies without setup or configuration fees for 30 days with this shared Openshift and Kubernetes cluster.
    • Try at no cost
  • Technologies

    Featured

    • AI/ML
      AI/ML Icon
    • Linux
      Linux Icon
    • Kubernetes
      Cloud icon
    • Automation
      Automation Icon showing arrows moving in a circle around a gear
    • View All Technologies
    • Programming Languages & Frameworks

      • Java
      • Python
      • JavaScript
    • System Design & Architecture

      • Red Hat architecture and design patterns
      • Microservices
      • Event-Driven Architecture
      • Databases
    • Developer Productivity

      • Developer productivity
      • Developer Tools
      • GitOps
    • Secure Development & Architectures

      • Security
      • Secure coding
    • Platform Engineering

      • DevOps
      • DevSecOps
      • Ansible automation for applications and services
    • Automated Data Processing

      • AI/ML
      • Data Science
      • Apache Kafka on Kubernetes
      • View All Technologies
    • Start exploring in the Developer Sandbox for free

      sandbox graphic
      Try Red Hat's products and technologies without setup or configuration.
    • Try at no cost
  • Learn

    Featured

    • Kubernetes & Cloud Native
      Openshift icon
    • Linux
      Rhel icon
    • Automation
      Ansible cloud icon
    • Java
      Java icon
    • AI/ML
      AI/ML Icon
    • View All Learning Resources

    E-Books

    • GitOps Cookbook
    • Podman in Action
    • Kubernetes Operators
    • The Path to GitOps
    • View All E-books

    Cheat Sheets

    • Linux Commands
    • Bash Commands
    • Git
    • systemd Commands
    • View All Cheat Sheets

    Documentation

    • API Catalog
    • Product Documentation
    • Legacy Documentation
    • Red Hat Learning

      Learning image
      Boost your technical skills to expert-level with the help of interactive lessons offered by various Red Hat Learning programs.
    • Explore Red Hat Learning
  • Developer Sandbox

    Developer Sandbox

    • Access Red Hat’s products and technologies without setup or configuration, and start developing quicker than ever before with our new, no-cost sandbox environments.
    • Explore Developer Sandbox

    Featured Developer Sandbox activities

    • Get started with your Developer Sandbox
    • OpenShift virtualization and application modernization using the Developer Sandbox
    • Explore all Developer Sandbox activities

    Ready to start developing apps?

    • Try at no cost
  • Blog
  • Events
  • Videos

Open innovation: Red Hat’s impact on the Kafka and Strimzi ecosystem

June 26, 2024
Simon Woodman
Related topics:
Data integrationKafkaKubernetesStream processing
Related products:
Streams for Apache Kafka

Share:

    At Red Hat, open source is at the heart of everything we do. We are the world's largest open source software company, and staying true to our roots is something that is incredibly important to us. From the first versions of Red Hat Enterprise Linux, to the latest release of Red Hat OpenShift, all our code is open and available to our customers. We work with people across a wide spectrum of open source projects, some overseen by foundations such as the Apache Software Foundation (ASF), Cloud Native Computing Foundation (CNCF) and the Eclipse Foundation, and some independently run by their community. 

    This article explores how Red Hat engineers contribute to open source projects in the data streaming landscape. 

    The Strimzi project: Apache Kafka on Kubernetes

    In 2018 Red Hat identified an opportunity to develop a new team, and community, to improve the experience of deploying and operating Apache Kafka on Kubernetes. The Strimzi project was created and in 2019 it was donated to CNCF, entering at the sandbox level. The project continues to grow with a diverse set of users and contributors, and was rewarded with Incubating status in 2024 (indicating a stable project used in production by a number of organizations). 

    Strimzi has been at the heart of innovation at the intersection of Kubernetes and Apache Kafka, and significantly reduces the operational burden of deploying and managing Kafka on Kubernetes. Custom resources enable declarative management of components such as Kafka clusters, topics, users and Kafka Connect connectors. These allow administrators to manage Kafka clusters in a manner consistent with other workloads on Kubernetes. Strimzi goes further in helping with certificate management and security, and in the path to Kafka 4.0 will automate the migration from ZooKeeper to KRaft mode.

    Apache Kafka

    Engineers from Red Hat contribute heavily to not only Strimzi but Apache Kafka, too. Over the past year, Red Hat has had 5 active contributors in the project who have improved the code, documentation, tests or in some other way. There are 3 committers (Tom Bentley, Mickael Maison, and Luke Chen) who are Red Hat associates—these people have the ability to merge changes to the software and cast binding votes on the Kafka Improvement Proposals (KIPs)—the process by which API-affecting changes are introduced to the project. The three committers from Red Hat are also members of the Project Management Committee (PMC) who oversee the governance of the project. In 2023, Mickael Maison was voted Chair of the PMC following the resignation of Jun Rao.

    Apache Kafka utilizes a process known as Kafka Improvement Proposals (KIPs) to discuss and agree on architectural and API level changes in the project. These design documents require a review and support from three committers in order to be approved—a process Red Hat engineers are heavily involved in. Following approval, the code can be written to implement the feature and will be available in a future release of Kafka. Engineers at Red Hat have proposed many KIPs across a wide scope in the Kafka project.

    To the core Kafka broker: 

    • KIP-827: Expose log dirs total and usable space via Kafka API
    • KIP-707: The future of KafkaFuture
    • KIP-788: Allow configuring num.network.threads per listener
    • KIP-978: Allow dynamic reloading of certificates with different DN / SANs

    To the clients: 

    • KIP-830: Allow disabling JMX Reporter
    • KIP-894: Use incrementalAlterConfigs API for syncing topic configurations

    To the Kafka Connect runtime:

    • KIP-769: Connect APIs to list all connector plugins and retrieve their configuration definitions
    • KIP-581: Value of optional null field which has default value

    Contributions to Apache Kafka from Red Hat engineers are not limited to feature implementations, but also to maintaining the health of the project and community. For instance, engineers from Red Hat contribute to the migration from Scala to Java for a number of components, and provide numerous bug fixes. In 2022, engineers from Red Hat were responsible for discovering, reporting, and responsibly disclosing CVE-2022-34917. This involved liaising with the Apache Software Foundation security group, helping to fix the issue, and coordinating embargoed releases prior to the vulnerability being made public.

    Code must be also reviewed prior to being merged into the project. Engineers from Red Hat are among the most active code reviewers in the project. Releasing any piece of software is a complex operation and handled by the committers in Apache Kafka. To date, engineers from Red Hat have been the Release Manager for over 40% of releases in the Kafka 3.x series, the most of any company.

    Other data streaming projects

    Considering the wider data streaming ecosystem, engineers from Red Hat actively contribute to many projects in this space. Some of them include:

    • Cruise Control: A tool for balancing workloads (eg. disk utilization, network bandwidth, etc.) across a set of Apache Kafka brokers.
    • Kroxylicious: A Kafka protocol proxy which enables use cases such as topic encryption.
    • Debezium: Changes data capture for a variety of databases. This allows an application to consume a list of change events from database tables.
    • SmallRye Reactive Messaging with Kafka: Bridges the gap between Reactive Messaging and Apache Kafka.
    • Apicurio Registry: A schema registry for storing schemas in formats such as Apache Avro and JSON Schema.
    • kcctl: A command-line client for Kafka Connect.

    Engineers from Red Hat are also actively engaged in the Apache Kafka and data streaming communities. They shared their expertise at conferences such as Kafka Summit, Devoxx UK, and jFokus about Kafka and data streaming topics. Mickael Maison releases a monthly blog post highlighting notable changes to Kafka, and other engineers contribute to blogs such as Strimzi and Grafana and provide regular support on mailing lists and Stack Overflow. 

    Kate Stanley and Mickael Maison recently wrote a book on Kafka Connect (free download available from Red Hat Developer). While writing it, they identified and addressed bugs in Kafka Connect and fixed numerous gaps in the documentation.

    Summary

    This article details the contributions of engineers from Red Hat to open source projects in the data streaming space. In some of those communities, the contributions are bug fixes and incremental improvements. However, in others such as Apache Kafka and Strimzi, engineers from Red Hat play leading roles. 

    Note: Red Hat Streams for Apache Kafka includes supported versions of many of these projects. If you are interested in finding out more, please get in touch.

    Related Posts

    • Building Apache Kafka Streams applications using Red Hat AMQ Streams: Part 1

    • Kubernetes-native Apache Kafka with Strimzi, Debezium, and Apache Camel (Kafka Summit 2020)

    • Introduction to Strimzi: Apache Kafka on Kubernetes (KubeCon Europe 2020)

    • HTTP-based Kafka messaging with Red Hat AMQ Streams

    • How to manage and preserve Kafka Connect offsets smoothly

    • Introduction to Kafka Connect

    Recent Posts

    • GuideLLM: Evaluate LLM deployments for real-world inference

    • Unleashing multimodal magic with RamaLama

    • Integrate Red Hat AI Inference Server & LangChain in agentic workflows

    • Streamline multi-cloud operations with Ansible and ServiceNow

    • Automate dynamic application security testing with RapiDAST

    What’s up next?

    Read Kafka Connect to discover how to build data pipelines between Kafka clusters and a variety of data sources and sinks.

    Get the e-book
    Red Hat Developers logo LinkedIn YouTube Twitter Facebook

    Products

    • Red Hat Enterprise Linux
    • Red Hat OpenShift
    • Red Hat Ansible Automation Platform

    Build

    • Developer Sandbox
    • Developer Tools
    • Interactive Tutorials
    • API Catalog

    Quicklinks

    • Learning Resources
    • E-books
    • Cheat Sheets
    • Blog
    • Events
    • Newsletter

    Communicate

    • About us
    • Contact sales
    • Find a partner
    • Report a website issue
    • Site Status Dashboard
    • Report a security problem

    RED HAT DEVELOPER

    Build here. Go anywhere.

    We serve the builders. The problem solvers who create careers with code.

    Join us if you’re a developer, software engineer, web designer, front-end designer, UX designer, computer scientist, architect, tester, product manager, project manager or team lead.

    Sign me up

    Red Hat legal and privacy links

    • About Red Hat
    • Jobs
    • Events
    • Locations
    • Contact Red Hat
    • Red Hat Blog
    • Inclusion at Red Hat
    • Cool Stuff Store
    • Red Hat Summit

    Red Hat legal and privacy links

    • Privacy statement
    • Terms of use
    • All policies and guidelines
    • Digital accessibility

    Report a website issue