Red Hat continues to increase the features available for users looking to implement a 100% open source, event-driven architecture (EDA) through running Apache Kafka on Red Hat OpenShift and Red Hat Enterprise Linux. The Red Hat Integration Q4 release provides new features and capabilities, including ones aimed at simplifying usage and deployment of the AMQ streams distribution of Apache Kafka.
Features available in this new version include:
- Service Registry as Technical Preview
- HTTP-Kafka bridge is now generally available
- Secure HTTP-Kafka bridge with 3scale API Management
The addition of the service registry and the HTTP Kafka bridge improves Red Hat positioning as the 100% open source platform for cloud-native Kafka workloads. These new features become a suitable complement to the usage of the Kafka Operator for OpenShift based on the CNCF sandbox project Strimzi.
Schema service registry for event-driven architecture
Red Hat Integration’s service registry, based on the Apicurio project registry, provides a way to decouple the schema used to serialize and deserialize Kafka messages with the applications that are sending/receiving them. The registry is a store for schema (and API design) artifacts providing a REST API for access management and a set of optional rules for enforcing content validity and evolution.
The Apicurio service registry handles the following data formats:
- Apache Avro
- JSON Schema
- Protobuf (protocol buffers)
- OpenAPI
- AsyncAPI
In addition to the registry itself, users can leverage the included custom Kafka serializers and deserializers (SerDes). These SerDes Java classes allow Kafka applications to pull relevant schemas from the Service Registry instead of requiring the schemas to be bundled with the applications.
Correspondingly, the registry has its own REST API to create, update, and delete artifacts as well as managing global and per-artifact rules. The registry API is compatible with another Kafka provider’s schema registry to facilitate a seamless migration to AMQ Streams as a drop-in replacement.
For the upcoming technical preview, only the Avro format will be included in the service registry for the Red Hat Integration release.
Connecting to Kafka through HTTP
Apache Kafka uses a custom protocol on top of TCP/IP for communication between applications and the cluster. Clients are available for many different programming languages, but there are many scenarios in which a standard protocol such as HTTP/1.1 is more appropriate.
The Red Hat AMQ Streams Kafka Bridge provides an API for integrating HTTP-based clients with a Kafka cluster running on AMQ Streams. Applications can perform typical operations such as:
- Sending messages to topics.
- Subscribing to one or more topics.
- Receiving messages from the subscribed topics.
- Committing offsets related to the received messages.
- Seeking to a specific position.
Users can deploy the Kafka Bridge into an OpenShift cluster by using the AMQ Streams Operator or similar to an AMQ Streams installation, and users can download the Kafka Bridge files for installation on Red Hat Enterprise Linux.
Users can provide TLS support, authentication, and authorization using the API Management capabilities of Red Hat Integration by securing the Kafka Bridge with the 3scale component. Integration with API Management also means that additional features such as metrics, rate limiting, and billing are available.
Summary
The Red Hat Integration Q4 release makes Red Hat’s AMQ Streams distribution a better open source platform for cloud-native Kafka workloads. The Service Registry technical preview provides a common ground for data governance in the ever-changing domain of the event management bus. The general availability of the HTTP Bridge in Red Hat Integration enhances the options available to developers when building applications with Apache Kafka.
Last updated: March 29, 2023