Skip to main content
Redhat Developers  Logo
  • Products

    Featured

    • Red Hat Enterprise Linux
      Red Hat Enterprise Linux Icon
    • Red Hat OpenShift AI
      Red Hat OpenShift AI
    • Red Hat Enterprise Linux AI
      Linux icon inside of a brain
    • Image mode for Red Hat Enterprise Linux
      RHEL image mode
    • Red Hat OpenShift
      Openshift icon
    • Red Hat Ansible Automation Platform
      Ansible icon
    • Red Hat Developer Hub
      Developer Hub
    • View All Red Hat Products
    • Linux

      • Red Hat Enterprise Linux
      • Image mode for Red Hat Enterprise Linux
      • Red Hat Universal Base Images (UBI)
    • Java runtimes & frameworks

      • JBoss Enterprise Application Platform
      • Red Hat build of OpenJDK
    • Kubernetes

      • Red Hat OpenShift
      • Microsoft Azure Red Hat OpenShift
      • Red Hat OpenShift Virtualization
      • Red Hat OpenShift Lightspeed
    • Integration & App Connectivity

      • Red Hat Build of Apache Camel
      • Red Hat Service Interconnect
      • Red Hat Connectivity Link
    • AI/ML

      • Red Hat OpenShift AI
      • Red Hat Enterprise Linux AI
    • Automation

      • Red Hat Ansible Automation Platform
      • Red Hat Ansible Lightspeed
    • Developer tools

      • Red Hat Trusted Software Supply Chain
      • Podman Desktop
      • Red Hat OpenShift Dev Spaces
    • Developer Sandbox

      Developer Sandbox
      Try Red Hat products and technologies without setup or configuration fees for 30 days with this shared Openshift and Kubernetes cluster.
    • Try at no cost
  • Technologies

    Featured

    • AI/ML
      AI/ML Icon
    • Linux
      Linux Icon
    • Kubernetes
      Cloud icon
    • Automation
      Automation Icon showing arrows moving in a circle around a gear
    • View All Technologies
    • Programming Languages & Frameworks

      • Java
      • Python
      • JavaScript
    • System Design & Architecture

      • Red Hat architecture and design patterns
      • Microservices
      • Event-Driven Architecture
      • Databases
    • Developer Productivity

      • Developer productivity
      • Developer Tools
      • GitOps
    • Secure Development & Architectures

      • Security
      • Secure coding
    • Platform Engineering

      • DevOps
      • DevSecOps
      • Ansible automation for applications and services
    • Automated Data Processing

      • AI/ML
      • Data Science
      • Apache Kafka on Kubernetes
      • View All Technologies
    • Start exploring in the Developer Sandbox for free

      sandbox graphic
      Try Red Hat's products and technologies without setup or configuration.
    • Try at no cost
  • Learn

    Featured

    • Kubernetes & Cloud Native
      Openshift icon
    • Linux
      Rhel icon
    • Automation
      Ansible cloud icon
    • Java
      Java icon
    • AI/ML
      AI/ML Icon
    • View All Learning Resources

    E-Books

    • GitOps Cookbook
    • Podman in Action
    • Kubernetes Operators
    • The Path to GitOps
    • View All E-books

    Cheat Sheets

    • Linux Commands
    • Bash Commands
    • Git
    • systemd Commands
    • View All Cheat Sheets

    Documentation

    • API Catalog
    • Product Documentation
    • Legacy Documentation
    • Red Hat Learning

      Learning image
      Boost your technical skills to expert-level with the help of interactive lessons offered by various Red Hat Learning programs.
    • Explore Red Hat Learning
  • Developer Sandbox

    Developer Sandbox

    • Access Red Hat’s products and technologies without setup or configuration, and start developing quicker than ever before with our new, no-cost sandbox environments.
    • Explore Developer Sandbox

    Featured Developer Sandbox activities

    • Get started with your Developer Sandbox
    • OpenShift virtualization and application modernization using the Developer Sandbox
    • Explore all Developer Sandbox activities

    Ready to start developing apps?

    • Try at no cost
  • Blog
  • Events
  • Videos

Quick links: redhat.com, Customer Portal, Red Hat's developer site, Red Hat's partner site.

  • You are here

    Red Hat

    Learn about our open source products, services, and company.

  • You are here

    Red Hat Customer Portal

    Get product support and knowledge from the open source experts.

  • You are here

    Red Hat Developer

    Read developer tutorials and download Red Hat software for cloud application development.

  • You are here

    Red Hat Partner Connect

    Get training, subscriptions, certifications, and more for partners to build, sell, and support customer solutions.

Products & tools

  • Ansible.com

    Learn about and try our IT automation product.
  • Red Hat Ecosystem Catalog

    Find hardware, software, and cloud providers―and download container images―certified to perform with Red Hat technologies.

Try, buy, & sell

  • Red Hat Hybrid Cloud Console

    Access technical how-tos, tutorials, and learning paths focused on Red Hat’s hybrid cloud managed services.
  • Red Hat Store

    Buy select Red Hat products and services online.
  • Red Hat Marketplace

    Try, buy, sell, and manage certified enterprise software for container-based environments.

Events

  • Red Hat Summit and AnsibleFest

    Register for and learn about our annual open source IT industry event.

Visualize your Apache Kafka Streams using the Quarkus Dev UI

December 7, 2021
Daniel Oh
Related topics:
Event-DrivenJavaKafkaQuarkus
Related products:
Red Hat build of Quarkus

Share:

Share on twitter Share on facebook Share on linkedin Share with email
  • Reactive messaging and Apache Kafka
  • Example: Using the Quarkus Dev UI
  • Where to learn more

This article shows how you can visualize Apache Kafka Streams with reactive applications using the Dev UI in Quarkus. Quarkus, a Java framework, provides an extension to utilize the Kafka Streams API and also lets you implement stream processing applications based directly on Kafka.

Reactive messaging and Apache Kafka

Reactive messaging and Apache Kafka

With the rise of event-driven architectures, many developers are adopting reactive programming to write business applications. The requirements for these applications literally specify that they not be processed in real-time because end users don't really expect synchronous communication experiences through web browsers or mobile devices. Instead, low latency is a more important performance criterion, regardless of data volume or concurrent users.

You might be wondering how reactive programming could meet this very different goal. The secret is an asynchronous communication protocol that decouples senders from the applications that consume and process events. In this design, a caller (e.g., end user) sends a message to a recipient and then keeps processing other requests without waiting for the reply. Asynchronous processing can also improve high-volume data performance, security, and scalability.

However, it's not easy to implement everything involved in asynchronous communication capabilities with just reactive programming. This is the reason that message-queue platforms have also come to occupy a critical role in event-driven applications. Apache Kafka is one of the most popular platforms for processing event messages asynchronously to support reactive applications. Kafka Streams is a client library that abstracts changing event data sets (also known as streams) continuously in Kafka clusters to support high throughput and scalability. A stream is a collection of data records in the form of key-value pairs.

Example: Using the Quarkus Dev UI

Example: Using the Quarkus Dev UI

Take a look at the following getMetaData() method to see how Quarkus lets you issue interactive queries to Kafka Streams using a KafkaStreams injection. Find the complete code in the Quarkus Kafka Streams Quickstart.

    @Inject
    KafkaStreams streams;

    public List<PipelineMetadata> getMetaData() {
        return streams.allMetadataForStore(TopologyProducer.WEATHER_STATIONS_STORE)
                .stream()
                .map(m -> new PipelineMetadata(
                        m.hostInfo().host() + ":" + m.hostInfo().port(),
                        m.topicPartitions()
                                .stream()
                                .map(TopicPartition::toString)
                                .collect(Collectors.toSet())))
                .collect(Collectors.toList());
    }
Copy snippet

Kafka Streams also lets you build a process topology that represents a graph of sources, processors, and sinks in Kafka topics. Of course, you can monitor the streams using command-line tools (such as kcat), but the text-based output doesn't make it easy to understand how the streams are processing and consuming messages across Kafka topics.

Take a look at another example. The buildTopology() method lets you build the stream's topology. Find the complete code in the Quarkus Kafka Streams Quickstart.

    @Produces
    public Topology buildTopology() {
        StreamsBuilder builder = new StreamsBuilder();

        ObjectMapperSerde<WeatherStation> weatherStationSerde = new ObjectMapperSerde<>(WeatherStation.class);
        ObjectMapperSerde<Aggregation> aggregationSerde = new ObjectMapperSerde<>(Aggregation.class);

        KeyValueBytesStoreSupplier storeSupplier = Stores.persistentKeyValueStore(WEATHER_STATIONS_STORE);

        GlobalKTable<Integer, WeatherStation> stations = builder.globalTable(
                WEATHER_STATIONS_TOPIC,
                Consumed.with(Serdes.Integer(), weatherStationSerde));

        builder.stream(
                TEMPERATURE_VALUES_TOPIC,
                Consumed.with(Serdes.Integer(), Serdes.String()))
                .join(
                        stations,
                        (stationId, timestampAndValue) -> stationId,
                        (timestampAndValue, station) -> {
                            String[] parts = timestampAndValue.split(";");
                            return new TemperatureMeasurement(station.id, station.name, Instant.parse(parts[0]),
                                    Double.valueOf(parts[1]));
                        })
                .groupByKey()
                .aggregate(
                        Aggregation::new,
                        (stationId, value, aggregation) -> aggregation.updateFrom(value),
                        Materialized.<Integer, Aggregation> as(storeSupplier)
                                .withKeySerde(Serdes.Integer())
                                .withValueSerde(aggregationSerde))
                .toStream()
                .to(
                        TEMPERATURES_AGGREGATED_TOPIC,
                        Produced.with(Serdes.Integer(), aggregationSerde));

        return builder.build();
    }
Copy snippet

Visualize the Kafka Streams topology

To visualize the Kafka Streams topology, developers traditionally needed additional visualizer tools that run in the cloud or local development environments separately from Kafka clusters. But Quarkus's built-in Dev UI lets you see all the extensions currently loaded with relevant documentation. When you run Quarkus Dev Mode (e.g., ./mvnw quarkus:dev) and add a quarkus-kafka-streams extension in a project, the Dev UI shows the Apache Kafka Streams extension graphically (Figure 1).

The Developer UI shows the Apache Kafka Streams extension, with a Topology button.
Figure 1. The Developer UI shows the Apache Kafka Streams extension, with a Topology button.

When you click on the Topology icon, it brings you to the Kafka Streams topology UI (Figure 2).

The Topology screen for Apache Kafka Streams shows details, including active topics.
Figure 2. The Topology screen for Apache Kafka Streams shows details, including active topics.

The topology UI shows how the event streams sink in topics (e.g., temperature-values) and how Quarkus applications consume the streams from the topics. Also, you can understand how the application eventually aggregates streams from multiple topics (temperature-values and weather-stations) to one topic (temperatures-aggregated). The Topology UI also showcases the sequences on how the streams are sourced, joined, and aggregated continuously in Kafka clusters.

Where to learn more

Where to learn more

This article has shown how to visualize Kafka Streams with Quarkus applications and the Dev UI. Quarkus also provides awesome features to improve your productivity through continuous testing, the Quarkus command-line interface (CLI), and Dev Services. To learn more about Kafka and reactive messaging programming, see the following articles:

  • Getting Started to SmallRye Reactive Messaging with Apache Kafka
  • How do I run Apache Kafka on Kubernetes?
  • Level-up your gaming telemetry using Kafka Streams
  • Outbox pattern with OpenShift Streams for Apache Kafka and Debezium
  • Kafka at the Edge: an IoT scenario with OpenShift Streams for Apache Kafka
Last updated: May 8, 2024

Related Posts

  • Game telemetry with Kafka Streams and Quarkus, Part 1

  • Build an API using Quarkus from the ground up

  • Build a data streaming pipeline using Kafka Streams and Quarkus

  • HTTP-based Kafka messaging with Red Hat AMQ Streams

  • Kubernetes-native Apache Kafka with Strimzi, Debezium, and Apache Camel (Kafka Summit 2020)

Recent Posts

  • Meet the Red Hat Node.js team at PowerUP 2025

  • How to use pipelines for AI/ML automation at the edge

  • What's new in network observability 1.8

  • LLM Compressor: Optimize LLMs for low-latency deployments

  • How to set up NVIDIA NIM on Red Hat OpenShift AI

Red Hat Developers logo LinkedIn YouTube Twitter Facebook

Products

  • Red Hat Enterprise Linux
  • Red Hat OpenShift
  • Red Hat Ansible Automation Platform

Build

  • Developer Sandbox
  • Developer Tools
  • Interactive Tutorials
  • API Catalog

Quicklinks

  • Learning Resources
  • E-books
  • Cheat Sheets
  • Blog
  • Events
  • Newsletter

Communicate

  • About us
  • Contact sales
  • Find a partner
  • Report a website issue
  • Site Status Dashboard
  • Report a security problem

RED HAT DEVELOPER

Build here. Go anywhere.

We serve the builders. The problem solvers who create careers with code.

Join us if you’re a developer, software engineer, web designer, front-end designer, UX designer, computer scientist, architect, tester, product manager, project manager or team lead.

Sign me up

Red Hat legal and privacy links

  • About Red Hat
  • Jobs
  • Events
  • Locations
  • Contact Red Hat
  • Red Hat Blog
  • Inclusion at Red Hat
  • Cool Stuff Store
  • Red Hat Summit

Red Hat legal and privacy links

  • Privacy statement
  • Terms of use
  • All policies and guidelines
  • Digital accessibility

Report a website issue