Stream Processing

Building Apache Kafka Streams applications using Red Hat AMQ Streams: Part 2

Building Apache Kafka Streams applications using Red Hat AMQ Streams: Part 2

acattermole
By Adam Cattermole

The Apache Kafka project includes a Streams Domain-Specific Language (DSL) built on top of the lower-level Stream Processor API. This DSL provides developers with simple abstractions for performing data processing operations. However, how one builds a stream processing pipeline in a containerized environment with Kafka isn’t clear. This second article in a two-part series uses the basics from the previous article to build an example application using Red Hat AMQ Streams.

Continue reading “Building Apache Kafka Streams applications using Red Hat AMQ Streams: Part 2”

Share
Building Apache Kafka Streams applications using Red Hat AMQ Streams: Part 1

Building Apache Kafka Streams applications using Red Hat AMQ Streams: Part 1

acattermole
By Adam Cattermole

The Apache Kafka project includes a Streams Domain-Specific Language (DSL) built on top of the lower-level Stream Processor API. This DSL provides developers with simple abstractions for performing data processing operations. However, how to build a stream processing pipeline in a containerized environment with Kafka isn’t clear. This two-part article series describes the steps required to build your own Apache Kafka Streams application using Red Hat AMQ Streams.

Continue reading “Building Apache Kafka Streams applications using Red Hat AMQ Streams: Part 1”

Share
Upcoming Book: Vert.x in Action (MEAP)

Upcoming Book: Vert.x in Action (MEAP)

jponge
By Julien Ponge

I am pleased to announce that my upcoming book “Vert.x in Action: Asynchronous and Reactive Applications in Java” is now available from the Manning early-access program (MEAP): (See below for the exclusive Red Hat Developer discount code)

As enterprise applications become larger and more distributed, new architectural approaches like reactive designs, microservices, and event streams are required knowledge. The Eclipse Vert.x framework provides a mature, rock-solid toolkit for building reactive applications using Java, Kotlin, or Scala. Vert.x in Action teaches you to build responsive, resilient, and scalable JVM applications with Vert.x using well-established reactive design patterns.

Vert.x in Action teaches you to build highly-scalable reactive enterprise applications. In this practical developer’s guide, Vert.x expert Julien Ponge gets you up to speed in the basics of asynchronous programming as you learn to design and code reactive applications. Using the Vert.x asynchronous APIs, you’ll build services including web stack, messaging, authentication, and access control. You’ll also dive into deployment of container-native components with Docker, Kubernetes, and OpenShift. Along the way, you’ll check your app’s health and learn to test its resilience to external service failures.

Continue reading “Upcoming Book: Vert.x in Action (MEAP)”

Share
How to run Kafka on Openshift, the enterprise Kubernetes, with AMQ Streams

How to run Kafka on Openshift, the enterprise Kubernetes, with AMQ Streams

hguerrero
By Hugo Guerrero

On October 25th Red Hat announced the general availability of their AMQ Streams Kubernetes Operator for Apache Kafka. Red Hat AMQ Streams focuses on running Apache Kafka on Openshift providing a massively-scalable, distributed, and high performance data streaming platform. AMQ Streams, based on the Apache Kafka and Strimzi projects, offers a distributed backbone that allows microservices and other applications to share data with extremely high throughput. This backbone enables:

  • Publish and subscribe: Many to many dissemination in a fault tolerant, durable manner.
  • Replayable events: Serves as a repository for microservices to build in-memory copies of source data, up to any point in time.
  • Long-term data retention: Efficiently stores data for immediate access in a manner limited only by disk space.
  • Partition messages for more horizontal scalability: Allows for organizing messages to maximum concurrent access.

One of the most requested items from developers and architects is how to get started with a simple deployment option for testing purposes. In this guide we will use Red Hat Container Development Kit, based on minishift, to start an Apache Kafka cluster on Kubernetes.

Continue reading “How to run Kafka on Openshift, the enterprise Kubernetes, with AMQ Streams”

Share
EventFlow: Event-driven microservices on OpenShift (Part 1)

EventFlow: Event-driven microservices on OpenShift (Part 1)

hhidenmwessendswoodman
By Hugo Hiden and Matthias Wessendorf and Simon Woodman

This post is the first in a series of three related posts that describes a lightweight cloud-native distributed microservices framework we have created called EventFlow. EventFlow can be used to develop streaming applications that can process CloudEvents, which are an effort to standardize upon a data format for exchanging information about events generated by cloud platforms.

The EventFlow platform was created to specifically target the Kubernetes/OpenShift platforms, and it models event-processing applications as a connected flow or stream of components. The development of these components can be facilitated through the use of a simple SDK library, or they can be created as Docker images that can be configured using environment variables to attach to Kafka topics and process event data directly.

Continue reading “EventFlow: Event-driven microservices on OpenShift (Part 1)”

Share
Smart-Meter Data Processing Using Apache Kafka on OpenShift

Smart-Meter Data Processing Using Apache Kafka on OpenShift

hhidenmwessendswoodman
By Hugo Hiden and Matthias Wessendorf and Simon Woodman

There is a major push in the United Kingdom to replace aging mechanical electricity meters with connected smart meters. New meters allow consumers to more closely monitor their energy usage and associated cost, and they enable the suppliers to automate the billing process because the meters automatically report fine-grained energy use.

This post describes an architecture for processing a stream of meter readings using Strimzi, which offers support for running Apache Kafka in a container environment (Red Hat OpenShift). The data has been made available through a UK research project that collected data from energy producers, distributors, and consumers from 2011 to 2014. The TC1a dataset used here contains data from 8,000 domestic customers on half-hour intervals in the following form:

Continue reading “Smart-Meter Data Processing Using Apache Kafka on OpenShift”

Share