data streaming

Build a data streaming pipeline using Kafka Streams and Quarkus

Build a data streaming pipeline using Kafka Streams and Quarkus

kashukla
By Kapil Shukla

In typical data warehousing systems, data is first accumulated and then processed. But with the advent of new technologies, it is now possible to process data as and when it arrives. We call this real-time data processing. In real-time processing, data streams through pipelines; i.e., moving from one system to another. Data gets generated from static sources (like databases) or real-time systems (like transactional applications), and then gets filtered, transformed, and finally stored in a database or pushed to several other systems for further processing. The other systems can then follow the same cycle—i.e., filter, transform, store, or push to other systems.

Continue reading Build a data streaming pipeline using Kafka Streams and Quarkus

Share
Intro to Apache Kafka and Kafka Streams for Event-Driven Microservices on DevNation Live

Intro to Apache Kafka and Kafka Streams for Event-Driven Microservices on DevNation Live

burrsutter
By Burr Sutter

Scalability is often a key issue for many growing organizations. That’s why many organizations use Apache Kafka, a popular messaging and streaming platform. It is horizontally scalable, cloud-native, and versatile. It can serve as a traditional publish-and-subscribe messaging system, as a streaming platform, or as a distributed state store. Companies around the world use Apache Kafka to build real-time streaming applications, streaming data pipelines, and event-driven architectures.

Continue reading Intro to Apache Kafka and Kafka Streams for Event-Driven Microservices on DevNation Live

Share
Smart-Meter Data Processing Using Apache Kafka on OpenShift

Smart-Meter Data Processing Using Apache Kafka on OpenShift

hhidenmwessendswoodman
By Hugo Hiden and Matthias Wessendorf and Simon Woodman

There is a major push in the United Kingdom to replace aging mechanical electricity meters with connected smart meters. New meters allow consumers to more closely monitor their energy usage and associated cost, and they enable the suppliers to automate the billing process because the meters automatically report fine-grained energy use.

This post describes an architecture for processing a stream of meter readings using Strimzi, which offers support for running Apache Kafka in a container environment (Red Hat OpenShift). The data has been made available through a UK research project that collected data from energy producers, distributors, and consumers from 2011 to 2014. The TC1a dataset used here contains data from 8,000 domestic customers on half-hour intervals in the following form:

Continue reading “Smart-Meter Data Processing Using Apache Kafka on OpenShift”

Share
SCTP Stream Schedulers and User Message Interleaving

SCTP Stream Schedulers and User Message Interleaving

mleitnerredhat-com
By Marcelo Ricardo Leitner

As some may have noticed already, a new request for comments (RFC) regarding the Stream Control Transmission Protocol (SCTP), RFC8260, has been published recently. This RFC defines two major changes for the SCTP protocol, originally defined in RFC4960:

1) Stream schedulers, which control which stream gets served next when sending a data chunk over the wire.
2) I-Data chunk, which extends DATA to overcome some of its limitations.

This blog post will go over the two changes, pointing out the benefits of using the stream schedulers, and especially when using them together with the new I-Data chunks.

Continue reading “SCTP Stream Schedulers and User Message Interleaving”

Share