Scalability is often a key issue for many growing organizations. That’s why many organizations use Apache Kafka, a popular messaging and streaming platform. It is horizontally scalable, cloud-native, and versatile. It can serve as a traditional publish-and-subscribe messaging system, as a streaming platform, or as a distributed state store. Companies around the world use Apache Kafka to build real-time streaming applications, streaming data pipelines, and event-driven architectures.
Continue reading Intro to Apache Kafka and Kafka Streams for Event-Driven Microservices on DevNation Live
There is a major push in the United Kingdom to replace aging mechanical electricity meters with connected smart meters. New meters allow consumers to more closely monitor their energy usage and associated cost, and they enable the suppliers to automate the billing process because the meters automatically report fine-grained energy use.
This post describes an architecture for processing a stream of meter readings using Strimzi, which offers support for running Apache Kafka in a container environment (Red Hat OpenShift). The data has been made available through a UK research project that collected data from energy producers, distributors, and consumers from 2011 to 2014. The TC1a dataset used here contains data from 8,000 domestic customers on half-hour intervals in the following form:
Continue reading “Smart-Meter Data Processing Using Apache Kafka on OpenShift”
As some may have noticed already, a new request for comments (RFC) regarding the Stream Control Transmission Protocol (SCTP), RFC8260, has been published recently. This RFC defines two major changes for the SCTP protocol, originally defined in RFC4960:
1) Stream schedulers, which control which stream gets served next when sending a data chunk over the wire.
2) I-Data chunk, which extends DATA to overcome some of its limitations.
This blog post will go over the two changes, pointing out the benefits of using the stream schedulers, and especially when using them together with the new I-Data chunks.
Continue reading “SCTP Stream Schedulers and User Message Interleaving”
At the beginning of October, I attended JavaOne in San Francisco to present on WildFly Swarm and Apache Kafka. For those of you that weren’t able to attend the session, or for those that did and saw first hand the issues with the demo, I will be covering all the details of how the demo should work!
Continue reading Data Streaming with WildFly Swarm and Apache Kafka