Smart-Meter Data Processing Using Apache Kafka on OpenShift

There is a major push in the United Kingdom to replace aging mechanical electricity meters with connected smart meters. New meters allow consumers to more closely monitor their energy usage and associated cost, and they enable the suppliers to automate the billing process because the meters automatically report fine-grained energy use.

This post describes an architecture for processing a stream of meter readings using Strimzi, which offers support for running Apache Kafka in a container environment (Red Hat OpenShift). The data has been made available through a UK research project that collected data from energy producers, distributors, and consumers from 2011 to 2014. The TC1a dataset used here contains data from 8,000 domestic customers on half-hour intervals in the following form:

Continue reading “Smart-Meter Data Processing Using Apache Kafka on OpenShift”

Share

Introducing the Kafka-CDI Library

Using Apache Kafka in modern event-driven applications is pretty popular. For a better cloud-native experience with Apache Kafka, it’s highly recommended to check out Red Hat AMQ Streams, which offers an easy installation and management of an Apache Kafka cluster on Red Hat OpenShift.

This article shows how the Kafka-CDI library can handle difficult setup tasks and make creating Kafka-powered event-driven applications for MicroProfile and Jakarta EE very easy.

Continue reading “Introducing the Kafka-CDI Library”

Share

Unlock your Hadoop data with Hortonworks and Red Hat JBoss Data Virtualization

Welcome to this first episode of this series: “Unlock your [….] data with Red Hat JBoss Data Virtualization (JDV).”

This post will guide you through an example of connecting to a Hadoop source via the Hive2 driver, using Teiid Designer. In this example we will demonstrate connection to a local Hadoop source.  We’re using the Hortonworks 2.5 Sandbox running in Virtual Box for our source, but you can connect to another Hortonwork source if you wish using the same steps.

Hortonworks provides Hive JDBC and ODBC drivers that let you connect popular tools to query, analyze and visualize data stored within the Hortonworks Data Platform (HDP).

Note: we support HBase as well, stay tuned for an episode of Unlock your HBase data with Hortonworks and JDV.

Continue reading “Unlock your Hadoop data with Hortonworks and Red Hat JBoss Data Virtualization”

Share

Webinar: How to Stay Agile with Big Data: A Roadmap – 10 September

Agility is the key for benefiting from the use of Big Data for operational excellence and improved profitability. Ovum Research finds that organizations that take an iterative approach to refining analytic models, consolidating data sources, and transitioning to the cloud, tend to find more success with Big Data.

Attend this webinar to learn how to:

  • Consolidate your data sources
  • Build open, flexible Big Data ecosystems
  • Find success with Big Data

Continue reading “Webinar: How to Stay Agile with Big Data: A Roadmap – 10 September”

Share

DevNation 2014: Scott McClellan – Hadoop and Beyond

Abstract: Historically, the term “Hadoop” has been considered synonymous with its core technologies: MapReduce and the Hadoop Distributed File System (HDFS). But today the definition of Hadoop is rapidly evolving.

The Hadoop community is generalizing the application runtime model beyond MapReduce. On the storage front, we’re seeing the emergence of many alternative Hadoop-compatible file systems. Red Hat has built an interface layer for its Red Hat Storage Server product. This complete implementation of the Hadoop file system interface lets Hadoop-related projects run transparently, directly on a Red Hat Storage Server cluster.

Continue reading “DevNation 2014: Scott McClellan – Hadoop and Beyond”

Share