Running Spark Jobs On OpenShift

Introduction:

A feature of OpenShift is jobs and today I will be explaining how you can use jobs to run your spark machine, learning data science applications against Spark running on OpenShift.  You can run jobs as a batch or scheduled, which provides cron like functionality. If jobs fail, by default OpenShift will retry the job creation again. At the end of this article, I have a video demonstration of running spark jobs from OpenShift templates against Spark running on OpenShift v3.

Continue reading “Running Spark Jobs On OpenShift”


Join Red Hat Developers, a developer program for you to learn, share, and code faster – and get access to Red Hat software for your development.  The developer program and software are both free!

 


For more information about Red Hat OpenShift and other related topics, visit: OpenShift, OpenShift Online.

Applying API Best Practices in Fuse

API plays a huge part in modern integration architecture design, a good design will allow your application to thrive, a bad design will end up on the cold stone bench and eventually vanishes. 🙁

Continue reading “Applying API Best Practices in Fuse”


Join Red Hat Developers, a developer program for you to learn, share, and code faster – and get access to Red Hat software for your development.  The developer program and software are both free!

 


Download and learn more about Red Hat JBoss Fuse, an innovative modular, cloud-ready architecture, powerful management and automation, and world class developer productivity. It is Java™ EE 7 certified and features powerful, enterprise-grade features such as high availability clustering, distributed caching, messaging, transactions, and a full web services stack.

Getting Started with Fuse Integration Service 2.0 Tech preview

To get started with FIS 2.0, for people who are just getting to know the technology, here is how I interpret it. Basically, it’s divided into two aspects.

1. Integration development: FIS uses Apache Camel as the core technology that creates, orchestrates, and composes microservices into a super lightweight thin integration layer, and becomes the API provider and service orchestrator through exposing RESTful or messaging service endpoints. And you can choose to either package and run it with Spring-Boot or Karaf.

2. Application Deployment and Management: FIS takes advantages of the OpenShift platform, and allows you to separately deploy the micro-integration service among a distributed environment, at the same time it takes care of the failover, high availability, load balancing and look up available services for you.

So, now we know what is in FIS 2.0, it’s time to take a closer look at how it is achieved, as a developer, you first need to decide to go with either Karaf (OSGi) or Spring-Boot framework, I personally prefer the Spring-Boot, because it matches closest to the microservice concept of a lighter deployment package. But it is up to you, the developer, after all, you are the god and creator of your application. After the framework is chosen, the developer will start developing the micro-integration services, composing between microservices or even use it to create a microservice. (With two frameworks, the development experience of the route itself is basically the same, by configuring camel components.)

Once the developer is ready to deploy the integration service, we can then decide how to deploy it onto the OpenShift platform. In FIS 2.0 there are two options, Binary S2i will build the entire application locally, and push it onto OpenShift, so OpenShift platform will use it to create and build the container image that it will run on, and for Source S2i, everything is build on top of OpenShift, so developer need to set the location of the source code in order for OpenShift to retrieve it to build the application and the container image.

And that is all. It is actually much more powerful, it’s hard to describe it in one go, dive into it, and you will soon find how fascinating the technology is, and how it can help you to resolve your current problems.

Here is a quick video that shows you how to get your first FIS 2.0 running.

The steps are as follows,

  • Install and start up OpenShift on your local machine 
  • Install FIS image stream definition into OpenShift (raw.githubusercontent.com/jboss-fuse/application-templates/application-templates-2.0.redhat-000026/fis-image-streams.json) 
  • In JBoss developer studio, create a Camel Spring-boot using the new archetype in FIS 2.0 
  • (Archetype catalog url: https://maven.repository.redhat.com/earlyaccess/all/io/fabric8/archetypes/archetypes-catalog/2.2.180.redhat-000004/archetypes-catalog-2.2.180.redhat-000004-archetype-catalog.xml)
  • Deploy to OpenShift using the Binary S2i tool (maven plugin)

 


Join Red Hat Developers, a developer program for you to learn, share, and code faster – and get access to Red Hat software for your development.  The developer program and software are both free!

 


Download and learn more about Red Hat JBoss Fuse, an innovative modular, cloud-ready architecture, powerful management and automation, and world class developer productivity. It is Java™ EE 7 certified and features powerful, enterprise-grade features such as high availability clustering, distributed caching, messaging, transactions, and a full web services stack.


For more information about Red Hat OpenShift and other related topics, visit: OpenShift, OpenShift Online.

Automate integration CI/CD process

Red Hat Fuse Integration Service 2.0 tech preview was released a few weeks ago and as it’s based on Red Hat OpenShift 3.3, which has pipeline capability on top of it (tech preview on OpenShift as well), you are able to get one step closer to a more automated and agile continuous integration. As well as, a deployment one-stop platform for us, the integration developer.

For the pipeline to work on OpenShift, you need Jenkins installed and running. OpenShift uses it to build, process and handle all the workflows. If you are familiar with developing in OpenShift, building the pipeline is pretty simple and straight-forward. The pipeline is defined as a build configuration in OpenShift, just create a build config then import it to the namespace you want it to be in. And that is it.

This is what the build config looks like, note the strategy type is called JenkinsPipeline.  This will trigger the interaction with Jenkins, and pushes the defined Jenkinsfile onto the server itself. The Jenkins Server will then interact with Openshift and start the automated CI/CD process.

kind: BuildConfig
apiVersion: v1
metadata:
 name: pipelinename
 labels:
 name: pipelinename 
spec:
 triggers:
 - type: GitHub
 github:
 secret: secret101
 - type: Generic
 generic:
 secret: secret101
 strategy:
 type: JenkinsPipeline
 jenkinsPipelineStrategy:
 jenkinsfile: "
 node('maven') { 
 stage('build') { 
 print 'build'
 openshiftBuild(buildConfig: 'buildconfigname', showBuildLogs: 'true')
 } 
 stage('staging') {
 print 'stage'
 openshiftDeploy(deploymentConfig: 'deploymentconfigame') 
 } 
 }"
As you can see on the above Jenkinsfile in the build configuration, it’s interacting with OpenShift itself through the OpenShift and Jenkins plugin. For instance, you could trigger build an image, deploy the application through calling the deployment config, tag an image or even scale up and down the number of containers.

This upper part of the blog is pretty generic to most of the applications running on OpenShift, and Fuse Integration Service is just another application running on top of it. But this application just simply contains PATTERN BASE integration technology that has 160+ built-in components in it, so we don’t have to waste time and energy on repetitive stuff, no big deal. 🙂

No matter what version you are using, this pipeline capability can help you automate your integration microservice.

Here is a quick demo video that takes you through the entire process.


Join Red Hat Developers, a developer program for you to learn, share, and code faster – and get access to Red Hat software for your development.  The developer program and software are both free!

 


Download and learn more about Red Hat JBoss Fuse, an innovative modular, cloud-ready architecture, powerful management and automation, and world class developer productivity. It is Java™ EE 7 certified and features powerful, enterprise-grade features such as high availability clustering, distributed caching, messaging, transactions, and a full web services stack.

Unlock your PostgreSQL data with Red Hat JBoss Data Virtualization

And here we go for another episode of the series: “Unlock your [….] data with Red Hat JBoss Data Virtualization.” Through this blog series, we will look at how to connect Red Hat JBoss Data Virtualization (JDV) to different and heterogeneous data sources.

JDV is a lean, virtual data integration solution that unlocks trapped data and delivers it as easily consumable, unified, and actionable information. It makes data spread across physically diverse systems — such as multiple databases, XML files, and Hadoop systems — appear as a set of tables in a local database. By providing the following functionality, JDV enables agile data use:

  1. Connect: Access data from multiple, heterogeneous data sources.
  2. Compose: Easily combine and transform data into reusable, business-friendly virtual data models and views.
  3. Consume: Makes unified data easily consumable through open standards interfaces.

It hides complexities, like the true locations of data or the mechanisms required to access or merge it. Data becomes easier for developers and users to work with. This post will guide you step-by-step on how to connect JDV to a PostgreSQL database using Teiid Designer. We will connect to a PostgreSQL database using the PostgreSQL JDBC driver.

Continue reading “Unlock your PostgreSQL data with Red Hat JBoss Data Virtualization”


Join Red Hat Developers, a developer program for you to learn, share, and code faster – and get access to Red Hat software for your development.  The developer program and software are both free!

 

Installing Red Hat Developer Studio 10.2.0.GA through RPM

With the release of Red Hat JBoss Developer Studio 10.2, it is now possible to install Red Hat JBoss Developer Studio as an RPM. It is available as a tech preview. The purpose of this article is to describe the steps you should follow in order to install Red Hat JBoss Developer Studio.

Red Hat Software Collections

JBoss Developer Studio RPM relies on Red Hat Software Collections. You don’t need to install Red Hat Software Collections but you need to enable the Red Hat Software Collections repositories before you start the installation of the Red Hat JBoss Developer Studio.

Enabling the Red Hat Software Collections base repository

The identifier for the repository is rhel-server-rhscl-7-rpms on Red Hat Enterprise Linux Server and rhel-workstation-rhscl-7-rpms on Red Hat Enterprise Linux Workstation.

The command to enable the repository on Red Hat Enterprise Linux Server is:

sudo subscription-manager repos --enable rhel-server-rhscl-7-rpms

The command to enable the repository on Red Hat Enterprise Linux Workstation is:

sudo subscription-manager repos --enable rhel-workstation-rhscl-7-rpms

For more information, please refer to the Red Hat Software Collections documentation.

JBoss Developer Studio repository

As this is a tech preview, you need to manually configure the JBoss Developer Studio repository.

Create a file /etc/yum.repos.d/rh-eclipse46-devstudio.repo with the following content:

[rh-eclipse46-devstudio-stable-10.x]
name=rh-eclipse46-devstudio-stable-10.x
baseurl=https://devstudio.redhat.com/static/10.0/stable/rpms/x86_64/
enabled=1
gpgcheck=1
upgrade_requirements_on_install=1
metadata_expire=24h

Red Hat developer signing key

As this is a tech preview, you need to accept the Red Hat developer signing key that has been used for producing the JBoss Developer Studio RPM.

Execute the following command:

sudo rpm --import "https://www.redhat.com/security/a5787476.txt"

Install Red Hat JBoss Developer Studio

You’re now ready to install Red Hat JBoss Developer Studio through RPM.

Enter the following command:

sudo yum install rh-eclipse46-devstudio

Answer ‘y’ when asked and after all required dependencies have been downloaded and installed, Red Hat JBoss Developer Studio is available on your system through the standard update channel !!!

You should see messages like the following:

rh eclipse46 devstudio.log

Launch Red Hat JBoss Developer Studio

From the system menu, mouse over the Programming menu, and the Red Hat Eclipse menu item will appear.

programming menu

Select this menu item and Red Hat JBoss Developer Studio user interface will appear then:

devstudio

Enjoy!

Jeff Maury 


Join Red Hat Developers, a developer program for you to learn, share, and code faster – and get access to Red Hat software for your development.  The developer program and software are both free!

 

Take advantage of your Red Hat Developers membership and download RHEL today at no cost.

How to containerize your Camel route on Karaf within OpenShift

The Red Hat JBoss Fuse solution offers a new approach of ESB, both lightweight and modular. It is perfectly suited to allow you to implement light integrations.

JBoss Fuse is fully supported, based on the power of Apache Karaf — Karaf allows for the easy deployment of your ActiveMQ Broker, your CXF web services, or your own Apache Camel routes.

Most of us are more familiar with the OSGI Environment, and what it offers: things like control of classloader behavior, module isolation, and APIs within a single app/JVM process.

For this post, we are gonna to setup a simple camel-route using a FIS (Fuse Integration Service) based on a Karaf image (jboss-fuse-6/fis-karaf-openshift), with which we will containerize your camel route on Karaf within OpenShift!


Join Red Hat Developers, a developer program for you to learn, share, and code faster – and get access to Red Hat software for your development.  The developer program and software are both free!

 


Download and learn more about Red Hat JBoss Fuse, an innovative modular, cloud-ready architecture, powerful management and automation, and world class developer productivity. It is Java™ EE 7 certified and features powerful, enterprise-grade features such as high availability clustering, distributed caching, messaging, transactions, and a full web services stack.


For more information about Red Hat OpenShift and other related topics, visit: OpenShift, OpenShift Online.

Red Hat JBoss Data Virtualization on OpenShift: Part 3 – Data federation

Welcome to part 3 of Red Hat JBoss Data Virtualization (JDV) running on OpenShift.

JDV is a lean, virtual data integration solution that unlocks trapped data and delivers it as easily consumable, unified, and actionable information. JDV makes data spread across physically diverse systems such as multiple databases, XML files, and Hadoop systems appear as a set of tables in a local database.

When deployed on OpenShift, JDV enables:

  1. Service enabling your data
  2. Bringing data from outside to inside the PaaS
  3. Breaking up monolithic data sources virtually for a microservices architecture

Together with the JDV for OpenShift image, we have made available several OpenShift templates that allow you to test and bootstrap JDV.

Continue reading “Red Hat JBoss Data Virtualization on OpenShift: Part 3 – Data federation”


Join Red Hat Developers, a developer program for you to learn, share, and code faster – and get access to Red Hat software for your development.  The developer program and software are both free!

 


For more information about Red Hat OpenShift and other related topics, visit: OpenShift, OpenShift Online.

Red Hat Releases New Versions of DevStudio, CDK, and DevSuite

As the interest in container application development continues to grow, so does our expansion of development tools and features.

Today, Red Hat released new versions of the following:

Here’s a listing of the new features:

Continue reading “Red Hat Releases New Versions of DevStudio, CDK, and DevSuite”

Securing Fuse 6.3 Fabric Cluster Management Console with SSL/TLS

Introduction

Enabling SSL/TLS in a Fabric is slightly more complex than securing a jetty in a standalone Karaf container. In the following article, we are providing feedback on the overall process. For clarity and simplification, the article will be divided into two parts.

 

Part1: The Management Console

Part2: Securing Web Service:including gateway-http

 

For the purpose of this PoC, the following environment will be used.

Continue reading “Securing Fuse 6.3 Fabric Cluster Management Console with SSL/TLS”


Download and learn more about Red Hat JBoss Fuse, an innovative modular, cloud-ready architecture, powerful management and automation, and world class developer productivity. It is Java™ EE 7 certified and features powerful, enterprise-grade features such as high availability clustering, distributed caching, messaging, transactions, and a full web services stack.


Join Red Hat Developers, a developer program for you to learn, share, and code faster – and get access to Red Hat software for your development.  The developer program and software are both free!