The out-of-the-box monitoring stack in Red Hat OpenShift provides a comprehensive set of tools and services to monitor various aspects of your workloads, including metrics, logs, events, and traces. With this monitoring stack, you can gain valuable insights into the health and performance of your applications, identify and troubleshoot issues, and optimize resource utilization. This allows you to deliver a seamless user experience and confidently meet your business objectives.

In this article, we will explore the monitoring stack in Red Hat OpenShift and how to use it to monitor a sample application effectively using Prometheus and Grafana.

How to monitor a sample application

We will use an OpenShift cluster with version 4.10.36 in this example.

Step 1: Create a ConfigMap

Create a ConfigMap in the openshift-monitoring namespace.

In OpenShift, user workload refers to the applications and services that users deploy onto the cluster. Before we begin, we need to create a ConfigMap in the openshift-monitoring namespace to enable user workload monitoring.

apiVersion: v1
kind: ConfigMap
  name: cluster-monitoring-config
  namespace: openshift-monitoring
  config.yaml: |
    enableUserWorkload: true

Step 2: Check for Prometheus

Check openshift-user-workload-monitoring namespace for the new Prometheus server pod (Figure 1).

A screenshot of the Prometheus server pod in OpenShift Container Platform..
Figure 1: The Prometheus server pod is located in the Pod page.

After creating the ConfigMap, check the openshift-user-workload-monitoring namespace for a new Prometheus server pod.

Step 3: Deploy Java application

Deploy the example Java application as a deployment with an OCP route.

A sample Java application has been developed that provides an API for coin-tossing. This API exposes the number of times the heads side of the coin showed, represented as a coin_heads_total metric of counter type in Prometheus. The MicroMeter Quarkus extension has been utilized to expose this metric. The metrics can be accessed via the http://localhost:8080/q/metrics endpoint if the application starts locally. The application has been containerized and is available to use under

Next, deploy the coin-toss-api as a deployment with an OpenShift route. Use the following command to deploy the application:

oc new-app --name=coin-toss-api

oc expose svc/coin-toss-api

Step 4: Create a ServiceMonitor instance

Create a ServiceMonitor instance to register the Prometheus target.

To monitor the coin-toss-api application, we must create a ServiceMonitor instance to register the Prometheus target. Run the following command to create the ServiceMonitor:

kind: ServiceMonitor
  name: my-service-monitor
  namespace: my-project
    - interval: 5s
      path: /q/metrics
      port: 8080-tcp
      scheme: http
      - my-project
      app: coin-toss-api

Step 5: Filter targets

After creating the ServiceMonitor, navigate to the Prometheus UI by clicking on the Observe tab in the OpenShift console. Under the Targets menu, filter the targets by the user (Figure 2).

A screenshot of the filter tool under the targets tab.
Figure 2: Click on the Observe and Targets tab to filter metrics targets.

Step 6: Observe metrics

Navigate to the Observe tab in the OpenShift console, then select Metrics options (Figure 3).

In the expressions input field, you can type coin_heads_total, which is our custom metric name.

Then click the Run queries button.

Custom metric in Openshift console
Figure 3: Run Query

At this point, we have two different Prometheus server instances. One is for default OpenShift monitoring, and the other is for user workload monitoring. In OpenShift, to observe metrics, we can see metrics scraped through both the Prometheus server instances (Figure 4).

A diagram showing the two Prometheus server instances.
Figure 4: The two Prometheus server instances for workload monitoring.

Step 7: Install the Grafana operator

Install the Grafana operator in a different namespace. To install the Grafana operator, we recommend creating a new namespace. Run the following command to create the namespace as follows:

oc create namespace my-grafana

Then, install the Grafana operator using OperatorHub.

Step 8: Create a Grafana instance

Create a Grafana instance from the operator.

After installing the Grafana operator (v4.9.0), create a Grafana instance by running the following command:

kind: Grafana
  name: my-grafana
  namespace: my-grafana
      admin_user: admin
      admin_password: my-password
      - ReadWriteOnce
    class: gp2
    size: 1Gi
    enabled: true
      enabled: true

Configure Grafana to use an ingress with TLS enabled and setting an admin username and password. Make sure to replace "my-password" with a strong password.

Note: Please ensure that you include the available storage class in your cluster within the dataStorage section.

Step 9: Create a cluster role

Create a cluster role binding to the grafana-serviceaccount.

To allow Grafana to access monitoring data in the OpenShift cluster, we need to create a cluster role binding to the grafana-serviceaccount.

Run the following command:

oc create clusterrolebinding grafana-view --clusterrole=cluster-monitoring-view --serviceaccount=my-grafana:grafana-serviceaccount

This command creates a cluster role binding named grafana-view that provides the cluster-monitoring-view role to the grafana-serviceaccount in all namespaces.

Step 10: Generate a token

Generate a token using the grafana-serviceaccount.

To authenticate with the OpenShift API, Grafana needs a token generated from the grafana-serviceaccount. Run the following command to generate a token:

export TOKEN=$(oc create token grafana-serviceaccount -n my-grafana)

Then view the token using:

echo $TOKEN

Step 11: Create a GrafanaDataSource instance

Create a GrafanaDataSource instance from the operator.

Finally, create a GrafanaDataSource instance from the operator to connect Grafana to Prometheus. Run the following command:

kind: GrafanaDataSource
  name: prometheus
  namespace: my-grafana
    - basicAuthUser: internal
      access: proxy
      editable: true
        httpHeaderValue1: >-
          Bearer <<paste-your-generated-token-here>>
      name: Prometheus
      url: 'https://thanos-querier.openshift-monitoring.svc.cluster.local:9091'
        httpHeaderName1: Authorization
        timeInterval: 5s
        tlsSkipVerify: true
      basicAuth: false
      isDefault: true
      version: 1
      type: prometheus
  name: test_name

Log in to the Grafana dashboard and check data sources. You should see an entry like Figure 5. This is the data source we created using the custom resource in the operator.

Ensure that the Grafana can extract metrics from the Prometheus servers.

A screenshot of the Grafana dashboard.
Figure 5: The data source in the Grafana dashboard is displayed.

Once everything is working as expected, you can create your own Grafana dashboard. Figure 6 shows a simple graph I created inside the Grafana. It shows how many times the heads have appeared so far when tossing the coin in our Java application.

A Grafana graph created to show the number of coins tossed.
Figure 6: Grafana graph

The OpenShift monitoring stack is easy to use

In conclusion, using a monitoring stack on the Red Hat OpenShift platform is easy and straightforward. One of the major advantages of this platform is that it comes out of the box with all the essential components necessary to enable effective monitoring of your applications and infrastructure. With just a few configuration settings, you can easily set up monitoring tools like Prometheus, Grafana, and alert manager to provide insights into the performance and health of your applications.

Overall, the OpenShift platform provides a powerful and easy-to-use monitoring stack that simplifies the process of monitoring complex applications and infrastructure. With the ability to customize and extend the platform's monitoring capabilities, you can easily create a monitoring solution that meets the specific needs of your organization.

Last updated: September 19, 2023