Skip to main content
Redhat Developers  Logo
  • Products

    Featured

    • Red Hat Enterprise Linux
      Red Hat Enterprise Linux Icon
    • Red Hat OpenShift AI
      Red Hat OpenShift AI
    • Red Hat Enterprise Linux AI
      Linux icon inside of a brain
    • Image mode for Red Hat Enterprise Linux
      RHEL image mode
    • Red Hat OpenShift
      Openshift icon
    • Red Hat Ansible Automation Platform
      Ansible icon
    • Red Hat Developer Hub
      Developer Hub
    • View All Red Hat Products
    • Linux

      • Red Hat Enterprise Linux
      • Image mode for Red Hat Enterprise Linux
      • Red Hat Universal Base Images (UBI)
    • Java runtimes & frameworks

      • JBoss Enterprise Application Platform
      • Red Hat build of OpenJDK
    • Kubernetes

      • Red Hat OpenShift
      • Microsoft Azure Red Hat OpenShift
      • Red Hat OpenShift Virtualization
      • Red Hat OpenShift Lightspeed
    • Integration & App Connectivity

      • Red Hat Build of Apache Camel
      • Red Hat Service Interconnect
      • Red Hat Connectivity Link
    • AI/ML

      • Red Hat OpenShift AI
      • Red Hat Enterprise Linux AI
    • Automation

      • Red Hat Ansible Automation Platform
      • Red Hat Ansible Lightspeed
    • Developer tools

      • Red Hat Trusted Software Supply Chain
      • Podman Desktop
      • Red Hat OpenShift Dev Spaces
    • Developer Sandbox

      Developer Sandbox
      Try Red Hat products and technologies without setup or configuration fees for 30 days with this shared Openshift and Kubernetes cluster.
    • Try at no cost
  • Technologies

    Featured

    • AI/ML
      AI/ML Icon
    • Linux
      Linux Icon
    • Kubernetes
      Cloud icon
    • Automation
      Automation Icon showing arrows moving in a circle around a gear
    • View All Technologies
    • Programming Languages & Frameworks

      • Java
      • Python
      • JavaScript
    • System Design & Architecture

      • Red Hat architecture and design patterns
      • Microservices
      • Event-Driven Architecture
      • Databases
    • Developer Productivity

      • Developer productivity
      • Developer Tools
      • GitOps
    • Secure Development & Architectures

      • Security
      • Secure coding
    • Platform Engineering

      • DevOps
      • DevSecOps
      • Ansible automation for applications and services
    • Automated Data Processing

      • AI/ML
      • Data Science
      • Apache Kafka on Kubernetes
      • View All Technologies
    • Start exploring in the Developer Sandbox for free

      sandbox graphic
      Try Red Hat's products and technologies without setup or configuration.
    • Try at no cost
  • Learn

    Featured

    • Kubernetes & Cloud Native
      Openshift icon
    • Linux
      Rhel icon
    • Automation
      Ansible cloud icon
    • Java
      Java icon
    • AI/ML
      AI/ML Icon
    • View All Learning Resources

    E-Books

    • GitOps Cookbook
    • Podman in Action
    • Kubernetes Operators
    • The Path to GitOps
    • View All E-books

    Cheat Sheets

    • Linux Commands
    • Bash Commands
    • Git
    • systemd Commands
    • View All Cheat Sheets

    Documentation

    • API Catalog
    • Product Documentation
    • Legacy Documentation
    • Red Hat Learning

      Learning image
      Boost your technical skills to expert-level with the help of interactive lessons offered by various Red Hat Learning programs.
    • Explore Red Hat Learning
  • Developer Sandbox

    Developer Sandbox

    • Access Red Hat’s products and technologies without setup or configuration, and start developing quicker than ever before with our new, no-cost sandbox environments.
    • Explore Developer Sandbox

    Featured Developer Sandbox activities

    • Get started with your Developer Sandbox
    • OpenShift virtualization and application modernization using the Developer Sandbox
    • Explore all Developer Sandbox activities

    Ready to start developing apps?

    • Try at no cost
  • Blog
  • Events
  • Videos

How to monitor workloads using OpenShift monitoring stack

August 8, 2023
Jude Niroshan
Related topics:
Containers
Related products:
Red Hat OpenShift

Share:

    The out-of-the-box monitoring stack in Red Hat OpenShift provides a comprehensive set of tools and services to monitor various aspects of your workloads, including metrics, logs, events, and traces. With this monitoring stack, you can gain valuable insights into the health and performance of your applications, identify and troubleshoot issues, and optimize resource utilization. This allows you to deliver a seamless user experience and confidently meet your business objectives.

    In this article, we will explore the monitoring stack in Red Hat OpenShift and how to use it to monitor a sample application effectively using Prometheus and Grafana.

    How to monitor a sample application

    We will use an OpenShift cluster with version 4.10.36 in this example.

    Step 1: Create a ConfigMap

    Create a ConfigMap in the openshift-monitoring namespace.

    In OpenShift, user workload refers to the applications and services that users deploy onto the cluster. Before we begin, we need to create a ConfigMap in the openshift-monitoring namespace to enable user workload monitoring.

    apiVersion: v1
    kind: ConfigMap
    metadata:
      name: cluster-monitoring-config
      namespace: openshift-monitoring
    data:
      config.yaml: |
        enableUserWorkload: true

    Step 2: Check for Prometheus

    Check openshift-user-workload-monitoring namespace for the new Prometheus server pod (Figure 1).

    A screenshot of the Prometheus server pod in OpenShift Container Platform..
    Figure 1: The Prometheus server pod is located in the Pod page.

    After creating the ConfigMap, check the openshift-user-workload-monitoring namespace for a new Prometheus server pod.

    Step 3: Deploy Java application

    Deploy the example Java application as a deployment with an OCP route.

    A sample Java application has been developed that provides an API for coin-tossing. This API exposes the number of times the heads side of the coin showed, represented as a coin_heads_total metric of counter type in Prometheus. The MicroMeter Quarkus extension has been utilized to expose this metric. The metrics can be accessed via the http://localhost:8080/q/metrics endpoint if the application starts locally. The application has been containerized and is available to use under quay.io/jnirosha/coin-toss-api.

    Next, deploy the coin-toss-api as a deployment with an OpenShift route. Use the following command to deploy the application:

    oc new-app --name=coin-toss-api quay.io/jnirosha/coin-toss-api

    oc expose svc/coin-toss-api

    Step 4: Create a ServiceMonitor instance

    Create a ServiceMonitor instance to register the Prometheus target.

    To monitor the coin-toss-api application, we must create a ServiceMonitor instance to register the Prometheus target. Run the following command to create the ServiceMonitor:

    apiVersion: monitoring.coreos.com/v1
    kind: ServiceMonitor
    metadata:
      name: my-service-monitor
      namespace: my-project
    spec:
      endpoints:
        - interval: 5s
          path: /q/metrics
          port: 8080-tcp
          scheme: http
      namespaceSelector:
        matchNames:
          - my-project
      selector:
        matchLabels:
          app: coin-toss-api

    Step 5: Filter targets

    After creating the ServiceMonitor, navigate to the Prometheus UI by clicking on the Observe tab in the OpenShift console. Under the Targets menu, filter the targets by the user (Figure 2).

    A screenshot of the filter tool under the targets tab.
    Figure 2: Click on the Observe and Targets tab to filter metrics targets.

    Step 6: Observe metrics

    Navigate to the Observe tab in the OpenShift console, then select Metrics options (Figure 3).

    In the expressions input field, you can type coin_heads_total, which is our custom metric name.

    Then click the Run queries button.

    Custom metric in Openshift console
    Figure 3: Run Query

    At this point, we have two different Prometheus server instances. One is for default OpenShift monitoring, and the other is for user workload monitoring. In OpenShift, to observe metrics, we can see metrics scraped through both the Prometheus server instances (Figure 4).

    A diagram showing the two Prometheus server instances.
    Figure 4: The two Prometheus server instances for workload monitoring.

    Step 7: Install the Grafana operator

    Install the Grafana operator in a different namespace. To install the Grafana operator, we recommend creating a new namespace. Run the following command to create the namespace as follows:

    oc create namespace my-grafana

    Then, install the Grafana operator using OperatorHub.

    Step 8: Create a Grafana instance

    Create a Grafana instance from the operator.

    After installing the Grafana operator (v4.9.0), create a Grafana instance by running the following command:

    apiVersion: integreatly.org/v1alpha1
    kind: Grafana
    metadata:
      name: my-grafana
      namespace: my-grafana
    spec:
      config:
        security:
          admin_user: admin
          admin_password: my-password
      dataStorage:
        accessModes:
          - ReadWriteOnce
        class: gp2
        size: 1Gi
      ingress:
        enabled: true
        tls:
          enabled: true

    Configure Grafana to use an ingress with TLS enabled and setting an admin username and password. Make sure to replace "my-password" with a strong password.

    Note: Please ensure that you include the available storage class in your cluster within the dataStorage section.

    Step 9: Create a cluster role

    Create a cluster role binding to the grafana-serviceaccount.

    To allow Grafana to access monitoring data in the OpenShift cluster, we need to create a cluster role binding to the grafana-serviceaccount.

    Run the following command:

    oc create clusterrolebinding grafana-view --clusterrole=cluster-monitoring-view --serviceaccount=my-grafana:grafana-serviceaccount

    This command creates a cluster role binding named grafana-view that provides the cluster-monitoring-view role to the grafana-serviceaccount in all namespaces.

    Step 10: Generate a token

    Generate a token using the grafana-serviceaccount.

    To authenticate with the OpenShift API, Grafana needs a token generated from the grafana-serviceaccount. Run the following command to generate a token:

    export TOKEN=$(oc create token grafana-serviceaccount -n my-grafana)

    Then view the token using:

    echo $TOKEN

    Step 11: Create a GrafanaDataSource instance

    Create a GrafanaDataSource instance from the operator.

    Finally, create a GrafanaDataSource instance from the operator to connect Grafana to Prometheus. Run the following command:

    apiVersion: integreatly.org/v1alpha1
    kind: GrafanaDataSource
    metadata:
      name: prometheus
      namespace: my-grafana
    spec:
      datasources:
        - basicAuthUser: internal
          access: proxy
          editable: true
          secureJsonData:
            httpHeaderValue1: >-
              Bearer <<paste-your-generated-token-here>>
          name: Prometheus
          url: 'https://thanos-querier.openshift-monitoring.svc.cluster.local:9091'
          jsonData:
            httpHeaderName1: Authorization
            timeInterval: 5s
            tlsSkipVerify: true
          basicAuth: false
          isDefault: true
          version: 1
          type: prometheus
      name: test_name

    Log in to the Grafana dashboard and check data sources. You should see an entry like Figure 5. This is the data source we created using the custom resource in the operator.

    Ensure that the Grafana can extract metrics from the Prometheus servers.

    A screenshot of the Grafana dashboard.
    Figure 5: The data source in the Grafana dashboard is displayed.

    Once everything is working as expected, you can create your own Grafana dashboard. Figure 6 shows a simple graph I created inside the Grafana. It shows how many times the heads have appeared so far when tossing the coin in our Java application.

    A Grafana graph created to show the number of coins tossed.
    Figure 6: Grafana graph

    The OpenShift monitoring stack is easy to use

    In conclusion, using a monitoring stack on the Red Hat OpenShift platform is easy and straightforward. One of the major advantages of this platform is that it comes out of the box with all the essential components necessary to enable effective monitoring of your applications and infrastructure. With just a few configuration settings, you can easily set up monitoring tools like Prometheus, Grafana, and alert manager to provide insights into the performance and health of your applications.

    Overall, the OpenShift platform provides a powerful and easy-to-use monitoring stack that simplifies the process of monitoring complex applications and infrastructure. With the ability to customize and extend the platform's monitoring capabilities, you can easily create a monitoring solution that meets the specific needs of your organization.

    Last updated: September 19, 2023

    Related Posts

    • Monitor Node.js applications on Red Hat OpenShift with Prometheus

    • Change deployments on the fly in OpenShift 4.8

    • Testing memory-based horizontal pod autoscaling on OpenShift

    • Generate automated Grafana metrics dashboards for MicroProfile apps

    Recent Posts

    • How to run a fraud detection AI model on RHEL CVMs

    • How we use software provenance at Red Hat

    • Alternatives to creating bootc images from scratch

    • How to update OpenStack Services on OpenShift

    • How to integrate vLLM inference into your macOS and iOS apps

    Red Hat Developers logo LinkedIn YouTube Twitter Facebook

    Products

    • Red Hat Enterprise Linux
    • Red Hat OpenShift
    • Red Hat Ansible Automation Platform

    Build

    • Developer Sandbox
    • Developer Tools
    • Interactive Tutorials
    • API Catalog

    Quicklinks

    • Learning Resources
    • E-books
    • Cheat Sheets
    • Blog
    • Events
    • Newsletter

    Communicate

    • About us
    • Contact sales
    • Find a partner
    • Report a website issue
    • Site Status Dashboard
    • Report a security problem

    RED HAT DEVELOPER

    Build here. Go anywhere.

    We serve the builders. The problem solvers who create careers with code.

    Join us if you’re a developer, software engineer, web designer, front-end designer, UX designer, computer scientist, architect, tester, product manager, project manager or team lead.

    Sign me up

    Red Hat legal and privacy links

    • About Red Hat
    • Jobs
    • Events
    • Locations
    • Contact Red Hat
    • Red Hat Blog
    • Inclusion at Red Hat
    • Cool Stuff Store
    • Red Hat Summit

    Red Hat legal and privacy links

    • Privacy statement
    • Terms of use
    • All policies and guidelines
    • Digital accessibility

    Report a website issue