Monitor OpenShift Virtualization using user-defined projects and Grafana

August 19, 2024
Leonardo Araujo
Related topics:
PerformanceVirtualization
Related products:
Red Hat OpenShiftRed Hat OpenShift Container PlatformRed Hat OpenShift Virtualization

This article will teach you how to install and configure the Grafana Operator and integrate it with the openshift-monitoring stack using user-workload-monitoring. At the end of the article, we will import a dashboard with general metrics about Red Hat OpenShift Virtualization, which will give us an overview of the amount of resources in use, virtual machines (VMs) created, and their consumption.

In this article we use the following versions:

Prerequisites

  • User with the cluster-admin cluster role.
  • Grafana Operator.
  • User-Defined Projects enabled.

Enable user-defined projects

Execute this command to add enableUserWorkload: true under data/config.yaml:

oc -n openshift-monitoring patch configmap cluster-monitoring-config -p '{"data":{"config.yaml":"enableUserWorkload: true"}}'

Validate that the prometheus and thanos-ruler pods were created in the openshift-user-workload-monitoring project:

$ oc get pods -n openshift-user-workload-monitoring
NAME                                   READY   STATUS    RESTARTS   AGE
prometheus-operator-675f9d4b96-f9zxd   2/2     Running   0          8d
prometheus-user-workload-0             6/6     Running   0          8d
prometheus-user-workload-1             6/6     Running   0          8d
thanos-ruler-user-workload-0           4/4     Running   0          8d
thanos-ruler-user-workload-1           4/4     Running   0          8d

Install Grafana Operator

  • Using the WebConsole, in the left side menu, select OperatorHub and then in the search field, search for Grafana Operator.
  • Make sure to change the project context to openshift-user-workload-monitoring at the top.
  • Click the operator, then click Install.
  • In Update Channel, select v5.
  • In Installation Mode, select A specific namespace on the cluster and choose openshift-user-workload-monitoring below.
  • In Update approval, select Automatic
  • Click Install.

Now let's create a service account and assign permission to read metrics:

oc project openshift-user-workload-monitoring
oc create sa grafana-sa
oc adm policy add-cluster-role-to-user cluster-monitoring-view -z grafana-sa
 
Let's collect the grafana-sa service account token and create a secret for our Grafana instance:
SECRET=`oc -n openshift-user-workload-monitoring describe sa grafana-sa | awk '/Tokens/{ print $2 }'`

TOKEN=`oc -n openshift-user-workload-monitoring get secret $SECRET --template='{{ .data.token | base64decode }}'`

cat <<EOF > grafana-secret-creds.yaml
kind: Secret
apiVersion: v1
metadata:
  name: credentials
  namespace: openshift-user-workload-monitoring
stringData:
  GF_SECURITY_ADMIN_PASSWORD: grafana   <------ Set the password you want to authenticate with Grafana
  GF_SECURITY_ADMIN_USER: root          <------ Set the desired user to authenticate in Grafana
  PROMETHEUS_TOKEN: '${TOKEN}'          <------ This variable will receive the token collected above
type: Opaque
EOF

oc create -f grafana-secret-creds.yaml

Now let's create our Grafana instance and it will read the credentials defined in the previously created secret:

cat <<EOF > grafana-instance.yaml
apiVersion: grafana.integreatly.org/v1beta1
kind: Grafana
metadata:
  name: grafana
  labels:
    dashboards: "grafana"
    folders: "grafana"
spec:
  deployment:
    spec:
      template:
        spec:
          containers:
            - name: grafana
              env:
                - name: GF_SECURITY_ADMIN_USER
                  valueFrom:
                    secretKeyRef:
                      key: GF_SECURITY_ADMIN_USER
                      name: credentials
                - name: GF_SECURITY_ADMIN_PASSWORD
                  valueFrom:
                    secretKeyRef:
                      key: GF_SECURITY_ADMIN_PASSWORD
                      name: credentials
  config:
    auth:
      disable_login_form: "false"
      disable_signout_menu: "true"
    auth.anonymous:
      enabled: "false"
    log:
      level: warn
      mode: console
EOF
Let's apply and validate our created instance:
oc -n openshift-user-workload-monitoring create -f grafana-instance.yaml

oc -n openshift-user-workload-monitoring get pods -l app=grafana
NAME                                READY   STATUS    RESTARTS   AGE
grafana-deployment-dd8d8d5d-vpnjh   1/1     Running   0          8d
Now let's expose our Grafana service using an edge type route, we will use the service called grafana-service:
oc get svc | awk '/grafana/'
grafana-service                           ClusterIP   172.30.77.139    <none>        3000/TCP                      8d


oc -n openshift-user-workload-monitoring create route edge grafana --service=grafana-service --insecure-policy=Redirect
Displaying the route exposed to Grafana:
oc -n openshift-user-workload-monitoring get route grafana -o jsonpath='{.spec.host}'
Let's create our Grafana Datasource, which will connect to thanos-querier in the openshift-monitoring project and will use the grafana-sa service account token that is stored in secret credentials:
cat <<EOF > grafana-datasource.yaml
apiVersion: grafana.integreatly.org/v1beta1
kind: GrafanaDatasource
metadata:
  name: grafana-ds
  namespace: openshift-user-workload-monitoring  
spec:
  valuesFrom:
    - targetPath: "secureJsonData.httpHeaderValue1"
      valueFrom:
        secretKeyRef:
          name: "credentials"
          key: "PROMETHEUS_TOKEN"
  instanceSelector:
    matchLabels:
      dashboards: "grafana"
  datasource:
    name: Prometheus
    type: prometheus
    access: proxy
    url: https://thanos-querier.openshift-monitoring.svc:9091
    isDefault: true
    jsonData:
      "tlsSkipVerify": true
      "timeInterval": "5s"
      httpHeaderName1: 'Authorization'
    secureJsonData:
      "httpHeaderValue1": "Bearer \${PROMETHEUS_TOKEN}"
    editable: true
EOF
Let's apply and validate our created Datasource:
oc -n openshift-user-workload-monitoring create -f grafana-datasource.yaml

oc -n openshift-user-workload-monitoring get GrafanaDatasource  
NAME         NO MATCHING INSTANCES   LAST RESYNC   AGE
grafana-ds                           56s           8d

Creating Grafana dashboard

Now let's create a Grafana dashboard, which will fetch the JSON externally from Github:

cat <<EOF > grafana-dashboard-ocp-v.yaml
apiVersion: grafana.integreatly.org/v1beta1
kind: GrafanaDashboard
metadata:
  name: grafana-dashboard-ocp-v
  labels:
    app: grafana
spec:
  instanceSelector:
    matchLabels:
      dashboards: grafana  
  folder: "Openshift Virtualization"      
  url: https://raw.githubusercontent.com/leoaaraujo/articles/master/openshift-virtualization-monitoring/files/ocp-v-dashboard.json
EOF
Let's apply and validate our created Grafana dashboard:
oc -n openshift-user-workload-monitoring create -f grafana-dashboard-ocp-v.yaml


oc -n openshift-user-workload-monitoring get grafanadashboard
NAME                      NO MATCHING INSTANCES   LAST RESYNC   AGE
grafana-dashboard-ocp-v                                         3m54s

Viewing the dashboard

  • Access Grafana, and in the left side menu, click Dashboards and then Browse.
  • A folder with the name OpenShift Virtualization and the dashboard OCP Virt—Metrics will be displayed, click dashboard.
  • Dashboard (see Figure 1 and Figure 2).
OpenShift Virtualization Dashboard displaying General Information
Figure 1: The OpenShift Virtualization dashboard displaying various metrics.

 

OpenShift Virtualization Dashboard showing Resources Information
Figure 2: The OpenShift Virtualization dashboard displaying resources information.

In this dashboard, you can view the following metrics:

  • Number of VMs running
  • Number of Allocable Nodes
  • Number of VMs by status
  • Number of VMs per OS
  • Number of VMs per Resource Flavor
  • List of Top CPU Consumers
  • List of Top Memory Consumers
  • List of Top IOps Consumers
  • List of Top Storage Traffic Consumers
  • List of Top Network Consumers
  • Read request times in the Kubevirt API
  • Write request time in the Kubevirt API
  • VM creation time

Conclusion

In this article, we demonstrate how it is possible to monitor your OpenShift Virtualization in a simple way, using the openshift-monitoring and user-workload-monitoring stacks, creating and customizing your own dashboard and having a more managerial view of your environment.

References

Last updated: September 13, 2024