Step-by-step guide to configuring alerts in Cluster Observability Operator

December 16, 2024
Christina Zhang
Related topics:
ObservabilityOperators
Related products:
Red Hat OpenShiftRed Hat OpenShift Container Platform

The first installment in this series provided a brief introduction to the Cluster Observability Operator (COO). This article serves as a detailed guide for configuring alerts within the COO namespace. In the following sections, I will present specific examples for two structural configurations within the COO monitoring stack.

Case 1: One monitoring stack for multiple namespace. 

Create a namespace to host the COO instance and utilize this monitoring stack to monitor the namespaces coo-ns2, coo-ns3, and coo-ns4 as shown in Figures 1 and 2.

COO Monitoring Stack structure.
Figure 1: COO Monitoring Stack structure.
One Monitoring Stack for multiple namespace structure.
Figure 2: One Monitoring Stack for multiple namespace structure.

Create a project to host the COO monitoring stack. Give a label for the namespace monitoring.rhobs: coo-stack:

apiVersion: v1
kind: Namespace
metadata:
  name: coo-namespace
  labels:
    monitoring.rhobs:coo-stack

Create a COO monitoring stack, ensuring that the namespaceSelector matches the project label monitoring.rhobs:coo-stack. The resourceSelector should be set to k8s-app: prometheus-coo-example-monitor. This label should match the ServiceMonitor label. See below:

apiVersion: monitoring.rhobs/v1alpha1
kind: MonitoringStack
metadata:
  name: my-coo-monitoring-stack
  namespace: coo-namespace
spec:
  alertmanagerConfig:
    disabled: false
  logLevel: debug
  namespaceSelector:
    matchLabels:
      monitoring.rhobs: coo-stack
  prometheusConfig:
    replicas: 2
  resourceSelector:
    matchLabels:
      k8s-app: prometheus-coo-example-monitor
  resources:
    limits:
      cpu: 500m
      memory: 512Mi
    requests:
      cpu: 100m
      memory: 256Mi
  retention: 16d

Set the label app: prometheus-coo-example for the deployment/pod you wish to monitor:

apiVersion: monitoring.rhobs/v1
kind: ServiceMonitor
metadata:
 labels:
   k8s-app: prometheus-coo-example-monitor
 name: coo-servicemonitor-forcoonamespace
 namespace: coo-namespace
spec:
 endpoints:
 - interval: 30s
   port: web
   scheme: http
 selector:
   matchLabels:
     app: prometheus-coo-example

Check the info below namespace coo-namespace, shown in Figure 3.

Info below coo-namespace.
Figure 3: Info below coo-namespace.

Create the namespace coo-ns2, also with label monitoring.rhobs: coo-stack:

apiVersion: v1
kind: Namespace
metadata:
  name: coo-ns2
  labels:
    monitoring.rhobs: coo-stack

Run a Deployment:

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: prometheus-coo-example
  name: coo-example-app2
  namespace: coo-ns2
spec:
  replicas: 1
  selector:
    matchLabels:
      app: prometheus-coo-example
  template:
    metadata:
      labels:
        app: prometheus-coo-example
    spec:
      containers:
      - image: ghcr.io/rhobs/prometheus-example-app:0.4.2
        imagePullPolicy: IfNotPresent
        name: coo-example-app2

Create a Service:

apiVersion: v1
kind: Service
metadata:
  labels:
   app: prometheus-coo-example
  name: coo-example-service-app2 # alter 条件是这个
  namespace: coo-ns2
spec:
  ports:
  - port: 8080
    protocol: TCP
    targetPort: 8080
    name: web
  selector:
    app: prometheus-coo-example
  type: ClusterIP

Also create a Service Monitor:

apiVersion: monitoring.rhobs/v1
kind: ServiceMonitor
metadata:
  labels:
    k8s-app: prometheus-coo-example-monitor
  name: prometheus-coo-example-servicemonitor2
  namespace: coo-ns2
spec:
  endpoints:
  - interval: 30s
    port: web
    scheme: http
  selector:
    matchLabels:
      app: prometheus-coo-example

Create a PrometheusRule and place the label k8s-app: prometheus-coo-example-monitor:

apiVersion: monitoring.rhobs/v1
kind: PrometheusRule
metadata:
  name: coo-alert2
  namespace: coo-ns2
  labels:
    k8s-app: prometheus-coo-example-monitor
spec:
  groups:
  - name: example
    rules:
    - alert: VersionAlert
      for: 1m
      expr: version{job="coo-example-service-app2"} > 0 
      labels:
        severity: warning

Use the same method to create coo-ns3 and coo-ns4.

To verify it the alert has been configured correctly, try oc get all. Find the svc. In my case, mine is my-coo-monitoring-stack-prometheus. See Figure 4.

Locate my-coo-monitoring-stack-prometheus.
Figure 4: Locate my-coo-monitoring-stack-prometheus.
oc port-forward svc/my-coo-monitoring-stack-prometheus 9090:9090 --address 0.0.0.0 -n coo-namespace

Try:

localhost:9090

You will see a Prometheus UI Page. Click Alert and you should find the alert there. See Figure 5.

Prometheus Alerts page.
Figure 5: Prometheus Alerts page.

Next, let's move on and examine case 2. The structure in this example is shown in Figures 6 and 7.

Monitoring Stack for single project structure.
Figure 6: Monitoring Stack for single project structure.
A COO Monitoring Stack project structure.
Figure 7: A COO Monitoring Stack project structure.

Case 2: Deploy the monitoring stack at the namespace you wish to monitor

In this instance, deploy the monitoring stack below namespace coo-ns1.

If you already had the namespace, assign a label for the namespace. If not, create a namespace, then make deployment and configure alerts for the namespace. Give a label for the namespace monitoring.rhobs: coo-stack:

apiVersion: v1
kind: Namespace
metadata:
  name: coo-ns1
  labels:
    monitoring.rhobs: coo-stack

Next, create a monitoringStack. Make sure the namespaceSelector is monitoring.rhobs: coo-stack, the same as the namespace label. Give a label k8s-app: prometheus-coo-example-monitor to resourceSelector:

apiVersion: monitoring.rhobs/v1alpha1
kind: MonitoringStack
metadata:
  name: my-coo-monitoring-stack
  namespace: coo-ns1
spec:
  alertmanagerConfig:
    disabled: false
  logLevel: debug
  namespaceSelector:
    matchLabels:
      monitoring.rhobs: coo-stack
  prometheusConfig:
    replicas: 2
  resourceSelector:
    matchLabels:
      k8s-app: prometheus-coo-example-monitor
  resources:
    limits:
      cpu: 500m
      memory: 512Mi
    requests:
      cpu: 100m
      memory: 256Mi
  retention: 16d

Create a ServiceMonitor. Give the label k8s-app: prometheus-coo-example-monitor. Also give a label app: prometheus-coo-example used for the deployment, service, and pod:

apiVersion: monitoring.rhobs/v1
kind: ServiceMonitor
metadata:
 labels:
   k8s-app: prometheus-coo-example-monitor
 name: coo-servicemonitor-forcoonamespace
 namespace: coo-ns1
spec:
 endpoints:
 - interval: 30s
   port: web
   scheme: http
 selector:
   matchLabels:
     app: prometheus-coo-example

Create a Service. Use the same label app: prometheus-coo-example:

apiVersion: v1
kind: Service
metadata:
  labels:
   app: prometheus-coo-example
  name: coo-example-service-app1 
  namespace: coo-ns1
spec:
  ports:
  - port: 8080
    protocol: TCP
    targetPort: 8080
    name: web
  selector:
    app: prometheus-coo-example
  type: ClusterIP

Create a Deployment. Use the same label  app: prometheus-coo-example:

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: prometheus-coo-example
  name: coo-example-app1
  namespace: coo-ns1
spec:
  replicas: 1
  selector:
    matchLabels:
      app: prometheus-coo-example
  template:
    metadata:
      labels:
        app: prometheus-coo-example
    spec:
      containers:
      - image: ghcr.io/rhobs/prometheus-example-app:0.4.2
        imagePullPolicy: IfNotPresent
        name: coo-example-app1

Create a PrometheusRule. It is to configure the alert you wish to fire once you trigger the condition. Use the label k8s-app: prometheus-coo-example-monitor  from the previous configuration:

apiVersion: monitoring.rhobs/v1
kind: PrometheusRule
metadata:
  name: coo-alert1
  namespace: coo-ns1
  labels:
    k8s-app: prometheus-coo-example-monitor
spec:
  groups:
  - name: example
    rules:
    - alert: VersionAlert
      for: 1m
      expr: version{job="coo-example-service-app1"} > 0 
      labels:
        severity: warning

To verify it the alert has been configured correctly, try oc get all. Find the svc. In my case, mine is my-coo-monitoring-stack-prometheus:

kubectl port-forward svc/my-coo-monitoring-stack-prometheus 9090:9090 --address 0.0.0.0 -n coo-ns1

Try:

localhost:9090

You will see a Prometheus UI Page. Click Alert and you should find the alert there. 

This guide provides you with a step-by-step understanding of the procedure. You can modify the the configuration to satisfy your business requirements.