How to monitor Openshift Virtualization VMs with Zabbix

With the Item Prototype, we will create specific queries such as CPU, memory, network, uptime, and phase using the discovery rule's Macros (variables), VM, and namespace.

Now click on Item Prototype within the discovery rule created earlier or Configuration > Templates > Template Name created earlier > Discovery Rule created earlier > Item prototypes > Create Item prototype.

Fill in the following fields:

• Name: VM {#VM} CPU usage
• Type: Select HTTP agent
• Key: kubevirt.vm.cpu[{#NAMESPACE},{#VM}]
• Type of Information: Numeric (float)
• URL: {$PROM_URL}/api/v1/query
• Query fields: Promql to collect CPU usage from VMs
  • Name: query
  • Value: sum by (name,namespace)(rate(kubevirt_vmi_cpu_usage_seconds_total{name="{#VM}",namespace="{#NAMESPACE}"}[5m]))
• Headers: Here we will add our token for authentication to the API using our Token Macro.
  • Name: Authorization
  • Value: Bearer {$PROM_TOKEN}
• Units: cores
• Update interval: 1m, this is how often our item will be collected.

Click on Preprocessing > Add . 

In Name, select Javascript and click on Parameters to add the following script:

var obj = JSON.parse(value);

if (obj.data.result.length === 0) {
  return 0;
}

return obj.data.result[0].value[1];

Click Add to save our preprocessing and click Add again to save our item.

To speed up the creation of new items, let's clone this one by clicking on the created item. At the bottom of the page, click Clone.

In this Item, we will update the following fields:

• Name: VM {#VM} CPU total
• Key: kubevirt.vm.cpu.request[{#NAMESPACE},{#VM}]
• Type of Information: Numeric (unsigned)
• Query fields: Promql to collect CPU Total from VMs
  • Name: query
  • Value: max by (namespace, name) (kubevirt_vm_resource_requests{resource="cpu", name="{#VM}", namespace="{#NAMESPACE}"} )

Click Add to save our preprocessing and click Add again to save our item.

Repeat the cloning process and add the following items.

Memory usage:

• Name: VM {#VM} Memory Usage
• Key: kubevirt.vm.memory[{#NAMESPACE},{#VM}]
• Type of Information: Numeric (unsigned)
• Query fields: Promql to collect Memory Usage from VMs
  • Name: query
  • Value: sum by (name,namespace)(kubevirt_vmi_memory_used_bytes{name="{#VM}",namespace="{#NAMESPACE}"})

• In Preprocessing, select JavaScript and in parameter, add the following script:

  var obj = JSON.parse(value);
  
  if (obj.data.result.length === 0) {
    return 0;
      }
  
  return obj.data.result[0].value[1];

Memory total:

• Name: VM {#VM} Memory total
• Key: kubevirt.vm.mem.request[{#NAMESPACE},{#VM}]
• Type of Information: Numeric (float)
• Query fields: Promql to collect memory total from VMs
  • Name: query
  • Value: max by (namespace, name)(kubevirt_vm_resource_requests{resource="memory", name="{#VM}", namespace="{#NAMESPACE}"})
• In preprocessing, select JSONPath and add  $.data.result[0].value[1] in parameter.

For this use case, we will monitor: CPU, memory, network, and VM status. You must create all the items you deem necessary to monitor, according to the needs of the environment.

With the Trigger Prototype, we will create alerts based on the collected consumption items.

Now click on Trigger Prototype within the Discovery Rule created earlier or Configuration > Templates > Template Namecreated earlier > Discovery Rule created earlier > Trigger prototypes > Create trigger prototype

We will fill in the following fields:

• Name: VM {#NAMESPACE}/{#VM} CPU usage high >= 90%
• Severity: High

• Expression:

  (
  avg(/Template OpenShift Virtualization/kubevirt.vm.cpu[{#NAMESPACE},{#VM}],5m)
  /
  last(/Template OpenShift Virtualization/kubevirt.vm.cpu.request[{#NAMESPACE},{#VM}])
  ) > 0.90
  and 
  last(/Template OpenShift Virtualization/kubevirt.vm.status[{#VM}])=1
  

   
• Description: VM {#NAMESPACE}/{#VM} CPU utilization: {ITEM.LASTVALUE}

Figure 4 illustrates the Zabbix trigger prototype.

With Graph Prototype, we can create specific graphs, such as a network graph.

Click on Graph Prototype within the discovery rule created earlier or Configuration > Templates > Template Name created earlier > Discovery Rule created earlier > Graph prototypes > Create graph prototype.

Fill in the following fields:

• Name: VM {#NAMESPACE}/{#VM} Network throughput
• Items:
  • Add prototype:
    • Select the prototype items created for network RX and TX.

After adding the items, change the Function field to avg, adjust the item colors as desired, and click Add.

Now, to make our monitoring more complete, let's create an LLD of nodes for virtualization. That is, nodes where the VMs run.

In the left side menu, click on Configuration > Templates 

Click on the template we created earlier.

Click on Discovery rules in the top tab. 

Then click on Create discovery rule.

Fill in the following fields as follows:

• Name: OpenShift - Node discovery
• Type: Select HTTP agent
• Key: openshift.node.discovery
• URL: {$PROM_URL}/api/v1/query
• Query fields: This field is responsible for passing our query promql to the endpoint in the URL field.
  • Name: query
  • Value: count by (node) (kube_node_labels{label_kubevirt_io_schedulable="true"})
• Headers: Here we will add our token for authentication to the API using our Token Macro.
  • Name: Authorization
  • Value: Bearer {$PROM_TOKEN}
• Update interval: 1h, this is the frequency at which our VM discovery will be run, to discover new VMs.

Click on Preprocessing > Add.

In Name, select Javascript. 

Click on Parameters and add the following script:

var obj = JSON.parse(value);
var out = [];

for (var i = 0; i < obj.data.result.length; i++) {
  out.push({ "{#NODE}": obj.data.result[i].metric.node });
}

return JSON.stringify({ "data": out });

To validate that our Preprocessing is working correctly, click Test all steps (Figure 6).

Check the Get value from host box and add the Macros values, adding the Thanos endpoint and the bearer token. Then click Get value and test.

Click Add to save our preprocessing and click Add again to save the item.

With the LLD of nodes, we can also create ITEM Prototype such as CPU, memory, network, uptime, and phase using the discovery rule's Macros (variables), NODE.

Now click on Item Prototype within the discovery rule created earlier or Configuration > Templates > Template Name created earlier > Discovery Rule created earlier > Item prototypes > Create Item prototype.

Fill in the following fields:

• Name: Node {#NODE} Ready status
• Type: Select HTTP agent
• Key: node.ready[{#NODE}]
• Type of Information: Numeric (unsigned)
• URL: {$PROM_URL}/api/v1/query
• Query fields: Promql to collect CPU usage from VMs
  • Name: query
  • Value: max by (node) (kube_node_status_condition{ condition="Ready", status="true", node="{#NODE}" })
• Headers: Here we will add our token for authentication to the API using our token macro.
  • Name: Authorization
  • Value: Bearer {$PROM_TOKEN}
• Update interval: 30s, this is how often our item will be collected.

Click on Preprocessing > Add 

In Name, select Javascript.

Click on Parameters and add the following script:

var obj = JSON.parse(value);

if (obj.data.result.length === 0) return 0;

return obj.data.result[0].value[1];

In this LLD, we create items and triggers for the CPU, memory, network, boot time and ready status. Download the XML template if needed by clicking this link.

Now that we have created our host cluster and template with the LLDs and their items, let's check if the data was collected correctly by going to Monitoring > Latest Data.

Figure 7 shows the items collected by LLD KubeVirt - VM discovery.

Figure 8 shows the items collected by LLD OpenShift - Node discovery.

With all our items collected correctly, we can view our dashboard and identify if there are any active alerts (Figure 9).

Monitoring OpenShift Virtualization with Zabbix demonstrates how easily external tools can integrate with the platform's native observability stack. By using Prometheus and Thanos, OpenShift exposes comprehensive metrics that can be consumed by Zabbix through simple HTTP queries.

This enables effective monitoring of virtual machines and infrastructure without additional agents, using features such as LLD for dynamic discovery and automation. Ultimately, this approach combines the flexibility of Zabbix with the power of OpenShift's integrated monitoring, providing a scalable and efficient solution for modern environments.

For more details and other configurations, start with these reference documents:

• Openshift Virtualization - Monitoring
• Openshift 4.18 - Accessing metrics
• KubeVirt.io - Monitoring
• Zabbix 6.0 - Low Level Discovery
• Zabbix 6.0 - Item HTTP Agent
• Zabbix 6.0 - Export and Import Templates