Best Practice Configuration and Tuning for Linux and Windows VMs

In this guide we’ll walk through some critical configuration details and some further “last mile” tuning options when running workloads in both Linux and Windows VMs on OpenShift Virtualization.

As always, you can find more details on all available tuning options in the OpenShift Virtualization - Tuning & Scaling Guide, and keep up with what the Red Hat Performance & Scale Engineering team is up to on our blog.  

VM Definition

OS Preferences

When defining a VM (Virtual Machine) using a provided Template or InstanceType, some critical best practice configurations are automatically applied which optimize performance for the guest OS type. This is especially important for Windows VMs because of some fundamental optimizations related to the guest clock, hyper-v enlightenments, and other bus type preferences.  

If you choose to implement custom VM definitions that don’t follow these provided settings, especially for Windows VMs it is highly recommended to apply the appropriate VirtualMachinePreference to the VM which will configure the necessary tuning. 

To see all provided VM Preferences: 

oc get VirtualMachineClusterPreference

An example of applying a Preference to a VM definition:

spec:
  preference:
    name: windows.2k25.virtio
  runStrategy: [...]

Note: if the VM is already running, a reboot will be required to pick up the configuration changes. 

In order to see all the settings automatically applied by the Preference, you can query the VMI (running VirtualMachineInstance) definition, ex:

 oc get vmi <vm_name> -o yaml

If there is a need to customize these VirtualMachinePreferences, the recommendation would be to create new/modified Preference names so that the default provided settings are always available as a reference. This article provides an example workflow of customizing a VirtualMachinePreference. 

Disk

In general bus: virtio is the best performing option, Windows VMs should not configure bus: sata for best performance.

For multi-threaded storage workloads, especially for larger VMs with many vCPUs, consider enabling the supplementalPool ioThreadsPolicy (starting in 4.19).

By default, all Block volumes will apply an io: native optimization. In order to utilize this optimization for Filesystem volumes, use preallocation.

Network

Use model: virtio, Windows VMs should not configure model: e1000e for best performance. 

In most scenarios, to achieve maximum network throughput it is recommended to include networkInterfaceMultiqueue: true. 

Host Tuning

Tuned Profile

By default OpenShift nodes apply a “throughput performance” based Tuned profile, which will perform well for a wide variety of cases. Other Tuned profiles are available for specific workload scenarios and can be managed using the default Node Tuning Operator.

CPU Sleep States

If your workload is sensitive to CPU wakeup latency, or in some cases if your hardware does not support optimized sleep state transitions, it can be useful to limit how “deep” CPUs are allowed to sleep. This tuning can be applied using the default Node Tuning Operator and does not require any reboots, for instance in order to limit CPUs to the “C1” state, apply this change: 

apiVersion: tuned.openshift.io/v1
kind: Tuned
metadata:
  name: c1-lowlatency
  namespace: openshift-cluster-node-tuning-operator
spec:
  profile:
  - data: |
      [main]
      summary=Pins to C1 cstate for low latency

      include=openshift-node
      [cpu]
      force_latency=1
    name: c1-lowlatency
  recommend:
  - machineConfigLabels:
      machineconfiguration.openshift.io/role: "worker"
    priority: 20
    profile: c1-lowlatency

CPU Allocation Ratio

The default allowed CPU overcommit ratio is 10:1, this ratio can be configured to the desired overcommit level by changing the cluster CPU Allocation Ratio, or CPU resource requests can be configured per-VM if desired.

High Performance Workload Tuning

Hugepages

By default RHEL host kernels use THP (Transparent HugePages) which will try to auto promote page sizes when possible, however using statically reserved 1GB hugepages may improve performance for applications that are sensitive to having pages “locked down” or hardware TLB (Translation Lookaside Buffer) misses. See the documentation for more information on backing VM memory with hugepages. 

Isolation and Pinning

In most cases, the above tuning will be enough for applications to perform well, but in some cases workloads are very sensitive to any scheduler disruptions or require very low latency configurations. Further pinning and isolation of the VM may be required in these scenarios, see the Pinning sections in the Tuning Guide for more details.