Over the past few years, achieving high-density virtualization has required broad re-engineering across the technology stack. To analyze this transformation, this article provides a comprehensive overview of the recent — and not so recent — memory management evolutions at both the node and cluster levels in Red Hat OpenShift Virtualization. These optimizations efforts span from the lower echelons of the Linux kernel up to the orchestration and management planes, encompassing core Kubernetes primitives, Red Hat OpenShift components, and specialized operators. By integrating these disparate layers, the platform transitions from static to a more dynamic, utilization-based memory management model.
This evolution allows OpenShift to use existing memory capacity more efficiently, and to introduce additional memory tiers by providing fast secondary storage (such as enterprise NVMe) to further drive up overall infrastructure utilization.
By safely increasing the workload density per physical node, your organization can effectively manage its hardware footprint, helping to stabilize infrastructure budgets and reduce long-term compute costs.
While the following sections focus on virtual machines, many of these optimizations (unless specifically tied to virtualization mechanics) apply equally to standard containers, highlighting the distinct benefits of managing both on one unified platform.
Scheduler memory handling and reservations
Memory management within OpenShift historically relies on a straightforward accounting model. A workload requests a specific amount of memory, and the scheduler identifies the best node by matching this memory request to all nodes memory capacity. The accounting is done in memory, and is not visible on the OpenShift API. Rather than rigidly dedicating or locking physical RAM for a specific VM, these requests function as a foundational ledger system, allowing the cluster to maintain resource visibility and predictability on workload placement across infrastructure. The node allocatable capacity is calculated using a standard formula:
node allocatable = node capacity - system reserved - kubelet reserved
Given this, the following relationship is always true for a given node:
sum(pod memory requests) node allocatablenode capacity
When running a virtual machines (VM) in OpenShift Virtualization, the atomic compute unit remains a pod. Specifically, VMs execute inside a dedicated pod, which is dynamically managed by the KubeVirt control plane. Because the hypervisor (QEMU/KVM) requires memory for its own runtime, caches, and device emulation, the overall memory request for the VM's pod is inherently larger than the raw memory allocated to the guest OS through the memory.guest specification.
For more information about scheduling fundamentals, see the Kubernetes resource scheduling documentation. For a detailed walkthrough of how the scheduler is handling memory, see the Red Hat Developer blog post on memory management in OpenShift Virtualization.
For details on configuring memory settings, refer to the OpenShift Virtualization post-installation documentation.
Default memory accounting and overcommit policies
Historically, OpenShift Virtualization is by default conservative and has memory overcommit set to 0 by default. As a result, 100% of a VM's assigned RAM is requested for VM — a 1:1 mapping between the allocation and request — to help ensure safe placement and avoid resource contention.
However, the administrator has the flexibility to reconfigure these thresholds, overriding the default behavior by adjusting the hard memory overcommit limit when maximizing density is a priority. While the 1:1 request approach provides stability, it can introduce inefficiencies at scale. This is because VMs are built with sufficient memory to cover the worst case scenario of the application inside the VM consuming almost all or all memory. However, this is just the worst case, and worst cases are exceptions. Aside from exceptions, most VMs run with a lot of unused memory. Operational data supports this, indicating that many virtualized workloads actively utilize only about 20% of allocated virtual RAM. This delta between requested capacity and actual runtime utilization can lead physical infrastructure to scale linearly with workload definitions rather than actual compute demand. Configuring a use-case specific memory overcommit hard limit is necessary in order to reduce this delta and use available resources more efficiently.
To explore real-world evaluations of overcommitment, refer to evaluating memory overcommitment in OpenShift Virtualization and boosting database VM density with memory overcommit.
Node-level efficiency: Memory saving and tiering
The technical foundation for higher virtual machine density lies at the node level, requiring the hypervisor to distinguish between active (hot) memory that should remain in physical RAM and inactive (cold) memory that might be safely evicted, deduplicated, or returned.
Guest-cooperative memory reclamation
OpenShift Virtualization attempts to reduce the physical footprint of the VM by reclaiming memory that the guest operating system considers free. This relies primarily on cooperative mechanisms.
The primary mechanism is free page reporting (FPR), which is enabled by default. FPR utilizes a driver from the virtio family integrated into the guest operating system. When a virtual machine is running, the hypervisor has already allocated physical host memory to back the guest's pages. Over time, however, the guest OS releases some of those pages as processes exit, caches are evicted, buffers are freed. Although the hypervisor originally allocated those pages on the guest's behalf, it has no inherent way of knowing that the guest is no longer using them. FPR closes this visibility gap. The guest kernel periodically scans its own free-memory lists and reports unused pages back to the hypervisor. Once notified, the hypervisor can decommission those physical pages and return them to the host's general allocation pool, making them available for other workloads or virtual machines.
This is in contrast to traditional memory ballooning, which actively inflates a balloon device inside the guest to force-allocate pages — effectively tricking the guest into surrendering memory. Ballooning can cause performance degradation during sudden spikes in memory consumption, as the guest is forced to rapidly allocate pages to the balloon, leading to guest swap usage, latency spikes. It also conflicts with guest-level memory compression subsystems. FPR avoids these problems by operating asynchronously, observing which pages the guest has already freed on its own rather than forcing allocation to discover them.
To learn more about how free page reporting works in detail, read saving memory in OpenShift Virtualization using free page reporting.
Guest-agnostic memory reclamation
To handle scenarios where the guest OS does not support FPR, OpenShift utilizes a feature called kernel samepage merging (KSM) zero pages. Unlike legacy KSM, which scans for identical memory pages across the system, KSM zero pages evaluates pages filled with zeros. This restricted scope helps address security concerns by preventing the deduplication of arbitrary application data across different virtual machines. Because it focuses on zeroed-out pages (which are common in Windows workloads), this approach tends to be faster and less CPU-intensive than traditional KSM, which tries to find duplicates across all pages. It operates independently of the guest OS as a hypervisor-side fallback, deduplicating these empty pages into a single physical reference until they are needed again.
Hugepages exception for high-performance workloads
While mechanisms like FPR efficiently return unused memory to the host, they are bypassed when a virtual machine is configured to use hugepages. Hugepage-backed memory is pre-allocated and pinned to the VM to reduce address translation overhead and support memory performance. Because this memory generally needs to remain dedicated to the guest workload, the hypervisor disables FPR, KSM zero pages, and other reclamation heuristics for these instances, aiming to ensure that high-performance VMs experience fewer latency penalties from dynamic de-allocation or page faults.
Tiers of memory
Even after minimizing the physical memory footprint of workloads using FPR and KSM zero pages, a host node must handle situations where the collective active memory of its workloads surpasses physical RAM limits. Because an out-of-memory (OOM) event on a node results in workload termination, physical RAM can be expanded with additional tiered memory. These tiers generally differ in technology and access speeds, with higher tiers exhibiting higher latency. Moving memory to supplementary tiers is part of memory reclamation. These mechanics are initiated when a node's kernel searches for available memory, operating both asynchronously in the background based on node-level configurations and synchronously in response to application memory requests.
Memory management across tiers of memory
Since OpenShift Virtualization 4.18, the platform leverages the multi-generational least-recently-used (MGLRU) algorithm as the default memory reclaim mechanism in order to decide what memory pages should be transferred to the next memory tier complementing physical RAM.
MGLRU significantly restructures how the Linux kernel identifies cold memory, replacing legacy heuristics that could struggle under dense virtualization workloads. Classic LRU implementations relied on a simple bifurcated structure consisting of an active and an inactive list. This design was prone to "scan cost" (the CPU overhead required to walk the inactive list to find reclaimable pages) and "one-time access pollution", where a bulk sequential read operation (such as a backup or tar execution) might flood the active list, pushing genuinely active application memory out to swap.
Stage: Youngest (hot)
- Classic LRU: Active list (prone to pollution)
- MGLRU model: Generation 0 (protected with Bloom)
Stage: Aging
- Classic LRU: Inactive list (high CPU scan cost)
- MGLRU model: Generation 1
Stage: Oldest (cold)
- Classic LRU: Evicted from inactive list
- MGLRU model: Generation 2 / 3 (target for evict)
MGLRU simplifies this by grouping pages into distinct generations based on access recency. The kernel tracks accessed pages to keep active memory in younger generations, while untouched pages naturally age into older generations, making them prime targets for eviction. This provides 3 main architectural advantages for dense virtualization:
- Per-VM memory tracking: OpenShift isolates each VM within its own Linux control group. MGLRU gives each of these isolated environments its own distinct set of generations (lruvec), allowing the kernel to reclaim memory from a specific, overloaded VM without needlessly scanning the broader host system.
- Thrashing prevention: To help prevent performance degradation from rapidly swapping the same pages in and out, MGLRU uses Bloom filters to fingerprint evicted pages. If an evicted page is quickly requested again, it is promoted to a protected generation to shield the active working set.
- Lower CPU overhead: By evicting the older pre-sorted generation rather than constantly scanning a massive list, MGLRU can cut the background memory reclaim (
kswapd) CPU usage significantly.
For more information, see the Linux kernel documentation for MGLRU.
Unlocking memory overcommitment through node-level optimizations
Given the suite of node-level memory optimizations discussed above, it is often no longer necessary to rigidly enforce a 1:1 request mapping between requested memory and physical capacity.
Instead, OpenShift can reduce the aggregate memory request mapping based on the probability that not all virtual machines will demand their peak memory allocations simultaneously. Sharing a fixed capacity based on probabilistic demand is referred to as statistical multiplexing. For example, if 10 virtual machines each request 8 GiB of memory, a traditional reservation model requires 80 GiB of physical RAM on the host. However, if operational data indicates that these VMs typically consume only 2 GiB during standard operation, statistical multiplexing permits the host to support the entire group with less physical RAM, such as 32 GiB. The architecture relies on the likelihood that all 10 workloads will not spike to their maximum 8 GiB requests at the exact same moment.
Because this density model relies on statistical averages, edge cases can occur. If multiple workloads on a single host experience simultaneous memory spikes, the statistical model temporarily breaks down. In these scenarios, the ability to push memory to a secondary memory tier and swap pages provides a necessary safety net, serving as a local buffer to absorb the shock without immediately triggering emergency eviction thresholds. Yet, relying solely on node-local storage tiering and emergency evictions may not be a sustainable long-term strategy. To mitigate the risk of individual node overload and help ensure consistent performance, continuous cluster-level monitoring and proactive balancing are utilized to redistribute active memory demands across the broader infrastructure, neutralizing local pressure points.
Emergency node protection and eviction thresholds
Even with memory saving and tiering techniques, physical nodes could still face memory saturation during emergency situations. When a node's memory utilization spikes, memory reclaim attempts to shed cold pages and smooth out the transient demand. However, if this reclaim mechanism is insufficient to stabilize the host and active memory reaches OpenShift's configured node-pressure soft or hard eviction thresholds, the system enters a protective state. To shield the node from resource exhaustion, the kubelet evicts (shuts down) running VMs.
sum(hot memory pages)ThresholdNodePressureEviction ProtectNode()
This is a protection mechanism and the very last layer of defense in order to protect a node, appropriate system sizing and monitoring is preventing this from ever happening. For further details on eviction mechanics, refer to the Kubernetes documentation for node-pressure eviction.
Cluster-level architecture: Cluster-wide orchestration and balancing
While node-level optimizations like MGLRU, FPR, and KSM zero pages delay the onset of severe memory pressure, pushing for density across a cluster typically requires global orchestration. If the aggregate hot memory (the active working set) of all virtual machines on a single node exceeds the physical RAM limits, thrashing between media tiers will likely degrade performance. Therefore, the cluster architecture coordinates node-level thresholds with proactive migration strategies.
Managing the implications of reduced memory requests
By embracing statistical multiplexing and node-level overcommit, memory requests submitted to the scheduler can be lowered, meaning requests can be set to less than 100% of a virtual machine's configured guest memory. Lowering this request parameter instructs the scheduler to pack a higher number of workloads onto a single node, driving up overall cluster density.
However, lowering memory requests is memory overcommitment and introduces an operational consideration: The node's allocatable ledger no longer reflects the potential peak memory demand of its resident VMs. To safely operate in this reduced-request (memory overcommitted) model, the cluster requires a mechanism to monitor actual utilization, and proactively disperse workloads in order to avoid triggering an emergency node-level eviction.
Proactive balancing with descheduler
Building upon this need for proactive orchestration, the descheduler in OpenShift Virtualization changed its default and now rebalances workloads according to utilization and workload pressure (impact) in order to account for actual dynamic demand. The OpenShift descheduler has to be configured with the KubeVirtRelieveAndMigrate strategy in order to evaluate nodes using real-time active memory and CPU utilization and pressure metrics.
Utilization and pressure
Under normal conditions, the descheduler balances workloads based on standard CPU and memory utilization metrics to maintain an even distribution across the infrastructure. However, raw utilization can sometimes be an unreliable metric in dense environments. For example, high memory usage might simply indicate healthy page caching rather than genuine workload starvation. Because of this, when cluster-wide utilization reaches high saturation points (such as exceeding 80% capacity), the descheduler shifts its decision-making to prioritize pressure stall information (PSI) metrics. PSI measures the time workloads spend actively waiting for a memory operation. Waiting for memory can occur in saturated systems when the hypervisor is for example busy dropping caches, compacting memory, or transferring memory in or out of a secondary memory tier. By pivoting to PSI during peak loads, the descheduler can migrate workloads away from nodes where applications are experiencing performance degradation due to resource contention.
In other words, if every node is saturated, then the descheduler optimizes for the least workload impact (pressure).
When a node's pressure deviates significantly from the cluster average, the descheduler intervenes. Unlike standard OpenShift pod node-pressure eviction, which terminates the workload, KubeVirtRelieveAndMigrate interfaces with KubeVirt's live-migration controller. This integration allows the platform to seamlessly move running virtual machines from hot nodes to underutilized nodes without losing guest state or network connectivity. To help maintain cluster stability, the system aims to prevent "migration churn" — a cycle of repeatedly moving VMs back and forth across the infrastructure. The descheduler discourages this loop by enforcing asymmetric deviation thresholds (such as AsymmetricLow). These thresholds ensure the balancing logic focuses primarily on relieving nodes that are genuinely over-utilized. Concurrently, to protect nodes that are already burdened, the descheduler dynamically applies a soft taint (PreferNoSchedule) to them. This taint acts as a hint to the default OpenShift scheduler, steering newly created VMs, or VMs currently in migration, away from saturated environments. By combining PSI-aware metrics, live migration, and workload steering, the descheduler transforms cluster balancing into a proactive orchestration engine.
For more information on pressure stall information, refer to Understand Pressure Stall Information (PSI) Metrics. For further details on how the descheduler balances according to PSI data, see Load aware rebalancing with OpenShift Virtualization.
For benchmarking results on rebalancing performance, see Dynamic VM CPU Workload Rebalancing with Load Aware Descheduler | Red Hat Developer
Virtualization platform autopilot
Historically, navigating the boundaries of memory overcommitment required administrators to manually tune kernel reclaim parameters, calculate descheduler thresholds, install and configure OLM operators, and monitor live-migration budgets. Because orchestrating this across the whole stack is complex, tedious, and prone to error, the virtualization platform autopilot was introduced to help address this operational burden.
The autopilot is primarily a configuration tool. It doesn't actively manage or proactively migrate workloads. Instead, it tunes and configures a system to adjust the hard memory overcommit limit by default. By calculating and setting the optimal node-level MGLRU parameters, control group throttling limits, and cluster-level descheduler thresholds, it provides a foundation for the platform's independent components to handle transient memory spikes and execute cluster-wide migrations as needed.
For an overview of how the autopilot simplifies cluster and operator configuration, read the Introducing Virtualization Platform Autopilot.
Hardware evolution and further optimizations
As of the current design baseline (incorporating updates through August 2026), OpenShift Virtualization continues to shift from conservative resource guarantees to utilization-based orchestration. With the maturation of MGLRU, FPR, and the autopilot, future OpenShift versions transition away from rigid 1:1 memory reservations, defaulting to node-level overcommitment ratios of 20%. This default ratio can be configured higher or lower as needed by your specific use-case. This is paired with enhanced observability dashboards that allow operators to track memory utilization, enabling granular tuning of density policies.
Exploring CXL and hardware-tiered memory
Looking forward, the architectural roadmap targets the physical constraints of local node memory. While MGLRU provides software-defined tiering to secondary storage using hardware-agnostic NVMe or SSD drives, paging through the Linux kernel I/O stack introduces latency that can bottleneck high-throughput databases and real-time analytics. To resolve this, future iterations of OpenShift Virtualization will explore the integration of compute express link (CXL) technologies.
CXL is an open standard, cache-coherent interconnect built over the PCIe interface. Specifically, the CXL.mem protocol enables the attachment of disaggregated expansion modules that provide CPU byte-addressable memory. Instead of the kernel paging cold memory through the I/O stack to a storage drive, CXL allows the CPU to interact with expansion memory directly through standard load and store instructions, largely bypassing the kernel I/O stack and minimizing the latency cliffs of traditional swap operations.
Unlike the current storage-agnostic tiering flows, this transition requires specialized CXL-compatible hardware, but it facilitates a faster, hardware-driven tiered memory architecture. By disaggregating memory from the physical compute socket, CXL may enable OpenShift workloads to transparently access footprints vastly larger than local DRAM limits without application refactoring, while software heuristics like MGLRU continue to orchestrate the tiering logic.
Cloud-native closure
While the underlying KVM ecosystem has supported memory overcommitment for well over a decade, translating these node-level optimizations into a cloud-native architecture required significant structural adaptation.
At the node level, foundational hypervisor mechanics were refined. Legacy memory ballooning has been superseded by free page reporting. KSM zero pages mitigates performance and security risks to enable default deduplication. MGLRU, combined with high-performance secondary NVMe storage, makes page swapping a viable operational strategy. However, realizing these benefits at scale required evolving the Kubernetes control plane. By expanding Kubernetes resource accounting models and enhancing the descheduler with dynamic, PSI-aware monitoring, OpenShift Virtualization drove this change, and now closes the loop between hypervisor-level memory efficiency and cluster-wide orchestration.
For further details on Kubernetes swap management, see the Kubernetes documentation on swap memory management.
For more details about Kubernetes swap watermarks, refer to Handling Kubernetes Memory Pressure and Achieving Workload Stability With NodeSwap.
Try it out
With OpenShift Virtualization 4.22, all of the memory optimizations are available, and the virtualization platform autopilot configures Gereally Available features automatically, with Developer Preview and Technology Preview capabilities requiring manual opt-in activation. All features are expected to reach general availability in upcoming OpenShift versions.
To enable the complete feature set in OpenShift Virtualization 4.22 on a pre-production cluster, ensure that the OpenShift Virtualization operation and the descheduler operator are installed. Then enable autopilot:
$ oc annotate hyperconverged kubevirt-hyperconverged \
-n openshift-cnv platform.kubevirt.io/autopilot="true" --overwriteClear the full KSM and then enable KSM zero pages:
$ oc patch hyperconverged kubevirt-hyperconverged -n openshift-cnv \
--type='json' -p='[{"op": "remove", "path": "/spec/virtualization/ksmConfiguration"}]'
$ oc annotate hyperconverged kubevirt-hyperconverged -n openshift-cnv \
platform.kubevirt.io/enable-ksm-zero-only="true" --overwriteDisable the autopilot (this doesn't disable the processes it has enabled, such as KSM zero pages):
$ oc annotate hyperconverged kubevirt-hyperconverged \
-n openshift-cnv platform.kubevirt.io/autopilot="false" --overwriteFor more details about the virtualization platform autopilot features in OpenShift Virtualization 4.22, refer to upstream release-4.22 branch.
Conclusion
Shifting away from rigid, 1:1 memory reservations required establishing a new baseline of efficiency, beginning within the host operating system. The platform had to first minimize the raw physical footprint of each virtual machine through guest-cooperative mechanisms like FPR and guest-agnostic deduplication using KSM zero pages. Building upon that foundation, MGLRU and control group integration granted individual nodes the localized capability to swap memory to additional memory tiers.
Unlocking greater density meant extending these node-local defenses into a cohesive, cluster-wide strategy. By elevating orchestration to prioritize PSI-aware metrics, the descheduler monitors the environment and leverages live migration to proactively disperse active memory demand across the infrastructure. To ensure this complexity does not compromise usability, the virtualization platform autopilot aligns the stack, automatically tuning low-level kernel heuristics alongside high-level migration policies to deliver a seamless user experience.
This architectural evolution sheds some light on the power of open source collaboration. By bringing together contributions from various vendors across different layers of the Linux, Kubernetes, and KubeVirt ecosystems, we have built a highly flexible and powerful system (and ecosystem) capable of safely meeting modern infrastructure demands.
Special thanks to Andrew Sullivan, Raffaele Spazzoli, Simone Tiraboschi, and Michal Skrivanek for reviewing early drafts of this post and offering valuable feedback.
