Skip to main content
AI
  • Get started with AI

    • Red Hat AI

      Accelerate the development and deployment of enterprise AI solutions.

    • AI learning hub

      Explore learning materials and tools, organized by task.

    • AI interactive demos

      Click through scenarios with Red Hat AI, including training LLMs and more.

    • AI/ML learning paths

      Expand your Red Hat OpenShift AI knowledge using these learning resources.

    • AI quickstarts

      Focused AI use cases designed for fast deployment on Red Hat AI platforms.

    • No-cost AI training

      Foundational Red Hat AI training.

  • Featured resources

    • Red Hat OpenShift AI
    • Open source AI for developers
    • AI product application development
    • Open source-powered AI/ML for hybrid cloud
    • AI and Node.js cheat sheet

Red Hat AI Factory with NVIDIA

Red Hat AI Factory with NVIDIA is a co-engineered, enterprise-grade AI solution for building, deploying, and managing AI at scale across hybrid cloud environments.

Learn
  • Self-guided

    • Documentation

      Find answers, get step-by-step guidance, and learn how to use Red Hat products.

    • Learning paths

      Explore curated walkthroughs for common development tasks.

    • Guided learning

      Receive custom learning paths powered by our AI assistant.

    • See all learning
  • Hands-on

    • Developer Sandbox

      Spin up Red Hat's products and technologies without setup or configuration.

    • Interactive labs

      Learn by doing in these hands-on, browser-based experiences.

    • Interactive demos

      Click through product features in these guided tours.

  • Browse by topic

    • AI/ML
    • Automation
    • Java
    • Kubernetes
    • Linux
    • See all topics
  • Training & certifications

    • Courses and exams
    • Certifications
    • Skills assessments
    • Red Hat Academy
    • Learning subscription
    • Explore training
Build
  • Get started

    • Red Hat build of Podman Desktop

      A downloadable, local development hub to experiment with our products and builds.

    • Developer Sandbox

      Spin up Red Hat's products and technologies without setup or configuration.

  • Download products

    • Red Hat Enterprise Linux
    • Red Hat AI
    • Red Hat OpenShift
    • Red Hat Ansible Automation Platform
    • See all products
  • Featured

    • Red Hat build of OpenJDK
    • Red Hat JBoss Enterprise Application Platform
    • Red Hat OpenShift Dev Spaces
    • Red Hat Developer Toolset
  • References

    • E-books
    • Documentation
    • Cheat sheets
    • Architecture center
Community
  • Get involved

    • Events
    • Live AI events
    • Red Hat Summit
    • Red Hat Accelerators
    • Community discussions
  • Follow along

    • Articles & blogs
    • Developer newsletter
    • Videos
    • Github
  • Get help

    • Customer service
    • Customer support
    • Regional contacts
    • Find a partner

Join the Red Hat Developer program

Download Red Hat products and project builds, access support documentation, learning content, and more.

ConsoleDocsSupport Search
New For you

Resources to explore

Check out popular resources, tools, and information from Red Hat. We'll recommend more as you browse.

  • Product downloads
  • Cheat sheets
  • Red Hat Developer learning portal
  • Developer blog
  • E-books
  • Developer Sandbox
Log in

Get more with a Red Hat Developer account

  • Product downloads
  • Console access
  • Event registration
  • Training & trials
Explore more account benefitsLog in or register

A subscription may be required for some services.

Contact us

High Scale Performance Testing of OpenShift Virtualization : An Introduction

November 20, 2026
Shekhar Berry Jenifer Abrams
Related topics:
Performance and Scale EngineeringVirtualizationMigration
Related products:
Red Hat OpenShift Virtualization

    Introduction

    As the OpenShift Virtualization Performance & Scale team, we get access to hardware from the Red Hat Scale Lab that lets us push OpenShift Virtualization well beyond what most environments will ever see. Our job is straightforward: break things at scale before customers do, work with engineering to fix what we find, and publish results so that architects and administrators can plan their deployments with confidence.

    In 2025, we conducted a deeper high-scale performance re-validation against OpenShift (OCP) 4.20, extending our previous scalability studies. Along the way, we uncovered scaling bugs in various controllers, Prometheus resource usage, the UI, and the descheduler. Thanks to strong efforts from multiple engineering teams, many were fixed in time to continue testing. We published our VM density findings for that round.

    For OCP 4.22, we are expanding this effort, leveraging the bug fixes resolved after the previous test cycle to perform high-scale testing on a more resilient platform. Our officially published and fully supported scale limit is 10,000 VMs per single cluster, our job is to find the absolute breaking points to ensure that customers operating at or near those limits have a great product experience. Overall, our 4.22 high scale testing has proven stabilization across all Virtualization components, including for large scale VM migrations and node drain events, which are critical for cluster maintenance and upgrades.

    This blog introduces our 2026 high scale testing campaign, a multi-month effort that we'll be documenting across a series of detailed posts covering everything from raw VM density to database performance at scale.

    What the 4.20 Campaign Taught Us - And Why It Matters

    Before diving into the 4.22 effort, it's worth looking back at what the 2025 round of high-scale testing revealed. The scale and scope of bugs found during the 4.20 campaign underscores exactly why this testing is critical - these issues simply don't surface at smaller scales.

    During the 4.20 campaign, over 30 bugs were filed spanning operator installs, scale deployment failures, descheduler issues, and virt-controller problems. Here are some highlights of the bug areas our previous scale testing uncovered, and thanks to close coordination with various Engineering teams have since been fixed in 4.20.zstreams and later releases:

    • Advanced Cluster Management (ACM) Issues: Resolved search memory limits at 10K VMs and metric collector pod failures at higher VM scale. 
    • Virtualization Issues: Resolved migration and virt-controller deadlocks and failures at scale including a migration completion detection issue, webhook scaling issues, multi namespace kubemacpool scaling errors, and node eviction/drain stalls.
    • Descheduler Issues: Identified cases where stale node taints were left behind and resolved issues with incorrect eviction success signals.
    • OpenShift Data Foundation (ODF) Issues: Resolved block-only installation issues and identified components that require memory limit increases to resolve scaling limitations.

    The volume and severity of these findings demonstrate the value of high-scale testing. Several of these fixes directly improved the production experience for customers running at scale.

    Validating the Next Level of Resiliency

    Building on the lessons from the 2025 campaign, our validation efforts require looking at the platform holistically:

    • With the foundational fixes from previous cycles in place, our focus shifts from basic scalability to comprehensive platform resiliency, ensuring that the user experience remains flawless even when operating at the 10,000 VM limit.
    • We must account for how dense physical servers handle hundreds of VMs per node, ensuring that localized performance doesn't degrade as hardware utilization peaks.
    • Virtualization workloads possess unique operational characteristics - such as live migration traffic and continuous VM status updates - that we must rigorously evaluate to guarantee stability for enterprise deployments.
    • Validating loaded upgrades across 100 nodes and 10,000 active VMs to guarantee seamless live migrations and node drains without workload interruption.

    Our 2026 high scale testing goals reflect these realities:

    • Cluster scale: Target stability at 10,000 VMs across up to 250 nodes
    • Node density: Scale up from 100 VMs per node to 500+, validating kubelet and container runtime behavior at that density
    • Namespace scale: Scale up from ~175 namespaces to 500+, testing controller behavior with high namespace counts
    • Migration concurrency: Raise limits from the conservative 2 per-node / 5 per-cluster to higher ranges, to find and fix any new scalability issues in the migration components
    • Real workloads: Focus on customer-relevant workloads - MSSQL on Windows, database benchmarks, actual I/O patterns - not just synthetic boot storms

    Our Test Environment

    The Red Hat Scale Lab allocated hardware starting the last week of April 2026. We configured three clusters:

    • Primary Cluster: Started at 92 Dell R650 worker nodes (56cores,112 threads, 512 GiB RAM each), later expanded to 200+ workers. Hyperconverged storage using OpenShift Data Foundation across all workers with NVMe drives, zone-based topology for parallel node drains.
    • Secondary Cluster: 10 worker nodes used for lower scale and per node density testing.
    • ACM Hub Cluster: Used for Observability scale testing. 

    All clusters ran OCP 4.22 with OpenShift Virtualization 4.22, starting from pre-GA nightly builds and progressing through release candidates. We installed and configured ODF with a block-only storage cluster, local storage for monitoring persistence, and configured dedicated migration networks where available.

    Phased Execution

    This isn't a one-shot effort. The scale lab is a shared resource, and running 200+ nodes for months requires careful planning. We adopted a phased approach:

    • Phase 1 (April–May): Smaller clusters stressing per-node scaling before adding more nodes. In this phase we prototyped high VM density per node, validated tooling, and ran initial cluster-wide density tests up to the current supported limits. This is where we discovered that a single namespace with thousands of VMs could drive virt-controller memory to extreme levels (~100 GB for one pod) for which we filed a bug to work with Engineering, and that the default cluster network subnet silently caps per-node VM count at ~512.
    • Phase 2 (June–September): Scaling out to 200+ nodes for the full assault - replicating large customer configurations, running sustained workloads, testing upgrades, and analyzing the resulting data for this blog series. Migration testing, storage I/O benchmarking, and descheduler validation all happen in this phase.

    What We Covered in This High Scale Effort

    We tested the below features/scenarios in this round of testing and we will publish blog posts for some of them where we will explore the in-depth testing area with full methodology, cluster configuration, results, and operational guidance.

    •  VM Density: Pushing OpenShift Virtualization beyond officially supported thresholds such as scaling up to 40,000 VMs past the 10,000 VM limit is essential for uncovering hidden control plane bottlenecks, evaluating etcd and API server saturation under intense load, and validating the overall robustness and resilience of the platform in extreme enterprise scale scenarios. We looked at boot latency, control plane resource consumption, the challenges encountered at this scale (including elevated alert counts and control plane pressure), and the impact of namespace partitioning on controller performance. 
    • Storage I/O, FIO and Database Workloads: Sustained storage performance at scale using FIO for synthetic I/O characterization and real database workloads (MariaDB, PostgreSQL, MSSQL) running inside VMs backed by ODF. We tested configurations from 100 to 3,000 VMs under I/O load, characterizing aggregate IOPS, throughput, ODF utilization, and the stability boundaries where Ceph OSDs begin to struggle.
    •  Networking UDN & CUDN Scalability: User-Defined Networks and Cluster User-Defined Networks are newer networking models for OpenShift Virtualization. We test the network fabric's ability to handle large numbers of UDNs and CUDNs under load, compare migration performance between UDN and the default pod network, and examine ingress/egress behavior.
    • Windows Performance, MSSQL: Windows VMs are a critical use case for organizations migrating off VMware and Hyper-V. We evaluated Windows Server 2022 virtual machines running Microsoft SQL Server 2022 at scale on OpenShift Virtualization 4.22 using HammerDB 4.12 (TPC-C workload) across density tiers ranging from 100 to 1,500 concurrent VMs.
    • Live Migration at Scale: Migration is fundamental to upgrades, node maintenance, and workload rebalancing. We assessed node evacuation routines and active migration efficiency across a 100-worker environment hosting 10,000 active instances subject to targeted guest memory load on OpenShift Data Foundation storage. In parallel, we also assessed live migration scalability and overall responsiveness across a 240-worker environment using higher limits, and creating thousands of VMIMs (VirtualMachineInstanceMigrations) in parallel to stress test the live migration capabilities.
    • Descheduler and Memory Overcommit at Scale: We ran the descheduler with both CPU and memory profiles active on a 10,000+ VM cluster - monitoring eviction patterns, validating convergence, and documenting the oscillation scenarios we found with very large VMs. Combined with memory overcommit via per-node swap, these features enable higher consolidation ratios but introduce new stability considerations at scale.
    •  Upgrade and Node Drain Scenarios: What happens when you need to roll through an upgrade with tens of thousands of VMs running? We tested parallel node drain behavior, the interaction between ODF Pod Disruption Budgets and drain parallelism, Ceph PG recovery timing, and the total duration for cluster-wide rollouts. We covered high-scale OpenShift cluster upgrades under active workloads defined as 100 nodes and 10,000 virtual machines running dirty-memory write loads.
    • ACM Observability and UI at Scale: We set up managed clusters and scaled VMI objects to understand ACM scaling limitations, provided cluster access to the ACM UI team for observation at large VM scale, and evaluated RBAC behavior. We completed an ACM virtualization performance observability study and documented lessons learned for multi-cluster management at high VM counts.

    A Note on What We've Accomplished

    While preserving more details for subsequent deep-dives, our recent validation cycle achieved two core objectives: demonstrating the platform's inherent stability under intense operational pressure and exposing critical edge constraints that happen exclusively at scale.

    These efforts directly helped us improve the platform in several ways. Working together with the engineering teams, we fixed critical issues throughout the stack of the product. We will share more technical details and data about our testing in future blog posts.

    Running tests at such a large scale brought unique challenges, like a high number of alerts and temporary stress on the control plane during busy workloads. Fixing these issues with our development teams proved that the platform stays reliable at the official 10,000 VM limit. Each bug fix and design improvement makes the system stronger and more reliable under heavy use.

    Finding and fixing these issues in our lab before customers run into them is the main goal of our work. It ensures the software remains reliable, predictable, and ready for production use.

    Disclaimer: Please note the content in this blog post has not been thoroughly reviewed by the Red Hat Developer editorial team. Any opinions expressed in this post are the author's own and do not necessarily reflect the policies or positions of Red Hat.

    Related Posts

  • High Scale Performance Testing: Virt Density

  • Best Practice Configuration and Tuning for Linux and Windows VMs

  • Use kube-burner to measure Red Hat OpenShift VM and storage deployment at scale

  • Performance analysis of storage live migration feature in Red Hat OpenShift Virtualization

  • Recent Posts

    • High Scale Performance Testing of OpenShift Virtualization : An Introduction

    • Automate Lightwell updates with Mend Renovate

    • Configure HikariCP for Spring Boot on Red Hat OpenShift 4

    • OpenShift Pipelines and Test Containers

    • The ultimate guide to analyzing and troubleshooting critical issues on Red Hat Enterprise Linux 8 and 9

    Red Hat Developers logo

    Platforms

    • Red Hat AI
    • Red Hat Enterprise Linux
    • Red Hat OpenShift
    • Red Hat Ansible Automation Platform
    • See all products

    Build

    • Developer Sandbox
    • Developer tools
    • Interactive tutorials
    • API catalog

    Quicklinks

    • Learning resources
    • E-books
    • Cheat sheets
    • Blog
    • Events
    • Newsletter

    Communicate

    • About us
    • Contact sales
    • Find a partner
    • Report a website issue
    • Site status dashboard
    • Report a security problem

    Red Hat Developer

    Build here. Go anywhere.

    We serve the builders. The problem solvers who create careers with code.

    Join us if you’re a developer, software engineer, web designer, front-end designer, UX designer, computer scientist, architect, tester, product manager, project manager or team lead.

    Sign me up

    Red Hat legal and privacy links

    • About Red Hat
    • Jobs
    • Events
    • Locations
    • Contact Red Hat
    • Red Hat Blog
    • Inclusion at Red Hat
    • Cool Stuff Store
    • Red Hat Summit
    © 2026 Red Hat

    Red Hat legal and privacy links

    • Privacy statement
    • Terms of use
    • All policies and guidelines
    • Digital accessibility
    Ask AI