Validating new architectures at scale is a critical step in enterprise software deployment. We conducted a performance and scale test to validate that the new argocd-agent pull architecture scales to robust enterprise requirements. Teams across Red Hat in performance and scale engineering, Red Hat Advanced Cluster Management for Kubernetes engineering, and Red Hat OpenShift GitOps engineering conducted the testing.
The evolution: From push to pull
Moving from a push-based to a pull-based model fundamental shifts how GitOps controllers manage cluster credentials and network traffic.
The traditional push model (hub-and-spoke)
In a standard software deployment setup, a central GitOps controller (hub) "pushes" configurations to remote clusters (spokes).
The risk of this mode is that the hub requires high-level administrative credentials for every spoke cluster. This model also creates bottlenecks because the hub must constantly observe the state of every resource across the fleet, which leads to high memory usage and heavy network traffic for deployments with many spoke clusters.
The modern pull model (Argo CD Agent)
The Argo CD Agent model decentralizes workload deployment. A lightweight agent lives on each managed cluster and "pulls" its configuration from a central principal—the main hub cluster. Some benefits of this mode include:
- Security by design: Clusters initiate outbound connections to the hub. No inbound administrative ports need to be open on the spoke clusters.
- Distributed compute: The heavy lifting of reconciliation (comparing Git to the live state) happens locally on the spoke, not the hub.
- Network efficiency: Testing shows up to an 80% reduction in cross-cluster network traffic.
The test environment and setup
To assess the limits of Red Hat OpenShift GitOps, an extensive performance lab environment was built. It included:
- Hardware: 42 physical machines, 56 Xeon Gold 5420+ cores (Hyper-Threading), 512 GB RAM.
- Network: 25 Gbps per second network connectivity between nodes.
- Clusters: 1 hub cluster (3 nodes) and 500 spoke clusters (Red Hat OpenShift virtual machine clusters).
- Software stack: Red Hat OpenShift Container Platform 4.21, Red Hat Advanced Cluster Management 2.16, OpenShift GitOps 1.20 (Argo CD 3.3.z), and Gogs.
Test scenarios: Bootstrap vs. sync-to-commit
The testing team set up 65 ApplicationSets on the hub cluster, each generating 1 application per target cluster. Every application was configured to source 100 resources from a Git repository.
Two primary scenarios were tested across multiple load phases:
- Bootstrap scenario: This involved generating all ApplicationSets and Applications on the hub and measuring the exact time until all Applications were marked "synced" and "healthy."
- Sync-to-commit scenario: This involved changing 1 resource in Git, pushing the commit, and measuring the time until all applications successfully synced to the new commit secure hash algorithm (SHA) on the hub cluster. Because Argo CD is configured to poll Git repositories for new changes, the total sync-to-commit time is affected by Argo CD's reconciliation timeout, so it can take up to 3 minutes for a new change to be picked up by Argo CD. This is reflected in the measurement.
Performance benchmarks
Testing was executed in a phased approach to gradually increase the load and observe the system's behavior across 2 primary scenarios: Bootstrap (generating applications on the hub and syncing) and sync-to-commit (pushing a Git commit and syncing to the new SHA).
| Phase | Number of clusters | Total applications | Total resources | Sync time (bootstrap) | Sync to commit time |
|---|---|---|---|---|---|
| Phase 1 | 50 | 3,250 | 325,000 | 9m 31s | 12m 58s |
| Phase 2 | 100 | 6,500 | 650,000 | 9m 50s | 12m 39s |
| Phase 3 | 250 | 16,250 | 1,625,000 | 10m 44s | 13m 48s |
| Phase 4 | 480 | 31,200 | 3,120,000 | 13m 03s | 13m 00s |
As a bonus test, the team deleted all 31,200 applications across the 480 clusters using cascading deletion. This massive cleanup operation was completed in 4 minutes and 53 seconds.
Findings and bottlenecks
While the argocd-agent architecture proved highly capable of handling immense scale, the tests revealed several important bottlenecks and rate-limiting factors:
- Kubernetes API limits: The throughput on the Kubernetes API and etcd on the hub cluster became a primary rate-limiting factor. At the highest scale in phase 4, the team had to limit the principal's Kube API queries per second (QPS) to avoid overwhelming the API during list operations.
- Informer processing: The principal's application informer callbacks were not processing fast enough under heavy load, which led to frequent watch restarts and general slowdowns.
- Event writer retries: The agent's event writer retry mechanism proved problematic under heavy load. It was designed to resend an event after 1 second if processing wasn't acknowledged, but because most events took longer than 1 second to process at scale, resent events filled up queues and starved the system.
- ApplicationSet controller overhead: The ApplicationSet controller on the hub cluster struggled during the later phases, consuming a significant amount of RAM when generating thousands of applications.
Conclusion
Scale testing confirmed that the pull architecture in OpenShift GitOps stabilizes large-scale deployments that previously overloaded central hub controllers. Syncing more than 3.1 million resources across 480 clusters in roughly 13 minutes is a major achievement for the platform's capabilities. The insights and minor bugs revealed by this exercise are already informing currently unreleased code changes and will pave the way for further optimizations in future OpenShift GitOps releases.
Explore the interactive demonstration, or review the Argo CD Agent architecture documentation and Argo CD Agent installation guide to learn more.
Prerequisites:
- Red Hat OpenShift GitOps 1.21
- A Red Hat OpenShift Platform Plus subscription on each cluster that runs the OpenShift GitOps agent. The control plane of OpenShift GitOps is still available with Red Hat OpenShift Container Platform, but using the agent requires an OpenShift Platform Plus subscription.
