At Red Hat, our CI/CD pipelines are the heartbeat of our development process. However, as we scaled the Red Hat Developer Hub performance testing framework, we hit a wall. Our environment setup time was ballooning, turning what should have been a seamless verification step into a long waiting game. For performance engineering in catalog-dependent applications, deployment and catalog population speed directly dictate your feedback loops. Here's how we solved this problem.
Our performance bottleneck was runtime user generation. Within Red Hat Developer Hub, users and groups represent critical catalog entity types, serving as the foundation for the application's role-based access control (RBAC) across various pages and actions. A primary objective in our performance engineering involves modulating the entity volume within the Red Hat Developer Hub database. Furthermore, simulating high levels of concurrent active users requires a specific volume of pre-existing entries. At massive scales, our legacy simulation methods simply were not feasible, necessitating a fundamental architectural rethink.
We've identified the bottleneck: The "old way".
Previously, our testing framework relied on generating thousands of users and groups on-the-fly using the Keycloak Admin REST API . Every single user entry necessitated a serialized chain of REST operations. When handling massive catalog scales, we attempted to bypass the single-entry limitations of the Keycloak API by leveraging hardware-level concurrency. However, even with multi-threaded workers, the environment provisioning remained a time-intensive marathon. After populating the database, we had to get back to the performance framework to continue with its next steps for the added sync processes (Red Hat Developer Hub to keycloak synchronization). This is where Red Hat Developer Hub used to make paginated HTTP calls to Keycloak to sync users to its database.
This tedious process, executed only to start the deployment, left us with increased failures. We had to cope up with complex code structures and fallback logic because the tests might get network timeouts, rate limits, and so on, leading to an incomplete catalog, and failed test runs.
More components means more potential failure points.
The pivot: Enhanced infrastructure
We decided to replace runtime user generation with pre-built LDAP container Images. Instead of creating users during test deployment, we treat our LDAP environment as immutable infrastructure. We moved from an on-demand generation model to a pre-built artifact model.
Importing 10,000 users with ldapadd (online, using the protocol) is slow because it validates access control and schema for every entry. So we switched to slapadd, which writes directly to the LMDB database files while the server is offline. Deploying these environments is streamlined: By defining the user and group requirements through the <RBAC_POLICY>-<USER_COUNT>u-<GROUP_COUNT>g convention, the appropriate pre-baked images are automatically retrieved.
This shift required us to pre-compute our state into artifacts rather than generating data at runtime. We also reached a state where keycloak shifted from data source for the catalog to authentication gateway only, which is (most of the time) a customer behaviour.
Our Containerfile now uses a multi-stage build:
Stage 1: Builder
The first stage uses slapadd to populate the OpenLDAP database:
# Stage 1: populate the OpenLDAP database offline
FROM docker.io/osixia/openldap:1.5.0 AS builder
COPY slapd.conf /etc/ldap/slapd.conf
COPY seed.ldif /tmp/seed.ldif
RUN mkdir -p /var/lib/ldap && \
slapadd -f /etc/ldap/slapd.conf -l /tmp/seed.ldifStage 2: Runtime
The second stage copies the populated database files into the final image:
# Stage 2: ship only the populated database
FROM docker.io/osixia/openldap:1.5.0
COPY --from=builder /var/lib/ldap /var/lib/ldapUsing slapadd bypasses the network layer completely and writes directly to the database files. This makes it ideal for seeding databases during builds. As a result, when our test environment starts, the LDAP server boots with 10,000+ users already indexed in Keycloak, ready to query. For the Red Hat Developer Hub part, we wait for the first sync of the catalog to complete.
| Users | Groups | Populate Time Before | Populate Time After |
|---|---|---|---|
| 1,000 | 10,000 | 37 Min | 21 Sec |
| 5,000 | 50,000 | 5 Hour 34 Min | 1 Min 1 Sec |
| 10,000 | 150,000 | 22+ Hours | 2 Min 8 Sec |
By moving to this pre-built, immutable architecture, we fundamentally altered our performance matrix. The old workflow was serial, as shown in figure 1:
The new workflow, illustrated in figure 2, introduced parallel processes and was significantly faster:
The benefits
The change moved the user and group population from a runtime dependency into an immutable build artifact, which made the test environment deterministic, the catalog provider realistic, and the setup time constant regardless of scale.
- Resolution of "flaky test" errors caused by incomplete provisioning and guaranteed data consistency across thousands of test cycles
- Improved trend analysis through stable performance baselines, reduced deployment durations, and unlocked capabilities for very high catalog population counts for performance testing
- Faster troubleshooting and scenario reproduction since the dataset remains constant while Red Hat Developer Hub evolves
Core principles for engineering success
To overcome challenges in large-scale test environment setups, we recommend applying these 3 principles validated during our migration:
- Favor binary protocols: Moving from JSON/REST to the binary LDAP protocol reduced network latency and serialization overhead by great margins.
- Pre-compute or compute-at-runtime: If your test data is static (or predictable), then don't generate it during the deployment phase. Use static data sources like containers, storage drives (and so on) to bake in your state.
- Optimize for read-heavy workloads: By adding the
memberOfattribute, we essentially pre-computed our graph relationships, eliminating the need for expensive N+1 queries during catalog ingestion.
Conclusion
Our architectural shift didn't just save us time, it provided us with reproducible baselines. Whether we're testing with 1,000 or 100,000 users, we now have a consistent, reliable environment that accurately mirrors the enterprise reality our customers have in production.
