Identity and access management (IAM) is at the very core of your enterprise's runtime infrastructure. When authentication services fail, otherwise healthy application workloads instantly lock up, blocking users and disconnecting platform services. But high-availability setups for these services have historically been operationally complex, rigid, and restricted to heavy cloud architectures.
The on-premises challenge
In enterprise environments, migrating from legacy IAM systems like Red Hat;s single sign-on (SSO) technology to the newer Red Hat build of Keycloak has often raised crucial architecture questions regarding multi-site high availability (HA). Enterprise IT teams operating across independent on-premises failure domains need to preserve authentication continuity. Starting with Red Hat build of Keycloak 26.4, we have expanded HA multi-site support to include on-premises private data centers and non-AWS cloud providers (such as Microsoft Azure, Google Cloud Platform, and IBM Cloud) using an Red Hat OpenShift single-cluster deployment spanning multiple availability zones with a synchronous database.
For teams managing custom, private OpenShift clusters, enterprise load balancers, and on-premises databases, this gap was a critical friction point. A gap needed to be supported, the goal was clear: Deliver HA-enabled product capabilities that empower customers to deploy fallback deployments across independent clusters using the supported databases and load-balancing infrastructure.
The "stateless" breakthrough in high availability
Multi-datacenter Keycloak deployments depended heavily on cross-site replication of in-memory caches using embedded Infinispan. This model required tight coupling between sites, low latency (less than 10 ms). If the connection degrades, one of the sites would be taken offline. The low latency is still a must.
The introduction of the new stateless feature completely changes this landscape. This architecture shifts away from in-memory cross-site cache replication in favor of several key principles:
- Externalized state: Cross-site session data is persisted directly into a shared, highly consistent database, rather than being held across live Infinispan memory pools.
- Isolated local caches: Infinispan embedded caches remain completely isolated and local to each site, maximizing processing speed and eliminating WAN-latency synchronization issues.
- Unified OIDC issuer: A single, unified hostname and public issuer are presented behind an external load balancer, making site failover completely transparent to OpenID Connect (OIDC) clients, important for custom resource (CR) definitions.
- Active health probing: An active health check endpoint (
/lb-check) allows external load balancers to query individual site status and route traffic away from a degraded site.
Strategic shift
Red Hat build of Keycloak is shifting from prescribing rigid supported architectures to providing robust supported HA features, like stateless. This transition empowers your team to build, own, and validate its own custom on-premises environments while remaining within a supported software model.
Dual-site high availability with Red Hat OpenShift Container Platform
To put this new stateless HA model to the test under conditions with a specific application, I designed a complete dual-site proof of concept (PoC) entirely within a local area network (LAN). Using Red Hat OpenShift Local, Podman, and HAProxy, I established the following local setup:
Site A (primary failure domain)
- Host environment: Linux
- Local OpenShift: CodeReady Containers (CRC)
- Database layer: PostgreSQL Primary (Podman container)
Site B (standby failure domain)
- Host environment: Windows 11
- Local OpenShift: CodeReady Containers (CRC)
- Database layer: PostgreSQL Synchronous Standby (Podman container)
A local instance of HAProxy, running on the Linux host and bound to port 8443, acted as the external load balancer. It was configured with an active health check that actively polled the /lb-check endpoint of both sites. By enforcing synchronous PostgreSQL replication between the Linux primary and the Windows standby, I ensured that the database stayed perfectly aligned. If either site went offline, OIDC clients could continue seamlessly, because the database served as the single, consistent source of truth.
Experiencing the failover: The active failover drill
To test the architecture, I launched a single page application (SPA) designed to perform constant OIDC authentication and token refreshes against the shared external hostname. During an active login session, I simulated a complete platform crash on Site A by scaling its Red Hat build of Keycloak operator deployment down to 0 replicas. The local HAProxy instantly detected the failure through /lb-check and marked Site A as down. Within seconds, the SPA redirected all traffic to the Windows-based Site B.
The OIDC session remained completely valid, the token refreshed successfully, and the user experienced 0 interruptions or logout prompts. The combination of stateless instances and a synchronized database had successfully bridged the gap between failure domains.
Explore the interactive simulation canvas
To help visualize how the traffic flow behaves during site outages, database primary failures, and load-balancer switches, you can run an interactive simulation directly in your browser. I've built a custom visual canvas that maps out the failover process step-by-step.
Access a fully rendered, interactive simulation to test OIDC failover scenarios, site outages, and database promotion paths at my demo site.
Note
While the raw GitHub HTML file of my demo site is useful for reviewing the source code, the GitHub HTML Preview link provided is much better suited for direct interaction, as it parses and renders the full CSS/JS simulator.
Lessons learned and upstream contributions
While the results were incredibly successful, building a production-grade active-standby deployment on a limited local sandbox revealed several nuances where the initial upstream documentation fell short. By collaborating closely with the upstream Keycloak maintainers, we were able to turn these findings into active documentation improvements in upstream pull request:
- Strict hostname alignment: The initial documentation didn't explicitly detail that both sites must use the exact same public URL and OIDC issuer behind the load balancer, which caused browser redirect mismatches until we manually mapped the routes to
auth.lan.local:8443. - Port specification in URLs: When using a custom external port (like 8443 in this lab), the port must be explicitly included in the
spec.hostname.hostnameparameter, or Red Hat build of Keycloak generates default issuer URLs pointing to port 443. - Disabling router stickiness: Standard OpenShift routers have IP-sticky cookies enabled by default. To allow the external load balancer to manage true site failover, we disabled route-level cookies and configured round-robin distribution.
- Sizing down for local labs: The official guide cited high pool counts and instances that overwhelmed local OpenShift Local environments. Testing proved that a minimal configuration (1 instance, pool size of 15) is fully functional for lab validation.
OIDC flows validations tested
Verification of authorization code and token exchange flows under cross-site failover. Public authorization code flow with PKCE and standard token exchange V2 (RFC 8693) remain fully operational during active cross-site outages without relying on session stickiness.
- Phase 1 - Authorization code flow results: The authorization code flow was validated using a single page application (poc-spa) enforcing PKCE S256. Cross-site failover was tested by manually scaling down StatefulSets on Site A and Site B to 0 replicas.
| ID | Test action / Condition | Expected behavior | Status |
|---|---|---|---|
| AC-1 | Login through https://auth.lan.local:8444 | Redirect over port 8443; valid token and issuer | PASS |
| AC-2 | Refresh access token in SPA | New access token issued under same issuer | PASS |
| AC-3 | Call protected API (/api/protected/profile) | HTTP 200 OK with valid bearer token | PASS |
| AC-4 | Scale Site A to 0 replicas (drop Site A) | HAProxy detects Site A down; refresh | PASS |
| AC-5 | Restore Site A; scale Site B to 0 replicas | HAProxy detects Site B down; refresh | PASS |
| AC-6 | Scale both Site A and Site B to 0 replicas | HAProxy returns HTTP 503; refresh/login fail gracefully | PASS |
| AC-7 | Round-robin distribution verification | No sticky cookies required; requests balanced across sites | PASS |
- Phase 2 - Token exchange flow results: Token exchange capabilities were evaluated against the standard V2 specification (RFC 8693), which is supported and enabled by default in upstream Keycloak 26.7. Tests confirmed that token exchange operations seamlessly survive cross-site failovers.
| ID | Test Action / Condition | Expected Behavior | Status |
|---|---|---|---|
| TE-1 | Both Site A and Site B active | Standard V2 token exchange succeeds on poc-exchange-requester | PASS |
| TE-2 | Site A DOWN (StatefulSet scaled to 0) | Token exchange request routed to Site B and succeeds | PASS |
| TE-3 | Site B DOWN (StatefulSet scaled to 0) | Token exchange request routed to Site A and succeeds | PASS |
| TE-4 | Token issued on Site A, then Site A dropped | Exchange of pre-issued token succeeds through Site B | PASS |
Note on Token exchange roadmap: The test run verified Red Hat build of Keycloak's implementation roadmap for token exchange capabilities:
| Version | Feature flag | Status in 26.7 | Architectural recommendation |
|---|---|---|---|
| V2 Standard (RFC 8693) | default on | Supported | Build all new workloads on Standard V2 |
| V1 Legacy | token-exchange | Deprecated | Migrate existing clients |
| V2 External-Internal | token-exchange-V2 | Removed | Replaced fully by Standard V2 |
Take the next step with Red Hat build of Keycloak
The shift toward "stateless" features represents a massive leap forward in simplifying high-availability architectures for Red Hat build of Keycloak. Rather than certifying rigid cloud-specific environments, Red Hat is empowering engineers to design robust, HA-enabled topologies on your own private cloud with OpenShift Container Platform and on your well known infrastructure providers (Red Hat Enterprise Linux on supported virtual machine hypervisors or other Kubernetes provider), both in the cloud and on-premises.
We highly encourage our enterprise customers and community developers to begin testing the "stateless" preview features using the upstream Keycloak 26.7 improvements. Share your feedback, report back with a support case, and help us refine this amazing new feature.
Learn more about deploying high availability Keycloak instances on Kubernetes using the official upstream guide.
Try out the upstream community release in your local test environment and explore the repository.
