What happens when you use AI to upgrade your AI platform? When the IT AI Platforms team at Red Hat needed to upgrade Red Hat OpenShift AI from 2.25.5 to 3.3.2, they faced a highly complex challenge. Version 3.3.2 is the most architecturally significant release, replacing core networking, the authentication layer affecting inferencing, workbenches, and data science pipeline workloads. Refer to OpenShift AI 3.3.3 release notes.
By using an AI coding assistant as their primary engineering partner, the team executed an unprecedented upgrade across 3 managed clusters. This approach reduced an estimated 6-month project requiring 3 to 4 engineers down to about 8 weeks with a single engineer (about 1.5 full-time equivalent (FTE) positions), representing a 3- to 4-fold compression in calendar time and an approximate 60% reduction in engineering effort. This guide walks through the methodology used to achieve zero data loss, minimal extended service outages, and minimal user impact during the migration.
Prerequisites
Before beginning an enterprise-scale Red Hat OpenShift AI upgrade using this approach, verify you have the following:
- Target clusters: Development, pre-production, and production OpenShift clusters
- Migration diagnostics: Access to rhai-cli, the official migration readiness diagnostic tool provided by Red Hat for OpenShift AI upgrades
- AI engineering partner: An AI coding assistant integrated into your workflow
- Technical knowledge: Familiarity with the new OpenShift AI 3.3.2 architectural replacements: Gateway API (replacing OpenShift Service Mesh 2),
kube-rbac-proxy(replacingoauth-proxy), andRawDeployment(replacingServerlessandModelMesh)
Step-by-step instructions
Follow these steps to prepare your architecture, build your tooling, and execute the migration.
Preparation and tooling architecture
The single most important decision in this project was investing upfront in architecture before writing a single line of implementation code. Rather than creating a manual runbook, which would drift with every new cluster and require constant human interpretation, we collaborated with our AI coding assistant to design a gate-based orchestration pipeline in the first working session. This pipeline follows a strict sequence: back up, pre-validate, upgrade, post-validate, and restore (if needed), with explicit go/no-go gates between each phase.
The result was a custom Go command-line interface (CLI) called rhoai-upgrade wrapping rhai-cli (Red Hat's official migration readiness diagnostic tool) inside this orchestration framework. The CLI grew to 9,384 lines of Go across 28 source files, implementing a hexagonal architecture for the OpenShift client abstraction layer. This architecture separates read-only operations (Run) from mutating operations (RunMutating) with full dry-run awareness, verifying safe testing before any cluster modification.
The per-cluster YAML configuration system allows development, pre-production, and production clusters to have different rollback thresholds, large language model (LLM) inference checks, and certificate authority (CA) bundle preservation requirements.
Alongside the Go CLI, we developed a Bash tooling suite called workbench-images-migration-to-rhoai-3.3.2 to discover which custom workbench container images need NGINX configuration patches for Gateway API path-based routing. The tool rebuilds affected images with the correct NGINX configuration and pushes the new -gw-tagged images to the internal registry.
This was critical because more than 50 custom VS Code and RStudio images built by Red Hat data science teams required these patches—a task otherwise error-prone and time-consuming to perform manually on each image.
Dev cluster upgrade and issue encoding
The development cluster serves as the crucible where undocumented issues surface safely. We ran the rhoai-upgrade CLI against the development cluster, and over 2 days of iterative debugging, uncovered 9 novel issues with no documentation anywhere—not in the official OpenShift AI migration guide, not in the rhai-cli output, and not in community forums.
For each issue, we used the AI coding assistant for real-time investigation rather than code generation alone. The workflow was: observe a failure, feed the exact oc command output to the assistant, let it form and test hypotheses against the live cluster state, and then write the fix.
For example, after the upgrade, all InferenceService routes returned HTTP 503 despite GatewayConfig showing Ready and Istio pods running. The AI assistant identified a NetworkPolicy podSelector still referencing the old HAProxy IngressController label (ingresscontroller.operator.openshift.io/deployment-ingresscontroller: internal-router-shard) while the new Istio Gateway Envoy pod used a completely different label (gateway.networking.k8s.io/gateway-name: data-science-gateway). The network was silently dropping traffic. The fix was a single oc patch command, but finding it manually would've taken hours of trial and error.
The critical discipline here is immediate encoding: every discovered issue and its fix was integrated directly into the automation tool in the same session it was found. This means the tool gets smarter with each cluster. By the time we reached production, all 9 dev-discovered issues were caught and resolved automatically, without requiring any human intervention or memory of the previous fix.
Pre-prod preparation and upgrade
With the development cluster clean and all known issues encoded, we executed the rhoai-upgrade CLI against the pre-production cluster. This environment mimics production workload patterns and is where we validated the migration of complex workloads at scale—specifically, 46 custom workbenches rebuilt with Gateway API-compatible NGINX configurations, multiple data science pipeline applications (DSPAs), and InferenceServices (ISVCs) under realistic traffic conditions.
The pre-production upgrade took 3 days, primarily because it surfaced workload-specific edge cases that didn't exist on the development cluster. Custom workbench images with nonstandard volume mount names, pipelines with hardcoded service mesh annotations, and InferenceService instances relying on deprecated Serverless autoscaling annotations all required attention. Each resolution was diagnosed collaboratively with the AI assistant by feeding it live oc query output from the cluster and immediately encoding it into the upgrade tool.
We also used the AI assistant to dynamically generate cluster-specific documentation and assessment reports based on live oc queries. Rather than writing these manually after the fact, each significant finding was documented in real time: workload migration plans, ISVC inventories, workbench migration logs, and post-upgrade testing plans. This parallel documentation approach meant we accumulated 32 technical documents over the course of the project without any dedicated writing time.
Production upgrade and verification
The production upgrade was the culmination of all the preparation, completing in a single day. The CLI tool, hardened for stability, executed the full upgrade sequence automatically: patching the DataScienceCluster, removing legacy operators (OpenShift Service Mesh 2, Serverless, Authorino) in the correct order, installing new operators (cert-manager, Red Hat Connectivity Link, Leader Worker Set, Kueue), running a final lint gate, switching the subscription channel, and approving the install plan. Every fix discovered on development and pre-production clusters was applied automatically.
After the upgrade completed, we executed parallel smoke tests against all 33 inference routes in the LLM namespace to confirm live AI traffic and user environments were fully operational. This test battery measures time-to-first-byte (TTFB) latency for each route and validates response structure. The results confirmed zero downtime for the 297 production users and 71 InferenceServices powering 29 production AI models.
The production smoke tests also surfaced unexpected API behavior differences: 3 reasoning models (gpt-oss-20b, both Nemotron Nano and Super variants) return content: null in the standard chat completion response because their output goes into a reasoning field instead. Qwen3-14B uses chain-of-thought mode, wrapping responses in <think> tags. Qwen3-Reranker requires POST /v1/score instead of /v1/chat/completions entirely. These behavioral differences are now documented for downstream consumer teams—integration bugs that would've been discovered much later without the comprehensive post-upgrade scan.
Common issues and troubleshooting
During a major architectural shift like the OpenShift AI 3.3.2 upgrade, you'll encounter novel, undocumented issues. The key is to use your AI coding assistant for diagnosis by feeding it live system state rather than asking generic questions. The following sections highlight the most significant issues we encountered and how they were resolved.
The OpenShift AI dashboard URL changes after the upgrade because the new Gateway API routing replaces the old ingress mechanism entirely. What was previously accessible at rhods-dashboard-redhat-ods-applications.$APPS_DOMAIN transitions to data-science-gateway.$APPS_DOMAIN.
For example, on our development cluster, the URL changed from https://rhods-dashboard-redhat-ods-applications.apps.ai-dev.example.com/ to https://data-science-gateway.apps.ai-dev.example.com/. This affects any bookmarks, documentation links, or hard-coded references across your organization. We addressed this by generating a URL migration table for each cluster and distributing it to all affected users before the upgrade window.
HTTP 503 errors on InferenceService instances were among the most challenging issues to diagnose because the cluster appeared healthy at every observable layer. The root cause was a NetworkPolicy using an outdated podSelector label from the old HAProxy IngressController (ingresscontroller.operator.openshift.io/deployment-ingresscontroller: internal-router-shard). After the upgrade, the new Istio Gateway Envoy pod carries a different label (gateway.networking.k8s.io/gateway-name: data-science-gateway), so the unchanged policy silently dropped traffic.
The fix is straightforward: use oc patch to update the NetworkPolicy podSelector to match the new gateway label. Without AI-assisted debugging, this silent label mismatch could take hours to trace through the networking stack.
HTTP 502 errors on workbenches with the underlying error EACCES: permission denied represent a naming convention incompatibility between OpenShift AI 2.x and 3.x. In OpenShift AI 3.x, the volume mount is only auto-added to the /opt/app-root/src path when the persistent volume claim (PVC) name matches the notebook name exactly. The older OpenShift AI 2.x convention used a <notebook-name>-pvc suffix, which causes the volume to be listed in the pod spec but never actually mounted.
As a result, the code-server process can't write to its expected file system path and fails with a permission error. The resolution requires 2 oc patch commands on the Notebook custom resource: 1 to rename the PVC reference and 1 to update the volume mount entry. This issue had zero documentation anywhere, and we diagnosed it entirely through AI-assisted tracing of the failure chain from the HTTP 502 response back through NGINX, the container file system, and the pod volume spec.
Tips and best practices
Start with architecture before writing code. The gate-based pipeline model we designed in the first session with our AI assistant required no structural changes across 8 weeks of iterative development. That initial architecture—the hexagonal OpenShift client layer, the per-cluster configuration system, and the explicit go/no-go gates—held up through adding the --chat-test flag, extending smoke tests to cover 33 inference routes, and adding per-command 60-second timeouts after a large cluster caused pipeline hangs. Investing that first conversation in design rather than jumping straight to implementation prevented costly structural rewrites later in the project.
Use AI for real-time investigation, not just code generation. The highest-value moments in this project weren't "write me a function" requests—they were "here is the oc output, what is wrong?" sessions. When you feed your AI assistant live system state, specific error messages, and concrete symptoms, it can form and test hypotheses in ways that dramatically accelerate debugging. The 9 undocumented issues we discovered were all resolved through this pattern: observe, provide context, diagnose collaboratively, then fix. Treat your AI assistant as a pair-debugging partner with unlimited patience and broad systems knowledge.
Encode findings immediately into your automation. Every manual fix discovered on the development cluster should become an automated check or remediation step before you move to the next environment. This compounding effect, in which development fixes apply automatically to pre-production and production clusters, was one of the most valuable properties of our approach. By the time we ran the production upgrade, all previously discovered issues were caught and resolved without human intervention. Your automation tool should get smarter with each cluster it touches.
Treat documentation as equal to code. The 32 technical documents produced during this project aren't hand-written summaries or afterthoughts. Each one is derived directly from live oc query output collected during the session with exact image tags, pod counts, TTFB latency measurements, and cluster-specific findings. Generating data-driven documentation in parallel with engineering work means you produce a complete record without the deferred documentation debt typically accumulating during complex infrastructure projects. This approach allowed a single engineer to produce the equivalent output of a dedicated technical writer working alongside the engineering team.
Next steps for your OpenShift AI environment
By relying on purpose-built automation and a collaborative AI partnership, the IT AI Platforms team successfully upgraded the models hosting OpenShift AI platforms. The automated approach protected 217 workbenches, 71 InferenceService instances, and 297 production users. Ultimately, this demonstrates using AI assistance for complex infrastructure upgrades isn't just faster—it's more reliable and better documented than comparable manual efforts.
- Explore the tools: Get hands-on with the Red Hat OpenShift AI CLI tool (rhai-cli).
- Upgrade to OpenShift AI 3.3: Follow our step-by-step assessment and migration guide to update from version 2.25.4 or later.
