Operationalize AI agents with OpenShift and Kubernetes primitives

In Demystifying agentic AI: How to build production-ready AIOps with open source models, we discussed why you should use open source models for production-ready agentic AI operations (AIOps) systems. Now you need to figure out where to run them and, more importantly, how to make them operationally manageable.

AI agents aren't static applications. They evolve. You'll need to:

• Swap models (frontier to open source, or between open source models).
• Update agent prompts and agentic routing rules.
• Add new domain specialists.
• Inject institutional knowledge as new failure patterns emerge.
• Roll out changes without downtime.

If your AI infrastructure requires rebuilding container images and redeploying every time you want to change a prompt or add a specialist, you've created an operational bottleneck. Your site reliability engineering (SRE) team won't tolerate that, and neither should you.

This is part 2 of a 3-part series on building production-ready agentic AIOps systems. Part 1 explained why open source models are ready. This article shows you why Red Hat OpenShift and, specifically, how Kubernetes primitives—and specifically Red Hat OpenShift— make AI agents as operationally manageable as any other workload.

The operational challenge

You've deployed an agentic AIOps pipeline. It's working. But then:

• You discover a gap. Package management failures go to a generic Linux specialist, but your organization has specific Red Hat Satellite content view policies the agent doesn't know about.
• You need to add institutional knowledge. A new standard operating procedure (SOP) for fast-track escalations.
• You want to test a different model. Switch from a 235 B parameter model to a lighter 120 B model for cost optimization.
• You need to update agentic routing rules. The ops_manager should delegate package failures to a new specialist instead of the generic Linux agent.

How long does this take? If the answer is "rebuild the image, push to registry, redeploy," you've lost.

Kubernetes gives you operational primitives

OpenShift is Kubernetes with an enterprise-grade operational layer. What makes it useful for AI workloads isn't some AI-specific feature. It's the same primitives your SRE team already uses:

• ConfigMaps. Externalized configuration updates without image rebuilds.
• PersistentVolumeClaims (PVCs). Durable storage for artifacts, skills, and context.
• Rolling restarts. Zero-downtime updates.
• Environment variables. Runtime configuration for model selection and behavior.

Here's how these solve the operational challenge.

Architecture: Configuration as data, not code

Our agentic AIOps pipeline separates logic (container image) from configuration (ConfigMaps and PVCs):

athena-agent (Deployment)
  ├── Container Image: quay.io/meridian/athena:1.2.3
  │   └── Python code for Deep Agents harness (unchanged)
  │
  ├── ConfigMap: athena-agent-config
  │   ├── AGENTS.md (ops_manager routing rules)
  │   └── subagents.yaml (specialist definitions: prompts, models, tools, skills)
  │
  ├── PVC: athena-skills
  │   └── /app/skills/
  │       ├── analyze-ansible-failure/SKILL.md
  │       ├── analyze-linux-failure/SKILL.md
  │       ├── analyze-package-management/SKILL.md  (added at runtime)
  │       ├── analyze-openshift-failure/SKILL.md
  │       ├── analyze-networking-failure/SKILL.md
  │       ├── create-ticket/SKILL.md
  │       ├── error-classifier/SKILL.md
  │       └── common/SKILL.md
  │
  └── Environment Variables
      ├── OPS_MANAGER_MODEL=qwen3-235b
      └── MAAS_ENDPOINT=https://maas.openshift-ai.svc.cluster.local

What this means in practice:

• To change a model: 

  oc set env deployment/athena OPS_MANAGER_MODEL=gpt-oss-120b

• To update agentic routing rules: edit ConfigMap, oc apply, and rolling restart.
• To add a new specialist: Update ConfigMap and copy skill to PVC, rolling restart.
• To inject new institutional knowledge: copy markdown file to PVC.

No image rebuild. No registry push. No extended downtime.

Example 1: Adding a new specialist agent

Let's walk through a real scenario. Our AIOps pipeline has a gap: package management failures go to sre_linux, which gives generic advice about dnf and repositories. But at our organization:

• We use Red Hat Satellite with per-team content views.
• Python packages for the AI/ML team require CodeReady Builder (CRB) and EPEL.
• There's an SOP (v2.3) for requesting content view updates.
• Fast-track escalation goes through #platform-satellite Slack channel.

We need a specialist agent that knows this context.

Step 1: Create the skill

First, encode institutional knowledge as a markdown file:

---
name: analyze-package-management-failure
description: Package management failure analysis with institutional knowledge
---

# Analyze Package Management Failure

## Institutional Knowledge: Satellite Infrastructure

**Satellite Servers:**
- Primary: satellite-primary.meridian.internal (RHEL 9 content)
- Sync schedule: Tuesdays 02:00 UTC

**Content View Model (per-team):**
- base-rhel9: All RHEL servers, core OS packages only
- ops-tooling: SRE team, monitoring, diagnostic tools
- aiml-workloads: AI/ML team, Python 3.11+, CUDA, data science libs (CRB + EPEL enabled)

**Content View Request Process:**
- Standard: Submit per SOP v2.3 (3-5 day turnaround)
- Fast-track: `#platform-satellite` Slack escalation (2-hour SLA for AI/ML workloads)

## Analysis Workflow

1. Identify the exact package(s) that failed to install from the dnf/yum error
2. Determine which team's content view is involved (check host group in AAP2)
3. Check whether the package requires CRB or EPEL
4. Reference the appropriate escalation path

Copy this to the PVC:

curl -sL https://gitea.example.com/configs/analyze-package-management-SKILL.md \
  | oc exec --stdin deployment/athena --container athena \
  -- sh -c 'cat > /app/skills/analyze-package-management/SKILL.md'

The skill is now on the persistent volume, available to any agent that loads it.

Step 2: Define the specialist agent

Add an entry to subagents.yaml in the ConfigMap:

sre_package_management:
  description: >
    Package management specialist. Delegate all package installation failures:
    dnf/yum errors, missing packages, Satellite content view gaps, EPEL/CRB
    requirements, repository sync issues, and content view request escalation.
  model: qwen3-235b
  system_prompt: |
    You are a senior SRE specializing in Red Hat package management and Satellite
    content delivery. You receive incident data from failed AAP2 jobs and perform
    root-cause analysis on package-related failures.

    Always:
    - Read the incident context (incident.json) first
    - Identify the exact package(s) that failed to install
    - Determine which team's content view is involved (check host group)
    - Reference the analyze-package-management skill for institutional knowledge
    - Check whether CRB or EPEL is needed and whether the team's content view includes it
    - Provide the SOP reference and escalation path for content view requests

    Use the create-ticket skill to structure your analysis as a TicketPayload.
    Set area to "linux" for all package management issues.
  tools:
    - web_search
  skills:
    - ./skills/analyze-package-management/
    - ./skills/create-ticket/
    - ./skills/common/

Download the updated config and apply it:

curl -sL https://gitea.example.com/configs/subagents-with-package-mgmt.yaml \
  -o /tmp/subagents-new.yaml

oc create configmap athena-agent-config \
  --from-file=subagents.yaml=/tmp/subagents-new.yaml \
  --from-file=AGENTS.md=/tmp/agents-new.md \
  --dry-run=client -o yaml | oc apply -f -

Step 3: Update the error classifier

The ops_manager uses an error classifier skill to identify failure domains. Update it to recognize package_management:

curl -sL https://gitea.example.com/configs/error-classifier-with-package-mgmt-SKILL.md \
  | oc exec --stdin deployment/athena --container athena \
  -- sh -c 'cat > /app/skills/error-classifier/SKILL.md'

Step 4: Rolling restart

To finalize your changes and ensure the new configuration is applied, perform a rolling restart:

oc rollout restart deployment/athena
oc rollout status deployment/athena --timeout=120s

Total time: 2 to 3 minutes. No downtime. The old pod keeps running until the new one passes health checks.

The result

Same infrastructure. Same error. Completely different analysis:

Before (sre_linux):

Root Cause: DNF package manager reports "No package python3.14 available"
Recommended Action:
1. Verify package name is correct
2. Check repository configuration
3. Consider using EPEL or enabling additional repos

After (sre_package_management):

Root Cause: DNF package manager reports "No package python3.14 available"
Analysis: Python 3.14 requires CRB + EPEL repositories. Host rhel-node-01 belongs
to the AI/ML team content view (aiml-workloads). This content view does not
currently include CRB or EPEL.
Recommended Action:
1. Submit Content View Request per SOP v2.3
2. Request CRB + EPEL enablement for aiml-workloads content view
3. Escalate via #platform-satellite fast-track (2-hour SLA for AI/ML workloads)
Satellite Server: satellite-primary.meridian.internal

The institutional knowledge lived in a markdown file. The agent definition lived in a ConfigMap. The specialist was added without touching the container image.

Example 2: Switching models at runtime

You want to test whether a lighter model (gpt-oss-120b instead of qwen3-235b) can handle specialist analysis at lower compute cost.

Traditional approach:

1. Update Helm values.
2. helm upgrade
3. Wait for image pull and pod restart.
4. Test.
5. Roll back if quality drops.

The OpenShift approach:

# Download updated config with new model
curl -sL https://gitea.example.com/configs/subagents-gpt-oss-120b.yaml \
  -o /tmp/subagents-lighter-model.yaml

# Apply to ConfigMap
oc create configmap athena-agent-config \
  --from-file=subagents.yaml=/tmp/subagents-lighter-model.yaml \
  --from-file=AGENTS.md=<(oc get configmap athena-agent-config -o jsonpath='{.data.AGENTS\.md}') \
  --dry-run=client -o yaml | oc apply -f -

# Rolling restart
oc rollout restart deployment/athena

Test the new model. If quality drops, roll back to the previous ConfigMap revision:

oc rollout undo deployment/athena

One command. Instant rollback. The previous pod configuration (including the original ConfigMap reference) is restored.

Example 3: Changing the orchestrator model

The ops_manager model is configured via an environment variable. To switch from a frontier model to open source:

# Check current model
oc get deployment athena -o jsonpath='{.spec.template.spec.containers[0].env}' \
  | jq '.[] | select(.name == "OPS_MANAGER_MODEL")'

# Switch to open source
oc set env deployment/athena OPS_MANAGER_MODEL=qwen3-235b

# Confirm rollout
oc rollout status deployment/athena --timeout=120s

That's it. No Helm. No ConfigMap editing. No image rebuild. The oc set env command updates the environment variable and triggers a rolling restart automatically.

This is the same command your SRE team uses to reconfigure any application on OpenShift. AI agents aren't special. They're just workloads.

Why this matters: AIOps velocity

Traditional ML/AI deployment pipelines:

Code Change → Build Image → Push to Registry → Update Deployment → Test → Rollback if Bad
(30-60 minutes per iteration)

OpenShift with externalized configuration:

Update ConfigMap or PVC → Rolling Restart → Test → Rollback if Bad
(2-3 minutes per iteration)

When you can iterate in minutes instead of hours, you can:

• Respond to new failure patterns quickly (add a specialist agent the same day).
• A/B test models in production (switch, test, compare, and decide).
• Continuously improve prompts (refine system prompts based on real incident data).
• Inject institutional knowledge as you learn (encode new SOPs immediately).

This is operational agility for AI systems.

Red Hat OpenShift AI integration

Red Hat OpenShift AI provides a Model-as-a-Service (MaaS) endpoint that serves open source models:

┌─────────────────────────────────────┐
│  Athena Agent (your namespace)      │
│  ┌───────────────────────────────┐  │
│  │ ops_manager                   │  │
│  │ └─> HTTP POST to MaaS         │  │
│  └───────────────────────────────┘  │
└─────────────────────────────────────┘
              ↓ HTTPS
┌─────────────────────────────────────┐
│  OpenShift AI MaaS                  │
│  ┌───────────────────────────────┐  │
│  │ vLLM Inference Engine         │  │
│  │ Models: qwen3-235b,           │  │
│  │         gpt-oss-120b,         │  │
│  │         llama-scout-17b, ...  │  │
│  └───────────────────────────────┘  │
│  Running on Nvidia/Intel GPUs       │
└─────────────────────────────────────┘

From your agent's perspective, MaaS is an endpoint. You change models by changing a string in the ConfigMap:

model: qwen3-235b # 235B parameters, strongest reasoning
model: gpt-oss-120b # 120B parameters, good balance
model: gpt-oss-20b # 20B parameters, fast and cheap

Red Hat OpenShift AI handles:

• Model loading and caching.
• GPU scheduling and acceleration (vLLM).
• Request routing and load balancing.
• Resource quotas and multi-tenancy.

You handle:

• Which model to use for which agent.
• What configuration to load.
• When to roll out changes.

Observability: It's still Kubernetes

Because your AI agents run as standard Kubernetes deployments, all your existing observability tooling works:

# Check agent pod logs
oc logs deployment/athena --tail=50 --follow

# Inspect resource usage
oc top pod -l app=athena

# View events
oc get events --field-selector involvedObject.name=athena

# Exec into the pod for debugging
oc exec deployment/athena -it -- /bin/bash

# Check ConfigMap contents
oc get configmap athena-agent-config -o yaml

# List files on the PVC
oc exec deployment/athena -- ls -la /app/skills/

Your SRE team doesn't need to learn "AI observability tools." They use oc logs, oc top, and oc get events (the same commands they use for every other workload).

The rolling restart primitive

One of the most useful patterns is the rolling restart:

oc rollout restart deployment/athena

What happens:

1. Kubernetes spins up a new pod with the updated ConfigMap/env vars.
2. Waits for the new pod to pass readiness checks.
3. Routes traffic to the new pod.
4. Terminates the old pod.

Zero downtime. You just reconfigured an AI operations team in production without dropping a single webhook.

This is what makes AI agents operationally manageable. The platform gives you:

• Immutable rollouts. New config = new pod, old pod stays until new pod is ready.
• Automatic rollback. oc rollout undo if something breaks.
• Health checking. Bad configs don't go live if the pod can't start.
• Audit trail. oc rollout history shows every deployment revision.

Comparison: OpenShift versus other approaches

This table highlights the operational efficiency gained by moving from traditional rebuild cycles to an externalized configuration approach on OpenShift.

The difference isn't speed. It's confidence. When you can roll back in 30 seconds, you experiment more. When experimentation costs 1 hour and requires an image rebuild, you experiment less.

Production checklist

If you're running agentic AI workloads on OpenShift, here's what you should have:

• ConfigMaps for agent definitions (subagent prompts, models, tools, and skills).
• PVCs for skills and artifacts (markdown skills, incident data, and audit logs).
• Environment variables for runtime config (model selection and API endpoints).
• Rolling restart strategy (zero-downtime updates).
• Rollback plan (oc rollout undo tested and documented).
• Health checks (readiness/liveness probes so bad configs don't go live).
• Resource limits (CPU/memory requests and limits per agent).
• Observability (logs aggregated, metrics scraped, and events monitored).

These aren't AI-specific. They're Kubernetes best practices. AI agents are workloads.

Next in this series

This article showed you why OpenShift (how Kubernetes primitives make AI agents operationally manageable). In part 3, we'll dive into skills-driven architecture: how to extend AI agents without writing code, encode institutional knowledge in markdown, and evolve your AIOps capabilities without retraining models.

Try it yourself

Want hands-on experience deploying agentic AI on OpenShift with externalized configuration? Try the complete workshop: Reach out to your Red Hat representative to try the full lab.

Resources

• Red Hat OpenShift
• Red Hat OpenShift AI
• Kubernetes ConfigMaps
• Kubernetes PersistentVolumeClaims
• LangChain Deep Agents

About this article: This post is based on a hands-on workshop created for Red Hat field demonstrations. The architecture patterns described are running in production SRE environments. Reach out to your Red Hat representative to try the full lab.