Integrate Claude Code with Red Hat AI Inference Server on OpenShift

Agentic coding tools help developers build software efficiently. Claude Code, Anthropic's terminal-based coding agent, improves productivity by letting you interact with your codebase through natural language—directly from the console.

One advantage of Claude Code is its flexibility. Rather than being locked to Anthropic's cloud models, you can connect it to any backend that uses the Anthropic Messages API. 

This article explores how to integrate Claude Code with a local model served by Red Hat AI Inference Server (a downstream version of vLLM) on Red Hat OpenShift. This approach keeps the inference process private on your infrastructure while retaining the full Claude Code workflow. By doing so, you keep all prompts and responses within your environment while benefiting from Claude Code's developer-focused workflows.

Prerequisites

You will need:

• An OpenShift cluster with GPUs enabled and the NVIDIA Operator installed. For a local OpenShift installation, follow the steps in How to enable NVIDIA GPU acceleration in OpenShift Local.
• A Hugging Face account and active API token.
• Access to the Red Hat image registry.

Environment

I executed the steps in this article using an environment with the following specifications:

• Single-node OpenShift 4.21
• GPU: NVIDIA RTX 4060 Ti
• CPU: Intel Core i7-14700 × 28
• Host machine operating system: Fedora 43

Deploy the Red Hat AI Inference Server

The first step is to deploy Red Hat AI Inference Server. For this demo, I created a Helm chart to simplify the deployment in an OpenShift 4.21 environment. You can alternatively follow the manual deployment procedure. 

Clone the project:

git clone https://github.com/alexbarbosa1989/rhai-helm

Set the minimal required environment variables:

export HF_TOKEN=hf_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
export AUTHFILE=$XDG_RUNTIME_DIR/containers/auth.json
export STORAGECLASS=<ocp-storageclass>

Alternatively, you can configure your own values in the rhai-helm/values.yaml file—for example, a different model from Hugging Face or a custom namespace.

Hint: Before setting AUTHFILE, verify whether auth.json already exists at the expected path. This file is created automatically when you authenticate using Podman in the terminal.

podman login registry.redhat.io

Once you define the required environment variables, you can install the Helm chart. For example, to use the default rhai-helm/values.yaml, run:

helm install rhai-helm ./rhai-helm \
--create-namespace --namespace rhai-helm \
--set persistence.storageClass=$STORAGECLASS \
--set secrets.hfToken=$HF_TOKEN \
--set-file secrets.docker.dockercfg=$AUTHFILE

Check the created resources:

oc get secrets
oc get pvc model-cache
oc get deployment
oc get svc
oc get route

Finally, check the running pod. This might take a few minutes, depending on hardware resources.

oc get pod

NAME                        READY   STATUS    RESTARTS   AGE
qwen-coder-5f6668b767-hp585   1/1     Running   0          5m11s

Install and configure Claude Code

Configure this on your developer workstation. Follow the official installation instructions, or install it directly using the convenience script for Linux and macOS:

curl -fsSL https://claude.ai/install.sh | bash

Claude Code uses environment variables for configuration. By overriding the default Anthropic settings, you can redirect requests to a local model served by vLLM. Use this example configuration:

ANTHROPIC_BASE_URL="<RHAI-Inference-exposed-route>" \
ANTHROPIC_API_KEY="vllm" \
ANTHROPIC_DEFAULT_OPUS_MODEL="qwen-coder" \
ANTHROPIC_DEFAULT_SONNET_MODEL="qwen-coder" \
ANTHROPIC_DEFAULT_HAIKU_MODEL="qwen-coder" \
CLAUDE_CODE_FILE_READ_MAX_OUTPUT_TOKENS="2000" \
CLAUDE_CODE_MAX_OUTPUT_TOKENS="4096" \
MAX_THINKING_TOKENS="0" \
claude

The ANTHROPIC_BASE_URL environment variable must point to the exposed OpenShift route of the Red Hat AI inference service. This is the endpoint Claude Code uses for all requests. 

Replace the example value with the route generated in your OpenShift cluster. Retrieve the route by running:

oc get route -n <namespace>

Also, the values for CLAUDE_CODE_FILE_READ_MAX_OUTPUT_TOKENS and CLAUDE_CODE_MAX_OUTPUT_TOKENS should be tuned according the hardware capabilities to avoid exhausting the context window.

Once you set the environment variables, launching Claude Code prompts an interactive setup to initialize the workspace (Figure 1).

Select ❯ 1. Yes, I trust this folder. At this point, Claude Code is fully initialized and ready for use, as shown in Figure 2.

In this example, the following instruction was provided:

❯ create a basic quarkus "hello" service

Claude Code immediately begins processing the request using the locally served model, as illustrated in Figure 3.

You can also verify the interaction directly from the vLLM backend pod in the OpenShift cluster. Successful requests appear in the logs as calls to the /v1/messages API endpoint:

(APIServer pid=1) INFO:     10.128.0.2:43662 - "POST /v1/messages?beta=true HTTP/1.1" 200 OK
(APIServer pid=1) INFO:     10.128.0.2:43664 - "POST /v1/messages?beta=true HTTP/1.1" 200 OK

This confirms that Claude Code successfully routes requests to the OpenShift-hosted inference service.

Key takeaways

By integrating Claude Code with a vLLM-based inference service on OpenShift, you gain access to effective AI-assisted coding workflows while keeping models, data, and inference under your control.

This demonstration uses a lightweight Qwen model. With specialized, higher-performance hardware, you can serve larger models that provide advanced coding and reasoning capabilities.

Overall, this approach combines the productivity of Claude Code with the security and scalability of OpenShift. It is a practical solution for organizations that need private, on-premises AI development environments.