If you've ever watched your cloud bill skyrocket because a single 15 GB model claimed an entire 80 GB NVIDIA H100 all to itself—or if you've had to wait in a long queue because a colleague locked down a whole GPU node for a light testing task—you know the pain of GPU waste. Standard Kubernetes allocation treats GPUs as an all-or-nothing resource. But when you are running modern models like Llama 3.1 8B, you shouldn't have to choose between massive hardware bills and frustrated developers.
By combining Kubernetes dynamic resource allocation (DRA) with NVIDIA Multi-Instance GPU (MIG) technology, platform teams can establish true hardware-isolated multitenancy. This pairing enables multiple AI workloads to run on a single GPU with hardware-level isolation, guaranteed resources, and no performance interference.
In this guide, we will run two concurrent Llama 3.1 8B inference services on a single NVIDIA H100 GPU. To do this, we will configure dynamic resource allocation (DRA) on Red Hat OpenShift 4.21 alongside the llm-d distributed inference framework. We'll explore how the DRA partitionable devices' shared counter mechanism prevents allocation conflicts while MIG provides physical isolation, strengthening workload isolation and resource efficiency.
MIG and DRA: Partners in GPU sharing
To understand how OpenShift coordinates this resource division, we must first look at how MIG and DRA split compute resources.
NVIDIA MIG: Hardware partitioning
Multi-Instance GPU (MIG) is NVIDIA's technology for physically dividing a GPU into isolated slices. Think of an H100 as a building with 8 floors—MIG lets you rent out individual floors:
- 1g.10gb slices: Small apartments for tiny models (up to 7 tenants)
- 3g.40gb slices: Spacious units for mid-sized models like Llama 8B (2 per GPU)
- 7g.80gb slice: The full penthouse for massive workloads (1 tenant gets everything)
Each tenant gets:
- Dedicated memory: Your memory can't be touched by other workloads.
- Dedicated compute: You get guaranteed streaming multiprocessors.
- Hardware isolation: You receive separate L2 cache partitions and memory controllers.
Dynamic resource allocation: Kubernetes-native scheduling
Traditional GPU allocation has limitations: the Kubernetes device plugin treats GPUs as opaque resources. When requesting "1 GPU," there's no visibility into which MIG slice, which memory partition, or whether the allocation conflicts with another pod.
DRA changes this with shared counters—a budget-tracking system built into the scheduler. An H100 publishes counters for:
Total memory: 81,152 MiB
Memory slices: 8 (numbered 0-7)
Multiprocessors: 132
Copy engines: 8
Decoders: 7When a pod requests a 3g.40gb MIG profile, the scheduler:
- Checks if memory slices 0-3 are available.
- Verifies the counters have enough budget.
- Allocates the device and decrements the counters.
- Binds the slice to the pod.
The next pod requesting 3g.40gb receives slices 4 through7 automatically. No conflicts. No manual tracking. Pure declarative allocation.
Prerequisites
Before running the demo, you need the following components:
Infrastructure
- Red Hat OpenShift 4.21 or later (includes Kubernetes 1.32 with DRA support)
- NVIDIA GPU node with MIG support (H100, A100, or similar)
- Google Cloud Platform account (or any cloud with GPU instances)
- At least 50 GB storage for model caching
Software components
- Node Feature Discovery (NFD): Detects GPU hardware and labels the node
- NVIDIA GPU Operator: Manages drivers and GPU resources
- NVIDIA DRA driver: Implements DRA device allocation
- llm-d: Distributed LLM inference framework (Cloud Native Computing Foundation Sandbox project)
Access tokens
- Hugging Face token for downloading Llama models (free account)
- OpenShift
cluster-adminaccess for enabling alpha features
The following demo sections cover all installation steps.
Enable DRA partitionable devices on your OpenShift cluster
DRA partitionable devices is an alpha feature in Kubernetes 1.34, so we need to enable it explicitly.
Step 1: Enable the feature gate
oc patch FeatureGate cluster --type='json' -p='[
{
"op": "add",
"path": "/spec/featureSet",
"value": "CustomNoUpgrade"
},
{
"op": "add",
"path": "/spec/customNoUpgrade",
"value": {
"enabled": ["DRAPartitionableDevices"]
}
}
]'What this does: Sets the cluster to CustomNoUpgrade mode and enables DRAPartitionableDevices. This triggers a control plane rolling restart (takes 2 to 3 minutes).
Verify it worked:
oc get featuregate cluster -o yaml | grep -A3 customNoUpgradeOutput:
customNoUpgrade:
enabled:
- DRAPartitionableDevicesNote
DRAPartitionableDevices is a Technology Preview feature in OpenShift Container Platform 4.22, and you can enable it with the TechPreviewNoUpgrade feature set. OpenShift Container Platform 5.0+ clusters enable it by default without requiring a feature gate.
Step 2: Install Node Feature Discovery
Via the OpenShift web console:
- Navigate to Operators → OperatorHub and search for
Node Feature Discovery. - Select Install → Accept defaults → Install.
Create an NFD instance:
cat <<EOF | oc apply -f -
apiVersion: nfd.openshift.io/v1
kind: NodeFeatureDiscovery
metadata:
name: nfd-instance
namespace: openshift-nfd
spec:
operand:
image: registry.redhat.io/openshift4/ose-node-feature-discovery:v4.21
servicePort: 12000
EOFStep 3: Install NVIDIA GPU Operator
Via the web console:
- Operators → OperatorHub and search for
NVIDIA GPU Operator. - Install from the stable channel.
Create the ClusterPolicy:
cat <<EOF | oc apply -f -
apiVersion: nvidia.com/v1
kind: ClusterPolicy
metadata:
name: gpu-cluster-policy
spec:
cdi:
enabled: true
default: false
driver:
enabled: true
use_ocp_driver_toolkit: true
upgradePolicy:
autoUpgrade: true
maxParallelUpgrades: 1
maxUnavailable: 25%
devicePlugin:
enabled: false # DISABLED for DRA
gfd:
enabled: true
dcgm:
enabled: true
dcgmExporter:
enabled: true
serviceMonitor:
enabled: true
toolkit:
enabled: true
installDir: /usr/local/nvidia
mig:
strategy: none
migManager:
enabled: false # DRA driver handles dynamic MIG
ccManager:
enabled: true
vfioManager:
enabled: true
sandboxDevicePlugin:
enabled: true
kataSandboxDevicePlugin:
enabled: true
nodeStatusExporter:
enabled: true
operator:
use_ocp_driver_toolkit: true
runtimeClass: nvidia
EOFImportant: Setting devicePlugin.enabled: false prevents conflicts with DRA.
Wait for driver installation:
watch -n 5 'oc get pods -n nvidia-gpu-operator'All pods should reach Running status (takes 3 to 5 minutes).
Step 4: Install NVIDIA DRA driver
helm install nvidia-dra-driver nvidia/nvidia-dra-driver-gpu \
--create-namespace \
--namespace nvidia-dra-driver \
--version="25.12.0" \
--set nvidiaDriverRoot=/run/nvidia/driver \
--set resources.gpus.enabled=true \
--set gpuResourcesEnabledOverride=true \
--set resources.computeDomains.enabled=false \
--set featureGates.DynamicMIG=true \
--wait --timeout=10mVerify the driver published DeviceClasses resources:
oc get deviceclassOutput:
NAME DRIVER
gpu.nvidia.com dra.nvidia.com
mig.nvidia.com dra.nvidia.com
vfio.gpu.nvidia.com dra.nvidia.comCheck ResourceSlices (these contain the shared counter budgets):
oc get resourcesliceThe output shows a resource slice for the GPU node containing device inventory.
One-time setup: Hugging Face token
Create a secret with your Hugging Face token for model downloads:
oc create secret generic huggingface-token \
--from-literal=token=hf_YOUR_TOKEN_HERE \
-n llm-inferenceHow to get a token:
- Sign up on Hugging Face.
- Go to Settings → Access Tokens.
- Create a Read token.
- Accept the Llama 3.1 model license on Hugging Face.
Demo: 2 Llama 3.1 models on 1 H100
This section demonstrates running 2 completely isolated Llama 3.1 8B inference services on a single GPU by integrating DRA with llm-d's deployment framework.
Architecture overview
Our stack:
- Red Hat OpenShift 4.21 on Google Cloud
- NVIDIA H100 80 GB GPU (a3-highgpu-1g instance)
- llm-d router: Intelligent request routing with prefix-cache awareness
- 2 vLLM model servers: Each running Llama 3.1 8B Instruct on a 3g.40gb MIG slice
- Gateway API v1.5.0: For inference routing and model management
Each model server gets:
- 40 GB dedicated GPU memory (the 3g.40gb MIG profile)
- 60 dedicated streaming multiprocessors
- Memory slices 0-3 or 4-7 (hardware isolation)
Why Llama 3.1 8B with 3g.40gb MIG slices? The model requires approximately 15 GB for weights and activations when using FP16 precision. The 3g.40gb MIG profile provides 40 GB total capacity, leaving about 25 GB for key-value (KV) cache during inference, enabling efficient handling of longer context windows and concurrent requests. This configuration allows 2 inference replicas to run on a single H100 GPU with complete hardware isolation.
Step 1: Create namespace
oc create namespace llm-inferenceStep 2: Install Gateway API Inference Extension CRDs
llm-d uses the Gateway API Inference Extension to manage inference routing, model discovery, and pool management. These custom resource definitions (CRDs) define resources like InferenceModel and InferencePool that llm-d's router uses to distribute requests across backend model servers.
Install the CRDs:
oc apply -f https://github.com/kubernetes-sigs/gateway-api-inference-extension/releases/download/v1.5.0/v1-manifests.yamlVerify the installed CRDs:
oc get crd | grep gatewayThe expected output includes:
gatewayclasses.gateway.networking.k8s.io
gateways.gateway.networking.k8s.io
httproutes.gateway.networking.k8s.io
inferencemodels.inference.networking.x-k8s.io
inferencepools.inference.networking.x-k8s.ioStep 3: Deploy llm-d router
# Clone the llm-d repository if not already done
git clone https://github.com/llm-d/llm-d.git
cd llm-d
# Install the router using Helm
helm install optimized-baseline \
oci://ghcr.io/llm-d/charts/llm-d-router-standalone-dev \
-f guides/recipes/router/base.values.yaml \
-f guides/optimized-baseline/router/optimized-baseline.values.yaml \
-n llm-inference \
--version v0Verify the router is running:
oc get pods -n llm-inference -l app.kubernetes.io/name=llm-d-routerThe expected output:
NAME READY STATUS RESTARTS AGE
optimized-baseline-epp-67b9c8f9d4-x8k2m 1/1 Running 0 30sStep 4: Deploy model servers with DRA
To integrate DRA with llm-d, we create a custom deployment overlay using Kustomize. This approach extends llm-d's base vLLM deployment with DRA-specific configurations.
Create the DRA deployment directory structure:
mkdir -p guides/optimized-baseline/modelserver/gpu/vllm/ocp-dra/
cd guides/optimized-baseline/modelserver/gpu/vllm/ocp-dra/Create the following files:
resourceclaim-template.yamldefines the MIG slice request:apiVersion: resource.k8s.io/v1 kind: ResourceClaimTemplate metadata: name: optimized-baseline-nvidia-gpu-vllm-llm-d-mig-gpu spec: spec: devices: requests: - name: mig-slice exactly: deviceClassName: mig.nvidia.com selectors: - cel: expression: "device.attributes['gpu.nvidia.com'].profile == '3g.40gb'" count: 1remove-gpu-limits.yamlremoves traditional GPU resource limits and requests:- op: remove path: /spec/template/spec/containers/0/resources/limits/nvidia.com~1gpu - op: remove path: /spec/template/spec/containers/0/resources/requests/nvidia.com~1gpupatch-ocp-dra.yamlis the strategic merge patch with full deployment configuration:apiVersion: apps/v1 kind: Deployment metadata: name: decode spec: replicas: 2 # Two replicas on single H100 template: spec: containers: - name: modelserver # Llama 3.1 8B model configuration args: - "meta-llama/Meta-Llama-3.1-8B-Instruct" - "--disable-access-log-for-endpoints=/health,/metrics,/v1/models" - "--tensor-parallel-size=1" - "--gpu-memory-utilization=0.9" - "--max-model-len=4096" env: - name: HF_TOKEN valueFrom: secretKeyRef: name: llm-d-hf-token key: HF_TOKEN # OpenShift compatibility - handle random UID - name: HOME value: /tmp - name: USER value: vllm - name: LOGNAME value: vllm - name: TORCHINDUCTOR_CACHE_DIR value: /tmp/torch-cache - name: TRITON_CACHE_DIR value: /tmp/triton-cache resources: limits: cpu: '8' memory: 64Gi requests: cpu: '4' memory: 32Gi # Reference DRA claim claims: - name: gpu-claim # Add DRA ResourceClaim at pod level resourceClaims: - name: gpu-claim resourceClaimTemplateName: optimized-baseline-nvidia-gpu-vllm-llm-d-mig-gpukustomization.yamlorchestrates the overlay:apiVersion: kustomize.config.k8s.io/v1beta1 kind: Kustomization # Build on top of base vLLM configuration resources: - ../base - resourceclaim-template.yaml # Override with OpenShift + DRA specific configuration patches: - path: patch-ocp-dra.yaml - path: remove-gpu-limits.yaml target: kind: Deployment name: decode # Use fully qualified image name for OpenShift images: - name: vllm/vllm-openai newName: docker.io/vllm/vllm-openai newTag: v0.19.1
Deploy the vLLM model servers with DRA:
oc apply -n llm-inference -k guides/optimized-baseline/modelserver/gpu/vllm/ocp-dra/This command deploys:
- A
ResourceClaimTemplaterequesting 3g.40gb MIG slices - A
Deploymentwith 2 replicas (each claiming a separate MIG slice) - A
Servicefor load balancing across replicas InferencePoolandInferenceModelcustom resources for llm-d routing
Watch the deployment progress:
watch -n 5 'oc get pods,resourceclaim -n llm-inference'The expected deployment timeline:
- 0 to 2 minutes: Image pull (vLLM image is approximately 22 GB)
- 2 to 3 minutes: Model download (Llama 3.1 8B weights from Hugging Face)
- 3 to 4 minutes: Model loading into GPU memory
Once ready, you'll see:
NAME READY STATUS RESTARTS AGE
pod/optimized-baseline-epp-67b9c8f9d4-x8k2m 1/1 Running 0 5m
pod/optimized-baseline-nvidia-gpu-vllm-decode-84dn6 1/1 Running 0 4m
pod/optimized-baseline-nvidia-gpu-vllm-decode-xr72n 1/1 Running 0 4m
NAME ALLOCATED RESERVED
resourceclaim/optimized-baseline-nvidia-gpu-vllm-decode-84dn6-gpu-claim true true
resourceclaim/optimized-baseline-nvidia-gpu-vllm-decode-xr72n-gpu-claim true trueNotably, both ResourceClaim resources show ALLOCATED and RESERVED, indicating the scheduler successfully allocated 2 non-conflicting MIG slices.
Step 5: Verify device isolation
To confirm each pod received a different MIG slice:
oc describe resourceclaim -n llm-inference | grep -E "Name:|Device:|State:"Output:
Name: optimized-baseline-nvidia-gpu-vllm-decode-84dn6-gpu-claim
Device: gpu-0-mig-3g40gb-9-0
State: allocated,reserved
Name: optimized-baseline-nvidia-gpu-vllm-decode-xr72n-gpu-claim
Device: gpu-0-mig-3g40gb-9-4
State: allocated,reservedThis is proof of isolation:
- First pod:
gpu-0-mig-3g40gb-9-0(uses memory slices 0 through 3) - Second pod:
gpu-0-mig-3g40gb-9-4(uses memory slices 4 through 7)
Different device IDs mean different hardware partitions. Zero memory overlap.
Check the model server logs to verify successful initialization:
oc logs -n llm-inference -l app=vllm --tail=20Look for messages indicating:
- Model loaded successfully
- vLLM server started on port 8000
- Ready to accept inference requests
Step 6: Test concurrent inference
Port-forward to both model server pods:
# Get pod names
POD1=$(oc get pods -n llm-inference -l app=vllm -o jsonpath='{.items[0].metadata.name}')
POD2=$(oc get pods -n llm-inference -l app=vllm -o jsonpath='{.items[1].metadata.name}')
# Port-forward to both pods
oc port-forward -n llm-inference pod/${POD1} 8001:8000 &
oc port-forward -n llm-inference pod/${POD2} 8002:8000 &Test pod 1:
curl -X POST http://localhost:8001/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "meta-llama/Meta-Llama-3.1-8B-Instruct",
"messages": [
{
"role": "user",
"content": "Explain Kubernetes Dynamic Resource Allocation in one sentence."
}
],
"max_tokens": 100
}' | jq -r '.choices[0].message.content'Response:
Dynamic Resource Allocation in Kubernetes is a feature that allows the cluster
to automatically manage and adjust resources allocated to containers based on
current workload demands, helping optimize resource utilization and preventing
resource contention.Test pod 2 simultaneously:
curl -X POST http://localhost:8002/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "meta-llama/Meta-Llama-3.1-8B-Instruct",
"messages": [
{
"role": "user",
"content": "What is NVIDIA MIG in one sentence?"
}
],
"max_tokens": 100
}' | jq -r '.choices[0].message.content'Response:
NVIDIA Multi-Instance GPU (MIG) is a technology that allows a single NVIDIA GPU
to be partitioned into multiple isolated instances, each with dedicated memory
and compute resources, enabling multiple workloads to run concurrently on the
same physical GPU.Result: Two completely isolated LLM services running concurrently on a single H100, each serving inference requests without interference. This demonstrates DRA's ability to enable a more efficient, security-focused GPU multitenancy environment with hardware-level isolation through MIG.
What's happening behind the scenes
When the deployment creates 2 pods requesting 3g.40gb MIG slices, the DRA scheduler automatically:
- First pod: Allocates
gpu-0-mig-3g40gb-9-0(uses memory slices 0 through 3). - Second pod: Allocates
gpu-0-mig-3g40gb-9-4(uses memory slices 4 through 7).
The scheduler prevents overlap by tracking which memory slices each allocation consumes. This prevents conflicts and enables hardware-isolated concurrent allocation across multiple pods.
Viewing actual MIG configuration
Exec into the NVIDIA driver DaemonSet pod to run nvidia-smi:
# Get the NVIDIA driver pod name (look for pods starting with nvidia-driver-daemonset)
DRIVER_POD=$(oc get pods -n nvidia-gpu-operator | grep nvidia-driver-daemonset | awk '{print $1}' | head -1)
# Exec into the pod and run nvidia-smi
oc exec -n nvidia-gpu-operator ${DRIVER_POD} -- nvidia-smiThe output shows two active MIG instances:
+-----------------------------------------------------------------------------+
| MIG devices: |
+------------------+----------------------+-----------+-----------------------+
| GPU GI CI MIG | Memory-Usage | Vol| Shared |
| ID ID Dev | BAR1-Usage | SM Unc| CE ENC DEC OFA JPG|
| | | ECC| |
|==================+======================+===========+=======================|
| 0 9 0 0 | 15023MiB / 40448MiB | 60 0 | 3 0 3 0 0 |
| | 0MiB / 65535MiB | | |
+------------------+----------------------+-----------+-----------------------+
| 0 9 4 1 | 15023MiB / 40448MiB | 60 0 | 3 0 3 0 0 |
| | 0MiB / 65535MiB | | |
+------------------+----------------------+-----------+-----------------------+
+-----------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=============================================================================|
| 0 9 0 12345 C python3 14.6GiB |
| 0 9 4 12389 C python3 14.6GiB |
+-----------------------------------------------------------------------------+Main observations:
- 2 MIG instances
- Each using about 15 GB for the loaded Llama 3.1 8B model
- Each has 40,448 MiB total capacity (about 40 GB)
- Different process IDs (complete isolation)
- Remaining memory (about 25 GB per slice) available for key-value (KV) cache during inference
The value: Better resource use, lower costs
Beyond the technical benefits, DRA delivers significant operational value. Consider a scenario where multiple teams need to run models with similar resource requirements to our Llama 3.1 8B example (about 15 GB per model). With traditional GPU allocation, each team would need a dedicated H100 GPU. With DRA's multitenancy and the 3g.40gb MIG profile, you can run 2 such workloads per GPU—effectively doubling your infrastructure efficiency.
This improved resource use translates to lower infrastructure costs, reduced power consumption, and more efficient use of expensive GPU resources. The exact gains depend on your specific models and workload patterns, but the principle remains: share GPUs safely when workloads don't need the full capacity.
