Benchmarking tiered KV cache offloading on H100 GPUs in OpenShift AI 3.5

Long-context workloads are quickly becoming the norm for enterprise AI. Agentic applications carry an ever-growing history of tool calls and intermediate results, retrieval-augmented generation prepends thousands of tokens of retrieved context, and multi-turn assistants resend the whole conversation with every turn. In all of these cases, the model sees large parts of the same prompt again and again.

That repetition should be cheap, because the model has already done the work once. In practice, it often isn't. The state that makes reuse possible, the KV cache, lives in GPU memory, and GPU memory runs out long before the context does. When it runs out, the work is discarded and recomputed from scratch the next time the same prefix shows up.

In Red Hat OpenShift AI 3.5, the inference platform offloads that state from GPU memory into cheaper, higher-capacity tiers: CPU DRAM, local NVMe, or any POSIX-compliant distributed and parallel file system. In this blog post, we walk through how we evaluated those tiers (using CephFS as our shared file system throughout), starting from raw storage latency, translating it into a break-even context length, and finally measuring what offloading does to throughput and latency on a real long-context workload with 3 models on NVIDIA H100 GPUs. We close with recommendations for each deployment topology and a procedure to validate them on your own workload.

Executive summary

On representative agentic workloads averaging roughly 50K input tokens, tiered KV cache offloading improved throughput by up to 5.8x and reduced mean time to first token (TTFT) from 60 s to 10 s for Nemotron-Ultra-253B on NVIDIA H100 GPUs. The magnitude of the gain depended on whether the model's KV working set exceeded the capacity of the CPU offload tier. When it did, a secondary storage tier was essential: CPU offload alone delivered 3.3x; adding NVMe or CephFS nearly doubled that to 5.8x. When the working set fit within the CPU tier, CPU offload alone captured most of the benefit.

Across our tests, CPU DRAM provided the first and most efficient extension beyond GPU HBM. Local NVMe delivered the lowest secondary-tier latency and is well suited to single-node deployments where cache locality can be preserved. CephFS introduced higher per-block latency, but its shared capacity, persistence, and cross-replica reuse became valuable in multi-node deployments and under high concurrency. In multi-replica tests, CephFS sustained cache reuse as concurrency increased while CPU-only configurations increasingly fell back to recomputation.

The practical recommendation is workload-driven evaluation:

1. Characterize storage performance by measuring p99 latency and throughput for each candidate tier under representative KV cache access patterns.
2. Calculate the restore-versus-recompute crossover point by quantifying the conditions under which KV cache restoration provides lower latency or cost than recomputation.
3. Validate the tier configuration experimentally using representative inference traces, workloads, and concurrency levels to assess end-to-end performance before production deployment.

What the KV cache holds, and why it fills up

To see why offloading helps, it's worth looking at how an LLM generates text. Inference runs in 2 distinct phases with very different cost profiles:

• Prefill processes the entire input prompt in one forward pass. Every token attends to every token before it, so the attention cost grows with the square of the context length. Prefill is compute-bound, and it is where TTFT comes from.
• Decode then generates output tokens one at a time. Each new token attends to all preceding tokens, but only has to compute its own query. Decode is memory-bandwidth-bound and comparatively cheap per token.

The catch is size. The KV cache grows linearly with context length, and its footprint per token depends on the number of attention layers, KV heads, head dimension, and data type (we provide the exact formula in the storage section). As an illustration, Llama 3.1 70B in FP8 needs about 5 MB per 16-token block across the server, or roughly 0.3 MB per token. A single 128K-token context therefore occupies about 40 GB of KV cache, before counting any other concurrent request, and the same GPU high-bandwidth memory (HBM) also has to hold the model weights. Under high concurrency, that pool of KV blocks fills quickly, and what happens next decides how well long-context inference performs.

Why KV cache capacity limits long-context inference

GPU HBM is the most expensive resource in the inference system, and vLLM's KV cache is one of its largest consumers. When the GPU KV cache pool fills, the system must either evict blocks and recompute the prefix later or move evicted blocks to a lower-cost tier and restore them on a cache hit. Your architectural decision is concrete: is restoring a KV block from the chosen storage tier faster than recomputing that block's prefix on the GPU?

If restoration is faster, offloading can reduce TTFT, increase throughput, and make larger context windows practical. If restoration is slower, offloading adds latency without recovering enough GPU work. The answer depends on model block size, concurrency, storage tail latency, context length, and the amount of KV data that can be retained outside HBM.

We refer to the KV cache footprint that concurrent requests are expected to access or reuse over a given time window as the working set. As KV blocks are produced, they can be asynchronously propagated across memory tiers. When the working set exceeds a tier’s capacity, that tier’s retention and eviction policies determine which blocks remain available at that level, while lower tiers can retain evicted blocks for future reuse and avoid recomputation.

The architectural challenge

Offloading only helps if the tier it uses fits the model, the workload, and the deployment. Here is the architecture we tested and the decisions it needs to support.

The proposed tiered architecture

The tested pattern uses GPU HBM as the active tier, CPU DRAM as a recently evicted tier, and local NVMe or CephFS as a larger cold tier. CPU offloading moves KV cache blocks from GPU HBM to system CPU DRAM, providing relatively low-latency access while freeing GPU memory. Local NVMe disk offloading moves KV cache blocks from GPU memory, and optionally CPU memory, to a locally attached NVMe SSD, providing substantially greater capacity at higher access latency. CephFS offloading stores KV cache blocks in a shared and distributed file system, allowing cache data to be accessed across inference replicas and providing larger shared capacity, but with additional network and file system overhead. vLLM's native file system offload connector writes KV blocks as individual files and restores them on demand (per the upstream multi-tier offloading work RFC #38260 and the object-store secondary tier).

Key architectural decisions

• Which storage tier is fast enough for your model and context distribution?
• At what context length does storage restore beat prefill recomputation?
• Which deployment topology should use CPU DRAM, local NVMe, CephFS, or a combination?

This blog post provides 2 linked measurements. FIO characterizes the storage cost of restoring a KV block under vLLM-like I/O. The inference benchmarking then measures whether that storage cost translates into lower TTFT and higher throughput under representative workloads.

The rest of this post follows these 3 questions in order: first the storage tier on its own, then its effect on inference, and finally what it means for each deployment topology.

Step 1: Measuring the storage cost with FIO

Before running a single inference request, we wanted to know how quickly each candidate tier can hand a KV block back to vLLM. That is a storage question, and FIO is the natural tool to answer it, as long as it is configured to look like the connector's real I/O pattern.

Why sequential bandwidth is not enough

Standard sequential 1 MB reads measure a bandwidth ceiling, but they do not reproduce the connector's critical path. vLLM manages fixed 16-token KV blocks. The byte size changes with model architecture, tensor parallelism, KV heads, head dimension, and data type. The connector also performs direct synchronous writes and concurrent reads and writes. The useful storage metric is therefore per-operation latency, especially p99 latency, under the same block size and concurrency that inference experiences.

For a tensor-parallel rank, the block size is approximated by:

bytes_{per token} = 2 · L_attn · (H_attn / TP) · d_head · b_kv

In this formula, L_attn represents attention layers, H_attn represents KV heads, TP is the tensor-parallel degree, d_head is the head dimension, and b_kv is bytes per element (that is, the precision). The leading 2 accounts for keys and values. Representative server-wide block sizes were approximately 1.20 MB for gpt-oss-120B, 2.25 MB for Qwen3-8B, 4.00 MB for Qwen3-32B-AWQ, and 5.00 MB for Llama 3.1 70B FP8.

Benchmark design

The benchmark compares a local PCIe Gen 4 NVMe reference against raw VPC block storage, CephFS, and Ceph RGW/S3 on Red Hat OpenShift and IBM Cloud VPC. The principal workload uses direct I/O, 16 jobs, and a mixed 75/25 read/write ratio to approximate steady-state serving. Representative command and typical models:

​​fio --rw=randrw --rwmixread=75 --bs=<model_block_size> --iodepth=1 --numjobs=16 --direct=1 --ioengine=libaio --runtime=60

Local NVMe provides the lowest and most stable latency. Raw VPC block has a predictable p99 close to p50 and saturates near 762 MB/s in the source tests. CephFS carries a protocol overhead ceiling of roughly 205 MB/s, with latency several times higher than raw VPC block. RGW/S3 can look competitive at p50, but its p99 tail reaches 1.13 seconds for the 5 MB block. Reference: natoscott/benchmarks storage-offload-eval report.

How to interpret the FIO results

FIO is not itself an inference benchmarking tool; it is the storage-side input to the crossover calculation. A tier is viable when its p99 restore cost is lower than the prefill work avoided by a cache hit.

Step 2: Where restoring beats recomputing

The crossover method

The crossover context length is the context at which storage restore and prefill recomputation have equal cost:

L_crossover = T_FIO,p99 / t_prefill   where   t_prefill = TTFT / context length

These thresholds are illustrative, not universal constants. Figure 1 plots the crossover context length L_crossover as a function of the prefill cost per token t_prefill across fixed storage tail latencies T_FIO,p99.

Step 3: Inference benchmarks with KV cache offloading

Benchmark configuration and workload

The benchmarking used Red Hat OpenShift AI 3.5, which comes with vLLM 0.24.0 and llm-d 0.8.0 for distributed inference, on an a Red Hat OpenShift 4.22.2 cluster with 8xH100 nodes. We deployed Qwen3.6-35B-A3B (MoE) on 2 H100 GPUs, Nemotron-Ultra-253B FP8 on 8 H100 GPUs, and Gemma 4 31B on 2 H100 GPUs.

Each deployment was given 300 GB of shared memory for the CPU offload tier. Within that, we set the connector's cpu_bytes_to_use to 64 GB for Qwen, 256 GB for Nemotron, and 64 GB for Gemma. The local NVMe and CephFS configurations add a secondary tier on top of that same CPU allocation. Local NVMe offload runs used a 7 TB Micron 7450 PRO NVMe SSD. CephFS offload runs used a 3 TiB PersistentVolumeClaim (PVC) backed by the CephFS storage class. The CephFS cache tier was built on Red Hat OpenShift Data Foundation 4.18, deployed as a 3-OSD cluster with each OSD on a dedicated VPC block device. KV cache data pools used a size of 1, storing a single copy of each cached block on 1 OSD, so the loss of that OSD means the affected cache entries are missed and recomputed by the GPU during recovery. This minimal-replication (size 1) configuration was the basis for all CephFS offload benchmark results in this article.

We evaluated the system and the different scenarios using AIPerf's AgentX MVP scenario and the SemiAnalysis Weka trace dataset (semianalysisai/cc-traces-weka-062126). Rather than generating independent synthetic prompts, AIPerf reconstructs and replays realistic multi-turn agent trajectories from these traces, preserving the long conversational histories and branching behavior characteristic of agentic applications. The workload used a maximum context length of 131,072 tokens, ran for 30 minutes with seed 42, and produced a strongly prefill-dominated request mix: average input length was approximately 50,600 tokens, while average output length was roughly 691 tokens.

Results

Throughout this section, we report throughput in completed requests per second, mean and 95th-percentile time to first token (TTFT), mean end-to-end (E2E) latency per request, and the external cache hit rate, which measures how often a prefix is found in an offload tier instead of being recomputed. We compare 4 configurations: no offload, CPU offload, CPU offload + local NVMe (called local NVME offload), and CPU offload + CephFS (called CephFS offload).

How offloading gains vary with working-set size

Across our 3 models, the throughput gains varied with one factor most directly: whether the model's KV working set exceeded the CPU tier's capacity.

• Nemotron-Ultra-253B required more KV cache capacity than the 256 GiB CPU tier could retain. Adding a secondary tier (NVMe or CephFS) nearly doubled the throughput gain over CPU offload alone.
• Qwen3.6-35B-A3B fits within its 64 GiB CPU allocation. CPU offload captured roughly 90% of the available gain, and a secondary tier added incremental improvement.
• Gemma 4 31B had high GPU memory utilization that constrained the offload tiers before they could fully engage. Offloading improved throughput, but GPU memory utilization limited the gains.

The following sections walk through each case, starting with the model that showed the greatest throughput gain from tiered offloading.

Nemotron-Ultra-253B: Working set exceeds CPU tier capacity

We deployed Nemotron-Ultra-253B FP8 across 8 H100 GPUs with a 256 GiB CPU offload tier. At approximately 50K average input tokens, the model's KV cache demands exceeded what the CPU tier could retain, so the GPU recomputed the full prefix on every cache miss.

In the single-replica test, Nemotron served 0.13 req/s with a mean TTFT of 59.7 s and mean E2E latency of 228.7 s without offload (Figure 2). CPU offload alone improved throughput to 0.44 req/s (3.3x). Adding a secondary tier nearly doubled that gain: local NVMe reached 0.76 req/s (5.7x) and CephFS reached 0.77 req/s (5.8x). Mean TTFT dropped to 9.6 s with either secondary tier, and mean E2E latency fell below 26 s.

The gap between CPU-only and CPU + secondary tier reflects working-set overflow. The CPU tier retained the most recently evicted blocks, but the Weka AgentX trace also revisited older blocks that the CPU tier had already evicted. NVMe or CephFS tiers retained those blocks, serving them as cache hits and avoiding recomputation.

With 2 replicas on separate nodes at matched concurrency, all 3 offload configurations delivered similar throughput (Figure 3). At this load, GPU and CPU memory were sufficient to retain the reusable working set, so the shared tier had little additional cache to contribute. CephFS throughput increased from 0.77 to 1.17 req/s (+53%), with mean TTFT dropping from 9.6 s to 1.8 s (5.3x faster). The concurrency scaling results in Figure 4 show where this changes: as concurrent requests grow, the CPU tier fills and the shared tier's capacity starts to matter.

Under concurrency scaling tests (Figure 4), at C=32, all 3 configurations delivered similar throughput (1.04 to 1.18 req/s), and external hit rates were still low, likely because GPU memory still served most reuse at that load. As concurrency increased, the configurations diverged:

• At C=64, the no-offload baseline fell to 0.28 req/s. CPU offload sustained 0.73 req/s. CephFS offload held at 1.06 req/s and reduced p95 TTFT to 27.7 s, compared with 52.4 s for CPU offload and 171.4 s without offloading.
• At C=128, the CPU tier's external cache hit rate dropped to 1.5% while CephFS offload still delivered 59%. The 256 GiB CPU tier could not retain the working set once enough concurrent long-context requests competed for it. CephFS offload sustained 0.46 req/s, roughly twice the throughput of either alternative.

At C=128, p95 TTFT exceeded 400 s in every configuration, a sign that the 2-replica deployment was saturated at that concurrency and additional replicas would be needed.

Tier capacity and loading rates from the inference benchmarking

Qwen3.6-35B-A3B: Working set fits within CPU tier

We deployed Qwen 3.6-35B-A3B (Mixture of Experts) on 2 H100 GPUs with a 64 GiB CPU offload tier. The model's KV working set fits within this allocation, so the secondary tier has less to contribute.

For a single replica, CPU offload alone improved throughput from 0.58 to 1.17 req/s (2x), with mean TTFT falling from 34.1 s to 15.4 s (Figure 2). Adding local NVMe or CephFS brought throughput to 1.29 and 1.28 req/s respectively, roughly 10% beyond CPU-only.

With 2 replicas,, all offload configurations performed similarly at matched concurrency. CephFS throughput improved from 1.28 to 1.72 req/s (+34%), with mean TTFT dropping from 10.9 s to 0.6 s (18x faster) and E2E from 20.6 s to 8.1 s (2.5x faster) (Figure 3). Even when the CPU tier is sufficient on a single replica, the shared cache still provides multi-replica scaling benefits.

Under concurrency scaling tests (Figure 5), both offload configurations held up as concurrency grew, rising from about 1.7 req/s at C=32 to 1.9-2.0 req/s at C=64 and C=128, while the no-offload baseline fell from 1.30 to about 1.0 req/s. CPU offload and CephFS tracked each other closely, confirming that the CPU tier captured most of Qwen's reusable working set. The difference appeared in cache hit rates: CephFS stayed between 75% and 79% at every concurrency level, while CPU offload dropped from 73% to 63% at C=128, suggesting the CPU tier was beginning to run low on capacity.

Gemma 4 31B: GPU memory as the binding constraint

We deployed Gemma 4 31B on 2 H100 GPUs with a 64 GiB CPU offload tier. With this model, available GPU memory bounded offloading's returns rather than cache tier capacity.

For a single replica, offloading produced meaningful gains: throughput rose from 0.06 req/s without offload to 0.12 with CPU (2x), 0.18 with NVMe (3x), and 0.18 with CephFS (3.1x) (Figure 2). Mean TTFT fell from 27.5 s to 3.3 s with CephFS, and mean E2E latency from 131.1 s to 28.9 s.

With 2 replicas, CephFS offload produced only a small throughput increase from 0.18 to 0.2 req/s (+8%), and mean TTFT regressed from 3.3 s to 4.4 s (Figure 3). We attribute this to high GPU memory utilization and limited remaining offload headroom, which constrained the benefit of scaling to additional replicas. In deployments where GPU memory is already the primary constraint, right-sizing the GPU allocation for the model is a prerequisite to realizing the full benefit of offload tiers.

What the storage layer sees

Finally, it's worth looking at the offload traffic from the storage side. Figure 6 shows CephFS read and write bandwidth and IOPS for both models during the 2-replica concurrency runs.

For both models, read traffic grows with concurrency as more prefixes are restored from the shared tier: Qwen's read bandwidth rises from 130 MB/s at C=32 to 462 MB/s at C=128, and Nemotron's from 61 MB/s to 2,700 MB/s, with read IOPS growing in step. Write bandwidth, by contrast, stays roughly flat, at about 260 to 350 MB/s for Qwen and 580 to 630 MB/s for Nemotron. Qwen's write IOPS fall from 424 to 121 while its write bandwidth rises, which means fewer but larger writes at higher load. Nemotron's read ramp is worth keeping in mind when sizing a shared tier for large models: under load, it has to sustain multi-GB/s reads, not just a low p99 latency.

Taken together, the results show that CPU offload captures most gains when the working set fits, while NVMe or CephFS helps when it does not. CephFS also improves cache reuse across replicas as concurrency rises.

What this means for your deployment

Local NVMe is the latency reference and is appropriate when a single node owns the workload. CephFS is the stronger production candidate when pods move, nodes fail, or multiple replicas can reuse the same prefixes. CephFS should still be rejected for short-context workloads if its p99 restore cost exceeds the recomputation cost or the application TTFT budget.

Beyond raw latency, the choice between the 2 secondary tiers comes down to operational behavior:

Validate on your own workload: PoC procedure

Every threshold in this post depends on the model, its block size, and the serving configuration, so the best next step is to reproduce the analysis on your own stack. A customer PoC follows 4 steps:

• Compute the actual KV block size for the customer model, tensor parallelism, and data type (dtype).
• Run FIO with direct I/O, the model block size, intended concurrency, and a mixed read/write profile. Record p50 and p99 latency.
• Measure uncached prefill time per token using the customer serving stack.
• Run no offload, CPU offload, CPU offload + local NVMe, and CPU offload + CephFS configurations with your trace. Compare throughput, TTFT, E2E latency, external cache hit rate, GPU utilization, and storage tail latency.

Conclusion and next steps

These measurements support a tiered approach for long-context inference. FIO identifies whether a storage tier is plausible; the crossover equation translates that latency into a context threshold; and the inference benchmarking shows whether the threshold produces real throughput and TTFT gains. CPU DRAM is the first capacity extension, local NVMe is the lowest-latency secondary tier, and CephFS is the shared production tier when persistence and cross-replica reuse matter.

The next step is a PoC: rerun FIO with your exact block size and validate the resulting crossover against a representative inference trace. If you are evaluating long-context inference on Red Hat OpenShift AI 3.5, you can use this procedure with your own model and a representative workload to identify the KV cache tier that best fits your deployment.

As next steps, we plan to compare RGW/S3 with CephFS and local NVMe using representative KV workloads, measuring cache reuse, TTFT, throughput, and tail latency. For agentic workloads, we will benchmark whether routing each session’s next turn to a replica with its cached context, and restoring offloaded KV when that context is not local, reduces recomputation under concurrency. We will then validate these approaches across replicas, eviction, recovery, and deployment-representative replication and data-protection configurations, and turn the results into guidance for choosing a storage tier and cache strategy.

Appendix: How vLLM's offloading connector works

vLLM’s OffloadingConnector asynchronously copies completed KV cache chunks from GPU memory into a larger CPU cache; in a multi-tier configuration, it cascades those chunks from CPU to secondary storage. GPU eviction remains independent of this process. When a later request has a matching prefix, vLLM checks the CPU and secondary tiers; a secondary-tier hit is first promoted to CPU and then loaded into newly allocated GPU blocks, avoiding recomputation.

The connector plugs into vLLM's KV transfer interface, the same extension point used for disaggregated prefill and for third-party caches such as LMCache. It is configured through --kv-transfer-config:

vllm serve <model> \
  --kv-transfer-config '{
    "kv_connector": "OffloadingConnector",
    "kv_role": "kv_both",
    "kv_connector_extra_config": {
      "spec_name": "TieringOffloadingSpec",
      "cpu_bytes_to_use": 274877906944,
      "secondary_tiers": [
        {
          "type": "fs",
          "root_dir": "/mnt/nvme-kv-cache",
          "n_read_threads": 64,
          "n_write_threads": 64
        }
      ]
    }
  }'

Three properties of the design matter for interpreting the benchmark results that follow.

Transfers are asynchronous and DMA-driven

GPU-to-CPU copies use cudaMemcpyAsync and can overlap with model computation. Stores are also deferred so that they begin after transfers associated with token sampling. This keeps CPU- and GPU-core overhead low, but offloading is not literally free: it still consumes accelerator-to-host bandwidth, host-memory bandwidth, and capacity in the pinned CPU cache. In our tests, those resources were not the bottleneck, so enabling CPU offloading added little overhead even when the cached data was rarely reused.

The CPU tier is privileged

Two built-in specifications are selected through spec_name. CPUOffloadingSpec, the default, provides a single CPU tier backed by pinned host memory. TieringOffloadingSpec adds an ordered list of secondary tiers, configured through secondary_tiers; each entry has a type plus tier-specific fields. Built-in secondary tiers include file system (fs), object store (obj), and peer-to-peer (p2p), and custom out-of-tree tiers can also be loaded.

Only the CPU primary tier has direct access to GPU memory, so it acts also as a gateway. Secondary tiers do not access GPU memory and are not arranged as a chain. They are independent tiers attached to the CPU tier:

GPU ←→ CPU primary tier ←→ secondary tier 0
                                                     ←→ secondary tier 1
                                                     ←→ secondary tier 2

On the store path, a completed chunk is first copied from GPU to CPU. After that copy completes, the tiering manager asynchronously cascades it from CPU to every configured secondary tier. On the load path, the manager checks the CPU tier first and then consults secondary tiers in configured order. A secondary-tier hit is promoted into the CPU tier before it can be copied back to newly allocated GPU blocks.

This is why the configurations we tested are CPU + NVMe and CPU + CephFS, rather than NVMe or CephFS alone. Both use the file system secondary tier, with the CPU cache acting as the mandatory staging layer. CPU sizing therefore matters even when most reusable KV data resides on secondary storage: promotions require an available CPU slot, and in-flight transfers temporarily protect those slots from eviction.

Recovery of preempted KV state

Offloading can also help without cross-request prefix sharing. Under memory pressure, vLLM can preempt a running request and release its GPU KV blocks. When the request is scheduled again, KV chunks that were copied to an offload tier can be restored instead of recomputed. This state is reused within the same request rather than shared across different requests.

The benefit is limited to chunks that were eligible for offload and whose stores completed successfully. By default, offload_prompt_only is enabled, so only prompt KV is retained; decode KV must still be recomputed after preemption unless offload_prompt_only is disabled. Incomplete trailing chunks, cache misses, and CPU-allocation failures can also require recomputation. Preemption recovery can therefore contribute to a widening advantage at high concurrency, but attributing most of the observed gap to it requires supporting metrics or a dedicated ablation.