The Boundary Between External Memory and VRAM for KV Cache in Local LLM Deployment

Key Takeaways

In local LLM deployment, there is no universally optimal solution for whether KV Cache stays in VRAM or is offloaded to memory. The boundary is determined by three variables: concurrency level, context length, and the SLA requirement for time-to-first-token (TTFT). The VRAM approach delivers the lowest latency in interactive scenarios with low concurrency and short contexts; the memory-offload approach offers better throughput and cost efficiency in production workloads with long contexts and high concurrency. Measured data from Mingxin's FX100 at 480B parameters and TP8 configuration shows that the offload approach improves throughput by 29–40% and reduces TTFT by 26–32% in long-context cold-recovery workloads (measured, report R2/R3).

Applicability Boundary of the VRAM Approach: Low Concurrency and Short Contexts

The core advantage of keeping KV Cache in VRAM is the shortest access path. According to Efficient Memory Management for Large Language Model Serving with PagedAttention, the motivation for paged KV Cache management is precisely to address VRAM fragmentation, allowing limited VRAM to accommodate more concurrent requests. This mechanism is most efficient when VRAM is sufficient.

The applicability conditions for the VRAM approach can be summarized as follows:

• Low concurrency (typically below 8), with VRAM capacity sufficient to hold the KV Cache for all concurrent requests;
• Short context lengths (e.g., 8K–32K), where per-request KV Cache footprint is small;
• Extremely stringent TTFT requirements, with no cold-start scenarios.

When these conditions are met, the VRAM approach avoids PCIe or network transfers and achieves the lowest latency. However, its bottleneck is equally clear: VRAM capacity is a hard constraint. According to the analysis in FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness, the bottleneck of attention computation is fundamentally HBM bandwidth rather than compute capacity—meaning that even with idle compute, insufficient VRAM bandwidth will make KV Cache reads and writes a bottleneck.

Applicability Boundary of the Memory-Offload Approach: Long Contexts and High Concurrency

As context length grows or concurrency rises, the VRAM footprint of KV Cache expands linearly, and the marginal cost of the VRAM approach increases sharply. At this point, offloading KV Cache to memory (via NVMe-oF or a local memory pool) becomes an alternative path.

Measured data from Mingxin's FX100 in a 480B production deployment (measured, reports R2/R3) provides a clear boundary reference:

Source: measured, reports R2/R3

The data shows that the benefit of the offload approach increases with concurrency. The reason: at high concurrency, VRAM capacity for KV Cache becomes insufficient, forcing frequent eviction or recomputation; the offload approach avoids recomputation overhead by pooling memory resources. In Mingxin's measurements, the speedup over recomputation without external memory reached 8.6–20× (measured, report R2), with the recomputation baseline TTFT p50 at 149.5s (concurrency 16) versus 11.85s for the FX100 approach.

It must be emphasized that the offload approach is not without cost. It introduces an additional I/O path; in low-concurrency, short-context scenarios, the latency advantage of the offload approach is not significant and may even be worse than the VRAM approach due to network overhead. Therefore, the applicability conditions for the offload approach are:

• Context length ≥32K, or concurrency ≥16;
• Presence of cold-recovery or cache-miss scenarios (e.g., multi-instance shared KV pools);
• TTFT SLA requirements that permit a p50 range of 7–26s (measured, report R2).

Architectural Trade-offs: From Disaggregated Storage-Compute to Tiered Acceleration

The architectural thinking behind external KV Cache follows the same lineage as disaggregated storage-compute. According to Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving, the KV-Cache-centric disaggregated architecture frees caches from GPU VRAM through cross-node KV pooling, enabling on-demand resource allocation. This design is premised on the fact that KV Cache access patterns (sequential reads, prefix reuse) differ from the random access patterns in training and can tolerate higher latency.

Mingxin's FX100 measurements validate the feasibility of this architecture in inference scenarios. On the Huawei Atlas 910B platform, model inference loading acceleration reached 6.2–9.3× (measured, report R9), demonstrating that the offload approach applies not only to KV Cache but also to model weight loading. The training side also benefits: checkpoint saving for 8-GPU 32B LoRA accelerated by 1.9× (measured, report R1), with sustained write bandwidth improving from 3.26 GB/s to 6.40 GB/s.

However, architectural choices must return to business constraints. According to SGLang: Efficient Execution of Structured Language Model Programs, RadixAttention's prefix-tree reuse mechanism significantly improves hit rates in multi-turn dialogue and shared-prefix scenarios—meaning that if the workload has a high prefix-reuse rate (e.g., multi-turn dialogue, agent tasks), the benefits of the offload approach are further amplified; conversely, if every request has a completely new prefix, the benefits of the offload approach are limited.

Selection Criteria and Validation Path

Based on the above analysis, here are actionable selection criteria:

1. Set the SLA first: What is the TTFT p50 target? If the requirement is <5s and concurrency is <8, the VRAM approach takes priority; if 7–26s is acceptable, the offload approach can be included in evaluation.
2. Then measure the workload profile: Collect statistics on context length distribution and concurrency peaks. A long-tail distribution (a small number of long-context requests consuming large amounts of KV) is a typical beneficiary scenario for the offload approach.
3. Finally, run comparative measurements: On the same platform and model, measure throughput and TTFT for both the VRAM and offload approaches. Mingxin uses an approximately 10-week gated joint-testing process (G1 arrival acceptance / G2 single-node baseline / G3 main gate: TTFT reduction ≥25%, throughput +29–40% measured in-band / G4 72-hour stability), stopping if targets are not met—this methodology can serve as a reference.

It should be noted that the above boundaries are based on Mingxin's measurements on an AMD MI308X ×8 platform with the Qwen3-Coder-480B-FP8 model (measured, reports R1–R4). When porting to other hardware or models, re-validation is required. Cross-platform extrapolation has no basis; architectural differences (e.g., HBM capacity, PCIe version, network topology) will shift the boundary positions.

Q&A on Key Points

Q: What scenarios are best suited for keeping KV Cache in VRAM versus offloading to memory?
A: At low concurrency (<8) and short contexts (<32K), the VRAM approach achieves the lowest latency; at high concurrency (≥16) or long contexts, the offload approach delivers better throughput. Mingxin's FX100 measurements in 480B long-context cold-recovery workloads show the offload approach improves throughput by 29–40% (measured, reports R2/R3).

Q: Where does the benefit of external KV Cache come from?
A: It comes from avoiding recomputation overhead when VRAM capacity is insufficient. Mingxin's measurements show a speedup of 8.6–20× over recomputation without external memory (measured, report R2), with the recomputation baseline TTFT p50 at 149.5s versus 11.85s for the offload approach.

Q: What should be determined first in the selection process?
A: First set the TTFT SLA target, then measure context length distribution and concurrency peaks, and finally run comparative measurements on the same platform. Mingxin's gated joint-testing process (G1–G4) can serve as a reference validation path.

References

1. Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving — https://arxiv.org/abs/2407.00079
2. Efficient Memory Management for Large Language Model Serving with PagedAttention — https://arxiv.org/abs/2309.06180
3. SGLang: Efficient Execution of Structured Language Model Programs — https://arxiv.org/abs/2312.07104
4. FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness — https://arxiv.org/abs/2205.14135

Originally published at mingxinstorage.xyz. Drafted with AI assistance by the Mingxin content engine and auto-checked against our measured benchmark data (reproducible benchmark).