KV Cache Context Window VRAM Inflation Calculator
Predicts execution memory overhead spikes inside runtime attention blocks as token context envelopes expand. Model the VRAM cost of long-context inference — linear in context length, multiplied by concurrency — to prevent out-of-memory failures during production serving.
🪟 Model Architecture Parameters
Configure your transformer model's architecture. The engine computes KV cache size using the standard attention memory formula and projects growth across context lengths.
KV Cache: The Hidden VRAM Killer in Long-Context LLM Inference
The Key-Value (KV) Cache is the single largest consumer of GPU memory during autoregressive LLM inference — often exceeding the model weights themselves. As context windows expand from 4K to 32K, 128K, and now 1M+ tokens, the KV cache grows linearly with sequence length, making it the dominant factor in VRAM provisioning for long-context serving. Understanding this growth curve is essential to avoid out-of-memory (OOM) crashes that silently kill inference requests under peak load.
How This Calculator Works
This tool applies the standard transformer KV cache formula and projects memory consumption across multiple context lengths so you can plan GPU capacity before deploying:
| Parameter | Symbol | Role in KV Cache |
|---|---|---|
| Context Window | C | The number of tokens stored in the cache. Every token in the sequence has a key and value entry. Doubling the context window doubles the KV cache. |
| Transformer Layers | L | Each transformer layer maintains its own independent KV cache. A 70B Llama-2 model has 80 layers — so the per-token KV cost is multiplied by 80. Deeper models pay a steeper KV cache penalty. |
| Hidden Dimension | d_model | The total width of the residual stream. The per-head dimension is d_model / num_heads, which determines the size of each key and value vector. |
| Attention Heads | n_heads | The number of query heads. Head dimension is d_model / n_heads — 128 for a 70B model at 8,192 / 64. The KV cache is sized by kv_heads × head_dim, so this value is not decorative: it is what makes the grouped-query reduction real rather than nominal. |
| KV Heads (GQA) | kv_heads | Grouped-Query Attention (GQA) shares key-value heads across query heads. Fewer KV heads drastically reduces cache size. Llama-2-70B uses 8 KV heads vs. 64 query heads — an 8× reduction in KV cache vs. full multi-head attention. |
| Batch Size | B | Each sequence in the batch has its own KV cache. Serving 16 concurrent users means 16× the KV cache memory. Batching is throughput-friendly but memory-hostile. |
The KV Cache Formula
The total KV cache memory in bytes is computed as:
KV Cache (bytes) = 2 × L × C × kv_heads × head_dim × 2 bytes × B
Breaking this down: the factor of 2 accounts for both keys and values; L multiplies across all layers; C is the number of cached tokens; kv_heads × head_dim is the per-token key/value width — and head_dim is d_model / attention_heads, not d_model / kv_heads. That distinction is the whole difference between grouped-query attention and full multi-head attention; writing the ratio against kv_heads cancels it out and overstates the cache by however many query heads the model has per KV head. 2 bytes is the FP16 storage per element; B accounts for batch parallelism. The expression simplifies to:
KV Cache (bytes) = 4 × L × C × kv_heads × head_dim × B
When kv_heads equals attention_heads — plain multi-head attention — that collapses to the familiar 4 × L × C × d_model × B, and the two forms agree. They diverge exactly when a model uses fewer KV heads than query heads, which is almost every model released since 2023.
Why Long Contexts Explode VRAM
Consider a 70B-class model — 80 layers, 8,192 hidden width, 64 attention heads and 8 KV heads, the shape Llama-2-70B and most 70B successors share — at FP16 KV, one sequence:
| Context Length | KV Cache (Batch=1) | Model Weights (FP16) | KV as % of Weights |
|---|---|---|---|
| 4,000 tokens | 1.31 GB | 140 GB | 0.9% |
| 8,000 tokens | 2.62 GB | 140 GB | 1.9% |
| 32,000 tokens | 10.49 GB | 140 GB | 7.5% |
| 128,000 tokens | 41.94 GB | 140 GB | 30.0% |
Read the last column again: at a single sequence and 128K context the cache is still under a third of the weights. Concurrency is what flips it. The same 128K context across 32 simultaneous requests is 1,342 GB of cache against 140 GB of weights — 9.6× the model, and that is the point at which serving long-context models needs GPUs for the cache rather than for the weights. Techniques like Multi-Query Attention (MQA), Grouped-Query Attention (GQA), KV cache quantization (INT8/FP8 KV), and sliding window attention (Mistral) are all architectural innovations designed to tame KV cache growth.
Mitigation Strategies
- GQA/MQA: Reducing KV heads from 64 to 8 (as in Llama-2-70B) cuts KV cache by 8×. This is the single most impactful architectural choice.
- KV Cache Quantization: Storing keys and values at INT8 instead of FP16 halves the KV cache. vLLM and TensorRT-LLM support KV cache quantization natively.
- PagedAttention: vLLM's block-based KV cache management reduces fragmentation and allows over-commit with near-zero waste. Logical KV cache may be 2× physical allocation.
- Sliding Window / Sparse Attention: Mistral's 4096-token sliding window bounds KV cache to a fixed size regardless of total context length. Long-form information is handled by dense layers, not attention.