The VRAM Ceiling

There is a model size that no vendor has an obvious answer for, and 70 billion parameters sits right on it. At FP16 the weights are 140 GB. That number is larger than every datacenter card on the market — the H100 and A100 both cap at 80 GB, the H200 at 141 GB, and the B200 at 192 GB. Only one of those fits a bare FP16 70B, and it is the most expensive card in the rack. So the 70B class is not a question of which GPU to rent. It is a question of how many, in what arrangement, at what cost in throughput lost to sharding.

Most VRAM calculators answer the question they were asked — how big is this model — and stop. That number is the easy half.

The 140 GB Gap Nobody Sells Into

GPU memory has clustered around two poles. Under 80 GB, where the A100, H100, L40S and the consumer cards live, and now 141–192 GB at the top end with H200 and B200. Between them sits a band with nothing in it — 90 to 140 GB — and the FP16 70B weights land at 140 GB, just above every one of them.

That gap is why a 70B deployment so often ends up on hardware nobody planned for. The obvious move is to quantize: INT8 halves the weights to 70 GB and fits on one H100 or A100-80GB. INT4 quarters them to 35 GB and fits anywhere, including a 48 GB L40S. Both of those are real options, and both introduce a second consideration that the weight arithmetic does not mention at all.

Three Numbers, Only One Obvious

A fit verdict needs three quantities, and the first is the only one people compute.

Figures for a dense 70B model at 8K context and a single concurrent request. Three things in that table are worth pausing on.

First, the KV cache column is flat — identical across all three quantization levels. That is deliberate and it is the part most people get wrong. Quantizing weights to INT4 does not quantize the KV cache; nearly every production serving stack keeps KV at FP16 regardless. A 4-bit model still pays 2 bytes per KV element.

Second, overhead is not negligible at these sizes. Eight point four gigabytes at FP16 is CUDA context, cuBLAS workspaces, activation buffers and the engine's own allocations. That is the margin that decides whether a 76 GB working set squeezes onto an 80 GB card, and it is routinely omitted.

Third, and most important: look at what happens when context grows. The weights do not move. Everything else does.

The KV Cache Is the Real Variable

Weights are a fixed cost paid once. The KV cache holds the key and value tensors for every token in the context window of every live request, and it grows along two axes at the same time. Context length extends it; concurrency multiplies it.

The same model, the same quantization, the same card — and the answer moves from "one card with room to spare" to "eighteen cards" without the model changing at all. Only the workload shape changed.

This is why long-context deployments are so much more expensive than their weight arithmetic suggests, and why provisioning reviews that start from parameter count end up surprised. The KV cache is the quantity that scales with the thing you are actually selling, which is concurrent users holding long conversations.

There is a genuine mitigation, and it is worth knowing about: grouped-query attention. A 70B model with 64 query heads but 8 KV heads carries an eighth of the KV footprint a full multi-head layout would need. That factor of eight is why modern long-context deployments are feasible at all, and why architecture matters as much as parameter count. A model with the same size and a less efficient attention layout can need eight times the cache for the same traffic.

MoE Breaks the Memory-Speed Link

For dense models, memory requirement and generation speed track each other: both are governed by parameter count, so a model that needs more memory also generates more slowly. Mixture-of-experts deliberately severs that link.

DeepSeek-V3 is the clearest case. It carries 671B parameters in total but activates 37B per token, routing each token to a small subset of experts. Throughput behaves like a 37B dense model. Memory requirement behaves like a 671B one, because every expert must stay resident — the router can pick any of them for any token, and paging an expert in from host memory mid-decode would stall the loop far longer than fetching the weights would cost.

Read that table the way a provisioning review would. The MoE model carries 9.6× the weight footprint of the dense 70B, and it generates tokens roughly twice as fast. If your constraint is memory, MoE is expensive. If your constraint is latency at high throughput, it is a bargain. The two models are not substitutes at the same price point — they are answers to different questions.

The KV cache, notably, does not follow the parameter count at all. Both models at 32K context with 32 concurrent requests carry a KV cache in the 336–1,342 GB range, driven by attention geometry rather than by how many parameters the model holds. Whether the MoE premium turns into a memory problem depends entirely on which of the two dominates your workload.

One practical consequence: MoE deployments are sized by VRAM capacity rather than GPU count. Adding cards does not help once memory is the binding constraint, and the ceiling is a hard one because there is no fallback short of spilling experts to host memory at a catastrophic latency cost.

FP8 Saves Nothing on the Wrong Card

FP8 looks like the clean answer to the memory problem: 1 byte per parameter, a tidy halving against FP16, and better throughput because fewer bytes move per token. The catch is that FP8 requires hardware support, and it does not fail loudly when absent.

On an A100 you can request FP8 and the framework will accept it, then quietly run BF16. The 2× memory saving you planned for never appears, and nothing in the logs tells you. This is the failure mode that makes spec-sheet planning dangerous: you check the model fits, deploy, and discover at runtime that the working set is twice what you budgeted for.

A related trap sits on the throughput side. FP8's advantage over FP16 is a peak rating, not a delivered result — the full argument is in The FP8 Trap, where the 2× spec-sheet figure lands closer to 1.15× on the invoice.

Why 10% Headroom Is Not Headroom

A fit verdict that leaves 3% of VRAM free is not a comfortable fit. It is a deployment that will break the first time something changes, and something always changes.

• Context growth. A product decision to raise the context limit from 8K to 32K multiplies the KV cache by four. If KV was 8% of the working set, it is now 26%.
• Concurrency creep. Traffic grows. Every additional concurrent request adds its own KV cache, and utilisation that looked comfortable at launch looks different at 3× load.
• Framework upgrades. Kernel changes, new attention implementations and updated CUDA versions all move the overhead line, usually upward.
• Batching policy changes. Raising the maximum batch size to improve throughput increases the resident KV footprint linearly.

The practical rule: below 10% headroom, treat the fit as provisional and identify the next card or the quantization step you would move to. Above 25%, you have room to absorb growth without re-provisioning. Between the two, monitor the actual VRAM high-water mark rather than the theoretical figure — the gap between them is where surprises live.

The 80 GB That Isn't 80 GiB

One more number causes more confusion than it should. An "80 GB" H100 holds 80,000,000,000 bytes of memory. It does not hold 80 × 1024³ = 85,899,345,920 bytes. GPU vendors label memory in decimal gigabytes, the same way storage vendors do, and the difference between the two conventions is 7.4%.

So when a tool reports 76.3 GiB for an 80 GB card, that is not a card that shipped short. It is the same number expressed under a different convention. Applying the conversion to a hardware label is a category error — the label already means decimal bytes, and converting it again subtracts memory that does not exist.

This matters for fit calculations because 7.4% is well inside the headroom margin we just said to treat as unsafe. If your planning tool subtracts 7% from a card's capacity and your other tool does not, you will get two different verdicts for the same model and spend an afternoon chasing a bug that is not there. The full explanation of when the GB–GiB gap is real and when it is a category error is in the marketing gap briefing.

A Five-Question Fit Check

Run this before committing to any hardware, in this order. Each question can end the evaluation on its own.

1. What is the working set, not the weight size? Weights plus KV cache plus overhead. If you only know the weight figure, you do not yet know whether the model fits.
2. What context length and concurrency will you actually serve? Not the maximum the model supports — the distribution you expect. This single answer moves the memory requirement by orders of magnitude.
3. Does the card support the precision you assumed? FP8 on Ampere silently becomes BF16. Check before you plan around a saving that will not materialise.
4. What is the smallest tensor-parallel group that holds one copy? That is your floor on card count, regardless of traffic. For a dense 70B at FP16, the floor is 2–4 cards; there is no configuration with fewer.
5. What does that group deliver, and what does it cost per token? Sharded throughput is not N× a single card — efficiency runs 92% at TP=2, 80% at TP=4 and 63% at TP=8. Multiply cards by the hourly rate and divide by delivered tokens, or you are comparing sticker prices rather than costs.

Question five is where most evaluations stop being about engineering and start being about money. It is also the question the fit matrix and the capacity planner are built to answer directly.

Frequently Asked Questions

How much VRAM does a 70B model need?

Weights: 140 GB at FP16, 70 GB at INT8, 35 GB at INT4. Working set at 8K context with one request: roughly 151 GB, 77 GB and 40 GB respectively. At 128K context with 32 concurrent requests the INT4 working set reaches about 1.38 TB. The spread between those numbers is the whole point — "how much VRAM" has no single answer without a workload shape attached.

Why does context length change whether a model fits?

Because weights are fixed and the KV cache is not. A 70B INT4 model always holds the same 35 GB of weights, but its KV cache runs from 2.6 GB at 8K context with one request to over 1,340 GB at 128K context with 32 requests. The model did not change; the deployment did.

Why do MoE models need VRAM sized by total parameters?

Because the router can select any expert for any token, so all of them must be resident. Paging an expert from host memory mid-decode would stall the loop. VRAM is therefore sized by total parameters while throughput is governed by active parameters — which is why a 671B MoE with 37B active needs more than one B200 yet generates like a dense 37B model.

Does 4-bit quantization always save half the memory of 8-bit?

For weights, yes. For the KV cache, no — INT4 weights still carry an FP16 KV cache in production serving stacks. On short-context, low-concurrency workloads the weight saving dominates and the distinction is academic. On long-context workloads the KV cache dominates and weight quantization does nothing for the number that blew the budget.

Methodology & Disclosure

Weight figures are parameter count × bytes per parameter at the stated precision, using vendor decimal gigabytes. KV cache is derived from each model's published attention configuration — layer count, grouped-query KV head count and head dimension — as 2 × layers × KV heads × head dimension × tokens × KV bytes, with KV precision held at FP16 regardless of weight quantization. Framework overhead is modelled as 6% of weight footprint with a 0.5 GB floor, a sizing heuristic rather than a measured constant. Tensor-parallel efficiency figures (92% at TP=2, 80% at TP=4, 63% at TP=8, 48% at TP=16) are representative published measurements for NVLink-interconnected deployments and are treated as upper bounds for PCIe. GPU memory capacities are vendor decimal labels and are not converted to GiB. This site takes no sponsorship, no affiliate commission and no vendor payment of any kind; the vendors named here have no relationship with jslet and did not review this briefing.

References & Further Reading

1. NVIDIA. H100, H200 and B200 Tensor Core GPU datasheets — HBM capacity and memory bandwidth ratings used throughout. nvidia.com
2. NVIDIA Transformer Engine. GPU compute-capability feature matrix: FP8 gating at SM 8.9 (Ada) and full FP8 datapath at SM 9.0 (Hopper) and SM 10.0 (Blackwell). github.com/NVIDIA/TransformerEngine
3. DeepSeek-AI. "DeepSeek-V3 Technical Report." Total and active parameter counts, expert routing, and KV head configuration. arxiv.org
4. Meta AI. Llama 4 model card — parameter counts, grouped-query attention configuration and layer geometry across the Scout, Maverick and Behemoth variants. ai.meta.com
5. vLLM documentation. PagedAttention and KV cache memory management, including the FP16-KV default under weight quantization. docs.vllm.ai
6. Related: GPU Model Fit Matrix · Inference Capacity Planner · LLM Inference Latency Estimator · The FP8 Trap: 2× Spec Sheet, 1.15× Invoice · GB vs GiB: The Marketing Gap

📜 Copyright & Attribution