GPU Model Fit Matrix

Executive Summary

The question "can I run this model?" has three answers, not two. It fits on one card. It fits if you shard it across N cards. Or it does not fit at all — not on this hardware, not with this quantization, not at this context length. Most VRAM calculators stop at the weights and hand you a single number; the answer you actually need is a verdict, because "138 GB" tells you nothing until you compare it against what you can rent.

A 70B parameter model is the canonical example. At FP16 the weights alone are 140 GB. No datacenter GPU has 140 GB of memory — the B200 tops out at 192 GB but the H100 sits at 80 GB and the A100 at 80 GB. So a 70B FP16 deployment is not a GPU choice, it is a topology choice: 2× H100-80GB with tensor parallelism, or 2× A100-80GB, or drop to INT8 and fit on one card. Each of those has a different throughput profile, and the throughput difference between them is larger than the VRAM difference.

This matrix computes the three-state verdict from first principles: weight size from parameter count and quantization, KV cache from context length and concurrency, framework overhead as a fixed fraction, and a vendor VRAM ceiling that is not the same number as the marketing name suggests. It also refuses to lie about FP8 — if the card you picked has no FP8 datapath, the tool tells you the framework will silently fall back to BF16 rather than quietly assuming a 2× memory saving that will never materialise.

Fit Calculator

Pick a model, a GPU and a quantization level, then set the context length and concurrency you actually plan to serve. All calculations run in your browser.

Capacity and Need are both in decimal GB. A card sold as "80 GB" holds 80 GiB of memory, which is 86 GB in this column — see why.

How the Verdict Is Computed

Three numbers decide the answer, and only the first one is obvious.

1. Weight size

Parameter count times bytes per parameter. A 70B model at FP16 is 70e9 × 2 = 140e9 bytes = 140 GB. Switch to INT8 and it halves to 70 GB; INT4 quarters it to 35 GB. The arithmetic is trivial, which is exactly why people stop here and get the provisioning wrong.

2. KV cache

The KV cache holds the key and value tensors for every token in the context window, for every concurrent request. It grows linearly with context length and linearly with batch size, and it is stored at the model's KV precision — 2 bytes per value at FP16/BF16 regardless of how aggressively you quantized the weights. A dense 70B model at INT4 weights still pays FP16 prices for its KV cache. At 8K context with a single request this is a few gigabytes; at 128K context with 32 concurrent requests it is tens of gigabytes, and it is the number that turns a comfortable fit into a failed one.

This calculator derives the cache from each model's published attention shape rather than from a per-parameter rule of thumb: 2 × layers × KV heads × head dimension × tokens × KV bytes. The KV head count is what decides the answer. A 70B model with 8 grouped-query KV heads carries an eighth of the cache the same layer count would need at 64 query heads — 2.6 GB instead of 21 GB at 8K — and that factor of eight is the difference between long context fitting on one card and not. Layer counts, KV head counts and head dimensions are published architecture figures rather than estimates, but the result is still a sizing model: validate it against your serving stack before committing to hardware.

3. Framework overhead

CUDA context, cuBLAS workspaces, activation buffers and the inference engine's own allocations consume VRAM that is not weights and not KV cache. In practice this runs about 6% of the weight footprint, with a 0.5 GB floor. It sounds small until you price it: a 70B model at FP16 carries more than 8 GB of overhead on top of 140 GB of weights, which is enough to push a deployment that looked comfortable past the ceiling.

The vendor VRAM convention

An "80 GB" H100 holds 80 GiB of memory — 85,899,345,920 bytes — which is exactly what nvidia-smi reports as 81,920 MiB. Nothing is missing and nothing is being held back: the label is binary, the same convention operating systems use for main memory, and the familiar "my 80 GB GPU only shows 74.5 GB" complaint comes from dividing that binary label by 1024³ as though it were a storage capacity. Weights and KV cache, by contrast, are computed in decimal gigabytes here, because bandwidth and price are both quoted that way. So this tool converts each card's label into decimal GB before comparing it with the working set: 80 GiB becomes 85.9 GB. Skipping that step would understate every card by 7.4%, which is well inside the headroom margin we tell you to treat as unsafe. See VRAM GB → MiB for the conversion, and GB vs GiB: the marketing gap for the storage case, where the opposite is true.

What Each Verdict Means in Practice

✅ Fits on one card

Weights, KV cache and overhead all sit under the ceiling with room to spare. Pay attention to the headroom figure: fitting with 3% margin means any context length increase, any extra concurrent request or any framework upgrade can push you over. Treat under 10% headroom as a fit that will not survive contact with production.

⚠️ Needs N cards

Tensor parallelism shards the weights across GPUs, and the KV cache along with them. The verdict lists the smallest N that works. Two warnings that do not appear in the VRAM arithmetic: first, throughput per GPU drops as N rises — a measured 92% efficiency at TP=2 falls to 80% at TP=4 and 63% at TP=8, because cross-GPU communication over NVLink eats into every layer. Second, those efficiency figures assume NVLink. Run tensor parallelism across PCIe and communication becomes the bottleneck rather than memory bandwidth.

❌ Does not fit

Even at maximum tensor parallelism the working set exceeds available VRAM. The realistic options: quantize more aggressively (the tool lets you try INT4 immediately), shorten the context window, reduce concurrency, or pick a different card. Note that quantization improves throughput and footprint at once for the weights, but does nothing for the KV cache — if KV is what blew the budget, quantization will not save you.

The MoE Trap: Size by Total, Speed by Active

Mixture-of-experts models break the intuition that VRAM requirement and generation speed are the same quantity. DeepSeek-V3 activates 37B of its 671B parameters per token, which makes it behave like a 37B model for throughput purposes — but all 671B must sit in VRAM because the router can select any expert on any token, and paging experts from host memory would stall the decode loop.

The result is a model that needs roughly 336 GB at INT4 while generating at something close to 37B speed. That combination — large memory footprint, modest compute per token — is why MoE deployments are constrained by VRAM capacity rather than by GPU count. This calculator sizes MoE models by total parameters and, in the verdict line, reports both the total and active counts so the throughput expectation is not left implicit.

FP8 Is Not Available Everywhere

FP8 is attractive on paper: 1 byte per parameter, a clean 2× reduction against FP16. But FP8 needs hardware support. Hopper (H100, H200) and Blackwell (B200) run the full FP8 datapath. Ada-generation cards (RTX 4090, L40S) accelerate FP8 with FP16 accumulation, close enough for inference. Ampere and Turing — A100, A10, T4, RTX 3090 — have no FP8 units at all.

On those cards the framework does not error out; it silently falls back to BF16. You ask for FP8, you get 2 bytes per parameter, and the 2× saving you planned for never appears. This tool detects the mismatch and computes the verdict at the real precision, stating the fallback explicitly rather than letting a phantom saving through. The same trap applies to spec-sheet throughput claims — see The FP8 Trap for why the 2× spec advantage lands closer to 1.15× on the invoice.

📜 Copyright & Attribution