GPU Model Fit Matrix

Executive Summary

The question "can I run this model?" has three answers, not two. It fits on one card. It fits if you shard it across N cards. Or it does not fit at all — not on this hardware, not with this quantization, not at this context length. Most VRAM calculators stop at the weights and hand you a single number; the answer you actually need is a verdict, because "138 GB" tells you nothing until you compare it against what you can rent.

A 70B parameter model is the canonical example. At FP16 the weights alone are 140 GB. No datacenter GPU has 140 GB of memory — the B200 tops out at 192 GB but the H100 sits at 80 GB and the A100 at 80 GB. So a 70B FP16 deployment is not a GPU choice, it is a topology choice: 2× H100-80GB with tensor parallelism, or 2× A100-80GB, or drop to INT8 and fit on one card. Each of those has a different throughput profile, and the throughput difference between them is larger than the VRAM difference.

This matrix computes the three-state verdict from first principles: weight size from parameter count and quantization, KV cache from context length and concurrency, framework overhead as a fixed fraction, and a vendor VRAM ceiling that is not the same number as the marketing name suggests. It also refuses to lie about FP8 — if the card you picked has no FP8 datapath, the tool tells you the framework will silently fall back to BF16 rather than quietly assuming a 2× memory saving that will never materialise.

Fit Calculator

Pick a model, a GPU and a quantization level, then set the context length and concurrency you actually plan to serve. All calculations run in your browser.

How the Verdict Is Computed

Three numbers decide the answer, and only the first one is obvious.

1. Weight size

Parameter count times bytes per parameter. A 70B model at FP16 is 70e9 × 2 = 140e9 bytes = 140 GB. Switch to INT8 and it halves to 70 GB; INT4 quarters it to 35 GB. The arithmetic is trivial, which is exactly why people stop here and get the provisioning wrong.

2. KV cache

The KV cache holds the key and value tensors for every token in the context window, for every concurrent request. It grows linearly with context length and linearly with batch size, and it is stored at the model's KV precision — 2 bytes per value at FP16/BF16 regardless of how aggressively you quantized the weights. A dense 70B model at INT4 weights still pays FP16 prices for its KV cache. At 8K context with a single request this is a few gigabytes; at 128K context with 32 concurrent requests it is tens of gigabytes, and it is the number that turns a comfortable fit into a failed one.

This calculator uses the same heuristic as our inference latency estimator: approximately 0.000025 bytes of KV per parameter per 1K tokens of context, scaled by KV precision. It is a sizing heuristic, not an architecture-exact formula — real KV footprint depends on layer count, KV head count and head dimension, which vary by model. It is calibrated to be conservative rather than optimistic.

3. Framework overhead

CUDA context, cuBLAS workspaces, activation buffers and the inference engine's own allocations consume VRAM that is not weights and not KV cache. In practice this runs about 6% of the weight footprint, with a 1 GB floor. It sounds small until you are trying to fit a 76 GB working set onto an 80 GB card — then it is the difference between fitting and not fitting.

The vendor VRAM convention

An "80 GB" H100 has 80,000,000,000 bytes of memory, not 80 × 1024³ = 85,899,345,920 bytes. GPU vendors label memory in decimal gigabytes. That is the opposite of what operating systems and file managers report, and it is the source of the persistent "my 80 GB GPU only shows 74.5 GB" confusion — a GB-to-GiB conversion that should never have been applied to a hardware label. This tool uses vendor decimal figures throughout. If you want the full explanation of when the gap is real and when it is a category error, see GB vs GiB: the marketing gap.

What Each Verdict Means in Practice

✅ Fits on one card

Weights, KV cache and overhead all sit under the ceiling with room to spare. Pay attention to the headroom figure: fitting with 3% margin means any context length increase, any extra concurrent request or any framework upgrade can push you over. Treat under 10% headroom as a fit that will not survive contact with production.

⚠️ Needs N cards

Tensor parallelism shards the weights across GPUs, and the KV cache along with them. The verdict lists the smallest N that works. Two warnings that do not appear in the VRAM arithmetic: first, throughput per GPU drops as N rises — a measured 92% efficiency at TP=2 falls to 80% at TP=4 and 63% at TP=8, because cross-GPU communication over NVLink eats into every layer. Second, those efficiency figures assume NVLink. Run tensor parallelism across PCIe and communication becomes the bottleneck rather than memory bandwidth.

❌ Does not fit

Even at maximum tensor parallelism the working set exceeds available VRAM. The realistic options: quantize more aggressively (the tool lets you try INT4 immediately), shorten the context window, reduce concurrency, or pick a different card. Note that quantization improves throughput and footprint at once for the weights, but does nothing for the KV cache — if KV is what blew the budget, quantization will not save you.

The MoE Trap: Size by Total, Speed by Active

Mixture-of-experts models break the intuition that VRAM requirement and generation speed are the same quantity. DeepSeek-V3 activates 37B of its 671B parameters per token, which makes it behave like a 37B model for throughput purposes — but all 671B must sit in VRAM because the router can select any expert on any token, and paging experts from host memory would stall the decode loop.

The result is a model that needs roughly 336 GB at INT4 while generating at something close to 37B speed. That combination — large memory footprint, modest compute per token — is why MoE deployments are constrained by VRAM capacity rather than by GPU count. This calculator sizes MoE models by total parameters and, in the verdict line, reports both the total and active counts so the throughput expectation is not left implicit.

FP8 Is Not Available Everywhere

FP8 is attractive on paper: 1 byte per parameter, a clean 2× reduction against FP16. But FP8 needs hardware support. Hopper (H100, H200) and Blackwell (B200) run the full FP8 datapath. Ada-generation cards (RTX 4090, L40S) accelerate FP8 with FP16 accumulation, close enough for inference. Ampere and Turing — A100, A10, T4, RTX 3090 — have no FP8 units at all.

On those cards the framework does not error out; it silently falls back to BF16. You ask for FP8, you get 2 bytes per parameter, and the 2× saving you planned for never appears. This tool detects the mismatch and computes the verdict at the real precision, stating the fallback explicitly rather than letting a phantom saving through. The same trap applies to spec-sheet throughput claims — see The FP8 Trap for why the 2× spec advantage lands closer to 1.15× on the invoice.

📜 Copyright & Attribution