Inference Capacity Planner

Executive Summary

Every throughput calculator on the internet runs the same direction: choose a GPU, read off tokens per second, close the tab. That answers "how fast is this card" and leaves the actual provisioning question untouched. Nobody is handed a GPU and asked how fast it is. They are handed a product requirement — this many users, this latency target — and asked how much hardware it takes.

So run it backwards. Start with target QPS and a latency budget, divide by the per-card throughput your model and quantization actually deliver, then correct for the two effects that make naive division wrong. The first is tensor-parallel efficiency: sharding a model across N cards gives you about N × 92%, 80%, 63% of one card's throughput at N = 2, 4, 8 — never N×. The second is that memory can force a shard count before throughput does. A 70B model at FP16 needs 151 GB of working set, so it cannot run on fewer than two 80 GB cards even if one card could serve the traffic.

The planner below returns four things: how many GPUs the load needs, the smallest valid tensor-parallel layout, the throughput that layout actually delivers after efficiency loss, and what the same token volume would cost as API calls. That last number is the one that usually changes the decision.

Capacity Planner

Enter your workload, then your hardware options. Everything is computed in your browser.

1 · Workload

2 · Model & Hardware

3 · Result

Self-host vs API for This Load

Why the Reverse Direction Is the One That Matters

Forward sizing is easy and almost useless. You pick a card, the tool tells you it does 2,000 tokens/sec, and you have learned something about the card rather than about your deployment. The question in a provisioning review is never "what can this GPU do" — it is "what does this product cost to run", and that question has the traffic on the left-hand side.

Reverse sizing turns a capacity target into a hardware count, and the difference is not cosmetic. It surfaces costs that forward sizing hides: a model that fits on one card but only serves 40% of your traffic, a tensor-parallel layout chosen for memory that leaves you paying for 3× the compute you need, a utilisation figure so low that the per-token cost lands an order of magnitude above the API price for the same model.

The four steps

1. Turn demand into peak tokens/sec. Monthly requests × output tokens per request ÷ 2.6 million seconds gives average token throughput. Multiply by the peak ratio — the planner makes you choose it explicitly, because average load provisioning is the single most common way inference services fall over.
2. Find per-card throughput. From the model's active parameters, the quantization level and the GPU's memory bandwidth: one decoded token requires reading every active parameter once, so tokens/sec ≈ bandwidth ÷ bytes-per-token-read. Apply the batch efficiency factor for the concurrency you expect.
3. Divide, then correct for sharding. Cards needed = peak throughput ÷ per-card throughput ÷ utilisation ceiling, rounded up. If memory forces a larger group than throughput does, take the larger of the two — and remember that aggregate throughput scales by the TP efficiency, not by N.
4. Price it both ways. Multiply cards by the hourly rate. Compute the same token volume at API rates. The ratio tells you which regime you are in, and it is far more sensitive to utilisation than to model size.

Tensor Parallelism Is Not Free Throughput

Sharding a model across N GPUs does not multiply throughput by N. Tensor parallelism splits every layer, which means every layer needs an all-reduce to combine the partial results before the next layer can begin. The communication is a fixed cost per layer that grows with group size, while the useful compute per card shrinks. Efficiency falls accordingly:

Going from 4 to 8 cards buys 1.58× throughput for twice the money — the marginal cost per token rises. These figures assume NVLink. Across PCIe interconnect the all-reduce cost is substantially higher and the practical ceiling arrives at a much smaller group size. If your hardware plan puts tensor parallelism over PCIe, treat the table above as an upper bound that you will not reach.

When sharding is forced by memory rather than speed

A 70B model at FP16 occupies 151 GB of working set including KV cache and framework overhead. Two H100-80GB cards hold 160 GB, so the model needs TP=2 whether or not one card's throughput would have covered the traffic. That is the case where you pay the efficiency penalty for a reason that has nothing to do with throughput — and it is invisible if you size from traffic alone.

Utilisation Decides Self-Host vs API

A rented GPU costs the same per hour whether it serves one request or a million. That single fact drives the self-host decision far more than model size does. At high sustained utilisation, owned or rented capacity beats per-token pricing comfortably. At low utilisation you are paying for idle silicon, and the API wins by a wide margin even though its per-token rate looks expensive.

Three practical notes the comparison table cannot capture. First, batch mode at 50% off applies only to OpenAI, Anthropic, Google and Mistral — DeepSeek and Together do not offer it, which changes the API side of the comparison by half. Second, cached input reads at up to 90–98% off on the major providers: for workloads with a large stable prefix, that discount moves the crossover further than any hardware choice. Third, the self-hosted side carries operational cost that does not appear in the GPU bill — the MLOps engineer, the evaluation harness, the upgrade cycle — and the breakeven briefing linked below puts numbers on it.

The honest summary: if your modeled utilisation lands under roughly 30%, treat the self-host column as a floor rather than an estimate, and check the API side first.

Reading the Plan

GPUs required

The larger of two constraints: throughput (peak tokens/sec ÷ per-card tokens/sec ÷ utilisation ceiling) and memory (working set ÷ VRAM per card, rounded up to the next valid tensor-parallel size). When these disagree, the memory constraint usually wins for models above 30B and the throughput constraint wins for small models under heavy load.

Tensor-parallel layout

Valid layouts are powers of two in practice, because all-reduce works most efficiently on power-of-two groups and the serving frameworks are tuned for them. A requirement of 5 cards becomes a 8-card layout or two 4-card replicas — the planner picks the smallest valid shard count and reports whether a single replica or several replicas is the better shape.

Resulting utilisation

Delivered capacity against peak demand. Below 30% the economics of renting dedicated silicon are poor and the API comparison will usually win. Above 70% you are running close to the ceiling you chose, which is a deliberate trade rather than an accident. Utilisation is the number to revisit when traffic grows — it tells you how much headroom is left before the next card is needed.

📜 Copyright & Attribution