The Reverse Sizing Problem

Search for how to size an LLM deployment and you will find hundreds of calculators that answer the same question: pick a GPU, pick a model, read off tokens per second. They are well built and they answer something real. They just are not answering the question that gets asked in a provisioning review.

Nobody is handed a GPU and asked how fast it is. They are handed a product requirement — this many users, this latency target, this growth curve — and asked how much hardware it takes, and what it costs per month. That question has the traffic on the left-hand side and the hardware on the right. Running it the usual direction means solving for the wrong unknown and then inverting by hand, which is exactly where the arithmetic goes wrong.

Everyone Sizes It Backwards

The forward question is comfortable because it has one variable. Change the GPU, watch the number move, form an intuition about which cards are fast. That intuition is genuinely useful — and it will not survive contact with a peak load figure.

The reverse question has four unknowns that interact. Demand comes in requests per month but hardware is priced per hour and provisioned per peak second. Throughput depends on a model choice, a quantization choice and a hardware choice simultaneously. Memory can force a configuration before throughput does. And the whole thing is bounded by a utilisation ceiling that is a business decision rather than an engineering constant.

The four steps, in order:

1. Demand → peak tokens/sec. Requests per month × output tokens per request ÷ 2.6 million seconds gives average token throughput. Then multiply by the peak ratio. Skipping this step is the single most common provisioning error, and the next section is about why.
2. Model + precision + GPU → per-card throughput. One decoded token requires reading every active parameter once, so throughput is bounded by memory bandwidth divided by bytes read per token. Quantization reduces bytes per token; larger batch sizes improve the constant factor by improving compute occupancy.
3. Divide, correct, round up. Cards needed is peak throughput ÷ per-card throughput ÷ utilisation ceiling — then corrected upward for sublinear scaling, and corrected again if memory demands a bigger group than throughput does.
4. Price it both ways. Cards × hourly rate against the same token volume at API rates. The ratio, not the absolute numbers, tells you which regime you are in.

Average Load Is Not a Target

Monthly volume divided by seconds in a month produces a number that describes nothing anyone experiences. Traffic is not uniform: it concentrates in working hours, in weekdays, in the minutes after a launch or an outage notification. A service that is comfortable at average load falls over at peak, and it falls over at the moment when failing is most expensive.

That last column is the point. Choosing 5× instead of 2× does not add a safety margin of a few percent — it multiplies the hardware bill by two and a half. The multiplier is one of the most expensive single decisions in an inference deployment, and it is usually made implicitly, as an unexamined "let us add some headroom" somewhere in a spreadsheet.

Two things make the decision easier. First, if the workload tolerates queueing — a batch job, an asynchronous summarisation service — the peak ratio can be low because excess demand waits rather than fails. Second, if the service can degrade gracefully, the effective peak ratio drops, because the consequence of being at capacity is a slower response rather than an error. The multiplier is a statement about acceptable failure modes, which is why it belongs to the product owner rather than the infrastructure engineer.

N Cards, 0.63N Throughput

This is the correction that most capacity plans miss entirely. Adding GPUs to a tensor-parallel group does not add throughput proportionally, because every layer in the sharded model needs an all-reduce to combine the partial results before the next layer can proceed. Communication cost scales with group size; useful work per card shrinks.

Read the last column. Going from four to eight cards doubles the hardware bill and increases throughput by 58%, so the cost per token rises by 27%. Going to sixteen cards more than doubles the cost per token relative to a single card. Sharding is sometimes necessary — for memory, usually — but it is never free, and past a certain group size it is actively counterproductive for cost efficiency.

The practical implication for planning: if a workload needs more throughput than one card can provide, the cheapest path is usually more replicas of a smaller group rather than a larger group. Four replicas of a two-card group deliver 7.36× a single card (4 × 1.84) using eight cards, versus 5.04× from a single eight-card group using the same eight cards. Same silicon, 46% more throughput, because replica-level parallelism has no per-layer communication cost.

The caveat is memory. If one copy of the model does not fit in two cards, a two-card group is not an option no matter how much better the arithmetic looks — which brings us to the case where the card count is set by something other than traffic.

When Memory Sets the Card Count

A dense 70B model at FP16 carries about 151 GB of working set at 8K context: 140 GB of weights, 2.6 GB of KV cache and 8.4 GB of framework overhead. Two H100-80GB cards hold 160 GB between them, so the model needs a group of at least two. Four cards gives comfortable headroom once the context grows.

Now suppose the traffic is modest — say a few hundred thousand requests a month, enough that a single card's throughput would have covered it several times over. That does not matter. The model cannot run on fewer than two cards, so the floor on the hardware bill is two cards, and the utilisation figure that results will be low by definition.

Both constraints must be satisfied, so the answer is the larger of the two. Systems sized by traffic alone will under-provision the memory side; systems sized by memory alone will under-provision replicas and run out of throughput at peak. The one that catches people out is the memory floor, because it produces a hardware count that looks wildly oversized relative to the traffic — and it is not oversized, it is the minimum the model can run on.

Utilisation Decides Self-Host vs API

Renting a GPU costs the same per hour whether it serves one request or a million. That single property drives the self-host decision far more than scale, model size or the headline per-token price on either side.

The failure mode is a deployment forced into a large group by memory, serving modest traffic, at an utilisation in the low teens. In that configuration the per-token cost lands an order of magnitude above the API price for a comparable model, and the reason is entirely that you are renting idle silicon.

The reverse failure mode is more comfortable: a well-utilised workload at a stable, predictable load, where owned or reserved capacity beats per-token pricing by a wide and durable margin. Reserved and committed-use discounts push that further, though they trade flexibility for rate.

Three qualifications the cost tables cannot capture.

• Operational cost is not in the GPU bill. Self-hosting carries the MLOps engineer, the evaluation harness, the upgrade cycle, the on-call rotation and the incidents. On small deployments this exceeds the hardware cost; it is the reason self-hosting looks cheaper on paper than it turns out to be.
• Latency and control are real goods. Self-hosting buys determinism — no provider-side rate limits, no deprecation schedule, no upstream change that alters behaviour overnight. That has value that per-token arithmetic does not price.
• The API side is not static either. Rates drop, models are retired, capacity is rationed at peak. A comparison that fixes both sides at today's numbers is a snapshot, not a forecast.

The Discounts That Move the Crossover

The gap between an API's sticker price and its effective price is often larger than the gap between self-hosting and the API. Two discounts do most of that work.

Note the asymmetry in the batch row. Batch mode at 50% off applies to four of the major providers — DeepSeek and Together do not offer it. For an asynchronous workload that is a factor of two on the API side of the comparison, which moves the crossover further than any realistic hardware choice. Anyone comparing a batch-tolerant workload should price the batch tier, not the standard tier.

Cached input is the more interesting lever because its impact scales with workload shape rather than volume. A service with a large stable prefix — a long system prompt, a fixed document corpus, a tool catalogue — pays full price for that prefix once and a heavily discounted rate thereafter. A service where every request is novel gains nothing. This is why two deployments with identical token volumes and identical models can land on opposite sides of the self-host crossover: they are not buying the same thing.

Three Worked Examples

All three assume 400 output tokens per request, a 2× peak multiplier, a 70% utilisation ceiling and H100-80GB rental at $3.90 per GPU-hour.

Small model, moderate traffic

An 8B dense model at INT4 serving 3 million requests per month. That is 1.15 average QPS, 2.3 QPS at peak, and 923 output tokens per second. One H100 delivers about 687 tokens/sec at interactive batch sizes, so a single card covers 74% of peak — close, but not enough with the utilisation ceiling applied. Two cards, running as two independent replicas, deliver 1,374 tokens/sec at 67% utilisation. Cost: roughly $5,700 per month.

Note the shape of the answer: two replicas rather than one two-card group. Replication adds capacity with no communication penalty, which is why the planner prefers it whenever memory allows.

Large model, modest traffic

The same 3 million requests, now on a dense 70B at FP16. Memory alone forces a group of four: the 151 GB working set needs at least two cards, and four gives headroom for context growth. Throughput is far below the group's capacity, but that is irrelevant — the model physically cannot run on less. Delivered capacity is about 1,380 tokens/sec against 923 needed, and the hardware count lands in the high double digits once replicas are counted. Cost: north of $250,000 per month.

This is the case where the API wins by an order of magnitude, and the reason is not the model's size but the utilisation implied by forcing a large group onto modest traffic. The same model at ten times the traffic would look entirely different.

The crossover case

An 8B model at INT4 serving 200 QPS directly — roughly 80,000 output tokens per second at peak. This needs a substantial replica fleet, and here the self-host side starts to win: the hardware is fully utilised, so the fixed hourly cost is amortised across a large token volume. The comparison narrows to within a few percent, and the decision turns on the qualifications rather than the arithmetic — operational cost, control, and the direction rates are moving.

The pattern across all three: small models at high traffic favour self-hosting, large models at low traffic favour the API, and the determining variable is utilisation rather than any property of the model.

A Provisioning Checklist

1. Convert monthly volume to peak tokens per second before touching any hardware numbers. If you cannot state your peak ratio, that is the first thing to resolve.
2. Establish the memory floor. Working set ÷ VRAM per card, rounded up to a valid tensor-parallel size. This is the card count you cannot go below.
3. Compute the throughput requirement at that group size, including TP efficiency loss, and divide by the utilisation ceiling you are willing to run at.
4. Prefer replicas over larger groups when memory allows. Four replicas of a two-card group beat one eight-card group on the same silicon by a wide margin.
5. Price both paths at effective rates, not sticker prices. Apply cached-input and batch discounts where your workload qualifies, and check whether your model's provider even offers them.
6. State the utilisation figure explicitly in the decision record. It is the number that explains the outcome, and the number that will change as traffic grows.
7. Re-run on a schedule. Provisioning is not a one-time calculation; it is a standing estimate that goes stale as traffic, models and rates move.

Steps two and three are what the capacity planner computes directly. Step two in isolation is the fit matrix.

Frequently Asked Questions

How do I calculate how many GPUs I need for an LLM service?

Four steps: convert demand to peak output tokens per second, find per-card throughput from active parameters and memory bandwidth, divide by the utilisation ceiling, then check that the memory side agrees. The answer is the larger of the throughput requirement and the memory floor, and both must be satisfied.

Why does throughput not scale linearly when I add GPUs?

Tensor parallelism requires an all-reduce at every layer to combine partial results. That communication grows with group size while work per card shrinks, so efficiency falls from 92% at two GPUs to 63% at eight and 48% at sixteen. Eight cards deliver about 5× a single card, not 8× — and the cost per token rises as a result.

Should I provision for average or peak load?

Peak, always. Average is a reporting metric. Real traffic peaks 3–10× above its daily mean, and choosing a peak ratio multiplies the hardware bill directly — 5× instead of 2× is a two-and-a-half-fold cost increase. It is a decision about acceptable failure modes, so it belongs to the product owner.

At what utilisation does self-hosting beat using an API?

There is no universal number, but below roughly 30% utilisation the API almost always wins because you are renting idle silicon. The crossover generally sits in the 30–60% band, and it moves further on effective rates than on hardware: cached input at 90–98% off and batch mode at 50% off can shift it by an order of magnitude.

Methodology & Disclosure

Throughput estimates derive from memory bandwidth divided by bytes read per decoded token, using each model's published active-parameter count, with a batch-dependent efficiency factor applied. Tensor-parallel efficiency figures (92% at 2, 80% at 4, 63% at 8, 48% at 16) are representative published measurements for NVLink-interconnected deployments and are treated as upper bounds for PCIe. Working sets combine weight size, a KV cache derived from published attention geometry at FP16, and framework overhead modelled at 6% of weight footprint. Rental rates are on-demand list prices for the named instance class in us-east-1, in USD, as published 2026-09-25, per GPU-hour, excluding storage and egress. API comparison figures use blended per-token rates for comparable hosted models; batch and cached-input discounts are excluded from headline figures unless stated. Self-hosting comparisons exclude operational cost. This site takes no sponsorship, no affiliate commission and no vendor payment of any kind; the vendors named here have no relationship with jslet and did not review this briefing.

References & Further Reading

1. NVIDIA. H100, H200 and B200 Tensor Core GPU datasheets — HBM capacity and memory bandwidth ratings used for per-card throughput. nvidia.com
2. NVIDIA Megatron-LM and Transformer Engine documentation. Tensor-parallel communication patterns, all-reduce cost per layer, and scaling efficiency measurements. github.com/NVIDIA/Megatron-LM
3. vLLM documentation. Continuous batching, PagedAttention KV memory management, and multi-GPU tensor-parallel deployment guidance. docs.vllm.ai
4. AWS and Google Cloud GPU instance pricing pages — on-demand rates for H100-class capacity in us-east-1, as published 2026-09-25. aws.amazon.com · cloud.google.com
5. OpenAI, Anthropic, Google and Mistral pricing documentation — per-token rates, batch tier availability and cached-input discount depths. provider pricing pages
6. Related: Inference Capacity Planner · GPU Model Fit Matrix · Self-Hosted LLM vs API Breakeven · LLM API Pricing Calculator · The FP8 Trap: 2× Spec Sheet, 1.15× Invoice

📜 Copyright & Attribution