The FP8 Trap

The Spec-Sheet Number Is a Peak, Not a Result

An H100 SXM5 does 989 TFLOPS of dense FP16 and 1,978 TFLOPS of dense FP8. That ratio — exactly 2×, by design, on the same silicon — is where every "FP8 doubles throughput" claim comes from. It is also close to irrelevant to your inference bill, for a reason that shows up the moment you look at what a decode step actually does.

Autoregressive decoding produces one token per step, and each step has to read the model's weights out of HBM before it can multiply anything. For Llama-3-8B at FP16 that is 16 GB of weights, and an H100 with 3.35 TB/s of memory bandwidth needs roughly 4.8 ms to stream them — about 210 tokens per second per sequence, before any overhead. The tensor cores sit almost idle during this, because a matrix-vector product at batch 1 has one column. You cannot speed it up by adding arithmetic throughput; you can only speed it up by moving fewer bytes.

That is the entire mechanism by which FP8 helps decode. One parameter in FP8 occupies one byte instead of two. The same model now needs 8 GB of weight traffic per step instead of 16 GB, so the ceiling on decode speed doubles. The ceiling — not the result. Everything else in this briefing is about the distance between that ceiling and what lands on the invoice, and that distance is set by choices that have nothing to do with the tensor-core ratio.

Four Switches Called "FP8"

"We turned on FP8" is an ambiguous sentence in 2026, because at least four distinct changes get described that way. They touch different tensors, they are enabled by different flags, and the returns are not close to each other.

The vLLM documentation is unusually direct about the confusion this produces, and the numbers are worth quoting precisely. Full weight-and-activation quantisation reduces model memory by about 2× with up to 1.6× throughput. The KV-cache flag alone, on the same model, returns about 1.15×. Both are legitimately "FP8," and a team that reads the first number, enables the second switch, and then measures a 15% gain will spend the afternoon looking for a bug that does not exist. The two settings compose — turning on both is the configuration that approaches the headline figure.

What NVIDIA Measured: 1.45× and 1.81×

The most useful published numbers here are NVIDIA's own, because they were produced by the people who sell the hardware, on a stack they control, and they still come out well below 2×. TensorRT-LLM throughput for Llama 3 at batch size 32, normalised per GPU:

Two rows matter more than the speedup column. The 70B FP8 engine runs on two H100s where the FP16 baseline needs four. Every quantised variant of that model fits on one card, which is where the 2× resource saving actually comes from — a saving in GPU count rather than in milliseconds. And the same measurement at batch size 1 reverses the ordering of the formats entirely.

Read the two tables together and the shape of the decision appears. FP8 gains more as the model gets bigger, because a bigger model spends more of each step on weight traffic. FP8 loses to INT4 as the batch gets smaller, because a smaller batch has less arithmetic to hide the memory traffic behind. Neither of those is a property of the format; both are properties of the workload.

Where INT4 Quietly Beats FP8

The row most people skim past is INT4 AWQ at 2.66× on Llama 3 70B at batch 1, against FP8's 1.84×. That is not a rounding artefact, and it is not a vendor preference. It follows from the bandwidth argument in the first section: at batch 1 there is no batching to amortise anything, so the step time is the time to read the weights, and four-bit weights are half the bytes of eight-bit weights. A 70B model needs 141 GB of weight traffic per token at BF16, 70 GB at FP8, and 35 GB at INT4. On a single H100 the last of those is simply faster, and no amount of tensor-core throughput changes it.

What FP8 sells instead is accuracy retention. Measured MMLU degradation across quantisation methods puts the formats in a clear order:

• FP8: 0.14 to 0.87 points. Frequently inside run-to-run noise on a single evaluation.
• INT4 AWQ: 0.85 to 2.11 points. Acceptable for chat and retrieval, visible on reasoning and arithmetic.
• INT8 SmoothQuant: 2.50 to 2.75 points. Counter-intuitively worse than INT4 AWQ on this metric, because the activation quantisation is where the damage happens.

So the honest framing of the choice is not "fastest versus slowest." It is: if the workload can absorb a point or two of accuracy — a support chatbot, a summariser, a classification pass — INT4 is the cheaper format and the faster one. If it cannot — code generation, multi-step reasoning, anything where a wrong token costs a retry — FP8 is the format that buys most of the memory saving while leaving the answers alone. INT8 SmoothQuant sits in an awkward middle: more accuracy loss than FP8 at the same one byte per parameter, which is why it has been losing ground since FP8 hardware arrived.

The 7,010-Token Break-Even

FP8 KV cache is not free at the margin. It adds a small constant cost per attention step — extra scale handling, and a conversion where the kernel has no native FP8 path — and that constant only becomes negligible once the cache is long enough that the halved memory traffic is the dominant term. For Llama-3.1-8B the two terms cross at 7,010 tokens of context.

Below that, FP8 KV cache is slower than leaving the cache in BF16. A deployment built on short prompts — a classification service, a routing layer, a JSON-extraction endpoint that sees 500-token inputs — is exactly the workload where "we enabled FP8" produces a regression and a confused engineer. On the same model at concurrency 1, the fitted inter-token-latency slope fell from 4.37×10⁻⁵ to 2.37×10⁻⁵ ms per token, which is 54% of the BF16 slope and close to the theoretical floor, while the intercept moved only from 6.44 ms to 6.58 ms. That intercept is the constant cost. At short contexts you are paying it without collecting the slope reduction.

Two structural cases never reach break-even at all:

• Models with head_dim = 256. These require two-level accumulation for FP8 attention, which raises time-to-first-token by roughly 1.6× at long context. Prefill in BF16 and quantise only the cache is the usual workaround, and it is worth validating accuracy separately because the two-level path is not bit-identical to the single-level one.
• Stacks with many small sliding-window layers. Each layer adds the fixed cost and contributes little cache to amortise it, so the overhead accumulates across layers without a corresponding traffic reduction.

Above the threshold the gain is dependable. The reference measurement — a single H100 with FlashAttention-3, Llama-3.1-8B, concurrency 8 — moved median inter-token latency from 15.18 ms to 12.93 ms, median time-to-first-token from 763.6 ms to 742.8 ms, and output throughput from 450.3 to 517.5 tokens per second. Two effects combine: each token decodes faster, and the 2× cache reduction lets the scheduler hold more concurrent requests in the same memory, so the card serves more sequences at once.

On A100, FP8 Is Free and Worth Nothing

This is the trap that costs the most engineering time, because it does not fail. Ampere's third-generation tensor cores accelerate FP16, BF16, TF32, INT8 and INT4. They do not accelerate FP8. Frameworks do not error out on this. They fall back to the nearest supported format with a conversion step around it, the job completes, the logs stay clean, and the throughput lands at approximately BF16 levels.

Two things are true simultaneously, and separating them is the whole point:

So on Ampere, FP8 is a memory decision, not a performance decision. It is a perfectly good way to fit a model that would otherwise require a second card — the weights genuinely shrink — and a completely ineffective way to make that card faster. If your fleet is A100-based and your problem is tokens per second, the quantisation menu is INT8 or INT4, and INT8 on A100 delivers roughly a 50% throughput gain against FP8's zero. This is not a driver limitation waiting on a patch. The silicon either has execution units for a format or it does not, and no framework update adds them.

Ada Has FP8. It Isn't Hopper's FP8.

Between "no FP8" and "full FP8" there is a third state that the specification sheets blur together. Ada Lovelace parts — the L40S, the L4, the RTX 4090 — do accelerate FP8, starting at compute capability 8.9 with CUDA 12.1 and cuBLASLt 12.1.3. What they do not have is Hopper's datapath.

The distinction is not academic. FP8's dynamic range is narrow, and keeping values inside it depends on scaling — per-tensor at minimum, per-block for the more aggressive recipes. Hopper's Transformer Engine performs that scaling on dedicated units and tracks a rolling maximum to compute the next step's scale factor without a synchronisation point at every layer. On a part without that hardware the scaling still happens, just on general-purpose paths, and each conversion round-trips through FP16. The result is that the same nominal FP8 precision returns less on Ada than on Hopper, and the gap widens as the scaling recipe gets finer-grained.

There is a practical consequence for capacity planning. "Supports FP8" on a specification sheet tells you which formats the multiply units accept. It does not tell you whether the scaling machinery that makes FP8 usable is the one the hardware accelerates. For a blockwise recipe — microscaling, or anything per-group rather than per-tensor — that second question is the one that decides whether the format pays for itself, and it is the one that datasheets leave unanswered.

Utilisation Outranks Precision

All of the above is second-order. Here is the same AWS instance — p5.48xlarge, eight H100s — priced four ways, with the implied cost of a million output tokens under each. Rates are from June 2026 listings:

The last row is the one to stare at. A three-year commitment at half utilisation costs $3.19 per million FP8 tokens. On-demand FP8 costs $3.69. The cheapest contract in the table becomes the second-most-expensive line the moment the card sits idle half the day — and it is a difference of fourteen cents per million tokens, which is larger than the entire gap between BF16 and FP8 on-demand. Meanwhile switching spot to reserved is worth $0.21 per million, and switching on-demand to spot is worth $1.88.

That ordering is the whole lesson. Fix utilisation, then fix the purchase model, then quantise. Most teams do it backwards, because quantisation is the part with a flag attached and a benchmark to run, and renegotiating a commitment or deleting an idle replica is the part that requires a conversation. The arithmetic does not care which was more fun. A pipeline that runs at 50% duty cycle has already spent more money than any precision change will recover, and no amount of FP8 closes a utilisation gap.

The same logic applies to the model itself. A quantisation decision that drops GPU count from four to two is worth more than any per-token improvement, because it removes a whole card from the bill. The 70B FP8 engine above, serving on two H100s instead of four, is the largest single saving in these tables.

What a Million Tokens Actually Costs

Combining the pieces gives a working estimate for a real deployment. Take Llama 3 70B, batch 32, one on-demand H100-class GPU-hour at $6.88, and the measured throughputs above collapsed to a single card for comparability:

The FP16 row is where the money hides. It is not that 256 tokens per second is slow — it is that 256 tokens per second requires two cards, and the bill counts cards, not tokens. FP8 on 70B wins here by a factor of 3.6× against the FP16 baseline, and the reason is almost entirely the GPU count: 1.81× more throughput per GPU, on half as many GPUs. That is what a "2× speedup" turns into once it is expressed in money, and it is why the model-size dependence in the earlier table matters so much. The same flag on the 8B model returns 1.45× on one card, which is $4.90 per million instead of $7.10 — real, but nowhere near a factor of three.

Two caveats keep these numbers honest. First, they price output tokens only; prompt processing happens at different rates and is what time-to-first-token measures, so a prompt-heavy workload should be modelled through the latency calculator rather than multiplied out from a single throughput figure. Second, they assume the card is doing useful work every hour it is billed, which is exactly the assumption the previous section spent its length dismantling. Run the top row of that table at 50% duty cycle and the FP8 figure becomes $8.24 per million — still better than FP16, and worse than simply running the same workload on spot.

For the memory side of the same decision, the parameter-to-VRAM converters handle FP16 and INT4 directly. The KV cache is modelled separately in the context window inflation calculator — the right place to check whether a long-context service can hold the concurrency its throughput implies.

A Four-Question Decision Procedure

Run these in order. The first one that produces a "no" ends the analysis, because every later question assumes the earlier ones passed.

1. Is the GPU generation native for FP8? Hopper (9.0), Blackwell (10.0), or Ada (8.9) — yes. Ampere (8.0) and older — stop here, and choose INT8 or INT4 instead. On A100 the throughput answer is no regardless of what the framework accepts without complaint.
2. Is the model large enough to care? The measured gain runs from 1.45× at 8B to 1.81× at 70B at batch 32. If the model is small, the weight traffic is not the dominant term and FP8 will disappoint. If it is large enough to need a second GPU at BF16 and only one at FP8, the win is structural rather than incremental.
3. What is the operating batch size? At batch 1, INT4 beats FP8 outright (2.66× against 1.84× on 70B) because decode is pure weight traffic. High-concurrency serving is where FP8's accuracy advantage becomes affordable; single-stream interactive use is where INT4's bandwidth advantage wins.
4. Does the context length clear the break-even? For FP8 KV cache specifically, below roughly 7,000 tokens the overhead dominates and the change is a regression. Check head_dim as well — 256-dimensional heads need two-level accumulation that costs about 1.6× on time-to-first-token at long context.

And before any of the four, check the utilisation number. A GPU that is idle half the month costs more than the precision decision can save, in every configuration in the table above.

Frequently Asked Questions

Methodology & Disclosure

Tensor-core peak figures (FP16 989 TFLOPS, FP8 1,978 TFLOPS dense on H100 SXM5) are the manufacturer's published dense ratings. Throughput tables are NVIDIA's published TensorRT-LLM results for Llama 3 8B and 70B at batch sizes 32 and 1, and are normalised per GPU as published. Inter-token-latency, time-to-first-token and the 7,010-token break-even come from the FlashAttention-3 FP8 KV-cache validation study published on the vLLM blog in April 2026, measured on a single H100 at concurrency 1 and 8. MMLU degradation ranges are the ranges reported across the same quantisation methods. Instance rates are June 2026 on-demand, spot and three-year reserved listings for AWS p5.48xlarge (eight H100s), divided by eight for the per-GPU figures. Cost-per-million-token figures are computed from those rates and the throughputs above, assume continuous useful utilisation, and cover output tokens only. This site takes no sponsorship, no affiliate commission and no vendor payment of any kind; the vendors named here have no relationship with jslet and did not review this briefing.

References & Further Reading

1. NVIDIA Developer Blog. "Post-Training Quantization of LLMs with NVIDIA NeMo and NVIDIA TensorRT Model Optimizer." Throughput tables for Llama 3 8B and 70B, batch sizes 32 and 1. developer.nvidia.com
2. vLLM Blog (Kübler, Budhathoki, Kurtić et al.). "FP8 KV cache with FlashAttention-3: validation study." April 2026. Inter-token latency, time-to-first-token, break-even context and failure modes. blog.vllm.ai
3. NVIDIA Transformer Engine. GPU compute-capability feature matrix: FP8 gating at SM 8.9 (Ada, CUDA 12.1 / cuBLASLt 12.1.3 minimum) and full FP8 at SM 9.0 (Hopper). github.com/NVIDIA/TransformerEngine
4. NVIDIA. H100 Tensor Core GPU architecture whitepaper — FP8 E4M3/E5M2 formats, Transformer Engine, and the dense FP16/FP8 throughput ratings. nvidia.com
5. Related: LLM Inference Latency Calculator · Self-Hosted LLM vs API Breakeven · Token Economics: Why Your LLM Bill Is 3× the Sticker Price · GPU Training Costs: The arXiv-to-Bill Gap

📜 Copyright & Attribution

Run the numbers on your own model

The LLM inference latency calculator models throughput against memory bandwidth across 12 models and 10 GPUs, including FP8. If the question is whether self-hosting beats an API at your volume, the breakeven calculator takes the throughput number this briefing produces and turns it into a decision.

← Back to all briefings