GPU Training Costs

The Number on the Paper Isn't the Number on Your Invoice

Every ML paper published on arXiv in the last two years has a methodology section that reads something like this: "We trained on 8×A100-80GB for 72 hours using PyTorch FSDP with mixed precision."

Multiply it out: 8 × 72 = 576 GPU-hours. Lambda Labs lists A100-80GB at $1.29/hr on-demand. 576 × $1.29 = $743. Seems reasonable. The grant proposal writes itself.

Then the invoices arrive and it's $2,100. Not because anyone lied — $743 is right there on the pricing page — but because a training run is not a car on a highway traveling at a constant speed. It's a machine with moving parts that break, and every time something breaks, the GPU clock keeps ticking. A training run at scale has more in common with operating a chemical plant than renting a server: uptime is a percentage, not an assumption, and the cost of downtime is billed by the minute.

This article is about the gap. The things that happen between "we rented 8 GPUs" and "the model converged" that nobody budgets for, because nobody wrote them down. I built a GPU training cost estimator to model this stuff because I got tired of explaining it from scratch every time a friend emailed me "how much should I budget for this run?" — and I want the numbers here to hold up when your CFO asks the same question.

Provider Pricing: The Raw Numbers

First, the baseline. Here is what public cloud GPU pricing looks like right now, on-demand, for the two GPUs that matter for most training workloads. These are list prices — enterprise commits and long-term reservations can reduce them, but for independent teams, this is what you'll actually pay.

Sources: Provider pricing pages and community benchmarks, June 2026. Vast.ai prices are spot-market averages. Lambda Labs, RunPod, and Vast.ai pricing is significantly more transparent than hyperscaler GPU pricing, which often requires contacting sales for committed-use discounts.

At first glance, the spread is straightforward: Vast.ai is cheapest, Lambda is the best value for reliability, the hyperscalers charge a premium that ranges from 2× (GCP) to 5× (Azure H100) over the market rate. If this were the whole story, the decision would take 30 seconds. It isn't.

These rates are for compute only. No storage. No network egress. No checkpoint retention. No data preprocessing instances. Just the GPU sitting in a server with its fans on. To get from a per-hour rate to an actual training budget, you need to model everything around the GPU.

The Five Leaks: Why Real GPU-Hours Always Exceed Paper GPU-Hours

I've talked to enough ML teams to notice a pattern: when you ask them to estimate their actual GPU spend for a training run, they take the paper's number and add 30–50%. No spreadsheet. No formal model. Just scar tissue.

The gap comes from five structural sources. Each one is individually modest. Together, they turn "576 GPU-hours" into 900–1,200.

1. GPU Utilization: No One Gets 100%

A GPU at 100% utilization means every Streaming Multiprocessor is occupied every clock cycle — compute-bound, no idle bubbles, no memory stalls. Real training never hits this. The best-case scenario for an optimized training loop with large matrix multiplications and high arithmetic intensity (a 70B model with batch size 4M tokens on H100) is about 85% utilization. The remaining 15% goes to checkpoint writes to disk, NCCL all-reduce communication between gradient accumulation steps, data loading from network-attached storage, and the brief moments when the CPU is preparing the next batch and the GPU is waiting.

For I/O-bound workloads — training on data stored in S3, or working with image/video datasets where decoding is a non-trivial cost — utilization drops to 60–70%. For debugging runs with small batch sizes (which under-utilize the GPU's parallelism), you'll see 40–50%.

This means 576 paper GPU-hours at 85% utilization = 678 billed GPU-hours. You paid for 102 hours of idle GPU time that the paper never mentioned. Our GPU training cost estimator includes a utilization slider that models this directly — set it to 70% for an I/O-heavy pipeline and watch the total jump 40%.

2. Xid Errors, NCCL Timeouts, and the Failure Budget

The NVIDIA driver reports hardware errors using codes called Xids. Xid 48 is a double-bit ECC error — the GPU detected an uncorrectable memory error and will reset itself. Xid 64 is a "GPU has fallen off the bus" — the PCIe link dropped and the device is unreachable until the node reboots. Xid 79 is a thermal shutdown. These codes are not hypothetical. At any scale beyond a single 8-GPU node, you will encounter them.

How frequently? Meta's public infrastructure talks suggest a GPU failure rate of roughly 1–2% per week for large-scale training clusters. That means on a 64-GPU training run (8 nodes × 8 GPUs), you should expect roughly one GPU to fail every week. If your training framework can't handle a single GPU failure without restarting the entire job, that's one full restart per week, wiping out however many GPU-hours had been accumulated since the last checkpoint.

Even on single-node training runs, transient NCCL timeouts (NCCL is NVIDIA's communication library for multi-GPU coordination) and CUDA OOM errors from memory fragmentation add an empirical 10–15% overhead to the total GPU-hours consumed. This is the failure budget. If your budget doesn't include it, your run will overrun it.

3. Hyperparameter Search: The Paper's Hidden Multiplier

The methodology section says "we trained with learning rate 3e-4 and batch size 4M tokens." It does not say "we tried learning rates {1e-4, 3e-4, 1e-3}, batch sizes {1M, 2M, 4M}, and three optimizer configurations before converging on this one."

The paper reports the final, converged, cherry-picked run. The team ran 3–10× more GPU-hours as exploratory sweeps to find the hyperparameters that produced that run. At a minimum, budget 3× the reported GPU-hours for hyperparameter search. If you're training a novel architecture where no prior art exists, budget 5×. The people who published the Llama papers are not guessing — they have institutional knowledge from training Llama-1 and Llama-2. You don't.

4. Checkpoint Storage: The Cost That Compounds

An FP16 70B-parameter model checkpoint is ~140 GB. With optimizer states (Adam momentum + variance in FP32), a full training state checkpoint is ~420 GB. If your script saves every 1,000 steps and you're training for 100,000 steps, that's 100 checkpoints × 420 GB = 42 TB of writes. On AWS EBS gp3 at $0.08/GB-month, retaining those checkpoints for 30 days costs ~$3,360/month in storage — which for a one-month training run adds 8–15% to the compute bill. On Lambda Labs, included local NVMe storage is free up to the instance's drive capacity, but object storage for long-term retention costs extra.

Most teams I've talked to don't include checkpoint storage in their GPU training budget because it's a "storage" line item, not a "compute" one. The cloud provider doesn't care how you categorize it. You're paying either way. Use our Cloud Storage Cost Comparison to model this separately — the numbers will surprise you.

5. Data Egress: The Thing No arXiv Paper Mentions

If your training data lives in AWS S3 and you're renting GPUs on Lambda Labs, every byte of training data crosses a network boundary. At ~$0.05–0.09/GB for cross-cloud egress, a 1 TB dataset costs $50–90 just to move to the GPU instance. Most training datasets compress well (text is highly redundant), and a 10:1 compressed transfer cuts this to manageable levels. But image and video datasets don't compress nearly as well, and multi-terabyte multimodal datasets can generate four-figure egress bills before a single training step runs.

The simplest mitigation: keep your training data in the same cloud as your GPUs. If your pipeline is already in AWS, the AWS GPU premium doesn't look as bad once you factor in zero egress between S3 and the training instance. The AWS Egress Cost Calculator and GPU training cost estimator together will give you the full picture.

The Spot Question

Spot instances are the closest thing the GPU cloud market has to a free lunch: 50–70% discount for the same hardware, in exchange for the provider's right to reclaim the instance with short notice. For a 72-hour training run that costs $743 on Lambda Labs on-demand, the spot equivalent is around $297–371. The savings are not marginal — they're the difference between "we can afford to experiment" and "we can't."

But spot economics have a structure, and if you don't understand it, you'll make one of two mistakes: overpaying on on-demand because you're scared of preemptions, or losing entire training runs to preemptions because you convinced yourself "it probably won't happen."

Preemption Frequency: What the Data Says

The decision tree is simpler than most people make it:

• Single-node, resumable checkpoints, run under 24 hours: Use spot. The preemption probability is low enough over a 24-hour window (on GCP: ~50% if you're unlucky, ~20% on a good day) that the expected cost — including wasted GPU-hours from restarts — is still well below on-demand. With 15-minute checkpoint intervals and a tested resume path, a preemption costs you 15 minutes of progress and 30 minutes of re-provisioning. Over a 24-hour run with one preemption, you lose 45 minutes out of 24 hours — a 3% overhead. The 60% spot discount demolishes that.
• Multi-node (4+ GPUs), any duration: Think hard. A single-node preemption in a multi-node NCCL topology brings down the entire communicator. All nodes restart from the last checkpoint. The overhead is proportional to the number of nodes. For an 8-node, 64-GPU training run with one preemption every 12 hours, you're restarting the entire cluster twice a day — and the wasted GPU-hours from those restarts (all 64 GPUs idle during re-provisioning and NCCL re-initialization) can exceed the spot savings. Elastic PyTorch and TorchElastic can mitigate this by dynamically resizing the topology, but most teams aren't running them in production. For multi-node, Lambda Labs on-demand or reserved is the pragmatic choice.
• Run over 72 hours without checkpointing infrastructure: On-demand. Period. A preemption at hour 67 of a 72-hour run without resumable checkpoints wipes the entire $3,000+ you've spent. You will not be refunded. The cloud provider's SREs will not feel bad about it. They reclaimed a GPU because someone else paid full price for it, and their job is to maximize utilization, not to protect your training progress.

For teams doing ongoing research with predictable GPU needs, 1-year reserved instances on AWS/GCP (~40% discount) can make sense. The math on 1-year reserved vs. spot depends on your annual GPU-hour volume. Model it with our RI vs Spot Breakeven Calculator — the crossover point varies by provider and GPU type, but a common threshold is around 3,000 GPU-hours/year. Below that, on-demand is simpler. Above it, the reserved discount pays back the commitment.

Real Workload Cost Models

Here are three realistic training scenarios, end-to-end, with all five leaks modeled. Prices use on-demand Lambda Labs rates (the pragmatic default for independent teams) with 85% utilization, 15% failure overhead, and checkpoint storage included.

Scenario A: Fine-Tuning Llama-3-70B with LoRA

Single-node 8×A100-80GB · ~20 hours · Hyperparameter factor: 3× · Low I/O overhead

Fine-tuning a 70B model with LoRA costs less than a monthly AWS bill for a mid-size SaaS application. This is why everyone is fine-tuning — the cost floor is low enough that experimentation isn't prohibitive. QLoRA on RTX 4090s (Vast.ai ~$0.25/hr) would push this under $150 total, with training ~30% slower per step.

Scenario B: Training a 7B Model From Scratch

8×H100-80GB · ~480 GPU-hours · Hyperparameter factor: 5× · 1 TB dataset

Training a 7B model from scratch costs roughly $6,000 on Lambda Labs — the annual GPU budget for a small research lab. On AWS, the same run costs more than double, which is a strong argument for keeping your training pipeline cloud-agnostic. The difference between Lambda and AWS is not marginal — it's the cost of an entire second training run.

Scenario C: The Full 70B Pretraining Run

64×H100 · ~3,840 GPU-hours · 15T tokens · Hyperparameter factor: 3× · 10 TB dataset

^* Lambda Labs on-demand H100 rate. Lower utilization (75%) reflects NCCL all-reduce overhead at 64-GPU scale. At this scale, reserved pricing or direct cluster contracts bring the effective rate down substantially — the on-demand number is a worst-case ceiling, not what you'd actually pay. Budget $80K–160K for the compute depending on provider and commitment level.

Full pretraining of a 70B model from scratch costs roughly what a senior ML engineer costs fully loaded for a year. This is not a coincidence — it's roughly the scale where it makes more sense to hire someone to fine-tune an existing model than to train from scratch. The crossover between "train" and "fine-tune" is the most important economic decision in applied ML right now.

VRAM: The Hard Ceiling on What You Can Train

Cost is one axis. The other is whether the model even fits. GPU VRAM determines the maximum model size you can train, and the relationship between parameter count and VRAM is well-understood but not always well-communicated.

The rule of thumb for full training in mixed precision (FP16 forward/backward, FP32 optimizer states):

A 7B model needs ~84 GB. A 70B model needs ~840 GB. On 80 GB GPUs with ZeRO-3 sharding (DeepSpeed / FSDP), that's 11×A100 for 7B or 11×A100 for 70B — except ZeRO-3 adds communication overhead, so round up to 12 or 16 for headroom. Use our FP16 VRAM Estimator to plug in your exact numbers.

LoRA changes this equation entirely: only the low-rank adapter matrices and their optimizer states need VRAM. The frozen base model stays in FP16 with no gradients. A 70B model can be LoRA fine-tuned on a single A100-80GB. QLoRA — which quantizes the frozen model to 4-bit — brings the threshold down to a single RTX 4090 with 24 GB. The INT4 VRAM Estimator models the quantized side of this tradeoff. The general principle: if you are not Google, you should be fine-tuning, not pretraining. The VRAM math alone tells you this.

And if your application involves long-context inference or training with large context windows, the KV Cache VRAM Estimator will tell you how much extra memory those attention blocks consume as you go from 4K to 128K token contexts.

How to Budget a Training Run: The Napkin Math Recipe

Here is the formula. Use it before you rent a single GPU:

Where:

• GPU_count: The number of GPUs your model needs to fit in VRAM. Not "what you want" — what fits.
• hours: Estimated wall-clock time for a single converged run. If you don't know, benchmark a small-scale run and extrapolate.
• sweeps: Hyperparameter search multiplier. 3× if you know what you're doing. 5× if the architecture is new.
• rate: The on-demand or spot $/GPU-hr for your chosen provider. Use actual pricing from our GPU training cost estimator, not the headline number from a blog post.
• util: Fractional utilization. 0.85 if you're GPU-bound and optimized. 0.70 for I/O-heavy pipelines. 0.50 if you're still debugging.
• fail%: Failure overhead. 0.15 for a single-node run with resumable checkpoints. 0.20–0.30 for multi-node. 0.50 if you've never run this pipeline at scale before.
• storage + egress: Use our Cloud Storage Cost and Egress Calculator for these.

Plug these numbers into the GPU training cost estimator) and compare providers side by side. The tool computes the full cost — not just GPU-hours — across 6 providers simultaneously.

One more thing: multiply the final number by 1.5 if this is your team's first training run at this scale. The first one always costs more. The second one costs what the first one should have. The third one costs what you budgeted for the first. This is not unique to ML — it's true of any engineering project where the failure modes are unfamiliar — but GPU training extracts the penalty at $100+ per hour, which concentrates the mind wonderfully on getting it right the third time.

Frequently Asked Questions

Methodology & Disclosure

Pricing data is based on publicly available provider rate cards accessed June 2026. Lambda Labs, RunPod, Vast.ai pricing is sourced from their respective public dashboards. AWS/GCP/Azure GPU pricing is sourced from on-demand list rates (us-east regions for AWS, us-central for GCP, East US for Azure). Enterprise negotiated discounts and long-term reserved-instance commitments can reduce hyperscaler rates significantly — the prices shown here are the ceiling, not what a committed enterprise customer pays.

GPU utilization estimates (85% best case, 70% I/O-bound, 50% debugging) are based on community benchmarks and operational experience. Failure overhead estimates (15% single-node, 20–30% multi-node) are drawn from community reports of training infrastructure operations at scale; they are not based on formal studies and should be treated as empirical heuristics. Checkpoint storage costs assume 30-day retention on object storage; actual costs vary with retention policy and storage tier. The VRAM estimation formula (params × 12 bytes for full training) reflects mixed-precision training with AdamW optimizer and is consistent with Hugging Face and DeepSpeed documentation.

Disclosure: jslet is an independent research project. We are not sponsored by any cloud provider, GPU vendor, or ML infrastructure company. The GPU training cost estimator on this site was built because the author spent too long manually computing training budgets in spreadsheets and decided to automate it. No affiliate links, no referral codes.

References & Further Reading

1. Lambda Labs (2026). "GPU Cloud Pricing — On-Demand Instances." Public pricing dashboard for A100, H100, and H200 instances. lambdalabs.com
2. RunPod (2026). "GPU Cloud — Secure and Community Cloud Pricing." runpod.io
3. Vast.ai (2026). "GPU Instance Marketplace — Live Pricing." vast.ai
4. NVIDIA (2026). "H100 Tensor Core GPU Datasheet." Specifications for H100 SXM with 80 GB HBM3, 990 FP16 TFLOPS. nvidia.com
5. NVIDIA (2026). "A100 Tensor Core GPU Datasheet." Specifications for A100 SXM with 80 GB HBM2e, 312 FP16 TFLOPS. nvidia.com
6. Meta AI Research (2024). "The Llama 3 Herd of Models." Training infrastructure appendix detailing GPU failure modes at scale and mitigation strategies. arxiv.org
7. Hugging Face (2026). "Model Memory Calculator — How Much VRAM Do You Need?" huggingface.co
8. DeepSpeed (2026). "ZeRO-3: Memory Efficient Training." Microsoft Research's ZeRO optimization stages for sharding optimizer states, gradients, and parameters across GPUs. deepspeed.ai

📜 Copyright & Attribution

"GPU Training Costs: The arXiv-to-Bill Gap — What Cloud GPU Pricing Pages Don't Tell You (2026)" — jslet Research, June 2026.
https://www.jslet.com/gpu-training-cost-real

📡 Enjoyed this? The arXiv paper forgot to mention the 47 failed training runs. RSS covers one hidden cost of doing infrastructure at scale, every week. Straight from engineers who've paid that bill. RSS Feed → | More options →