Token Economics

Executive Summary

Every LLM provider publishes a pricing table: $X per million input tokens, $Y per million output tokens. The napkin math is seductive. 10,000 requests/day × 2,000 input tokens × $2.00/M = $40/day on GPT-6 Sol. Annualized: $14,600. The CFO approves it. Three months later the bill is $120/day — $43,800/year — and nobody can explain the gap.

The gap isn't a billing error. It's five structural leaks between the pricing page and your credit card: workload ratio asymmetry (output tokens are priced 5–6× higher than input tokens, and your workload determines the ratio), tokenizer variance (the same text is 5–15% more tokens on one provider's tokenizer than another's), unused prompt caching (cache reads bill at 2–20% of the input rate depending on vendor — and most teams never configure it), batch processing discounts (50% off for async workloads — sitting on the table for any non-realtime use case), and rate-limit retry overhead (failed requests get retried, consuming tokens twice).

This briefing deconstructs each leak with real pricing data from OpenAI, Anthropic, Google Gemini, DeepSeek, Mistral, and Together AI. We run the breakeven math for self-hosted vs API across model scales. And we provide a practical decision framework that turns the five leaks into five cost levers — things you can configure today to cut your bill by 47% before changing a single line of application code.

Leak #1: Workload Ratio Asymmetry — Your Use Case Is the Multiplier

LLM pricing pages separate input and output token costs. The ratio between them is determined by your workload — and it's the single largest variable in your final bill. Output tokens are priced 5× higher than input tokens on most current models — 6× on Gemini 3.1 Pro, 3× at DeepSeek and Mistral Large 3. The workload that minimizes output tokens (summarization) is fundamentally cheaper than the workload that maximizes them (code generation), even at the same provider with the same daily request count.

The Workload Cost Spectrum

Consider five canonical workloads, each at 10,000 requests/day on GPT-6 Sol ($2.00/M input, $10/M output):

The same 10,000 requests/day on the same GPT-6 Sol model spans from $43,800/year (chat) to $131,400/year (translation) — a 3.0× spread — driven entirely by the input-to-output token ratio. This is before comparing providers. When you layer provider choice on top, the spread widens to 10× or more.

The fix: Measure your actual input-to-output token ratio in production before estimating costs. Most teams guess 1:1 because that's what the pricing page implies. Almost no real workload is 1:1. The LLM API Pricing Calculator includes workload presets for chat, summarization, code generation, RAG, and translation — select the one that matches your use case, then adjust token counts to your measured values.

Leak #2: Tokenizer Variance — 1,000 Tokens Isn't Universal

Every LLM provider uses a different tokenizer. OpenAI uses its tiktoken tokenizers. Anthropic uses a proprietary BPE tokenizer optimized for conversational English. Google Gemini uses SentencePiece. DeepSeek uses a BPE tokenizer optimized for Chinese+English bilingual text. Mistral uses a SentencePiece derivative. The same 10,000-character document produces a different token count on each — and that difference compounds with every request.

For English text, the variance is typically 5–15%. For multilingual text (especially CJK languages), the variance can reach 15–30%. A Chinese legal document with interspersed English technical terms hits the worst case on several tokenizers simultaneously.

Why this matters: When you compare per-token pricing across providers, you're comparing different units. Provider A at $2.00/M tokens with a 10% less efficient tokenizer is effectively the same price as Provider B at $2.20/M tokens — but only for your specific content type. A provider that's cheaper on English technical documentation may be more expensive on Chinese customer support transcripts.

The fix: Before committing to a provider, benchmark your actual production text through each candidate's tokenizer. The difference is measurable in hours. If you're doing 1M+ requests/day, a 10% tokenizer efficiency gap is thousands of dollars per month — more than enough to justify the benchmarking effort.

Leak #3: Prompt Caching — The 95% Discount Nobody Configures

Anthropic introduced prompt caching in August 2024. OpenAI followed with automatic caching in late 2024. Google Gemini launched context caching in early 2025. The discounts are substantial:

Now consider what's actually cacheable in a typical LLM application:

• System prompt: 500–2,000 tokens. Identical for every request. 100% cacheable.
• Few-shot examples: 500–3,000 tokens. Change rarely (when you update the prompt template). 100% cacheable between updates.
• RAG context: 2,000–8,000 tokens. Varies per query but often shares substantial overlap (same document chunks served to multiple users asking similar questions). Partially cacheable — 20–40% typical.
• Conversation history: 1,000–10,000 tokens. Unique per conversation. Not cacheable.

Real example: A customer support chatbot with a 1,500-token system prompt, 1,000 tokens of few-shot examples, and 3,000 tokens of retrieved knowledge base articles per query. Total input: 5,500 tokens per request. Cacheable: 2,500 tokens (45%). At 10,000 requests/day on Claude Sonnet 5, with caching configured:

• Naive annual cost: (5,500 × 10,000 × 365 / 1e6 × $2.00) + (800 × 10,000 × 365 / 1e6 × $10.00) = $40,150 + $29,200 = $69,350/year
• With caching: (3,000 × 10,000 × 365 / 1e6 × $2.00) + (2,500 × 10,000 × 365 / 1e6 × $0.20) + output = $21,900 + $1,825 + $29,200 = $52,925/year
• Savings: $16,425/year (24%) — from a single configuration change.

The fix: Identify your cacheable prefix tokens (system prompt, few-shot examples, static context). Structure your API calls so they appear at the beginning of every prompt — most caching implementations require the cacheable portion to be a prefix. Anthropic requires explicit cache point marking; OpenAI and Google handle it automatically for qualifying prefixes. Measure your cache hit rate in production and feed it into the LLM API Pricing Calculator to see the savings.

Leak #4: Batch Processing — Half Price If You Can Wait 24 Hours

OpenAI, Anthropic, Google and Mistral all publish batch (asynchronous) endpoints with 50% off standard pricing; DeepSeek and Together AI publish no batch tier at all. The tradeoff: up to 24-hour completion window instead of a real-time response. For offline workloads — evaluation runs, dataset labeling, embedding generation, nightly summarization jobs, synthetic data generation — there is literally no downside. The 50% discount is free money.

Combined with prompt caching, the stackable discounts are dramatic. On Anthropic, a batch workload with 45% cacheable input tokens achieves an effective rate of:

• Cached input: 50% batch discount × 90% caching discount = 5% of sticker price
• Uncached input: 50% batch discount = 50% of sticker price
• Output: 50% batch discount = 50% of sticker price

For the customer support chatbot example above, adding batch processing where applicable (say 60% of requests are non-realtime — ticket summarization, weekly analytics, knowledge base updates):

• Optimized annual: $52,925 × (0.4 + 0.6 × 0.5) = $37,048/year
• Total savings vs naive: $32,302/year (47%)

The fix: Segment your LLM traffic into realtime and async. Realtime: user-facing chat, code completion, live translation. Async: summarization, evaluation, labeling, data processing, report generation. Route async traffic to batch endpoints. The infrastructure change is an API endpoint swap — no model changes, no prompt changes, no application logic changes.

Leak #5: Rate Limits and Retry Overhead — Paying Twice for the Same Token

Every LLM API has rate limits: requests per minute (RPM) and tokens per minute (TPM). When your application hits the limit, the API returns a 429 error. Most client libraries automatically retry — but the retried request consumes tokens again. A request that succeeds on the third retry consumed tokens three times for one successful output.

The cost of retries is not theoretical. For GPT-6 Sol at $2.00/M input + $10/M output, each failed-and-retried 2,000-token prompt wastes $0.004 in input tokens. At 10,000 requests/day with a 2% retry rate, that's $0.80/day in wasted input tokens — $292/year. Small, but it compounds with scale.

The bigger cost is architectural: teams over-provision by deploying to multiple providers simultaneously to avoid rate limits, which means paying for capacity they don't need. Or they upgrade to higher rate-limit tiers (requiring prepaid commitments) before their usage justifies it.

The fix: Implement exponential backoff with jitter, not naive retry. Monitor your retry rate — if it exceeds 1%, you need either higher rate limits or a queuing layer. Batch endpoints have separate (and typically much higher) rate limits — moving async traffic to batch kills two birds: the batch discount and the rate limit headroom.

Provider-by-Provider Pricing: The 2026 Landscape

As of September 2026, the LLM API market has stratified into four tiers. The pricing below is public list pricing for on-demand API access. All prices are per 1 million tokens.

Tier 1: Premium Flagship ($20–$50/M output)

OpenAI GPT-6 Astra ($10/$50): The most expensive API on the market by output price. A 90% cache-read discount softens it for prefix-heavy workloads, and the batch tier takes another 50% off, but there is no workload on which Astra is the cheap choice — it is bought for capability. Best for: tasks where model quality is non-negotiable.

Anthropic Claude Opus 5.5 ($4/$20): Undercuts a $50-output competitor by 60% while sitting in the same quality tier. Its cache read is the deepest in the market at 95% off — an effective $0.20/M, or $0.10/M once batch is applied. For high-cache-hit, asynchronous workloads, Opus 5.5 can beat mid-tier models on effective price despite the flagship sticker. Best for: cacheable, high-stakes workloads where quality is non-negotiable.

Tier 2: Standard Flagship ($7.50–$15/M output)

Kimi K3 via Together AI ($3/$15): The top of the standard tier on output price — open weights served by a third-party host, so you buy the inference rather than the model. No batch tier. Best for: teams that want frontier-adjacent quality without a single-vendor commitment.

Google Gemini 3.1 Pro ($2/$12): The cheapest way to buy a 1M-token context window. Watch the context tiers: above 200K input tokens both the input and the output rate double. 90% cache-read discount. Best for: long-document processing, cost-sensitive workloads needing high capability.

OpenAI GPT-6 Sol ($2/$10): The reference point. Matches Gemini 3.1 Pro on input and sits below it on output, with automatic caching, a 50% batch tier, and no context-tier trap at its published limits. Best for: general-purpose workloads where you want a single provider and moderate cost.

Anthropic Claude Sonnet 5 ($2/$10): Price-matched to GPT-6 Sol on both axes. The 90% caching discount is what separates it — at a 45% cache-hit rate it is cheaper than GPT-6 Sol on input-heavy work. Best for: cacheable workloads, code generation, long-context reasoning.

Google Gemini 3.8 Flash ($1.50/$7.50): Priced 25% above the Pro tier's floor on input but with materially lower output cost. Carries introductory pricing of $0.75/$3.75 through 2026-12-31, which reverts on 2027-01-01 — plan long-horizon budgets on the standard rate. Best for: high-volume generation where output tokens dominate.

Mistral Medium 3.5 ($1.50/$7.50): Priced above Mistral's own Large 3 — a genuine inversion, not a typo. If you are shopping the Mistral line on price alone, Large 3 is the cheaper model. Best for: European deployments, multilingual workloads, sovereignty requirements.

Tier 3: Efficiency ($2.50–$5/M output)

Anthropic Claude Haiku 4.5 ($1/$5): 8× GPT-6 Luna's output price at full rate, but a 90% cache-read discount closes much of that gap on cacheable light tasks. Best for: cacheable lightweight tasks, multi-turn conversations with static system context.

GLM-5.3 via Together AI ($1.40/$4.40): Open weights, 3× output-to-input ratio, 81% cache-read discount. The best price-to-capability point in the open-weight set. No batch tier. Best for: self-hostable-class capability at hosted prices.

DeepSeek-V4-Pro ($1.32/$3.96 peak): Half price off-peak, which makes the effective rate $0.66/$1.98 if your traffic can shift. 98% cache-read discount. No batch tier. Best for: cost-sensitive workloads needing flagship-adjacent capability, Chinese-language applications.

Google Gemini 3.5 Flash-Lite ($0.30/$2.50): The odd shape in the market — cheap input, mid-tier output. Suits workloads where you send a lot and generate a little. Best for: classification, routing, and extraction at scale.

Tier 4: Disruptor / Budget ($0.50–$1.50/M output)

Mistral Large 3 ($0.50/$1.50): A Large-tier model at Small-tier prices, and cheaper than Mistral's own Medium 3.5. 3× output-to-input ratio, 90% cache-read discount, 10% surcharge for regional inference. Best for: high-volume generation where capability requirements are moderate.

DeepSeek-V4.1-Flash ($0.30/$1.20 peak; $0.15/$0.60 off-peak): The price disruption that forced multiple rounds of cuts across the market. Off-peak rates are half the peak rate; peak hours are 01:00–04:00 and 06:00–10:00 UTC, Monday to Friday. Chinese+English optimized tokenizer means English-only workloads may see slightly higher token counts. Best for: cost-sensitive workloads at volume.

MiniMax M3 via Together AI ($0.30/$1.20): Price-matched to DeepSeek-V4.1-Flash's peak rate, with a 80% cache-read discount and no peak/off-peak structure. Best for: steady-volume workloads that cannot shift to off-peak hours.

Mistral Small 4 ($0.15/$0.60): Matches the classic budget ratio with a 90% cache-read discount. Best for: high-volume filtering and simple Q&A.

OpenAI GPT-6 Luna ($0.10/$0.50): The new budget floor — the lowest published output price from any frontier lab, with automatic caching and a 50% batch tier behind it. Best for: classification, extraction, simple Q&A, high-volume filtering.

The Self-Hosted vs API Breakeven: When Does Buying GPUs Beat Paying Per Token?

The question every LLM-heavy startup asks at Series A: should we rent GPUs and self-host, or keep paying the API per token? The answer depends on model size, utilization rate, and which API tier you're comparing against.

The GPU Cost Baseline

GPU rental pricing (mid-2026, reserved instances, per GPU/month):

• NVIDIA H100 (80 GB): ~$1,800/month reserved ($2.50/hr), ~$3,000/month on-demand
• NVIDIA A100 (80 GB): ~$1,200/month reserved, ~$2,000/month on-demand
• NVIDIA L40S (48 GB): ~$800/month reserved

Model Sizing: What Fits Where

• 7–8B parameter model (small open-weight tier): 1× H100 or 1× L40S. VRAM: ~14 GB in FP16, fits comfortably.
• 70B parameter model (mid-size open weights): 2–4× H100 with tensor parallelism. VRAM: ~140 GB in FP16, requires multi-GPU.
• 400B+ parameter model (large open-weight MoE): 8–16× H100. Not practical for most teams to self-host.

Breakeven Math: 8B Model

Self-hosted cost: 1× H100 reserved = $1,800/month. At 70% utilization (realistic for a well-tuned inference server with continuous traffic), this delivers approximately 1,500 tokens/second output. At 800 output tokens per request: ~6,750 requests/hour, ~4.86M requests/month.

API equivalent (DeepSeek-V4.1-Flash at $0.30/$1.20 peak, same token volumes): (4.86M × 2,000 input tokens / 1e6 × $0.30) + (4.86M × 800 output tokens / 1e6 × $1.20) = $2,916 + $4,666 = $7,582/month.

API equivalent (GPT-6 Luna at $0.10/$0.50): (4.86M × 2,000 / 1e6 × $0.10) + (4.86M × 800 / 1e6 × $0.50) = $972 + $1,944 = $2,916/month.

Against DeepSeek and budget APIs, self-hosting an 8B model wins above roughly 20% utilization (~1,350 requests/hour). Against GPT-6 Luna, self-hosting breakeven is closer to 45% utilization (~3,000 requests/hour). Below that, the API is cheaper — and you don't manage infrastructure.

Breakeven Math: 70B Model

Self-hosted cost: 3× H100 reserved = $5,400/month. At 70% utilization: ~3,000 tokens/second output. At 800 output tokens/request: ~13,500 requests/hour, ~9.72M requests/month.

API equivalent (DeepSeek-V4.1-Flash): $5,832 + $9,331 = $15,163/month. Self-hosting wins at any utilization above 25%.

API equivalent (Claude Sonnet 5 at $2/$10): $38,880 + $77,760 = $116,640/month. Self-hosting wins at any utilization above 3%. The breakeven is essentially immediate for premium APIs at this scale.

The Utilization Reality Check

Most teams overestimate their GPU utilization. The theoretical breakeven is not the real breakeven. Self-hosted inference servers sit idle during traffic valleys (nights, weekends, holidays). The API charges zero for idle time. The three questions that determine which side of the breakeven you're on:

1. Is your traffic steady or bursty? Steady (continuous high throughput): self-hosted wins. Bursty (business hours only, spike-driven): API wins.
2. Can you batch requests? If yes, self-hosted utilization goes up and API costs go down (batch discount). The advantage shifts to API for most scales.
3. Do you have GPU infra expertise on staff? Self-hosting a 70B model across 3 GPUs requires understanding of tensor parallelism, quantization (AWQ/GPTQ/FP8), continuous batching (vLLM/TGI), and GPU node management. The engineering cost is real — allocate 0.25–0.5 FTE for a production self-hosted deployment.

Multi-Provider Architecture: Hedging Your Token Bets

Production LLM applications increasingly run across multiple providers — not for cost savings (though that's a side benefit), but for reliability. A single-provider architecture has a single point of failure: if the API is down, your application is down. Given that every major LLM provider has experienced multi-hour outages in 2025–2026, multi-provider is becoming table stakes for production deployments.

The architectural pattern is straightforward: a thin routing layer (LiteLLM, OpenRouter, or a hand-rolled proxy) that accepts OpenAI-format API calls and routes them to the configured backend. Fallback logic: try primary provider, if 429 or 5xx, retry on secondary, then tertiary. The routing layer also handles cost tracking per provider — essential for measuring whether your primary choice is still the right one.

Cost implications: Multi-provider architecture adds ~5–15% to your API bill (failed requests that get retried on a fallback provider, plus the overhead of routing). But it eliminates the far more expensive outage cost. The math is simple: if a single-provider outage costs you $X in lost revenue per hour, and the outage probability is Y% per year, multi-provider is worth it when the expected loss (X × Y) exceeds the multi-provider overhead. For most revenue-generating applications, the breakeven is unambiguous.

Multi-provider also enables cost-aware routing: send realtime requests to the cheapest provider that meets your latency SLA, send async batch requests to the cheapest period, and reserve the expensive flagship model for the 5% of requests that actually need it. A three-tier routing strategy (budget model for 80% of traffic, standard model for 15%, flagship for 5%) can cut your blended per-token cost by 60–80% compared to routing everything to the flagship.

Practical Decision Framework: Five Questions to Ask Before You Pick a Provider

1. What's your actual input-to-output token ratio? Measure it in production. Don't guess. The ratio is the single largest cost multiplier — and it determines which provider's pricing structure is cheapest for your specific workload.
2. What percentage of your input tokens are cacheable? Count your system prompt, few-shot examples, and static context. If it's above 20%, a 90–95% cache-read discount can make a nominally pricier Claude model cheaper than a cheaper competitor. Run the numbers — don't compare sticker prices, compare effective prices with your actual cache hit rate.
3. What percentage of your traffic can tolerate 24-hour latency? Offline processing, evaluation, labeling, report generation — anything that doesn't need a realtime response qualifies for batch pricing at 50% off. Moving 30% of traffic to batch cuts your blended cost by 15% with zero model quality impact.
4. Is your traffic steady or bursty? Steady, high-throughput traffic favors self-hosting (breakeven at surprisingly low volumes). Bursty traffic favors API (you pay zero for idle GPU time). Be honest about your utilization — most teams overestimate it.
5. Do you need multi-provider for reliability, not just cost? If yes, the cost conversation shifts: you're optimizing for blended cost across 2–3 providers with fallback logic, not the cheapest single provider. A three-tier routing strategy (budget/standard/flagship) yields the best blended cost for most production workloads.

Use the LLM API Pricing Calculator to model your specific workload across all 17 models from 6 providers, with caching and batch optimization. All calculations are client-side — your token counts and request volumes never leave your browser.

📜 Copyright & Attribution