LLM Parameters to INT4 Quant VRAM Calculator
70B at INT4: ~35 GB — barely fits on a single A100-80GB. At FP16 the same model needs 140 GB: two A100s at $6/hr each. Quantization stopped being about accuracy tradeoffs. Now it is about whether your model fits on a GPU at all.
🗜️ Quantization Configuration
Specify your model and quantization parameters. The engine computes compressed weights, scale factor overhead, and compares against the FP16 baseline.
INT4 Quantization: How 4-Bit Compression Shrinks LLM VRAM Footprints
Post-training quantization (PTQ) at 4-bit precision is the dominant strategy for running large language models on consumer and prosumer GPUs. Techniques like GPTQ (Generative Pre-Trained Transformer Quantization) and AWQ (Activation-Aware Weight Quantization) compress model weights from 16 bits to 4 bits — a 4× reduction — with minimal perplexity degradation. However, quantization introduces a second memory cost: scale factors stored at higher precision that grow as group size decreases.
How This Calculator Works
This tool models the two-component memory footprint of a group-wise quantized model:
| Component | Formula | Description |
|---|---|---|
| Quantized Weights | Parameters × Precision Bits ÷ 8 | The compressed weight matrix. At 4 bits, a 70B model stores weights in 35 GiB instead of 140 GiB at FP16 — a 75% reduction. |
| Scale Overhead | (Parameters ÷ Group Size) × 2 bytes | Each group of weights shares a 16-bit floating-point scale factor. Smaller group sizes improve accuracy but increase scale memory. At group size 128, a 70B model has ~546 million scale factors consuming ~1.02 GiB. At group size 32, that jumps to ~4.07 GiB. |
| Total VRAM | (Quantized Weights + Scale Overhead) × 1.15 | A 15% overhead buffer accounts for activation memory, CUDA context, and the inference framework runtime (e.g., ExLlamaV2, AutoGPTQ). |
Group Size Trade-Off: Accuracy vs. Memory
Group size is the central tuning knob in GPTQ/AWQ quantization. It defines how many weights share a single scale factor, creating a direct accuracy-memory trade-off:
| Group Size | Scale Overhead (70B Model) | Accuracy Impact | Use Case |
|---|---|---|---|
| 32 | ~4.07 GiB | Negligible perplexity increase. Near-FP16 quality. | High-quality chat, code generation, academic benchmarks. |
| 64 | ~2.04 GiB | Very minor perplexity increase (~0.1–0.3 PPL). | General-purpose inference, acceptable for most production use cases. |
| 128 | ~1.02 GiB | Moderate perplexity increase (~0.3–0.8 PPL). Standard GPTQ default. | Consumer GPU deployment, throughput-optimized serving. |
| 256 | ~0.51 GiB | Noticeable degradation on reasoning tasks. | Extreme compression, edge devices. |
INT4 vs. FP16: The Memory Savings Reality
While INT4 theoretically reduces weight memory by 75%, the effective VRAM savings are typically 65–72% once scale factors and runtime overhead are accounted for. This is still transformative: a 70B model that requires 2–4 A100 GPUs at FP16 can fit on a single RTX 4090 (24 GB) or dual RTX 3090s at INT4. For local inference enthusiasts and small teams, INT4 quantization is the only practical path to running frontier-scale models without enterprise GPU clusters.
Quantization Formats: GPTQ vs. AWQ vs. GGUF
Several INT4 quantization ecosystems exist, each with different trade-offs:
- GPTQ (GPU-optimized): One-shot weight quantization with Optimal Brain Quantizer. Best inference throughput on CUDA GPUs via ExLlamaV2 kernels. Requires calibration dataset but produces highly optimized CUDA kernels.
- AWQ (Activation-Aware): Similar memory footprint to GPTQ but uses activation statistics to protect salient weight channels. Often yields 0.1–0.5 better perplexity than GPTQ at the same bit-width and group size.
- GGUF (CPU/Unified Memory): llama.cpp's format supporting offloading to GPU. Uses k-quant strategies (Q4_K_M, Q4_K_S) that allocate variable bit-widths across layers. Not purely INT4 — some layers retain higher precision. Our calculator models uniform group-wise quantization (GPTQ/AWQ style) rather than mixed-precision GGUF schemes.