Why Quantization Matters for LLM Deployment
Large language models are compute- and memory-intensive. A 70B parameter model in full FP16 precision requires roughly 140 GB of GPU memory just to hold the weights, making it impractical for all but the largest server instances. Quantization reduces the numerical precision of model weights (and sometimes activations), shrinking the memory footprint and accelerating inference with minimal accuracy loss.
For production teams deploying LLMs at scale, quantization is often the difference between a viable service and an unaffordable one.
Common Quantization Formats
Several quantization methods have emerged, each with different tradeoffs in accuracy, speed, and tooling support:
- INT8: 8-bit integer quantization. Offers roughly 2x memory reduction over FP16 with minimal perplexity degradation. Widely supported by TensorRT-LLM, vLLM, and ONNX Runtime.
- INT4: 4-bit integer quantization. Pushes memory reduction to ~4x. Accuracy loss is more noticeable but often acceptable for chat and summarization workloads.
- GPTQ: A one-shot post-training quantization method that uses layer-wise approximate second-order information to minimize weight error. Excellent for INT4 with minimal calibration data.
- AWQ (Activation-aware Weight Quantization): Protects salient weights by analyzing activation distributions. Often outperforms GPTQ at the same bit width, particularly below 4 bits.
- GGUF: The format used by llama.cpp. Supports multiple quantization levels (Q2_Q8) and is optimized for CPU and consumer GPU inference. Ideal for edge and local deployment.
When to Use Each Method
| Method | Best For | Typical Memory Reduction | Tooling Support |
|---|---|---|---|
| INT8 | Production serving, minimal accuracy loss | ~2x | TensorRT-LLM, vLLM, ONNX |
| INT4 | Cost-sensitive serving, moderate accuracy tolerance | ~4x | AutoGPTQ, vLLM, ExLlamaV2 |
| GPTQ | One-shot PTQ, limited calibration data | ~3-4x | AutoGPTQ, Transformers |
| AWQ | Low-bit (3-4 bit) with high fidelity | ~4x | AutoAWQ, vLLM |
| GGUF | Edge, CPU, consumer GPU, local inference | ~2-8x (variable) | llama.cpp, Ollama |
There is no universal best choice. A cloud inference endpoint serving thousands of requests per second might use INT8 via TensorRT-LLM for maximum throughput, while a developer running a 7B model on a MacBook would use GGUF Q4_K_M through llama.cpp.
Practical Considerations
- Calibration data: Post-training quantization methods require a small representative dataset. Poor calibration can cause noticeable accuracy drops.
- Activation quantization: Quantizing weights alone is not enough for some workloads; activations may also need quantization for maximum speedup.
- KV-cache quantization: Quantizing the key-value cache can further reduce memory and improve TTFT, though support varies by engine.
Quantization is now a standard tool in the LLM deployment toolkit. Understanding which method fits your hardware, accuracy budget, and latency requirements is essential for cost-effective production inference.
