Quantization Explained
Learn how lower-precision weights reduce model size and what quality you may trade away.
The basic idea
Quantization represents model values with fewer bits than the original training representation. A lower-precision number needs less memory and can often be processed faster, especially on hardware designed for compact arithmetic. The model’s learned behavior is approximated rather than preserved exactly. Quantization is therefore an engineering trade-off between memory, speed, output quality, and compatibility with the chosen inference runtime.
Weights and activations
Weights are the parameters stored in the model file, while activations are intermediate values created during inference. Weight-only quantization is common because it greatly shrinks storage while keeping computation manageable. Quantizing activations can save additional memory but may be more sensitive to calibration and hardware support. Understanding which parts are quantized helps explain why two files with similar sizes can have different performance.
from pathlib import Path
for path in sorted(Path("models").glob("*.gguf")):
size_gb = path.stat().st_size / 1024**3
print(f"{path.name:32} {size_gb:.2f} GiB")Bit widths and naming
Labels such as eight-bit, six-bit, four-bit, or two-bit describe an approximate precision budget, but naming conventions vary between formats and tools. Some schemes use group-wise scales, mixed precision, or higher precision for sensitive layers. Do not compare labels as if they were universal quality grades. Inspect the format documentation and measure the exact file on the exact runtime you plan to use.
Quality trade-offs
Quantization can slightly change factual accuracy, instruction following, multilingual behavior, coding ability, or formatting reliability. The impact is often small for casual use and significant for narrow tasks that depend on exact tokens or long reasoning chains. Perplexity can provide a useful signal, but application-level tests are more informative. Include difficult examples, structured outputs, and safety cases in the comparison.
Memory and context
The model file is only one part of the memory budget. Runtime overhead, temporary buffers, tokenizer state, and the key-value cache grow with context length and concurrent requests. A quantized model may fit at startup but fail under a long prompt or multiple users. Measure peak memory during realistic requests, and leave headroom for the operating system and other processes.
Calibration and advanced methods
Some quantization methods use representative calibration data to estimate which values are most important, while others apply fixed or learned scales. The best method depends on architecture, runtime kernels, and workload. Calibration data should resemble real traffic without containing secrets. Preserve the calibration recipe and evaluation results, because a quantized artifact without its provenance is difficult to trust or reproduce.
Selecting a quantized model
Start with the highest-precision artifact that fits your latency and memory target, then test lower settings only if they provide a meaningful benefit. Compare answer quality, tokens per second, time to first token, peak memory, and failure rates. Keep the original model or a trusted baseline available. A smaller file is valuable only when it improves the complete product experience.