LLM Quantization Explained
Quantization reduces the precision used to represent model weights. The practical goal is to make inference smaller or faster while keeping acceptable model quality.
What is quantization?
Neural-network weights can be stored using different numerical representations. Quantization reduces the precision used for some or all of those values.
For local LLM inference, the immediate benefit is smaller model storage and lower memory requirements. llama.cpp's quantization tool takes GGUF input — typically higher-precision weights — and produces quantized GGUF output. Its documentation notes the core trade-off directly: reducing precision shrinks the model and may speed inference, but can introduce some loss in accuracy.
What do Q4, Q5 and Q8 mean?
Names beginning with Q identify quantized tensor encodings in the llama.cpp/GGML ecosystem. The number gives a useful indication of the bit range, but filenames such as Q4_0, Q4_K_M and Q5_K_M are not interchangeable labels for one identical algorithm.
The suffix describes the encoding/packing strategy or variant. llama.cpp documents classic quantizations, K-quants and other families, and its current quantizer exposes numerous options.
So “Q4” is a category-level description. When reproducing a benchmark or calculating memory, record the full quantization name.
The three-way trade-off
Quantization is usually a balance among:
Memory and disk size. Lower precision generally reduces the amount of storage required for model weights.
Performance. Smaller weights can reduce memory traffic and may improve inference performance, although the outcome depends on hardware and implementation.
Model quality. More aggressive quantization can alter outputs and degrade quality. The effect is model- and method-dependent; there is no responsible universal percentage that applies to every model and task.
Do not interpret a smaller quantization number as an automatic recommendation. The right variant depends on your memory limit, runtime, hardware and tolerance for quality loss.
How should you choose a quantization?
Start from the constraint you actually have.
If VRAM/RAM is the hard limit, shortlist variants that fit and then compare their quality and speed.
If quality is more important than memory, start with a higher-precision variant your hardware can handle and test whether a smaller quantization changes results that matter to your workload.
If speed is the priority, benchmark on your own backend. File size alone does not predict tokens per second.
For E—DOCEO Lab tests, quantization will always be part of the recorded environment. A result from Q4_K_M should not silently be presented as a result for every version of that model.