AI Model Quantization Explained
Quantization represents model values with lower precision to reduce storage and computation costs. The useful question is not only how many bits are named, but how the quantization method represents and reconstructs the values.
Quantization changes how numerical values are represented
Neural-network weights are numerical values. Quantization represents some of those values using lower precision than the representation used during training or higher-precision inference. The goal is usually to reduce storage, memory bandwidth, memory pressure or computational cost while retaining useful model behavior.
A simple intuition is that fewer representable levels require fewer bits, but also approximate the original values more coarsely. Real quantization methods add structure to make that approximation more useful than merely rounding every value in the same way.
Lower precision reduces the representation budget
A value stored in 16-bit form has a larger raw representation than one encoded with 8 or 4 bits. Across billions of parameters, that difference can substantially change weight storage.
However, labels such as 4-bit describe an important part of a quantization scheme, not necessarily every byte associated with the model. Some values or tensors may use different precision, and supporting metadata is required.
Scales, groups and metadata matter
Quantization methods can divide weights into groups and store additional information such as scales used to map quantized values back into a useful numerical range during computation. Different schemes make different choices about group size, calibration, which tensors are quantized and how outliers are handled.
That means two methods both described broadly as 4-bit can have different file sizes, speed characteristics and output behavior.
The bit label describes the encoded weights or parts of them. Metadata, mixed precision, context state and runtime buffers add other memory costs.
The parameter × bits calculation is a baseline, not the final file size
For intuition, multiplying parameter count by bits per parameter gives a theoretical raw-weight quantity. A 7-billion-parameter model at an idealized 4 bits per parameter would correspond to about 28 billion bits, or roughly 3.5 GB of raw encoded weight data.
A real quantized file can differ because the representation includes scales, metadata and potentially tensors stored at other precisions. Runtime memory can differ again because loading and inference introduce additional state.
Quantization is a trade-off that should be tested on the task
Reducing precision introduces approximation. The practical effect depends on the model, quantization method and workload. A lower-bit version may be attractive because it makes a model deployable on constrained hardware, while a higher-precision version may preserve behavior better for a sensitive workload.
There is therefore no universal rule that one quantization level is “best.” Compare variants under the tasks and hardware constraints that actually matter.
Conceptual quantization and local model labels are related but separate layers
This page explains the model-level concept. The Local AI track covers practical formats and labels such as GGUF quantizations, where names like Q4 or Q8 describe concrete packaging choices used by local runtimes.
Keeping these layers separate avoids a common confusion: quantization is a numerical representation technique; GGUF is a model file format and container ecosystem used by local inference tooling.