AI models contain large arrays of numbers: weights, activations, and temporary values used during inference. Those numbers are normally stored in a numeric format with a fixed number of bits.
Quantization represents some of those values with fewer bits. For example, a model may store weights as 8-bit or 4-bit integers instead of 16-bit floating-point numbers.
Using fewer bits can reduce memory, memory bandwidth, and sometimes latency. The tradeoff is approximation: the quantized values cannot represent every original value exactly.
Precision and model memory
A bit is one binary digit. More bits allow more distinct values.
Common formats include:
- FP32: 32-bit floating point.
- FP16 or BF16: 16-bit floating point.
- INT8: 8-bit integer.
- INT4: 4-bit integer.
Ignoring metadata and runtime overhead, weight memory is approximately:
memory = parameter_count x bits_per_parameter / 8
For a 7-billion-parameter model:
FP16: 7B x 16 / 8 = 14 GB
INT8: 7B x 8 / 8 = 7 GB
INT4: 7B x 4 / 8 = 3.5 GB
Real deployments need more memory for scales, zero points, activations, the KV cache, temporary buffers, and serving software. Quantizing the weights does not make all of those costs disappear.
How values are mapped
A simple integer quantizer maps a range of floating-point values into a limited set of integer levels.
For a symmetric 8-bit scheme:
quantized = round(original / scale)
restored = quantized x scale
The scale controls how much real numeric range each integer step represents. The restored value is close to the original, but usually not identical.
Example:
scale = 0.1
original 0.26 -> integer 3 -> restored 0.30
original -1.04 -> integer -10 -> restored -1.00
The difference between the original and restored value is quantization error.
Some schemes also use a zero point so an integer value maps exactly to real zero. Others use codebooks or non-uniform levels that place more representable values where weights occur most often.
Per-tensor, per-channel, and grouped quantization
One scale for an entire tensor is simple, but a few large outliers can force a wide range and reduce precision for most values.
Finer-grained schemes use different scales:
- Per-tensor: one scale for the whole weight array.
- Per-channel: one scale for each output channel.
- Grouped: one scale for each small group of weights.
More scales can reduce error because each group covers a smaller numeric range. They also add metadata and can require more complex kernels.
LLM 4-bit formats commonly use grouped quantization because one scale for billions of weights would be too coarse.
What can be quantized
Quantization may apply to different parts of a model:
- Weight-only quantization stores weights at lower precision but keeps activations in a floating-point format.
- Weight-and-activation quantization also lowers activation precision.
- KV-cache quantization reduces memory used by cached attention keys and values.
- Training quantization lowers precision during some training calculations.
Weight-only quantization is often the easiest starting point for LLM inference. Activations and KV-cache values can be more sensitive and depend strongly on hardware support.
Post-training quantization
Post-training quantization (PTQ) converts a trained model without retraining it from the beginning.
A typical process is:
- Load the trained floating-point model.
- Measure weight or activation ranges, sometimes using a calibration dataset.
- Choose scales, groups, and target formats.
- Convert values and save a quantized model.
- Evaluate quality and performance.
Weight-only PTQ can require little or no calibration data. Activation quantization usually needs representative examples so the system sees realistic ranges.
Methods differ in how they handle outliers and minimize error. Names such as GPTQ, AWQ, and SmoothQuant refer to particular approaches, not one universal quantization format.
Quantization-aware training
Quantization-aware training (QAT) simulates low-precision effects during training or fine-tuning. The model learns weights that are more robust to the rounding and clipping it will face after conversion.
QAT can preserve quality better at aggressive precision, but it requires training data, compute, and a more complex pipeline. It is not always necessary for weight-only 8-bit or 4-bit deployment.
Fine-tuning and quantization are separate ideas. Fine-tuning changes what the model has learned. Quantization changes how values are represented for storage and computation. A model can be fine-tuned first and quantized afterward.
Why speedups vary
Smaller weights reduce memory traffic, which is important because LLM decoding is often limited by how quickly weights can be moved to compute units.
But lower precision improves speed only when the hardware and serving engine have efficient kernels for that format.
Performance depends on:
- Accelerator support for INT8, INT4, or other formats.
- The cost of dequantizing values during computation.
- Batch size and sequence length.
- Whether the workload is memory-bound or compute-bound.
- Kernel layout and packing format.
- How much non-quantized work remains.
A model file that is half the size is not guaranteed to run twice as fast.
Quality tradeoffs
Quantization error can change model outputs. Effects may appear in:
- Perplexity or next-token prediction.
- Factual and reasoning tasks.
- Code generation.
- Rare languages or domains.
- Long-context behavior.
- Tool-call formatting.
Larger models often tolerate moderate weight quantization well, but there is no universal safe bit width. Some layers and outlier values are more sensitive than others. Mixed-precision schemes keep sensitive parts at higher precision.
Evaluate the tasks that matter for the product. A small average benchmark change can hide a serious regression in one high-value workflow.
Choosing a format
Start with deployment constraints:
- How much accelerator or system memory is available?
- Which low-precision kernels does the target hardware support?
- Is the goal lower cost, lower latency, larger batch size, or local execution?
- How much quality change is acceptable?
- Does the serving engine support the model architecture and quantization format?
Then compare candidates using the same prompts and serving configuration. Measure:
- Model and runtime memory.
- Time to first token.
- Output tokens per second.
- Throughput at realistic concurrency.
- Task quality and structured-output validity.
- Energy or cost per request when relevant.
Common misconceptions
- Quantization does not reduce the number of model parameters.
- A 4-bit model is not automatically four times faster than FP16.
- Lower file size does not include every runtime memory cost.
- Quantization and pruning are different: pruning removes values or structures; quantization changes numeric representation.
- Quantization and distillation are different: distillation trains a smaller model to imitate a larger one.
- One benchmark score is not enough to approve a production conversion.
The key idea
Quantization trades numeric precision for smaller, potentially faster model execution. The useful result depends on the format, grouping, hardware kernels, and measured quality on real tasks.