HQAI Engineering HQ
Search
Checking sign-in...

Quantization

Concept9 min readintermediate

What you’ll understand

  • Explain how quantization stores and computes model values with fewer bits
  • Estimate how numeric precision changes model memory requirements
  • Compare post-training quantization, quantization-aware training, and common tradeoffs

Helpful first: Parameters and weights in AI models · Inference — how models generate output

AI models contain large arrays of numbers: weights, activations, and temporary values used during inference. Those numbers are normally stored in a numeric format with a fixed number of bits.

Quantization represents some of those values with fewer bits. For example, a model may store weights as 8-bit or 4-bit integers instead of 16-bit floating-point numbers.

Using fewer bits can reduce memory, memory bandwidth, and sometimes latency. The tradeoff is approximation: the quantized values cannot represent every original value exactly.

Precision and model memory

A bit is one binary digit. More bits allow more distinct values.

Common formats include:

  • FP32: 32-bit floating point.
  • FP16 or BF16: 16-bit floating point.
  • INT8: 8-bit integer.
  • INT4: 4-bit integer.

Ignoring metadata and runtime overhead, weight memory is approximately:

memory = parameter_count x bits_per_parameter / 8

For a 7-billion-parameter model:

FP16: 7B x 16 / 8 = 14 GB
INT8: 7B x 8  / 8 =  7 GB
INT4: 7B x 4  / 8 =  3.5 GB

Real deployments need more memory for scales, zero points, activations, the KV cache, temporary buffers, and serving software. Quantizing the weights does not make all of those costs disappear.

Illustrative weight memory for a 7-billion-parameter model Three horizontal bars compare idealized weight-only memory. FP16 uses 14 gigabytes at 16 bits per parameter. INT8 uses 7 gigabytes at 8 bits per parameter. INT4 uses 3.5 gigabytes at 4 bits per parameter. A separate note lists runtime memory that remains outside these bars: quantization metadata, activations, KV cache, temporary buffers, and serving software. 7B parameters: idealized weight storage only FORMAT RELATIVE MEMORY FP16 14 GB 16 bits / parameter INT8 7 GB 8 bits / parameter INT4 3.5 GB 4 bits / parameter Extra runtime memory remains: scales, zero points, activations, KV cache, buffers, and serving software.
Lower precision can shrink model weights substantially, but total runtime memory includes more than the weight array.

How values are mapped

A simple integer quantizer maps a range of floating-point values into a limited set of integer levels.

For a symmetric 8-bit scheme:

quantized = round(original / scale)
restored  = quantized x scale

The scale controls how much real numeric range each integer step represents. The restored value is close to the original, but usually not identical.

Example:

scale = 0.1

original  0.26 -> integer  3 -> restored  0.30
original -1.04 -> integer -10 -> restored -1.00

The difference between the original and restored value is quantization error.

Some schemes also use a zero point so an integer value maps exactly to real zero. Others use codebooks or non-uniform levels that place more representable values where weights occur most often.

Mapping floating-point values to integer levels and back Two illustrative floating-point values, 0.26 and negative 1.04, enter a quantizer with scale 0.1. Dividing by the scale and rounding maps them to integer codes 3 and negative 10. Dequantization multiplies by the same scale to restore 0.30 and negative 1.00. The differences, positive 0.04 and positive 0.04 in absolute value, are labeled as quantization error. Illustrative symmetric quantization with scale = 0.1 FLOAT VALUE 0.26 -1.04 Quantize round(value / 0.1) 0.26 -> 3 -1.04 -> -10 INTEGER CODE 3 -10 RESTORED 0.30 error: 0.04 -1.00 abs error: 0.04 Fewer representable levels save space but introduce rounding error.
Quantization stores a nearby representable value rather than the exact original, so the scale determines the size of each rounding step.

Per-tensor, per-channel, and grouped quantization

One scale for an entire tensor is simple, but a few large outliers can force a wide range and reduce precision for most values.

Finer-grained schemes use different scales:

  • Per-tensor: one scale for the whole weight array.
  • Per-channel: one scale for each output channel.
  • Grouped: one scale for each small group of weights.

More scales can reduce error because each group covers a smaller numeric range. They also add metadata and can require more complex kernels.

LLM 4-bit formats commonly use grouped quantization because one scale for billions of weights would be too coarse.

What can be quantized

Quantization may apply to different parts of a model:

  • Weight-only quantization stores weights at lower precision but keeps activations in a floating-point format.
  • Weight-and-activation quantization also lowers activation precision.
  • KV-cache quantization reduces memory used by cached attention keys and values.
  • Training quantization lowers precision during some training calculations.

Weight-only quantization is often the easiest starting point for LLM inference. Activations and KV-cache values can be more sensitive and depend strongly on hardware support.

Post-training quantization

Post-training quantization (PTQ) converts a trained model without retraining it from the beginning.

A typical process is:

  1. Load the trained floating-point model.
  2. Measure weight or activation ranges, sometimes using a calibration dataset.
  3. Choose scales, groups, and target formats.
  4. Convert values and save a quantized model.
  5. Evaluate quality and performance.

Weight-only PTQ can require little or no calibration data. Activation quantization usually needs representative examples so the system sees realistic ranges.

Methods differ in how they handle outliers and minimize error. Names such as GPTQ, AWQ, and SmoothQuant refer to particular approaches, not one universal quantization format.

Quantization-aware training

Quantization-aware training (QAT) simulates low-precision effects during training or fine-tuning. The model learns weights that are more robust to the rounding and clipping it will face after conversion.

QAT can preserve quality better at aggressive precision, but it requires training data, compute, and a more complex pipeline. It is not always necessary for weight-only 8-bit or 4-bit deployment.

Fine-tuning and quantization are separate ideas. Fine-tuning changes what the model has learned. Quantization changes how values are represented for storage and computation. A model can be fine-tuned first and quantized afterward.

Why speedups vary

Smaller weights reduce memory traffic, which is important because LLM decoding is often limited by how quickly weights can be moved to compute units.

But lower precision improves speed only when the hardware and serving engine have efficient kernels for that format.

Performance depends on:

  • Accelerator support for INT8, INT4, or other formats.
  • The cost of dequantizing values during computation.
  • Batch size and sequence length.
  • Whether the workload is memory-bound or compute-bound.
  • Kernel layout and packing format.
  • How much non-quantized work remains.

A model file that is half the size is not guaranteed to run twice as fast.

Quality tradeoffs

Quantization error can change model outputs. Effects may appear in:

  • Perplexity or next-token prediction.
  • Factual and reasoning tasks.
  • Code generation.
  • Rare languages or domains.
  • Long-context behavior.
  • Tool-call formatting.

Larger models often tolerate moderate weight quantization well, but there is no universal safe bit width. Some layers and outlier values are more sensitive than others. Mixed-precision schemes keep sensitive parts at higher precision.

Evaluate the tasks that matter for the product. A small average benchmark change can hide a serious regression in one high-value workflow.

Choosing a format

Start with deployment constraints:

  • How much accelerator or system memory is available?
  • Which low-precision kernels does the target hardware support?
  • Is the goal lower cost, lower latency, larger batch size, or local execution?
  • How much quality change is acceptable?
  • Does the serving engine support the model architecture and quantization format?

Then compare candidates using the same prompts and serving configuration. Measure:

  • Model and runtime memory.
  • Time to first token.
  • Output tokens per second.
  • Throughput at realistic concurrency.
  • Task quality and structured-output validity.
  • Energy or cost per request when relevant.

Common misconceptions

  • Quantization does not reduce the number of model parameters.
  • A 4-bit model is not automatically four times faster than FP16.
  • Lower file size does not include every runtime memory cost.
  • Quantization and pruning are different: pruning removes values or structures; quantization changes numeric representation.
  • Quantization and distillation are different: distillation trains a smaller model to imitate a larger one.
  • One benchmark score is not enough to approve a production conversion.

The key idea

Quantization trades numeric precision for smaller, potentially faster model execution. The useful result depends on the format, grouping, hardware kernels, and measured quality on real tasks.

Saved in this browser only. Not synced to your account.