Quantization explained: GGUF, AWQ and running big models on small hardware

The idea

A full-precision weight might use 16 bits. Quantizing to 8 or 4 bits maps each weight to a coarser set of values. Memory falls roughly in proportion: a model at 4 bits needs about a quarter of the weight memory of the same model at 16 bits.

Formats you will meet

• GGUF: a single-file format used by llama.cpp and tools built on it, with many quantization levels.
• AWQ: a method that protects the weights that matter most to the output, designed for GPU inference.
• GPTQ: another post-training method, also aimed at GPU inference.

Choosing a level

1. Start from the memory you have.
2. Pick the highest precision that fits with room for context.
3. Test on your own examples; compare a lower level against it.
4. Stop lowering when quality drops in ways that matter.

Pitfalls

• Quantization loss shows up unevenly: maths, code and rare languages are often hit first.
• Leave memory for the context cache, which grows with the conversation.
• Re-test after changing quantization, like any other change.

IICTP is a new independent credential and is not ISO/IEC 17024 accredited.