Small language models explained: when smaller beats bigger

Why a smaller model can be the better engineering choice

• Cost and latency. Fewer parameters means less memory and less compute per token, so answers arrive faster and cost less to serve.
• Privacy and control. A model that fits on your own hardware can run where the data already lives, including offline.
• Predictability. A narrow task with a fixed output shape, such as classification, extraction or routing, rarely needs a general-purpose giant.
• Specialisation. A small model tuned on your own task can match a larger general one on that task, though it will be worse outside it.

Where a smaller model is the wrong choice

Open-ended reasoning across many topics, long and tangled documents, and tasks you cannot describe ahead of time still favour larger models. A small model is a specialist, and the cost of using one for the wrong job is wrong answers delivered cheaply and quickly.

How to decide

1. Define the task and write 50 real examples with the answers you expect.
2. Test the smallest candidate model first, then move up only as far as the examples demand.
3. Check the failure cases, not the average. A model that is usually right and occasionally dangerous may be unusable.
4. Add a verification step for anything that matters. See applied AI: verify what the machine produces.

Running the chosen model yourself is covered in how to run an LLM on your own server.

IICTP is a new independent credential and is not ISO/IEC 17024 accredited.