Choosing hardware for local AI: GPUs, NPUs and Apple Silicon

What matters, in order

1. Memory capacity: the model has to fit, with room for context.
2. Memory bandwidth: generating each token reads the weights, so bandwidth limits speed.
3. Compute: matters most for processing long prompts and for serving many users.
4. Software support: the runtime must support your hardware.

The families

• Discrete GPUs: the usual choice for servers; fast, with a fixed memory size.
• Apple Silicon: unified memory shared by CPU and GPU lets a laptop or desktop hold large models, at lower speeds than server GPUs.
• NPUs: low-power accelerators in new laptops and phones, good for small models; check what your runtime supports.
• CPU only: workable for small or heavily quantized models and for light use.

Sizing steps

1. Choose the smallest model that passes your tests.
2. Work out its memory at your chosen quantization, plus context.
3. Pick hardware with comfortable headroom.
4. Measure speed on your own prompts before you buy in volume.

IICTP is a new independent credential and is not ISO/IEC 17024 accredited.