IICTPIICTP

Blog / Guides

Local LLMs26 October 20265 min read

Edge AI inference: small models on devices, offline and in real time

Edge inference runs a model on or near the device that produces the data: a phone, a gateway, a vehicle, a factory machine. It cuts delay, reduces data sent over the network and keeps working when the connection drops.

In this post
  1. Why run at the edge
  2. What you give up
  3. Making it fit
  4. Related reading
  5. Sources and further reading

Why run at the edge

  • Responses in real time, without a network round trip.
  • Sensitive data stays on the device.
  • Operation continues offline.
  • Less data sent upstream.

What you give up

  • Smaller models and less capability.
  • Limited memory, power and cooling.
  • Harder updates across many devices.

Making it fit

  • Use a small model suited to one task.
  • Quantize, and test the quality loss.
  • Use the device’s accelerator if its software supports it.
  • Plan updates, rollback and monitoring from the start.

See small language models explained and edge computing explained.

Sources and further reading

  1. IICTP. Local LLM Practitioner credential page. iictp.org/certifications/llmp.

IICTP is a new independent credential and is not ISO/IEC 17024 accredited.

Keep reading

Local LLMsLLMP

How to run an LLM on your own server: Ollama, llama.cpp and vLLM compared

GUIDE · 7 MIN READ

Running a language model on hardware you control means prompts and documents never leave your building, the cost is the hardware…

26 October 2026
Local LLMsLLMP

Small language models explained: when smaller beats bigger

GUIDE · 5 MIN READ

A small language model (SLM) is a language model small enough to run on modest hardware, such as one workstation, a phone or an e…

26 October 2026
TPBOK

What is the TPBOK? The Technical Practitioner Body of Knowledge, explained

GUIDE · 4 MIN READ

The TPBOK, the Technical Practitioner Body of Knowledge, is the written standard behind every IICTP exam. It describes what a tec…

26 October 2026

← All posts

Where do you stand?

Find the credential that fits the work you do.

Check your eligibility Exam facts and blueprints