How to run an LLM on your own server: Ollama, llama.cpp and vLLM compared

Start with memory, not the tool

A model is mostly a large table of numbers called weights. The memory they need is roughly the number of parameters times the bytes stored per weight. A 7-billion-parameter model at 16-bit precision needs about 14 GB for weights alone. At 4-bit precision it needs about 3.5 GB. Add room for the context (the conversation the model is holding) and for the runtime itself.

So the first question is never which tool, but whether the model you want fits in the memory you have. Quantization, which stores weights at lower precision, is how large models are made to fit. Formats such as GGUF carry quantized weights in one file.

The three starting points

A sensible path

1. Write down the task and the smallest model that might do it. Smaller models are cheaper to run and often enough.
2. Try it on a workstation with Ollama or llama.cpp. Measure the answer quality on your own examples, not on a leaderboard.
3. Decide how many people will use it at once. One user and fifty users are different engineering problems.
4. If many users share it, move to a serving engine such as vLLM on server GPUs, and put it behind your normal authentication and logging.
5. Plan updates: models, runtimes and drivers all change, and a local deployment is yours to patch.

What “private” does and does not cover

Keeping inference local removes the third-party provider from the data path. It does not by itself secure the system. You still need access control, logging, network isolation and a plan for what the model is allowed to read. Treat the model server like any other service that touches sensitive data.

IICTP is a new independent credential and is not ISO/IEC 17024 accredited.