RAG on a local LLM: private document search that never leaves your building

The parts you host

• An embedding model to turn text into vectors.
• A vector store or a search engine.
• The language model that writes the answer.
• A thin service that ties them together and enforces permissions.

Design points

• Permissions: retrieve only documents the asking user may see. Do not leave this to the model.
• Ingestion: handle scanned pages, tables and updates.
• Citations: show the passages used so answers can be checked.
• Offline updates: plan how new models and software reach a machine with no internet.

Test it

Use real questions and measure retrieval and answers separately. See RAG explained for the pipeline.

IICTP is a new independent credential and is not ISO/IEC 17024 accredited.