RAG explained: retrieval-augmented generation, step by step

The pipeline

1. Ingest. Split documents into chunks and store them with an embedding for each, in a vector index or a search engine.
2. Retrieve. For a question, find the chunks most relevant to it.
3. Augment. Place those chunks in the prompt with the question.
4. Generate. The model answers, ideally citing the chunks it used.

Where quality is won or lost

• Chunking: chunks that cut a table or a thought in half retrieve badly.
• Retrieval: combining keyword and semantic search often beats either alone.
• Prompting: tell the model to answer only from the supplied text and to say when it cannot.
• Freshness: re-index when the source changes.

How to know it works

Build a set of real questions with known answers and measure two things separately: did retrieval find the right chunk, and did the model use it correctly. A wrong answer with the right chunk retrieved is a different bug from one where nothing relevant was found.

Limits

RAG reduces unsupported answers but does not remove them. Keep citations visible, restrict what each user can retrieve to what they may see, and test for prompts that try to pull in other people’s documents.

IICTP is a new independent credential and is not ISO/IEC 17024 accredited.