How to evaluate an LLM application: evals, hallucination tests and guardrails

Build the eval set from real use

1. Collect real or realistic inputs, including the awkward ones.
2. Write down what a good answer must contain and what it must never contain.
3. Add cases for each failure you find, so it cannot return unnoticed.

Ways to score

• Exact checks for structured output: valid format, required fields.
• Reference answers compared by a person or by another model.
• Rubrics for qualities such as tone, safety and use of sources.
• Human review of a sample, to keep the automated scores honest.

Hallucination tests

Include questions the system should refuse or say it cannot answer, and questions with a false premise. Check that answers stay within the supplied sources.

Guardrails

Add checks around the model, not only inside the prompt: validate output format, filter unsafe content, limit tool permissions, and log inputs and outputs for review. Run the eval set again whenever the model, the prompt or the data changes.

IICTP is a new independent credential and is not ISO/IEC 17024 accredited.