Observability for AI systems: tracing LLM calls, tokens and drift

What to trace

• Each model call: model, settings, latency and token counts.
• Retrieval steps: what was fetched and how relevant it was.
• Tool calls and their results.
• The full chain for one user request, as a single trace.

What to measure

• Cost per request and per feature.
• Quality signals: feedback, retries, refusals, evaluation scores.
• Drift: quality changing after a model, prompt or data change.

Privacy

Prompts and outputs can contain personal data. Decide what to store, mask sensitive fields and limit access and retention. OpenTelemetry has emerging conventions for generative-AI telemetry; check their current status before depending on them.

IICTP is a new independent credential and is not ISO/IEC 17024 accredited.