LLMOps: running AI in production, covering cost, monitoring and versioning

What to monitor

• Latency and error rates, as with any service.
• Token use and cost per request and per feature.
• Quality signals: user feedback, refusals, retries, escalations.
• Drift: answer quality changing after a model or data change.

What to version

• Prompts and prompt templates.
• The model and its settings.
• The retrieval index and its source data.
• The evaluation set.

Controlling cost

• Use the smallest model that passes your evals.
• Cache repeated work.
• Trim the context you send.
• Set per-user and per-feature limits.

Change safely

Treat a model upgrade as a release: run the eval set, roll out to a small share first, compare, and keep a way back. Log enough to reproduce a bad answer, while keeping personal data out of logs that do not need it.

IICTP is a new independent credential and is not ISO/IEC 17024 accredited.