Multimodal AI explained: text, image and audio in one system

Typical uses

• Reading documents with tables, forms and diagrams.
• Describing images for accessibility.
• Transcribing and summarising audio.
• Answering questions about a screenshot.

Design points

• Check the input quality. Blurry or cropped images and noisy audio lower accuracy.
• Verify the reading. Extracted numbers from a chart or form should be checked.
• Mind the privacy. Images and audio can contain people and personal data.
• Cost. Images and audio often cost more to process than text.

Testing

Build an evaluation set from your real inputs, including bad ones, and score extraction and description separately. See how to evaluate an LLM application.

IICTP is a new independent credential and is not ISO/IEC 17024 accredited.