Session 12

Multimodal AI & Data Pipelines

AI Product Engineering

Block 3: Agents, Evaluation & Deployment

Beyond Plain Text

Modern foundation models process multiple input types:

🖼️ Images

Photos, charts, screenshots, diagrams

📄 Documents

PDFs, forms, tables, invoices

🎙️ Audio/Video

Speech, recordings, video frames

The Compressed Pipeline

Before

Scanned doc → OCR → text cleaning → NLP pipeline → structured data

Complex, fragile, multiple tools

Now

Image → multimodal model + extraction prompt → structured JSON

One API call, one prompt

Extraction Prompt Pattern

Extract all line items from this invoice.
Return JSON matching this schema:
{
  "vendor": string,
  "date": string,
  "items": [{"name": string, "price": float}],
  "total": float
}

The model reads the image and produces the structured output directly.

Pipeline Architecture

Input Source (files, uploads, URLs)

Preprocessing (detect type, split PDFs)

AI Extraction (multimodal model + prompt)

Validation (check schema, flag missing fields)

Storage → Knowledge Base → RAG

Key Pipeline Decisions

Batch vs. real-time: process a folder or each file as it arrives?
Rate limiting: build in delays for large batches (API limits)
Error handling: what if AI cannot extract required fields?
Validation: check JSON structure before storing

Session 12 Summary

Multimodal AI processes images and documents directly
Structured extraction = image + JSON schema prompt
Pipelines automate raw files → structured knowledge
Rate limiting and error handling essential for production

Next Session: Evaluation & Debugging

Session 12 Slides: Multimodal AI & Data Pipelines