Session 12 Slides: Multimodal AI & Data Pipelines

--:-- --
↓ Scroll for more

Session 12

Multimodal AI & Data Pipelines

AI Product Engineering

Block 3: Agents, Evaluation & Deployment

Today’s Agenda

  • Beyond Plain Text
  • The Compressed Pipeline
  • Extraction Prompt Pattern
  • Pipeline Architecture
  • Key Pipeline Decisions
  • Interactive: Extraction Reality (3 min)
  • Interactive: Pipeline Decisions (3–4 min)
  • Lab: Lab (Core vs Stretch)

Beyond Plain Text

Modern foundation models process multiple input types:

🖼️ Images

Photos, charts, screenshots, diagrams

📄 Documents

PDFs, forms, tables, invoices

🎙️ Audio/Video

Speech, recordings, video frames

The Compressed Pipeline

Before

Scanned doc → OCR → text cleaning → NLP pipeline → structured data

Complex, fragile, multiple tools

Now

Image → multimodal model + extraction prompt → structured JSON

One API call, one prompt

Extraction Prompt Pattern

Extract all line items from this invoice.
Return JSON matching this schema:
{
  "vendor": string,
  "date": string,
  "items": [{"name": string, "price": float}],
  "total": float
}

The model reads the image and produces the structured output directly.

Interactive: Extraction Reality (3 min) (1/3)

True or false — cold-call, then tap.

Interactive: Extraction Reality (3 min) (2/3)

True or false — cold-call, then tap.

Interactive: Extraction Reality (3 min) (3/3)

True or false — cold-call, then tap.

Pipeline Architecture

Input Source (files, uploads, URLs)
Preprocessing (detect type, split PDFs)
AI Extraction (multimodal model + prompt)
Validation (check schema, flag missing fields)
Storage → Knowledge Base → RAG

Key Pipeline Decisions

  • Batch vs. real-time: process a folder or each file as it arrives?
  • Rate limiting: build in delays for large batches (API limits)
  • Error handling: what if AI cannot extract required fields?
  • Validation: check JSON structure before storing

Interactive: Pipeline Decisions (3–4 min) (1/3)

Choose the best practice.

Interactive: Pipeline Decisions (3–4 min) (2/3)

Choose the best practice.

Interactive: Pipeline Decisions (3–4 min) (3/3)

Choose the best practice.

Lab (Core vs Stretch) (1/2)

  1. 12.1: Extract two images → write Extraction-Notes.md (correct / missing / invented + one prompt revision)
  2. 12.2: npm run extract-folder → extracted-data.json (≥2 files)

Lab (Core vs Stretch) (2/2)

  1. 12.3: Partner share — one surprising failure from your notes

Stretch: browser FormData + multipart; vault ingest + re-embed.

Session 12 Summary

  • Multimodal AI processes images and documents directly
  • Structured extraction = image + JSON schema prompt
  • Pipelines automate raw files → structured knowledge
  • Rate limiting and error handling essential for production

Next Session: Evaluation & Debugging