📊 View Lecture Slides Full-screen presentation with navigation

Session 12: Multimodal AI & Data Pipelines

Session Duration: 2 Hours     Block: 3 — Agents, Evaluation & Deployment

Learning Objectives

By the end of this session, students will be able to:

  • Send images and documents to a multimodal AI model and extract structured information
  • Design a simple data ingestion pipeline that processes files and stores results
  • Identify appropriate use cases for multimodal AI in their project
  • Evaluate the accuracy and reliability of AI-extracted data

Hour 1: Processing More Than Text (Instructor-Led — 60 minutes)

1.1 What Is Multimodal AI?

A multimodal AI model can process multiple types of input beyond plain text. Current foundation models (including Gemini) support:

  • Images: Photographs, screenshots, charts, diagrams
  • Documents: PDFs, presentations (as images)
  • Audio: Speech transcription and audio understanding
  • Video: Frame-by-frame or temporal analysis (selected models)

The output can still be text, structured JSON, or in some cases generated images.

1.2 Why This Changes Data Pipelines

Previously, data pipelines for AI required all data to be in text format. If you had a scanned invoice, you needed an OCR (optical character recognition) system to extract the text first.

Multimodal models compress this pipeline. You can send the image directly:

  • “Extract all line items, quantities, and prices from this invoice as JSON”
  • “Describe what this chart is showing and identify the key trend”
  • “What does this error message in the screenshot say and what is the likely cause?”

This significantly reduces the infrastructure required to build AI applications that process real-world documents.

1.3 Sending Images to the API

import google.generativeai as genai
import PIL.Image
import json

model = genai.GenerativeModel('gemini-1.5-flash')

def extract_from_image(image_path: str, extraction_prompt: str) -> dict:
    """Extract structured information from an image using multimodal AI."""
    image = PIL.Image.open(image_path)

    response = model.generate_content([
        extraction_prompt,
        image
    ])

    # If we requested JSON, parse it
    try:
        return json.loads(response.text)
    except json.JSONDecodeError:
        return {'raw_response': response.text}

1.4 Designing a Data Ingestion Pipeline

A data ingestion pipeline automates the process of taking raw files and transforming them into structured, AI-ready data.

Simple pipeline architecture:

Input Source (files, uploads, URLs)
    ↓
Preprocessing (detect type, split PDFs into pages)
    ↓
AI Extraction (send to multimodal model with extraction prompt)
    ↓
Validation (check JSON structure, flag missing fields)
    ↓
Storage (save to JSON file, database, or knowledge base)
    ↓
Downstream Application (RAG, analysis, display)

Key design decisions:

  • Batch vs. real-time: Process a folder of files overnight, or process each file as it arrives?
  • Error handling: What happens if the AI cannot extract the required fields?
  • Rate limiting: APIs have per-minute token limits — build in delays for large batches.

Hour 2: Practical — Build a Document Processing Pipeline (60 minutes)

Lab 12.1 — Extract Data from an Image

Find an image that contains structured information — a chart, a table, a receipt, a form, or a screenshot of data. Use the function from section 1.3 to extract structured JSON.

Test prompts:

For a chart: “Describe this chart. Extract the data labels and approximate values as a JSON array. Identify the overall trend.”

For a table: “Extract all rows and columns from this table as a JSON array of objects, where each row is an object with the column headers as keys.”

For a receipt: {"vendor": string, "date": string, "items": [{"name": string, "price": float}], "total": float}

Lab 12.2 — Build a Simple Batch Pipeline

Create a script that processes a folder of files:

import os
import json
import time

EXTRACTION_PROMPT = """Extract the key information from this document.
Return a JSON object with the following fields:
- title: the document title or main heading
- summary: 2-3 sentence summary
- key_points: list of up to 5 key points
- entities: list of any named entities (people, organisations, products)"""

def process_folder(input_folder: str, output_file: str):
    """Process all images in a folder and save extracted data."""
    results = []
    files = [f for f in os.listdir(input_folder) if f.lower().endswith(('.png', '.jpg', '.jpeg'))]

    for i, filename in enumerate(files):
        print(f"Processing {i+1}/{len(files)}: {filename}")
        path = os.path.join(input_folder, filename)
        extracted = extract_from_image(path, EXTRACTION_PROMPT)
        results.append({'file': filename, 'data': extracted})
        time.sleep(1)  # Rate limiting

    with open(output_file, 'w') as f:
        json.dump(results, f, indent=2)
    print(f"Saved {len(results)} results to {output_file}")

process_folder('./sample-images', './extracted-data.json')

Lab 12.3 — Integrate with Your Knowledge Base

If your project involves processing documents or images, update your Obsidian vault and embedding pipeline to include AI-extracted data. The extracted JSON from lab 12.2 can be formatted as Markdown and added to your vault, then re-embedded for use in your RAG pipeline.

Key Takeaways

  • Multimodal AI can process images, PDFs, and documents directly — no separate OCR pipeline required
  • Structured extraction prompts with explicit JSON schemas produce machine-readable outputs
  • Data pipelines automate the transformation of raw files to structured, AI-ready knowledge
  • Rate limiting and error handling are essential for production pipelines
  • Multimodal extraction can feed directly into your RAG knowledge base

Further Reading