📊 View Lecture Slides Full-screen presentation with navigation

Session 12: Multimodal AI & Data Pipelines

Session Duration: 2 Hours     Block: 3 — Agents, Evaluation & Deployment

Session clock

Minutes Mode Focus
0–50 Lecture Theoretical Foundation & Concepts
50–110 Core lab Single Image & Batch Processing
110–120 Checkpoint Pair share / show artifact

Note: Stretch work starts only after the Core checkpoint is completed.

Learning Objectives

By the end of this session, students will be able to:

  • Format and send image data (base64) to a multimodal LLM alongside text instructions using Node.js.
  • Extract highly structured data (JSON) from unstructured visual inputs like receipts, forms, or charts.
  • Evaluate the quality of multimodal extraction, identifying common failure modes and hallucination risks in vision models.
  • Execute a batch processing pipeline that processes an entire folder of images into a single structured dataset.

Part 1: Theoretical Foundation — Processing More Than Text

1.1 The Shift to Multimodal

Historically, AI applications required disjointed pipelines: you needed a dedicated Optical Character Recognition (OCR) model to read text from an image, a computer vision model to detect objects, and an NLP model to make sense of the text.

Modern models like Gemini 1.5 Flash are natively multimodal. They process text, images, audio, and video within the same neural network architecture. This massively simplifies product engineering.

When Multimodal Solves Product Problems:

  • Digitization: Converting scanned paper forms, handwritten whiteboards, or receipts into database entries.
  • Analysis: Feeding business charts or dashboards to the AI to summarize trends.
  • Accessibility: Automatically generating rich alt-text for UI images.
  • Prototyping: Skipping the complexity of a dedicated OCR stack to prove a business concept quickly.

1.2 Anatomy of a Multimodal Request

To send an image to Gemini via the Node SDK, you do not just send a URL. You must read the file from your hard drive, convert it into a base64 encoded string, and define its MIME type (image/jpeg, image/png).

// A standard inline data object for the Gemini API
const imagePart = {
  inlineData: {
    data: base64EncodedString,
    mimeType: 'image/jpeg',
  },
};

// Send text AND the image part in the contents array
await ai.models.generateContent({
  model: 'gemini-2.5-flash',
  contents: ['Extract the total amount from this receipt', imagePart],
});

1.3 Common Multimodal Failure Modes

Vision-language models are powerful, but they fail in unique, frustrating ways:

  1. The “Looks Precise” Hallucination: If the image is blurry, the AI might invent a number that looks highly realistic instead of admitting it cannot read the text.
  2. Reading Order Failures: On dense, complex tables, the AI might read columns out of order, misaligning data.
  3. Format Disobedience: Even if you ask for raw JSON, the model might wrap the output in Markdown ( json ... ) or add conversational fluff (“Here is the JSON you requested:”).
  4. Rate Limiting: Images consume significantly more tokens than text. A batch process of 50 images will hit API rate limits much faster than text processing.

1.4 The Data Pipeline Architecture

When processing files at scale, you do not want a user waiting in a browser. You build a pipeline. Standard Pipeline Flow:

  1. Ingest files from a folder (or cloud bucket).
  2. Detect MIME type.
  3. Call generateContent with image bytes and strict JSON instructions.
  4. Validate: Attempt to JSON.parse() the output. Check for missing required fields.
  5. Store successful extractions in a database (or extracted-data.json).
  6. Log failures for manual human review.

Part 2: Practical Labs — Multimodal Extraction

Lab 12.1 — Single Image Extraction & Quality Audit (Core)

  1. Find two non-confidential images relevant to your product domain (e.g., a photo of a receipt, a screenshot of a chart, or a handwritten note).
  2. Place them in your starter kit under sample-images/ (e.g., sample-images/a.jpg and sample-images/b.png).
  3. The kit includes a pre-written Node script to test single-file extraction. Run it via the terminal:
npm run extract-image -- ./sample-images/a.jpg
npm run extract-image -- ./sample-images/b.png
  1. The Audit: The script should output a JSON object. Create a new file in your vault: 03-Project/Extraction-Notes.md.
  2. Review the JSON against the actual image. Document:
    • What data was extracted perfectly?
    • Was anything missing?
    • Did the model invent (hallucinate) any details that were not actually in the image?
  3. Iterate: Open scripts/extract-image.js, change the extraction prompt to be more specific, run the script again, and document if it improved the result.

Lab 12.2 — Batch Processing a Folder (Core)

Processing one image is a party trick; processing a folder is a product.

  1. Ensure you have at least two images in the sample-images/ folder. (If you only have one, just duplicate it so the script actually runs a loop).
  2. Use the provided kit helper to run the batch script:
npm run extract-folder

(Alternatively: node scripts/extract-folder.js ./sample-images)

  1. This script iterates through the folder, processes each image, and appends the results to an array.
  2. Open the resulting file: extracted-data.json.
  3. Verify that it contains valid JSON representing data from all processed images.

Lab 12.3 — Checkpoint

Turn to a partner (or note it down for your instructor) and share one surprising failure from your Extraction-Notes.md. Did the model misread handwriting? Did it hallucinate a decimal point?

Stretch Goal: The core lab relies on local files processed via terminal scripts. For a massive challenge, integrate this into your Express app:

  1. Create an HTML form using <input type="file">.
  2. Send the image to a new Express route using FormData.
  3. Use the multer npm package to intercept the upload, convert it to base64 in memory, send it to Gemini, and return the JSON directly to the browser UI.

Key Takeaways

  • Multimodal models reduce infrastructure complexity by eliminating the need for chained OCR and vision models.
  • Inline Data: Sending images via the API requires converting files to base64 strings and specifying accurate MIME types.
  • Validation is Mandatory: Always run JSON.parse() and verify required fields before saving LLM output to a database.
  • Batch processing images consumes tokens rapidly; implement robust error handling for API rate limits.

Further Reading & Resources

  • Gemini Vision API Documentation: Read up on the specific file size limits and supported video/image formats for the Gemini API.
  • Code Review: Thoroughly inspect scripts/extract-image.js and scripts/extract-folder.js in your kit to see how the Node fs (File System) module interacts with the GenAI SDK.