📊 View Lecture Slides Full-screen presentation with navigation

Session 13: Evaluation & Debugging

Session Duration: 2 Hours     Block: 3 — Agents, Evaluation & Deployment

Session clock

Minutes Mode Focus
0–50 Lecture Theoretical Foundation & Concepts
50–110 Core lab Build & Run an AI Evaluation Suite
110–120 Checkpoint Pair share / show artifact

Note: Stretch work starts only after the Core checkpoint is completed.

Learning Objectives

By the end of this session, students will be able to:

  • Apply the three core dimensions of AI evaluation: correctness (factual grounding), relevance (did it answer the prompt?), and consistency (does it format correctly?).
  • Design adversarial test inputs that intentionally expose hallucinations, out-of-scope failures, and RAG retrieval gaps.
  • Debug AI application failures systematically by isolating prompt issues, retrieval issues, and generation issues.
  • Create and execute a foundational JSON-based test suite for their capstone project.

Part 1: Theoretical Foundation — Testing AI Systems Is Different

1.1 The Non-Deterministic Nature of LLMs

Traditional software testing is deterministic: if you input calculate_tax(100, 0.20), the output is always 20. You write a unit test: assert output == 20.

Large Language Models are probabilistic. If you ask an LLM the same question three times, you might get three slightly different responses. You cannot easily write a unit test that says assert response == "The cat sat on the mat".

Therefore, testing AI products requires graded scoring against specific dimensions, rather than simple pass/fail assertions.

1.2 The Three Dimensions of LLM Evaluation

When reviewing an AI’s response, evaluate it across these three axes:

  1. Correctness (Grounding): Are the facts true? Does the answer rely only on the provided RAG context, or did it hallucinate data from its general training weights?
  2. Relevance: Did the model actually answer the user’s question, or did it go on a tangent and provide unrelated information?
  3. Consistency: Did the model follow your structural instructions? (e.g., If you asked for a JSON output, did it return Markdown? If you asked for the persona of a pirate, did it sound like a corporate lawyer?)

1.3 Designing Adversarial Tests

An adversarial test is a prompt specifically designed to break your system. To build a robust AI product, you must test its failure modes.

Categories of Adversarial Prompts:

  • Out-of-Scope / Knowledge Cutoff: Ask a question your vault explicitly does not contain. Goal: The system should safely say “I don’t know” rather than guessing.
  • False Premises: “Why is the sky green?” Goal: The system should correct the premise, not play along with the hallucination.
  • Numeric & Math Logic: Ask it to sum up three numbers found in different RAG documents. Goal: Check if the LLM fails at basic arithmetic despite retrieving the right documents.
  • Ambiguous Queries: Send just one word: “Refund”. Goal: Check if the system handles poor user input gracefully.

1.4 Systematically Debugging RAG Pipelines

When an answer is wrong, junior developers immediately start tweaking the System Prompt. This is often the wrong move.

The RAG Debugging Hierarchy:

  1. Did it Retrieve the right document? (Check your terminal logs. If the vector search missed the document, the LLM is innocent. Fix your chunking or embeddings).
  2. Was the document fed to the LLM? (Check the augmented prompt. Is the context too long? Was it truncated?)
  3. Did the LLM ignore the document? (If the document was there, but the LLM hallucinated anyway, now you fix the System Prompt to be stricter).

Part 2: Practical Labs — Break Your AI System

Note: For the final showcase (Session 15), you will need a test suite of at least 15 scored cases. Today’s in-session goal is 10 cases.

Lab 13.1 — Build the Test Suite (Core)

  1. In your starter kit, duplicate test-suite.example.json and rename it to test-suite.json.
  2. Expand this file to contain 10 distinct test cases relevant to your specific product.
  3. Ensure you have a mix of test types:
    • 4x “Happy Path” questions (things the vault easily answers).
    • 2x Out-of-Scope questions.
    • 2x False Premise / Ambiguous questions.
    • 2x Complex reasoning / summarization requests.

Format Example:

[
  {
    "question": "What is the cancellation policy?",
    "expectedBehavior": "Should accurately quote the 30-day window from policy.md.",
    "category": "Happy Path"
  },
  {
    "question": "Who won the Super Bowl last year?",
    "expectedBehavior": "Should explicitly refuse to answer as it is out of scope.",
    "category": "Out of Scope"
  }
]

Lab 13.2 — Execute and Score the Suite (Core)

We will use the kit helper script to automatically send these questions to your /query endpoint.

  1. Ensure your Express server (npm run dev) is running in one terminal.
  2. Open a second terminal and run:
npm run eval
  1. The script will print the Question, the Expected Behavior, and the AI’s actual Answer to your console.
  2. Manual Scoring: Read the output. On a piece of paper or a spreadsheet, score each of the 10 outputs on a scale of 1-5 for Correctness and 1-5 for Relevance.

Lab 13.3 — Identify Patterns and Document (Core)

Create a new file in your vault: 03-Project/Evaluation-Report.md.

Review your scores and answer these questions:

  1. Which category scored the lowest in correctness?
  2. Did the model hallucinate at all? If so, under what conditions?
  3. Which adversarial case caused the most catastrophic failure?

This report will be a required deliverable for the Session 15 showcase.

Lab 13.4 — Fix One Failure (Core)

Pick the most egregious failure from Lab 13.3. Apply the RAG Debugging Hierarchy (Section 1.4) to isolate the cause.

  1. If the retrieval failed, go into your vault, rewrite the note to have clearer headings, and re-run npm run build-embeddings (Session 8).
  2. If the LLM ignored the rules, update your System Prompt in server.js (Session 9).
  3. Re-run npm run eval and confirm that specific case now passes.

Stretch Goal: Add 5 more test cases (focusing on prompt injection or formatting attacks). Write a simple regression script that fails the build if the AI outputs the word “Error”.

Key Takeaways

  • Non-deterministic testing: You cannot use strict unit tests for LLMs. You must evaluate based on graded rubrics (Correctness, Relevance, Consistency).
  • Adversarial testing (deliberately trying to break the AI with false premises or out-of-scope requests) is essential for building a robust product.
  • RAG Debugging: Always check the retrieval logs before you blame the LLM or change the prompt.
  • An Evaluation Report is a professional engineering artifact that proves your AI system is ready for production.

Further Reading & Resources

  • RAGAS (RAG Assessment) / TruLens: If you want to automate this process in the future, research these Python libraries. They use “LLMs to evaluate LLMs” (LLM-as-a-judge). (Optional Stretch reading, not required for Core).