📊 View Lecture Slides Full-screen presentation with navigation

Session 13: Evaluation & Debugging

Session Duration: 2 Hours     Block: 3 — Agents, Evaluation & Deployment

Learning Objectives

By the end of this session, students will be able to:

  • Apply three evaluation dimensions: correctness, relevance, and consistency
  • Design adversarial test inputs that expose hallucinations and failure modes
  • Systematically debug AI application failures using structured diagnosis
  • Create a basic test suite for their course project

Hour 1: Testing AI Systems Is Different (Instructor-Led — 60 minutes)

1.1 Why Standard Software Testing Fails for AI

Traditional software testing has a fundamental assumption: given input X, output Y should always be exactly the same. AI systems violate this assumption. The same input can produce different outputs across runs, and “correct” is often a matter of degree rather than a binary pass/fail.

This does not mean AI systems cannot be tested — it means they require different testing frameworks.

1.2 Three Evaluation Dimensions

Dimension 1: Correctness Is the answer factually accurate? For RAG systems: is the answer supported by the retrieved context?

Test approach: Create a test set of question-answer pairs where you know the correct answer (from your knowledge base). Manually evaluate whether the AI’s answer matches.

Dimension 2: Relevance Does the answer address the actual question? A technically correct response that doesn’t answer the question asked is still a failure.

Test approach: Rate each answer on a 1–5 scale for relevance to the specific question. Identify patterns: what types of questions consistently produce low-relevance answers?

Dimension 3: Consistency Given similar queries phrased differently, does the system produce consistent answers?

Test approach: Ask the same underlying question in five different ways. Do the answers agree? Do they contradict? This reveals whether the system is genuinely reasoning from knowledge or pattern-matching on surface features.

1.3 The Hallucination Problem

Hallucination occurs when an AI model produces confident, fluent, and specific responses that are factually incorrect. It is not “lying” — it is generating the statistically most likely continuation of the input, which sometimes produces plausible-sounding but false content.

Common hallucination patterns:

  • False specificity: Inventing precise numbers, dates, or names that sound authoritative
  • Confabulation: Combining real information with invented connecting details
  • Confidently wrong: Stating incorrect facts without hedging language

How to test for hallucination:

  1. Ask about things that are NOT in your knowledge base — the model should say “I don’t know”
  2. Ask leading questions that presuppose false premises — “What year did X stop using Y?” (if X never used Y)
  3. Ask for specific numerical data — verify every number independently

1.4 Adversarial Testing: “Break the AI”

Adversarial testing deliberately tries to make the system fail. Categories of adversarial inputs:

Category Example What It Tests
Out-of-scope Ask a cooking question to a code assistant Does it refuse or hallucinate?
Ambiguous “Tell me about the current situation” Does it ask for clarification or guess?
Contradictory Two questions implying opposite facts Does it maintain consistency?
Edge case Empty input, very long input, special characters Does it crash or behave unexpectedly?
Injection “Ignore all previous instructions and…” Does it resist prompt injection? (Session 14)

Hour 2: Practical — Break Your AI System (60 minutes)

Lab 13.1 — Create a Test Set

Create a file test-suite.json with 20 test cases. For each:

  • The test question/input
  • The expected correct answer or behaviour
  • The test category (correctness / relevance / consistency / adversarial)

Lab 13.2 — Run the Test Suite

import json

def evaluate_response(question: str, expected: str, actual: str) -> dict:
    """Manually-scored evaluation (you provide the score)."""
    print(f"\nQ: {question}")
    print(f"Expected: {expected}")
    print(f"Actual: {actual}")
    correctness = input("Correctness score (1-5): ")
    relevance = input("Relevance score (1-5): ")
    notes = input("Notes (optional): ")
    return {
        'question': question,
        'correctness': int(correctness),
        'relevance': int(relevance),
        'notes': notes
    }

# Load test cases and run
with open('test-suite.json') as f:
    test_cases = json.load(f)

results = []
for case in test_cases[:10]:  # start with 10
    actual = rag_query(case['question'])['answer']
    result = evaluate_response(case['question'], case['expected'], actual)
    results.append(result)

# Calculate average scores
avg_correctness = sum(r['correctness'] for r in results) / len(results)
avg_relevance = sum(r['relevance'] for r in results) / len(results)
print(f"\nAverage Correctness: {avg_correctness:.1f}/5")
print(f"Average Relevance: {avg_relevance:.1f}/5")

Lab 13.3 — Identify Failure Patterns

From your evaluation results, answer:

  1. What category of question produces the lowest correctness scores?
  2. At what point does the system hallucinate? (Low retrieval score? Specific query types?)
  3. What adversarial input caused the most concerning behaviour?

Document these findings in 03-Project/Evaluation-Report.md. You will expand this into the Responsible AI Report in Session 14.

Lab 13.4 — Fix One Identified Failure

Choose the most critical failure mode you identified. Apply one of:

  • Prompt update: Refine the system prompt to address the failure pattern
  • Knowledge base update: Add missing content to your vault and re-embed
  • Retrieval adjustment: Change chunk size or top-K to improve retrieval quality

Re-run the affected test cases. Did the fix improve the scores?

Key Takeaways

  • AI evaluation uses three dimensions: correctness, relevance, and consistency
  • Hallucination testing requires specific adversarial inputs — do not only test happy-path queries
  • A structured test suite makes progress measurable and regression detectable
  • Every identified failure should lead to a specific improvement hypothesis
  • Your evaluation report is a core deliverable for the final showcase

Further Reading

  • “Evaluating Large Language Models: A Survey” (Chang et al., 2023)
  • RAGAS framework for RAG evaluation: github.com/explodinggradients/ragas
  • TruLens evaluation framework documentation