📊 View Lecture Slides Full-screen presentation with navigation

Session 9: Building the RAG Pipeline

Session Duration: 2 Hours  |  Block: 2 — AI-Assisted Engineering & Integration

Learning Objectives

By the end of this session, students will be able to:

  • Describe the complete RAG pipeline from query to grounded answer
  • Implement a query → retrieve → augment → generate workflow in Python
  • Integrate the RAG pipeline into the Flask application from Session 7
  • Evaluate the difference between RAG answers and ungrounded answers

Hour 1: RAG — The Complete Picture (Instructor-Led — 60 minutes)

1.1 Why RAG?

Retrieval-Augmented Generation (RAG) combines two capabilities:

  • Retrieval: Finding relevant information from a knowledge base
  • Generation: Using an LLM to produce a coherent answer from that information

The result is an AI system that answers questions grounded in specific, controllable, up-to-date knowledge — rather than from the model’s potentially outdated and hallucination-prone training data.

1.2 The RAG Pipeline

User Query
    ↓
[1. Embed the Query]
    → Convert the query to a vector using the embedding model
    ↓
[2. Retrieve Relevant Chunks]
    → Compare query vector to all document vectors
    → Select the top K most similar chunks (K=3 to 5)
    ↓
[3. Construct the Augmented Prompt]
    → System instruction: "Answer only from the provided context"
    → Context: the retrieved chunks
    → User question: the original query
    ↓
[4. Generate the Answer]
    → Send the augmented prompt to the LLM
    → LLM answers using the retrieved context
    ↓
[5. Return to User]
    → Display the answer (optionally: display the sources cited)

1.3 The Critical Prompt Design for RAG

The system instruction for a RAG query is crucial. It must:

  1. Instruct the model to answer only from the provided context
  2. Tell the model to say “I don’t know” if the context doesn’t contain the answer
  3. Optionally: ask the model to cite which section it used
RAG_SYSTEM_PROMPT = """You are a knowledge base assistant.
Answer the user's question using ONLY the information provided in the context below.
If the context does not contain enough information to answer the question, say:
"I don't have enough information in my knowledge base to answer this."
Do not use your general training knowledge. Do not invent facts.
Cite the source document name when relevant."""

This prevents the model from “helping” by supplementing retrieved knowledge with hallucinated training data.

1.4 Evaluating Retrieval Quality

Before worrying about answer quality, check retrieval quality:

  • Are the top-K retrieved chunks actually relevant to the query?
  • Is the relevant information in the knowledge base at all?
  • Are chunks too large (overwhelming) or too small (missing context)?

Log the retrieved chunks during development. If retrieval is wrong, no prompt engineering will fix it.

Hour 2: Practical — Build and Integrate the RAG Pipeline (60 minutes)

Lab 9.1 — The Retrieval Function

import json
import numpy as np
import google.generativeai as genai

def cosine_similarity(a, b):
    a, b = np.array(a), np.array(b)
    return float(np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b)))

def load_embeddings(path='embeddings.json'):
    with open(path) as f:
        return json.load(f)

def retrieve_relevant_chunks(query: str, top_k: int = 3) -> list[dict]:
    """Find the most relevant document chunks for a query."""
    embeddings_data = load_embeddings()

    # Embed the query
    query_result = genai.embed_content(
        model="models/text-embedding-004",
        content=query,
        task_type="retrieval_query"
    )
    query_embedding = query_result['embedding']

    # Score all chunks
    scored = []
    for item in embeddings_data:
        sim = cosine_similarity(query_embedding, item['embedding'])
        scored.append({'file': item['file'], 'content': item['content'], 'score': sim})

    # Return top K
    return sorted(scored, key=lambda x: x['score'], reverse=True)[:top_k]

Lab 9.2 — The Augmented Prompt Builder

RAG_SYSTEM_PROMPT = """You are a knowledge base assistant.
Answer the user's question using ONLY the information provided in the context below.
If the context does not contain the answer, say "I don't have enough information."
Do not invent facts. Cite the source document when relevant."""

def build_rag_prompt(query: str, chunks: list[dict]) -> str:
    """Build the augmented prompt with retrieved context."""
    context = "\n\n---\n\n".join([
        f"[Source: {c['file']}]\n{c['content'][:1500]}"
        for c in chunks
    ])
    return f"CONTEXT:\n{context}\n\nQUESTION:\n{query}"

Lab 9.3 — The Full RAG Query Function

def rag_query(user_question: str) -> dict:
    """Complete RAG pipeline: retrieve → augment → generate."""
    # Step 1: Retrieve
    chunks = retrieve_relevant_chunks(user_question, top_k=3)

    # Step 2: Build augmented prompt
    augmented_prompt = build_rag_prompt(user_question, chunks)

    # Step 3: Generate
    rag_model = genai.GenerativeModel(
        'gemini-1.5-flash',
        system_instruction=RAG_SYSTEM_PROMPT
    )
    response = rag_model.generate_content(augmented_prompt)

    return {
        'answer': response.text,
        'sources': [c['file'] for c in chunks],
        'top_score': chunks[0]['score'] if chunks else 0
    }

Lab 9.4 — Integrate into Flask

Update your Flask /query route to use rag_query() instead of the direct AI call. Update the frontend to display the sources alongside the answer.

Test with five questions:

  1. A question that is clearly in your knowledge base
  2. A question that is partially in your knowledge base
  3. A question that is not in your knowledge base at all
  4. A misleading question designed to trigger the model’s training knowledge
  5. Your most important real-world use case

Document: does the system say “I don’t know” correctly for case 3 and 4?

Key Takeaways

  • RAG = embed query → retrieve relevant chunks → augment prompt → generate grounded answer
  • The RAG system prompt must explicitly instruct the model to answer only from context
  • Evaluating retrieval quality (are the right chunks being retrieved?) is more important than prompt engineering
  • Logging retrieved chunks during development is essential for debugging
  • Your application is now a knowledge-grounded AI system

Further Reading

  • LangChain RAG documentation
  • “Improving RAG Pipelines” — Anthropic research blog
  • ChromaDB documentation (lightweight vector database for production)