Session 11 Slides: Tool Calling & Actions

--:-- --
↓ Scroll for more

Session 11

Tool Calling & Actions

AI Product Engineering

Block 3: Agents, Evaluation & Deployment

Today’s Agenda

  • Giving AI Hands
  • The Function Calling Protocol
  • Defining a Tool Schema
  • ⚠️ Security Warning
  • Tool + RAG Together
  • Interactive: Who Executes? (3 min)
  • Interactive: Tools + RAG (3–4 min)
  • Lab: Lab (Core)

Giving AI Hands

A language model only produces text. Tool calling lets it request execution of real functions — and use the results.

The model decides to call the tool. Your code executes it.

The Function Calling Protocol

User: "What is (15 / 100) * 2340?"
Model → emits: calculate("(15 / 100) * 2340")
Your code → executes → returns "351"
Model → "(15 / 100) * 2340 is 351"

Interactive: Who Executes? (3 min) (1/3)

Pick the strongest answer.

Interactive: Who Executes? (3 min) (2/3)

Pick the strongest answer.

Interactive: Who Executes? (3 min) (3/3)

Pick the strongest answer.

Defining a Tool Schema

name: 'calculate'
description: 'Evaluate a mathematical expression.'
parameters:
  expression:
    type: string
    description: 'Expression to evaluate, e.g. "(15 / 100) * 2340"'
    required: true

The model reads the description to decide when to use this tool. Write it clearly.

⚠️ Security Warning

  • Never expose unrestricted eval() to AI-driven tool calls
  • Restrict tool scope: math only, read-only file access, no system commands
  • Log all tool calls and their arguments
  • Validate and sanitise all tool inputs before execution

Tool + RAG Together

Your RAG retrieval function from Session 9 becomes a tool the agent can call.

User asks a question → agent decides to search knowledge base → retrieves relevant chunks → generates grounded answer.

Sessions 9 + 11 = a complete agentic knowledge assistant.

Interactive: Tools + RAG (3–4 min) (1/3)

True or false — call on a pair.

Interactive: Tools + RAG (3–4 min) (2/3)

True or false — call on a pair.

Interactive: Tools + RAG (3–4 min) (3/3)

True or false — call on a pair.

Lab (Core) (1/3)

  1. 11.1: Uncomment POST /tools + queryWithTools — ask (15 / 100) * 2340
  2. Keep RAG on /query — do not replace it with tools-only

Lab (Core) (2/3)

  1. 11.2: Implement search_knowledge_base in executeTool + register declaration
  2. If top score < 0.5 → return "nothing relevant found"

Lab (Core) (3/3)

  1. 11.3: Log one successful tool call (name, args, result) in prompts.md

Also test: explain an API (no tool) · vault question (search tool).

Session 11 Summary

  • Tool calling: model requests → your code executes
  • Tool descriptions decide when tools are used — write them clearly
  • Security: restrict scope, validate inputs, log calls
  • RAG + tools = complete agentic application

Next Session: Multimodal AI & Data Pipelines