📊 View Lecture Slides Full-screen presentation with navigation

Session 11: Tool Calling & Actions

Session Duration: 2 Hours     Block: 3 — Agents, Evaluation & Deployment

Learning Objectives

By the end of this session, students will be able to:

  • Explain how function calling enables AI to interact with external systems
  • Define tools with clear schemas that the AI model can understand and invoke
  • Implement a tool-calling loop in Python
  • Add a meaningful tool (search or calculator) to their AI application

Hour 1: Function Calling — Giving AI Hands (Instructor-Led — 60 minutes)

1.1 What Is Tool Calling?

A language model, in its pure form, can only produce text. Tool calling extends this by giving the model the ability to request that specific functions be executed, and then use the results of those functions in its response.

The key insight: the model does not execute the function — it decides to call it and your code executes it.

This is a critical architectural point. The model emits a structured request: “Call the search_web function with query=’current weather in Kathmandu’.” Your application code then executes the search, retrieves the result, and feeds it back to the model as additional context. The model then incorporates the real-time data into its response.

1.2 The Function Calling Protocol

[Turn 1]
User: "What is the population of Kathmandu?"
Model: [Decides it needs current data] → Emits function call: search_web("population of Kathmandu")

[Your code executes search_web("population of Kathmandu") → returns "1.44 million (2021)"]

[Turn 2 — function result fed back]
Context now includes: [search result: "1.44 million (2021)"]
Model: "The population of Kathmandu is approximately 1.44 million as of the 2021 census."

1.3 Defining Tools

Tools are defined as structured schemas that describe:

  • The function name
  • What the function does (description — the model reads this to decide when to use it)
  • The parameters it accepts (names, types, descriptions)

For Gemini:

import google.generativeai as genai

# Define the tool
calculator_tool = genai.protos.Tool(
    function_declarations=[
        genai.protos.FunctionDeclaration(
            name='calculate',
            description='Evaluate a mathematical expression and return the result.',
            parameters=genai.protos.Schema(
                type=genai.protos.Type.OBJECT,
                properties={
                    'expression': genai.protos.Schema(
                        type=genai.protos.Type.STRING,
                        description='The mathematical expression to evaluate, e.g. "2 + 2" or "sqrt(16)"'
                    )
                },
                required=['expression']
            )
        )
    ]
)

1.4 Implementing the Tool Executor

import ast
import math
import operator as op

_ALLOWED_BINOPS = {
    ast.Add: op.add,
    ast.Sub: op.sub,
    ast.Mult: op.mul,
    ast.Div: op.truediv,
    ast.Pow: op.pow,
    ast.Mod: op.mod,
}
_ALLOWED_UNARYOPS = {
    ast.UAdd: op.pos,
    ast.USub: op.neg,
}
_ALLOWED_FUNCS = {name: getattr(math, name) for name in (
    'sqrt', 'sin', 'cos', 'tan', 'log', 'log10', 'exp', 'fabs', 'floor', 'ceil'
)}

def safe_calculate(expression: str) -> float:
    """Safely evaluate a restricted math expression."""
    node = ast.parse(expression, mode='eval').body

    def _eval(n):
        if isinstance(n, ast.Constant) and isinstance(n.value, (int, float)):
            return float(n.value)
        if isinstance(n, ast.BinOp) and type(n.op) in _ALLOWED_BINOPS:
            return _ALLOWED_BINOPS[type(n.op)](_eval(n.left), _eval(n.right))
        if isinstance(n, ast.UnaryOp) and type(n.op) in _ALLOWED_UNARYOPS:
            return _ALLOWED_UNARYOPS[type(n.op)](_eval(n.operand))
        if isinstance(n, ast.Call) and isinstance(n.func, ast.Name) and n.func.id in _ALLOWED_FUNCS:
            args = [_eval(a) for a in n.args]
            return float(_ALLOWED_FUNCS[n.func.id](*args))
        raise ValueError('Unsupported expression')

    return float(_eval(node))

def execute_tool(tool_call) -> str:
    """Execute a tool call from the model and return the result."""
    name = tool_call.name
    args = dict(tool_call.args)

    if name == 'calculate':
        try:
            result = safe_calculate(args['expression'])
            return str(result)
        except Exception as e:
            return f"Error: {str(e)}"
    return f"Unknown tool: {name}"

Security note: Avoid eval() for user/model input. Prefer a strict parser/AST whitelist or a dedicated safe-expression library for expression evaluation.

1.5 The Tool-Calling Loop

def query_with_tools(user_message: str, tools: list) -> str:
    """Handle a query with potential tool calls."""
    model = genai.GenerativeModel('gemini-1.5-flash', tools=tools)
    chat = model.start_chat()

    response = chat.send_message(user_message)

    # Handle tool calls
    while response.candidates[0].content.parts:
        part = response.candidates[0].content.parts[0]
        if hasattr(part, 'function_call') and part.function_call.name:
            # Execute the tool
            result = execute_tool(part.function_call)
            # Send the result back to the model
            response = chat.send_message(
                genai.protos.Content(parts=[
                    genai.protos.Part(
                        function_response=genai.protos.FunctionResponse(
                            name=part.function_call.name,
                            response={'result': result}
                        )
                    )
                ])
            )
        else:
            break  # No more tool calls

    return response.text

Hour 2: Practical — Add a Tool to Your Application (60 minutes)

Lab 11.1 — Implement the Calculator Tool

Implement the calculator tool as defined in section 1.3. Test it with:

  1. "What is 15% of £2,340?" — should trigger the tool
  2. "Explain what an API is" — should NOT trigger the tool
  3. "What is the square root of 144 divided by 3?" — multi-step calculation

Verify the model correctly decides when to use the tool and when it is unnecessary.

Lab 11.2 — Add a Search Tool (Simulated)

Implement a simulated search tool (returns hardcoded results for this lab) and connect it to your application:

def search_knowledge_base(query: str) -> str:
    """Search the knowledge base for relevant information."""
    # In this lab, call your RAG retrieve function from Session 9
    chunks = retrieve_relevant_chunks(query, top_k=2)
    if not chunks or chunks[0]['score'] < 0.5:
        return "No relevant information found in knowledge base."
    return "\n\n".join([f"[{c['file']}]: {c['content'][:500]}" for c in chunks])

This connects your RAG system from Session 9 to the tool-calling framework from this session — the two components now work together.

Lab 11.3 — Relevant Tool for Your Project

Identify one tool that would be genuinely useful for your course project. Some examples:

  • Course Assistant: get_learning_objective(session_number) — returns the objectives for a given session
  • Document Navigator: list_documents() — returns the available document names
  • CV Matcher: score_cv_match(job_requirements, cv_text) — returns a percentage match

Define the tool schema, implement the function, and integrate it into your application.

Key Takeaways

  • Tool calling gives AI the ability to request execution of specific functions with structured arguments
  • The model decides when to call a tool; your code decides how to execute it safely
  • Tool descriptions are critical — the model reads these to decide when to use each tool
  • Security: never expose unrestricted eval() or system access to tool-calling AI
  • Tool calling + RAG together create a far more capable application than either alone

Further Reading