📊 View Lecture Slides Full-screen presentation with navigation

Session 3: The Personal Exocortex

Session Duration: 2 Hours     Block: 1 — Foundations & The Exocortex

Session clock

Minutes Mode Focus
0–50 Lecture Theoretical Foundation & Concepts
50–110 Core lab Obsidian Setup & Vault Architecture
110–120 Checkpoint Pair share / show artifact

Note: Stretch work starts only after the Core checkpoint is completed.

Learning Objectives

By the end of this session, students will be able to:

  • Define the concept of an exocortex and explain its critical role in modern AI-assisted software systems.
  • Set up and structure a local knowledge base (a vault) using Obsidian and standard Markdown conventions.
  • Create Maps of Content (MOCs) to organise knowledge hierarchically and dynamically.
  • Explain why clean, structured local knowledge in plain text is the necessary foundation for building a Retrieval-Augmented Generation (RAG) system.

Part 1: Theoretical Foundation — Externalising Memory

1.1 What Is an Exocortex?

The term “exocortex” (from the Greek exo- meaning outside, and cortex referring to the brain’s outer layer of neural tissue) refers to an external cognitive system. It is a digital extension of your own memory, reasoning, and information processing capacity.

The premise is straightforward: the human brain is highly creative and excellent at connecting disparate ideas, but it has severe limitations in working memory and is notoriously unreliable at long-term, high-fidelity data storage. An exocortex offloads the burden of storage and exact recall to an external system that you intentionally design and control.

For AI product engineers, an exocortex serves a vital second function: it becomes the pristine knowledge layer that your AI systems will query.

If you ask an AI model to answer a question relying solely on its internal, pre-trained weights, you face three massive problems:

  1. Knowledge Cutoff: The model knows nothing about events or data created after it was trained.
  2. Hallucination: If the model doesn’t know the exact answer, it will confidently invent a statistically plausible one.
  3. Lack of Proprietary Context: The model has no access to your private company data, your specific project requirements, or your personal notes.

By building an exocortex, you provide the AI with a grounded source of truth. Instead of asking the AI to guess, you retrieve relevant information from your exocortex and say to the AI: “Read these specific notes, and answer the user’s question using ONLY this information.”

This technique is known as Retrieval-Augmented Generation (RAG), and the exocortex we build today is the foundational database for the RAG pipeline we will engineer in Sessions 8 and 9.

1.2 Why Use Obsidian and Markdown?

There are countless note-taking applications (Notion, Evernote, Roam, OneNote). Why use Obsidian for this course?

Obsidian operates on a fundamentally different philosophy: it stores all notes as plain .md (Markdown) files in a local folder on your hard drive (referred to as a “vault”). There is no proprietary database, no forced cloud lock-in, and no hidden data structures.

This architectural choice has three profound consequences for AI engineering:

  1. AI-Ready by Default: Markdown is a lightweight markup language containing plain text with structural indicators (like # for headings). This is exactly the format LLMs parse best. Because there is no complex HTML or proprietary database schema to strip away, you can feed Markdown files directly into an AI model without expensive conversion pipelines.
  2. Ultimate Portability: Your notes are just a folder of text files. You can write scripts in Node.js, Python, or Go to read, search, modify, and process your vault seamlessly. Our Node/Express backend will read these files directly.
  3. Durability: In 20 years, regardless of what software companies exist, your notes will still be readable by any basic text editor.

1.3 The Anatomy of a High-Quality Vault

A vault is only useful if it is structured effectively. A well-designed exocortex relies on four core components:

1. Atomic Notes The principle of atomicity means that each note should contain exactly one clear idea or concept. The title should be descriptive, often a complete sentence or clear noun phrase. The body should be a self-contained explanation.

Example of an Atomic Note:

# Why Context Window Size Matters for RAG Systems

A context window dictates the maximum amount of text a language model can process in a single API call.
In a RAG system, the documents retrieved from your database must fit entirely within this window, alongside the system prompt and the user's original query.
Larger context windows allow you to retrieve and analyze more content simultaneously, but they significantly increase the API cost and the time it takes to generate a response (latency).

2. Bidirectional Links Obsidian uses [[double bracket notation]] to link notes together. If you mention “Context Window” in another note, you wrap it in brackets like this: [[Context Window Size]]. This creates a clickable link, but more importantly, it establishes a semantic relationship (a knowledge graph) that software can map and traverse.

3. Tags Using #tags creates broad categories that cut horizontally across your folder structure. For example, a note on API pricing might be tagged #deployment and #costs, allowing you to find it via multiple mental pathways.

4. Maps of Content (MOCs) As your vault grows, folders become insufficient. Maps of Content (MOCs) are index notes that serve as navigational hubs. An MOC for “AI Applications” is simply a note that contains links to your atomic notes on RAG, agents, embeddings, and evaluation, organised logically. MOCs act as the table of contents for your brain.

While Obsidian allows you to put everything in one folder and rely purely on links, a lightweight folder structure helps scripts easily target specific document types. We will use the following structure:

ai-product-vault/
├── 00-Inbox/           # Quick captures and unorganised thoughts.
├── 01-Concepts/        # Theoretical notes: definitions and "what things are".
│   ├── AI-Foundations/
│   ├── RAG-Theory/
│   └── Agents/
├── 02-How-To/          # Practical guides: tutorials, code snippets, "how to do things".
│   ├── Prompting-Guides/
│   ├── API-Integration/
│   └── Deployment/
├── 03-Project/         # Dedicated to your AI product build.
│   ├── Product-Spec.md
│   ├── Prompts.md      # (From Session 2)
│   └── Architecture.md
├── 04-References/      # External sources, downloaded papers, documentation.
└── MOC-Course.md       # The master index note of this entire vault.

Part 2: Practical Labs — Build Your Course Vault

Lab 3.1 — Install and Configure Obsidian

  1. Download Obsidian from obsidian.md. It is free and available for all major operating systems.
  2. Launch Obsidian and select “Create new vault”.
  3. Choose a dedicated location on your local machine (e.g., Documents/ai-product-vault/).
  4. Inside Obsidian, use the file explorer pane to create the folder structure outlined in section 1.4 (00-Inbox, 01-Concepts, etc.).
  5. Configure Core Plugins: Go to Settings (the gear icon) > Core plugins. Ensure the following are enabled:
    • Backlinks: Allows you to see which notes link to the current note.
    • Graph view: Provides a visual representation of your linked notes.
    • Templates: Allows you to create standard layouts for new notes.

Lab 3.2 — Create Your First Atomic Notes

Let’s populate your exocortex. Create three atomic notes based on the theory covered in this session.

Note 1: Create a file titled What is an Exocortex.md. Write a 3-5 sentence explanation in your own words. Note 2: Create a file titled Why Markdown is Ideal for AI.md. Explain the benefits of plain text for LLMs. Note 3: Create a file titled Maps of Content (MOC).md. Define what an MOC is and why it’s useful.

Crucial Step: Edit the content of these notes so they link to one another. For example, in Note 1, you might write: “An exocortex is often managed using tools that support [[Maps of Content (MOC)]].” Ensure the brackets create active links.

Lab 3.3 — Create Your Project MOC

Create a file inside the 03-Project/ folder called MOC-AI-Product.md. This will serve as the running index for your coursework and product development.

Copy the following template into the file and fill in the details based on your Session 1 product idea:

# MOC: AI Product Engineering Project

## Product Overview

- **Problem Statement:** [Insert the problem your product solves]
- **Target User:** [Insert your specific target audience]
- **AI Role:** [Insert what the AI will actually do]
- **Success Metric:** [Insert how you will measure success]

## Core Project Assets

- [[Product-Spec]]
- [[Prompts]] (Your prompt library from Session 2)
- [[Architecture]] (To be created in Session 6)

## Knowledge Base (The Exocortex)

- [[What is an Exocortex]]
- [[Why Markdown is Ideal for AI]]
- [[Maps of Content (MOC)]]

Action Item: Make a habit of adding to this MOC after every single session. By Session 15, this single file will serve as the map of your entire project and learning journey.

Lab 3.4 — Import External Content

An exocortex must interface with the outside world. Find one external resource relevant to the domain of your AI product (e.g., a Wikipedia article, a software documentation page, or an industry report).

Create a new note in 04-References/:

  • Summarise the key point of the article in 2–3 sentences (do not just copy and paste; use your own words to aid retention).
  • Paste the source URL at the bottom of the note.
  • Add at least one [[link]] connecting this external reference to a concept note in your vault.

Key Takeaways

  • An exocortex is a structured, external memory system. In AI engineering, it serves as the highly curated, proprietary knowledge layer that your AI models will query to prevent hallucinations.
  • Obsidian uses plain, local Markdown files. This format is lightweight, universally readable, and perfectly structured for Large Language Models to consume directly without complex parsing.
  • An effective vault relies on Atomic Notes (one idea per note), Bidirectional Links (creating a knowledge graph), and Maps of Content (acting as navigational hubs).
  • The structured knowledge you build in this vault will serve as the raw, embeddable data for the Retrieval-Augmented Generation (RAG) pipeline you will build in upcoming sessions.

Further Reading & Resources

  • Obsidian Official Documentation: help.obsidian.md - Guides on formatting, linking, and advanced plugin usage.
  • “Building a Second Brain” by Tiago Forte (2022) - The foundational modern text on personal knowledge management and externalizing memory.
  • The Zettelkasten Method: zettelkasten.de - Deep dive into the historical note-taking system that inspired bidirectional linking and atomic notes.