Chapter 4 — Memory the Model Was Never Given: Embeddings, Retrieval, and RAG

Tools gave the model the ability to act. But there's a different problem that tools don't solve: knowledge. The model was trained on text that existed before its training cutoff. It does not know your company's internal documentation, your customers' history, your product's current features, or anything that changed after training ended. You can't fix this with a better system prompt — the knowledge isn't there to be prompted. You can't fix it with fine-tuning for most use cases — fine-tuning teaches behavior patterns, not encyclopedic facts, and fine-tuning on proprietary data is expensive, slow, and still doesn't keep up with documents that change daily. The fix is to retrieve the relevant information at runtime and inject it into the context window, right before the model needs to answer. That's Retrieval-Augmented Generation. This chapter explains why it works, how to build it correctly, and — critically — how to know whether it's actually working.

Part 1: Embeddings — How Meaning Becomes Geometry

Before you can retrieve relevant documents, you need a way to measure relevance. Keyword search (grep, BM25) measures lexical overlap — are the same words present? Semantic search measures meaning — are the concepts related, even if the words differ? Embeddings are the mechanism that makes semantic search possible.

An embedding is a vector — a list of floating-point numbers, typically 768 to 3072 dimensions depending on the model — that represents the meaning of a piece of text. The embedding model is a neural network trained to map text to vectors such that texts with similar meaning end up close together in this high-dimensional space, and texts with different meaning end up far apart.

"The capital of France is Paris" and "Paris is France's largest city" end up close together. "The capital of France is Paris" and "The quarterly earnings exceeded forecasts" end up far apart. "Dog bites man" and "Man bitten by dog" are semantically similar (passive/active voice difference only) and end up close together.

Why can this work? Because the embedding model was trained on vast amounts of text and learned to map meaning to geometry — the directions in this high-dimensional space correspond to semantic dimensions. There's no human-designed coordinate system; the geometry emerged from the training process. What matters for you, as a builder, is that the property holds reliably: similar meaning → nearby vectors.

The Mathematics of Closeness

Given two vectors, how do you measure how close they are? The naive answer is Euclidean distance — the straight-line distance between two points in the embedding space. But Euclidean distance has a problem: it conflates magnitude with direction. Two documents might produce vectors pointing in the same direction but with different magnitudes (because one is longer, contains more tokens, or uses more emphatic language), even though they mean essentially the same thing.

The standard measure for semantic similarity is cosine similarity: the cosine of the angle between two vectors.

cosine_similarity(A, B) = (A · B) / (|A| × |B|)

This normalizes out magnitude — it measures only whether the vectors point in the same direction, which corresponds to whether they have the same meaning. Two identical-meaning vectors have cosine similarity 1.0 regardless of their magnitude. Orthogonal vectors (completely unrelated meaning) have cosine similarity 0. Opposite-meaning vectors have cosine similarity -1.0.

High-dimensional geometry intuition:

       "Paris is France's capital"
              /
             /  small angle → high cosine similarity
            /
           /____________ "France's largest city is Paris"
           
           vs.
           
       "Paris is France's capital"
              |
              | large angle → low cosine similarity
              |
              _________________ "Quarterly earnings exceeded forecasts"

The practical implication: when building a retrieval system, always use cosine similarity (or an equivalent like dot product on normalized vectors), not Euclidean distance.

Part 2: Vector Databases — Approximate Nearest Neighbour Search

You have a corpus of 10 million documents, each embedded into a 1536-dimensional vector. A user asks a question. You embed the question into the same 1536-dimensional space. Now you want to find the documents whose vectors are closest to the question vector.

The brute-force approach: compute the cosine similarity between the query vector and all 10 million document vectors. That's 10 million dot products per query. For 1536-dimensional vectors, that's on the order of 15 billion floating-point operations per query. Even on modern hardware this is too slow for low-latency applications.

Vector databases exist to make this feasible. They use Approximate Nearest Neighbour (ANN) algorithms — algorithms that find the approximately nearest neighbours much faster than brute force, at the cost of occasionally missing some true nearest neighbours.

The most common ANN algorithm in production vector databases is HNSW (Hierarchical Navigable Small World). Without going into the full algorithm, the intuition is: the database builds a multi-layer graph structure during indexing. Nodes at higher layers span large distances (allowing fast coarse navigation); nodes at lower layers span small distances (allowing fine-grained precise matching). A query starts at the top layer, navigates quickly to the right region of the space, then descends to lower layers for precise matching. This runs in O(log n) time rather than O(n), making it tractable for billions of vectors.

The key engineering tradeoff: ANN indices trade recall (finding all true nearest neighbours) for speed. An index configured for very high speed may miss 5-10% of true nearest neighbours. For most retrieval applications, this is acceptable — if you're retrieving the top 20 most relevant documents, missing 1 or 2 due to approximate search is tolerable. For high-stakes retrieval where you absolutely cannot miss relevant documents, you can configure the index for higher recall at the cost of query latency.

Popular vector databases: Pinecone, Weaviate, Qdrant, Chroma (local, development-friendly), pgvector (PostgreSQL extension — semantic search inside your existing Postgres instance). Each has different tradeoffs on scale, latency, filtering capabilities, and operational complexity.

Part 3: Chunking — Why This Decision Matters More Than You Think

Before you can embed documents, you have to decide what unit to embed. A document might be 50 pages long. You can't embed the whole thing as one vector — you lose too much specificity (one vector can't represent all the distinct concepts in a 50-page document). But you also can't split it arbitrarily.

Chunking is the process of splitting documents into units that will each get their own embedding. Getting chunking wrong is the most common reason RAG systems underperform.

Fixed-Size Chunking

Split every document into chunks of N tokens (e.g., 512 tokens) with an optional overlap (e.g., 50-token overlap between consecutive chunks so a concept that spans a boundary isn't completely severed).

This is the simplest approach and the most widely used in tutorials. It's also the most fragile. Consider a document where paragraph 7 contains a key insight. If your chunk boundary falls in the middle of that paragraph — which it will, because you're splitting blindly by token count — one chunk contains the beginning of the insight and another contains the end. Neither chunk contains the complete idea. When you embed these partial chunks, the vectors are weaker signals for the concept than if you'd embedded the complete paragraph.

Worse: in technical documentation, a code example often spans several paragraphs — setup, the example itself, and explanation. Fixed-size chunking will split these apart. The code example chunk doesn't explain what the code does. The explanation chunk doesn't include the code. Both chunks are less useful in isolation.

Fixed-size chunking works acceptably when the document structure is already regular (e.g., database records with fixed-length fields) or when the domain is tolerant of partial context. Avoid it for narrative text, technical documentation, or anything where concepts span paragraph boundaries.

Semantic Chunking

Split at structural boundaries: paragraph breaks, section headers, list items. A new chunk starts when the topic changes, not when a token count is reached. This preserves the semantic coherence of each chunk.

The implementation is document-type specific: for Markdown, split at ## headers; for PDFs, split at paragraph breaks detected by the PDF parser; for HTML, split at <section> or <article> boundaries. For unstructured text, you can use an embedding-based approach: compute embeddings for consecutive sentences and split when consecutive sentence embeddings drift significantly apart (indicating a topic shift).

Recursive Character Text Splitting

A practical middle ground, popularized by LangChain's RecursiveCharacterTextSplitter. Define a hierarchy of separators: try to split at double newlines first (paragraph boundaries), then single newlines, then sentences (periods), then words, then characters. This produces chunks that are approximately the target size but prefer natural linguistic boundaries over arbitrary character counts.

Parent-Child Chunking (Contextual Retrieval)

A more sophisticated strategy: create small chunks for retrieval (high specificity, better cosine similarity signal) but return a larger parent chunk to the model (more context for comprehension).

You index the small chunks (e.g., single paragraphs) and use them for retrieval. But when a small chunk matches the query, you return not just the chunk but the surrounding context (the full section it came from, or the preceding and following paragraphs). The model gets the precise match plus the context it needs to use that match correctly.

This is often the right architecture for complex documents where:

• Specific sentences are the retrieval targets
• But those sentences only make sense in their surrounding context

Part 4: Hybrid Search — Why Neither Vector Nor Keyword Alone Is Sufficient

Semantic (vector) search is powerful for conceptual queries — it finds documents about similar ideas even when the exact words don't match. Keyword (BM25/lexical) search is powerful for exact-match queries — if the user asks for "RFC 7231" or "order #ORD-20241101-7823", keyword search will find it exactly while semantic search might return vaguely related documents.

In practice, both failure modes hurt you:

Semantic search failure: The user asks for a specific product SKU "XJ-4420-R". Semantic search finds documents about similar products. The user wanted the exact SKU.

Keyword search failure: The user asks "how do I scale my database horizontally when writes are bottlenecked?" Keyword search looks for exact matches on those words. The documentation that answers this question might say "partitioning can distribute write load across nodes" — completely different words, same answer.

Hybrid search runs both searches and combines the results. The standard combination technique is Reciprocal Rank Fusion (RRF):

RRF score = Σ 1 / (k + rank_i)

Where:
  k = constant (typically 60) to dampen the effect of very high ranks
  rank_i = the document's rank in each individual search result list

Each document gets a score based on its rank in the semantic results list and its rank in the keyword results list. The scores are summed. Documents that rank highly in both searches get the highest combined scores. Documents that only rank in one search can still be retrieved if they rank very highly in that one.

def reciprocal_rank_fusion(semantic_results: list, keyword_results: list, k=60) -> list:
    scores = {}
    
    for rank, doc in enumerate(semantic_results):
        scores[doc.id] = scores.get(doc.id, 0) + 1 / (k + rank + 1)
    
    for rank, doc in enumerate(keyword_results):
        scores[doc.id] = scores.get(doc.id, 0) + 1 / (k + rank + 1)
    
    return sorted(scores.items(), key=lambda x: x[1], reverse=True)

Most production RAG systems use hybrid search. The only argument for pure semantic search is simplicity — a single vector database is easier to operate than a vector database plus an Elasticsearch/OpenSearch cluster. If operational simplicity is paramount and your domain doesn't have many exact-match queries, pure semantic search is defensible. In most cases, hybrid search is worth the complexity.

Part 5: Metadata Filtering — Scope Before Search

Semantic similarity is computed globally across your entire corpus unless you constrain it. For large corpora with multiple distinct data types, unfiltered semantic search returns irrelevant results that happen to be semantically similar to the query.

Example: your corpus contains product documentation, support tickets, and engineering specs. A user asks about "authentication error." The most semantically similar chunks might include a support ticket from three years ago, an engineering spec for a different product, and the actual relevant documentation section. The support ticket and engineering spec rank highly by cosine similarity but aren't useful for this query.

Metadata filtering constrains the search space before running vector similarity. Each chunk is stored with metadata (document type, product version, creation date, department, access level, etc.), and the retrieval query includes a filter that narrows to the relevant subset:

results = vector_db.query(
    query_vector=embed(user_query),
    filter={
        "document_type": "product_documentation",
        "product_version": {"$gte": "v2.0"},
        "access_level": {"$in": user.permissions}
    },
    top_k=10
)

Metadata filtering is also the mechanism for multi-tenant data isolation: a customer's agent should only search chunks belonging to that customer. Store tenant_id as metadata on every chunk, filter by tenant_id on every query. This is safer than relying on the semantic distance to naturally separate tenant data (it won't — semantically similar content from different tenants will mix).

Part 6: Re-ranking — The Speed/Quality Tradeoff

Your hybrid search returns the top 20 most relevant chunks. But "most relevant" according to the retrieval stage is approximate — both semantic similarity and BM25 are coarse signals. The actual most useful chunks for generating a correct answer might not be the top 3 by retrieval score.

Re-ranking is a second-pass scoring step that applies a more expensive but more accurate relevance model to reorder the top-K retrieved candidates.

Bi-Encoders (What Vector Search Uses)

Your vector database uses bi-encoder embeddings: the query is embedded independently, the document is embedded independently, and cosine similarity is computed between two pre-computed vectors. This is fast because document embeddings are pre-computed and stored — each query only requires embedding the query (one forward pass) and computing dot products against stored vectors.

The weakness: the query and document are embedded independently, so the similarity score can't capture complex query-document interactions. The model never sees both at once; it just measures the angle between two independently-computed vectors.

Cross-Encoders (What Re-ranking Uses)

A cross-encoder takes both the query and a document as input simultaneously, concatenated, and produces a relevance score. Because the model sees both the query and the document at the same time, it can model their interactions — "this word in the query corresponds to this specific sentence in the document" — producing much more accurate relevance judgments.

The weakness: cross-encoders cannot be pre-computed for all document pairs — you have to run a fresh forward pass for every (query, document) pair at query time. This is too slow to run against millions of documents, which is why you use bi-encoders for the initial retrieval and cross-encoders only for the re-ranking step (typically the top 20-50 candidates).

Retrieval pipeline with re-ranking:

Query
  ↓
Hybrid Search (bi-encoder + BM25)
  → Returns top 50 candidates (fast, approximate)
  ↓
Re-ranker (cross-encoder)
  → Scores each of the 50 candidates with query context
  → Returns top 5 re-ranked results (slow, precise)
  ↓
LLM context
  → Model answers with the top 5 most relevant chunks

Popular re-rankers: Cohere Rerank API, Voyage AI, open-source cross-encoders from HuggingFace (ms-marco series). The latency cost is typically 100-300ms for 20-50 candidates — acceptable for interactive applications, worth measuring before committing.

Part 7: Query Rewriting — Why the User's Literal Query Is Often the Wrong Retrieval Key

The user's query is optimized for human communication, not for retrieval. Humans communicate with assumed context, abbreviations, references to prior conversation, and casual language. Vector search needs a query that is a clear, complete, standalone expression of the information need.

Some specific failure modes:

Coreference: "What did they say about the database in the last section?" — "they" and "the last section" have no meaning without conversation context. The embedding of this query vector is nowhere near the embedding of the relevant document.

Keyword poverty: "Show me the timeout thing" — the relevant documentation section says "configuring request timeout thresholds." The words share no overlap and their semantic similarity is moderate at best.

Implicit knowledge assumed: "Why does it keep crashing when I have a lot of users?" — the word "it" refers to a specific system the user is working with; "a lot" has no scale; "crashing" might mean process death or slow responses. The query doesn't contain enough signal to find the relevant monitoring documentation.

Query rewriting transforms the user's input into a form that is more retrievable. Several approaches:

HyDE (Hypothetical Document Embeddings): Instead of embedding the query, ask the model to generate a hypothetical answer to the query (a document that would answer it), then embed that hypothetical document. The hypothesis lives in the same part of the embedding space as real documents that would answer the question, so cosine similarity retrieves better results. This works remarkably well in practice.

def hyde_retrieval(user_query: str) -> list[Document]:
    # Step 1: Generate a hypothetical document that would answer the query
    hypothesis = model.complete(
        f"Write a short paragraph that would directly answer this question: {user_query}"
    )
    
    # Step 2: Embed the hypothesis (not the query)
    query_vector = embed(hypothesis)
    
    # Step 3: Retrieve based on hypothesis embedding
    return vector_db.query(query_vector, top_k=10)

Conversational context injection: Prepend the relevant conversation history to the query before embedding. "What did they say about the database?" becomes "What do the company's engineering docs say about database configuration?" after the model rewrites it with context.

Multi-query expansion: Generate multiple reformulations of the query and retrieve for each, then deduplicate results. Catches documents that are relevant to one formulation but not another.

Part 8: Semantic Caching — Same Query, No Redundant Work

If your system receives high traffic, many queries will be semantically similar or identical. Running full retrieval + LLM inference for "what are your business hours?" every time it's asked is wasteful. Semantic caching serves cached results for similar queries.

Implementation: maintain a cache of (query_embedding, result) pairs. For each new query, compute its embedding, and check if any cached query has cosine similarity above a threshold (e.g., 0.95). If so, return the cached result without retrieval or inference. If not, run the full pipeline and cache the result.

SIMILARITY_THRESHOLD = 0.95

def semantic_cache_lookup(query: str, cache: list[CacheEntry]) -> Optional[str]:
    query_embedding = embed(query)
    
    for entry in cache:
        similarity = cosine_similarity(query_embedding, entry.query_embedding)
        if similarity >= SIMILARITY_THRESHOLD:
            return entry.result  # Cache hit
    
    return None  # Cache miss

def handle_query(query: str) -> str:
    cached = semantic_cache_lookup(query, cache)
    if cached:
        return cached
    
    result = full_rag_pipeline(query)
    cache.append(CacheEntry(query_embedding=embed(query), result=result))
    return result

The threshold calibration matters: too high (0.99) and you miss nearly-identical queries with slightly different wording; too low (0.85) and you return cached answers for questions that are topically similar but actually different. 0.92-0.95 is a reasonable starting point. For factual queries (where the same question always has the same answer) you can tune lower; for personalized or context-dependent queries, tune higher or disable caching entirely.

The cost savings can be significant in high-volume applications. A query like "how do I reset my password?" may constitute 5-15% of all support traffic and needs to be answered only once.

Part 9: The Six RAG Evals — How to Know If Your System Is Actually Working

This is the part most RAG tutorials skip, and it's the part that determines whether your system actually works in production.

Jason Liu's framework (from his work at Instructor and the RAG community) decomposes RAG quality into six measurements across three pairs. Understanding this framework requires understanding that RAG has two distinct failure modes: retrieval failure (you didn't find the right documents) and generation failure (you found the right documents but the model didn't use them correctly or made things up).

The Three Pairs

Pair 1: Retrieval Quality

• Context Relevance (C|Q): Given the query Q, is the retrieved context C actually relevant to answering it? This measures whether your retrieval pipeline is finding the right documents. A low C|Q score means your chunking, embedding, or search is returning noise.

• Answer Faithfulness (A|C): Given the context C that was retrieved, is the answer A faithful to that context? Does the model answer only from what's in the retrieved chunks, or does it supplement with hallucinated information? This is the classic RAG hallucination check.

• Answer Relevance (A|Q): Given the query Q, is the answer A actually relevant to what was asked? This catches a specific failure mode: the model uses the retrieved context correctly but wanders off-topic or answers a related but different question.

Pair 2: Retrieval Completeness

• Context Recall: Of all the relevant information needed to answer the question, what fraction is actually present in the retrieved context? Low context recall means your retrieval found some relevant chunks but missed others — the model will have incomplete information and may hallucinate the missing pieces.

• Context Precision: Of the retrieved context chunks, what fraction are actually relevant? High context noise (many retrieved chunks that aren't actually needed) dilutes the relevant signal and can confuse the model.

Pair 3: End-to-End

• End-to-end correctness: Is the final answer actually correct? This is the ground-truth check that catches failures that slip through the individual component metrics.

Implementing the Evals

For production evaluation, you typically use an LLM-as-judge for context relevance and faithfulness, because these require natural language reasoning that's difficult to reduce to a simple metric. You'll calibrate this judge against human labels in Chapter 7 — for now, understand that the eval exists and what it measures.

# Faithfulness eval using LLM-as-judge
def eval_faithfulness(question: str, context: list[str], answer: str) -> float:
    """
    Returns a score from 0.0 to 1.0.
    1.0 = answer is fully supported by the context, no hallucination.
    0.0 = answer contains claims not supported by the context.
    """
    prompt = f"""
    Given the following context and answer, evaluate whether every claim
    in the answer is supported by the context.
    
    Context: {' '.join(context)}
    Answer: {answer}
    
    Rate faithfulness from 0 to 1, where:
    1.0 = all claims are directly supported by context
    0.5 = some claims are supported, some are not
    0.0 = answer contains significant claims not in context
    
    Output only a number between 0 and 1.
    """
    score = float(model.complete(prompt).strip())
    return score

# Context relevance eval
def eval_context_relevance(question: str, context_chunk: str) -> float:
    prompt = f"""
    Given this question and context chunk, rate how relevant the context
    is to answering the question.
    
    Question: {question}
    Context: {context_chunk}
    
    Rate from 0 to 1, where 1.0 means the chunk directly helps answer the question.
    Output only a number.
    """
    return float(model.complete(prompt).strip())

The Ground-Truth Dataset

Running these evals requires a ground-truth dataset: a set of (question, expected answer, relevant documents) triples. Building this dataset is the hard part. For a production system, you want:

• 50-200 question-answer pairs at minimum for meaningful statistics
• Coverage of all major query types your system is expected to handle
• Coverage of known edge cases (questions where the answer isn't in the corpus, questions that require reasoning across multiple chunks)
• Ideally, a mix of human-curated and synthetically generated examples

You'll build the full synthetic generation approach in Chapter 7. For now, the key insight: without these evals, you are deploying a RAG system and hoping it works. With them, you know which component is failing and can fix it methodically.

Part 10: Why RAG Still Hallucinates and How to Catch It

RAG dramatically reduces hallucination compared to a model answering from parametric memory alone. But it does not eliminate it. Understanding why it still happens — and how to detect it — is the difference between a system you can trust and one that occasionally poisons your users' understanding.

The Three Remaining Hallucination Causes in RAG Systems

1. Retrieval failure hallucination. The relevant document wasn't retrieved — either because it wasn't in the corpus, the chunking severed it, or the query didn't match its embedding well. The model is asked to answer with no relevant context. It does what a model always does when it has weak signal: it predicts the most probable continuation, which looks like a confident answer but isn't grounded in your data.

Detection: context relevance eval (C|Q) will show low scores. Retrieval recall will be low. The model's answer may look plausible but reference your specific context incorrectly.

2. Context misuse hallucination. The relevant document was retrieved but the model used it incorrectly — misread a number, conflated two similar concepts, or answered with partial information while presenting it as complete. This is the subtlest hallucination type because it partially resembles the truth.

Detection: faithfulness eval (A|C) will catch cases where the model's answer diverges from the retrieved context. Hard cases require human review.

3. Extrapolation hallucination. The retrieved context answers part of the question, and the model fills in the gap by extrapolating. If the context says "Product X supports OAuth 2.0" and the user asks about SAML support, the model might extrapolate "Product X probably supports SAML too." The retrieval worked; the model invented the rest.

Detection: end-to-end correctness eval against known ground truth. Faithfulness eval may not catch this because the extrapolated claim is adjacent to (but not supported by) the context.

The Practical Defense

Run faithfulness evals (A|C) continuously on a sample of production traffic. When faithfulness scores drop, triage whether the cause is retrieval failure (context relevance also dropped) or model misuse (context relevance is fine but faithfulness dropped). Each cause has a different fix.

For high-stakes answers, add a "citation check" step: instruct the model to include, for each claim in its answer, the specific chunk from the context that supports it. Then programmatically verify that those citations exist in the retrieved context. Any claim without a citation is a potential hallucination.

Capstone 1 — Begin: RAG Over a Large Corpus

This is where you start building Capstone 1, which you'll complete with the evals in Chapter 7.

The system to build: a RAG pipeline over a documentation corpus (pick something real: your company's internal docs, a large open-source project's documentation, the AWS docs, or similar). The requirements:

• Document ingestion with semantic chunking (not fixed-size)
• Hybrid search: vector (your choice of embedding model and vector DB) + BM25
• Re-ranking using a cross-encoder or an API-based re-ranker
• Query rewriting (at minimum: HyDE or conversational context injection)
• Semantic caching for common queries
• Instrumentation to log: query, retrieved chunks with scores, final answer, latency, cost

The instrumentation is not optional. Chapter 7 will use those logs to run the six evals. A system you can't measure is a system you can't improve.

Vocabulary Map

Practice Problems

Problem 1 — Failure Analysis

Your RAG system answers questions about your company's engineering documentation. Users report that answers about security configurations are correct 90% of the time but wrong 10% of the time — specifically, answers that span two related documents tend to have errors. Walk through which of the six RAG eval metrics would catch this failure, which specific cause it maps to (retrieval failure, context misuse, or extrapolation), and what specific change to the retrieval pipeline would most likely fix it.

Problem 2 — Design from Scratch

You're building a RAG system for a legal document corpus (contracts, case law, regulatory filings). The corpus has 500,000 documents averaging 80 pages each. Users ask highly specific questions like "what does clause 7.3 of the 2023 Master Services Agreement with Vendor X say about data retention?" and also broad conceptual questions like "what are our data retention obligations under GDPR?" Design the full retrieval pipeline: chunking strategy, search approach (vector only vs hybrid), metadata schema, re-ranking decision, and semantic caching configuration. Justify every decision against the constraints of the domain.

Problem 3 — Synthesis

Explain why these three facts are all consequences of the same fundamental limitation: (a) cosine similarity measures angle, not content accuracy; (b) ANN search is approximate, not exact; (c) a model can produce a hallucinated answer even when the retrieved context is correct. Identify the one underlying characteristic of the RAG pipeline these three facts share, and explain what architectural decisions minimize the damage from this characteristic.

Problem 4 — Go Deeper

Consider a RAG system where the corpus is continuously updated — new documents are added hourly, existing documents are revised. Design the update strategy for: (a) the vector index (how do you handle additions, deletions, and updates to documents?), (b) the semantic cache (when does a cached answer become stale?), and (c) the eval dataset (how do you detect when corpus changes have invalidated your ground-truth eval answers?). This requires reasoning about systems design that the chapter doesn't explicitly cover.

Problem 5 — Connect to Previous Chapter

Chapter 3 introduced the Model Context Protocol (MCP) and explained that tools let the model reach outside its context. Explain precisely how RAG fits into or alongside the MCP architecture. Is RAG a tool (in the Chapter 3 sense)? Could you implement RAG as an MCP server? What would the tool schema look like? What are the tradeoffs of implementing RAG as an MCP tool versus implementing it as a pre-processing step that enriches the context before calling the model?

What's Coming Next

Your model can now be prompted precisely, it can use tools to act on the world, and it can retrieve knowledge it was never trained on. But every capability in the first four chapters is exercised in a single call. The model responds. The call ends. Whatever happened inside that call is complete. Real multi-step work — debugging a system, researching a topic across dozens of sources, autonomously completing a complex task — requires a loop: the model observes its output, decides what to do next, executes something, observes the result, and continues. That loop is what makes an agent an agent, and Chapter 5 builds it from first principles before you touch any framework.