Chapter 6 — Many Agents, One Goal: Orchestration, Memory, and Failure Modes

A single agent doing complex work runs into the same wall every time: the context window fills, the model loses focus on the original goal, and a generalist tackling a specialist's job underperforms. These aren't bugs in your prompt or your loop — they're physical limits of the architecture. A context window is finite memory. One set of instructions creates one behavioral persona. One agent can only work serially. When a task genuinely exceeds these limits, the answer is to distribute it: multiple agents, specialized for different parts of the task, coordinated by an orchestrator, sharing state through structured memory. This is where AI engineering starts to look like distributed systems design — and your instincts from that domain are directly applicable, with important AI-specific failure modes layered on top.

Part 1: Memory Architecture — The Four Layers

Before building multi-agent systems, you need a precise vocabulary for what "memory" means in this context. Not all memory is the same, and confusing the layers produces systems that lose critical state at inopportune moments.

Layer 1: Working Memory (Context Window)

This is the model's immediate, active memory — the message history that is present in the current inference call. It is:

• Fast to access (already in the LLM's forward pass)
• Finite (128k tokens for most frontier models, as of mid-2025)
• Ephemeral (gone when the context window is reset or the conversation ends)
• Stateless (the model has no memory across calls except what you inject)

Working memory is where the agent's active reasoning happens. Everything an agent "knows" during a given inference call is in the context window. When working memory is full, either older content gets truncated (catastrophic — the agent loses its history) or you must actively manage what to keep.

Layer 2: Long-Term Memory (External Store)

A persistent store outside the model — a database, a file, a vector store — that the agent writes to and reads from explicitly via tool calls. It is:

• Arbitrarily large (bounded only by your infrastructure)
• Persistent across agent runs, user sessions, and application restarts
• Slow to access relative to context (requires a tool call + query)
• Queryable (unlike the context window, you can search it)

Long-term memory is the agent's equivalent of a database. It holds facts, decisions, completed work, user preferences, and anything that needs to survive beyond the current context window.

Layer 3: Episodic Memory (Past Interactions)

A specific type of long-term memory: a record of past interactions, sessions, and agent runs. What tasks has this agent done before? What did it discover? What mistakes did it make? Episodic memory enables an agent to improve over time and to avoid repeating the same exploration when asked a similar task again.

Implementation: after each completed agent run, extract a structured summary (what was the task, what was the approach, what was the outcome, what was learned) and store it in a searchable episodic store. At the start of future runs, retrieve relevant past episodes and inject them as context: "You have done this type of task before — here's what worked."

Layer 4: Semantic Memory (Knowledge Base)

Factual, semantic knowledge that isn't tied to a specific past interaction — your company's documentation, a curated knowledge base, domain-specific facts. This is exactly what Chapter 4's RAG system provides. Semantic memory is retrieved via embedding similarity; it contains general knowledge relevant to the agent's domain rather than records of specific past events.

Memory layers in a production agent:

┌─────────────────────────────────────────────┐
│  Working Memory (Context Window)             │
│  [system] [user] [tools] [assistant] ...     │
│  Fast | Finite | Ephemeral                   │
└───────────────┬─────────────────────────────┘
                │ inject at runtime
    ┌───────────┼──────────────────────┐
    ▼           ▼                      ▼
Long-Term   Episodic               Semantic
(DB writes) (Past runs →           (RAG → facts,
            search and inject)     docs, knowledge)

Getting the memory architecture right is the highest-leverage architectural decision in multi-agent system design. A system that writes to long-term memory at the wrong granularity, doesn't retrieve from episodic memory, or over-populates the context window will underperform — not because the models are bad but because the state management is wrong.

Part 2: Working Memory Budgeting — What to Keep, Compress, or Discard

A context window running at 80% capacity is an agent heading for trouble. At 90%, quality typically degrades (lost-in-the-middle worsens). At 95%+ you're one large tool result away from either a truncation error or forcing the model to work with an incomplete picture.

Working memory budgeting is active context management: at each iteration, you decide what to keep in the context window, what to summarize (compress into fewer tokens), and what to write to long-term memory and drop from context.

The Token Budget Mental Model

128,000 token context window — budget allocation:

System prompt:               500  tokens  (constant, always present)
Task description:            500  tokens  (constant)
Long-term memory injections: 2,000 tokens  (relevant retrieved context)
Recent tool results:         5,000 tokens  (last 3-5 tool calls)
Compressed history:          3,000 tokens  (summary of earlier work)
Active reasoning space:     10,000 tokens  (current thought + next action)
Output headroom:             4,000 tokens  (expected response)
                           -------
Reserved budget:            25,000 tokens  (25% of 128k)
Available overflow:        103,000 tokens  (for unexpected large results)

You want substantial headroom because tool results are unpredictable. A database query might return 200 tokens or 20,000 tokens. A web fetch might return a short snippet or an entire article. If your budget is full before the first large tool result, you're in trouble.

Summarization Strategy

When earlier parts of the conversation history are no longer needed in detail, compress them. The model itself is the best summarizer of the model's own work:

def compress_history(old_messages: list[dict], keep_last_n: int = 5) -> list[dict]:
    if len(old_messages) <= keep_last_n:
        return old_messages
    
    to_compress = old_messages[:-keep_last_n]
    recent = old_messages[-keep_last_n:]
    
    # Ask the model to summarize the history being compressed
    summary = model.complete(f"""
    Summarize the following agent work log into 2-3 paragraphs.
    Include: what task was being done, what was discovered, 
    what decisions were made, what has been completed.
    Preserve specific facts, numbers, and identifiers that 
    may be needed later.
    
    Work log: {format_messages(to_compress)}
    """)
    
    compressed_message = {
        "role": "system",
        "content": f"[Earlier work summary]: {summary}"
    }
    
    return [compressed_message] + recent

The critical tension in summarization: you want to reduce token count, but you don't want to lose facts the agent will need later. A summary that omits a key finding is worse than no summary — the agent will confidently proceed without the context it needs. When in doubt, err toward preserving specific data (numbers, IDs, decisions) even at the cost of longer summaries.

Write-Then-Drop

For completed subtasks, the pattern is: execute the subtask, write the result to long-term memory, drop the detailed execution from the context window. Keep only the summary of what was done and a pointer to where the full result is stored.

Before write-then-drop (in context):
[50 messages of detailed web scraping for competitor A] ← 15,000 tokens

After write-then-drop (in context):
[Summary: Competitor A pricing analysis complete. Stored in long_term/competitor_a_pricing.json] ← 50 tokens

In long-term store:
competitor_a_pricing.json ← full detailed analysis, retrievable on demand

Part 3: The Orchestrator + Specialist Pattern

The most common multi-agent architecture has two roles: an orchestrator agent that plans and routes work, and specialist agents that execute specific subtasks.

Orchestrator + Specialist architecture:

User Request
     │
     ▼
┌────────────────┐
│  Orchestrator  │  "I need competitive analysis,
│  Agent         │   SEO research, and a summary.
│                │   I'll route to specialists."
└────┬───┬───┬───┘
     │   │   │
     ▼   ▼   ▼
  [Web] [SEO] [Writer]
  Specialist Specialist Specialist
     │   │   │
     └───┴───┘
          │
          ▼
  Results injected back
  to Orchestrator
          │
          ▼
  Orchestrator synthesizes
  final response

The orchestrator knows about the high-level task and the available specialists. It doesn't know how to do deep web research, SEO analysis, or writing — those are the specialists' domains. The orchestrator decomposes the task, routes subtasks to the right specialists, collects results, and synthesizes.

The specialist agents are narrow: they have a focused system prompt, a specific set of tools appropriate to their domain, and limited context about the broader task. A web research specialist knows about web searching and content extraction; it doesn't need to know about SEO ranking factors. This narrowness is a feature — specialized context produces more reliable output than a generalist trying to do everything.

Implementing Agent-to-Agent Communication

From the orchestrator's perspective, a specialist agent is a tool. It has a name, a description, and it returns a result. The agent-to-agent protocol (A2A, introduced in Chapter 3) formalizes this, but even without a formal protocol, the pattern is straightforward:

# Orchestrator's tool set includes "sub-agents" as tools
orchestrator_tools = [
    Tool(
        name="run_web_research_agent",
        description="Delegate web research tasks. Give it a specific research question. Returns a structured research report.",
        parameters={"research_question": "string", "max_sources": "integer"}
    ),
    Tool(
        name="run_seo_analysis_agent", 
        description="Analyze SEO performance for a URL or domain.",
        parameters={"target": "string", "analysis_type": "enum[technical,content,backlinks]"}
    ),
    # ...
]

# The tool execution dispatches to the sub-agent:
def execute_tool(tool_call):
    if tool_call.name == "run_web_research_agent":
        return run_agent(
            task=tool_call.arguments["research_question"],
            tools=WEB_RESEARCH_TOOLS,
            system_prompt=WEB_RESEARCHER_SYSTEM_PROMPT,
            max_iterations=15
        )

From the orchestrator's view, it made a tool call and got a result. The fact that the "tool" is itself running an agent loop with multiple LLM calls is an implementation detail. This is the composability property that makes multi-agent systems powerful.

Part 4: The Critic + Refiner Pattern

One agent's output quality is bounded by that agent's reasoning. A second agent reading the first agent's output and critiquing it is a powerful quality amplifier — but only if the critic has genuinely independent perspective.

The critic + refiner pattern:

1. Generator agent produces an initial output (a document, a code snippet, a plan)
2. Critic agent receives the output and evaluates it against specific criteria, producing structured feedback
3. Refiner agent (often the same as the generator) receives the original output + the critique and produces an improved version
4. Repeat until the critic's score is above a threshold or N iterations are reached

def critic_refiner_loop(task: str, criteria: list[str], max_rounds=3) -> str:
    output = generator_agent.run(task)
    
    for round in range(max_rounds):
        critique = critic_agent.run(
            f"Evaluate this output against these criteria: {criteria}\n\n"
            f"Output:\n{output}\n\n"
            f"For each criterion, score 0-10 and explain improvements needed."
        )
        
        if all_criteria_pass(critique, threshold=8):
            break
        
        output = refiner_agent.run(
            f"Original task: {task}\n"
            f"Current output:\n{output}\n"
            f"Critique:\n{critique}\n"
            f"Produce an improved version addressing the critique."
        )
    
    return output

The critic's value is proportional to how independently it evaluates. A critic agent with the same system prompt and context as the generator will reproduce the same blind spots. To make the critique genuine:

• Give the critic a different perspective in its system prompt ("You are reviewing this from a skeptical reader's perspective who knows nothing about internal context")
• Don't give the critic the generator's reasoning — only its output (to avoid the critic rationalizing the same reasoning)
• Use structured criteria rather than "is this good?" — vague criteria produce vague critiques

Part 5: Mixture-of-Agents

Mixture-of-Agents (MoA) takes a different approach: instead of one agent producing and another critiquing, you run the same task through multiple independent agents and aggregate their outputs.

Mixture-of-Agents:

Task
 ├─→ Agent A → Output A ─┐
 ├─→ Agent B → Output B ─┼─→ Aggregator → Final Output
 └─→ Agent C → Output C ─┘

The aggregator can be another model call that synthesizes the three outputs, or a voting/majority scheme for classification tasks, or a cross-encoder that scores each output against the task.

MoA is appropriate when:

• Individual model outputs have high variance (some are excellent, some are poor)
• The task is high-stakes enough to justify the cost of multiple runs
• You can aggregate meaningfully (easiest for classification and structured outputs, harder for free-form generation)

The theoretical basis: the outputs from multiple independent agents sample different regions of the model's output distribution. The aggregated output has lower variance than any individual output — same logic as ensemble methods in traditional ML. The cost is 3x the compute and latency of a single agent run.

Part 6: Supervisor/Worker and Hierarchical Graphs

For very complex tasks, a two-level orchestrator/specialist hierarchy isn't enough. A marketing campaign pipeline might need: a strategy orchestrator, with specialist orchestrators for content, analytics, and distribution, each with their own specialist agents.

The hierarchical graph model generalizes to any depth:

Tier 1: Campaign Orchestrator
    │
    ├─ Tier 2: Content Orchestrator
    │     ├─ Tier 3: Blog Writer Agent
    │     ├─ Tier 3: Social Media Agent
    │     └─ Tier 3: Email Copy Agent
    │
    ├─ Tier 2: Analytics Orchestrator
    │     ├─ Tier 3: Performance Data Agent
    │     └─ Tier 3: Competitive Analysis Agent
    │
    └─ Tier 2: Distribution Orchestrator
          ├─ Tier 3: Publishing Agent
          └─ Tier 3: Scheduling Agent

Each tier orchestrates the tier below it. The top-level orchestrator never talks directly to bottom-level agents — it delegates to mid-level orchestrators, which delegate further. Each orchestrator only needs to know about its direct reports.

In LangGraph, this maps naturally to subgraphs: the campaign orchestrator is a graph whose nodes include mid-level orchestrators, which are themselves compiled subgraphs. The state flows down through the hierarchy as tasks are decomposed and up through it as results are collected.

The engineering discipline required: explicit contracts between levels. Each orchestrator-specialist pair needs a well-defined interface — what the orchestrator sends down (task description, constraints, context), what the specialist sends back (structured result, status, errors). Without these contracts, changes to one level break others silently.

Part 7: Failure Modes — The Ones That Kill Production Systems

Multi-agent systems introduce failure modes that single-agent systems don't have. Understanding these before they happen in production is the difference between a system you can debug in an hour and one that costs you a weekend.

Deadlock — Agent A Waits for Agent B; Agent B Waits for Agent A

In a multi-agent system where agents communicate and await results from each other, circular dependencies produce deadlocks. Agent A needs output from Agent B to proceed. Agent B needs output from Agent A to proceed. Both wait indefinitely.

Prevention: enforce a strict DAG (directed acyclic graph) in your agent communication topology. An orchestrator delegates to specialists; specialists never call back to the orchestrator or to other specialists at the same level. If two specialists need each other's output, introduce a coordinator that sequences them.

Detection: every agent call must have a timeout. If a result isn't received within the timeout, the waiting agent fails with a "downstream timeout" error rather than waiting indefinitely.

Infinite Loop — Agent Retries a Broken Tool Indefinitely

An agent calls a tool. The tool fails. The agent, instructed to "retry on failure," retries. The tool fails again. The agent retries again. Indefinitely.

This is the max_iterations guard from Chapter 5, applied at the tool level. Every retry loop needs:

• A maximum retry count
• Backoff between retries (exponential, with jitter)
• A terminal failure state that propagates up rather than retrying forever

def call_tool_with_retry(tool_call, max_retries=3, base_delay=1.0):
    for attempt in range(max_retries):
        try:
            return execute_tool(tool_call)
        except RetryableError as e:
            if attempt == max_retries - 1:
                return ToolError(f"Tool {tool_call.name} failed after {max_retries} attempts: {e}")
            time.sleep(base_delay * (2 ** attempt) + random.uniform(0, 0.5))
    
    # Propagate failure upward — don't silently swallow it
    return ToolError(f"Permanent failure: {tool_call.name}")

Sub-Agent Timeout — The Specialist Never Returns

The orchestrator dispatches a task to a specialist. The specialist hangs — perhaps due to an infinite loop in its own execution, a network call that never resolves, or a model API rate limit that's being waited on. The orchestrator is waiting. The user is waiting. Nothing is happening.

Every sub-agent call must have a wall-clock timeout. When the timeout expires, the orchestrator receives a timeout error and must decide: retry with the same specialist, try a different approach, or fail the subtask and proceed without it (if the subtask was non-critical).

In an async architecture (which is how long-running agents should be designed), this looks like:

async def run_specialist_with_timeout(specialist, task, timeout_seconds=120):
    try:
        result = await asyncio.wait_for(
            specialist.run_async(task),
            timeout=timeout_seconds
        )
        return result
    except asyncio.TimeoutError:
        await specialist.cancel()  # Clean up the specialist's resources
        return SpecialistTimeoutError(f"Specialist timed out after {timeout_seconds}s")

State Corruption Across Tool Calls

Agent A reads a record from a shared database. Agent B modifies the same record. Agent A, operating on its stale read, makes a decision that was correct for the old state but wrong for the new state. The database now contains an inconsistency.

This is the classic read-modify-write race condition, and it applies directly to multi-agent systems that share mutable state. The mitigations are the same ones you know from distributed databases:

• Optimistic locking: read with a version token; write only succeeds if the version hasn't changed. The agent retries on version conflict.
• Pessimistic locking: the agent acquires an exclusive lock before reading, holds it through the write, releases after. Serializes concurrent agents at the cost of parallelism.
• Event sourcing: agents never modify shared state directly; they append events to an immutable log. State is derived from the log. Concurrent appends are safe; reads are consistent.

The right choice depends on conflict frequency and the cost of stale reads. For high-conflict scenarios, use pessimistic locking. For low-conflict, optimistic is fine. For audit-critical systems (anything financial, medical, legal), event sourcing is worth the complexity.

Context Drift — Agent Forgets Its Original Objective

Over a long agent run with many iterations, the original task recedes in the context window. The agent has been reading tool results, writing summaries, making decisions — and somewhere around iteration 25, it's executing sub-tasks that were derived from earlier work but have drifted from the original user intent. The final output doesn't answer what was asked.

Context drift is insidious because each individual decision in the loop looked reasonable. The drift accumulates gradually.

Mitigations:

• Anchor the original task explicitly in every model call. Don't bury it at position 0 of a long message history — re-inject it at the end of the context immediately before the model's response: "Remember, your original task is: [task]".
• Periodic alignment checks. Every N iterations, inject a check: "Does your current plan of action still serve the original goal? Confirm yes or explain any divergence."
• Goal-tagged memory. When writing to long-term memory, tag each stored item with the specific subtask it serves. When retrieving, retrieve only items tagged to the current subtask.

The Silent Hallucination Problem at Scale

In a multi-agent system, one agent's hallucinated output becomes another agent's input. If the first agent invents a fact, the second agent reasons on that fact as if it were true. The hallucination propagates and compounds. By the time the final output is generated, the hallucinated fact has been "confirmed" by multiple agents reasoning consistently from the same false premise.

This is why evals (Chapter 7) become mandatory at the multi-agent level. A single-agent hallucination might be caught by a human reading the output. A multi-agent hallucination that's been reinforced across multiple agents looks like a confident, well-reasoned conclusion and is much harder to identify as wrong.

The structural defense: at every point where one agent's output becomes another agent's input, apply faithfulness evaluation. Does the next agent's reasoning trace back to grounded sources, or is it extending a chain of inference that has no external anchor?

Part 8: Context Engineering at Scale — What Goes In When You Have 10 Agents Running

At the multi-agent level, context engineering (Chapter 2's principle that what you put in the context window and where matters enormously) becomes a coordination problem.

Each agent has its own context window, and those context windows need to be populated with the right information. The orchestrator's context is different from the specialist's. A specialist doing web research needs current retrieval; an orchestrator coordinating five specialists needs results summaries and agent status, not raw web content.

Design patterns for context management at scale:

Agent-specific system prompts. Each agent has a system prompt tuned for its role. The orchestrator's system prompt says "you coordinate specialists and synthesize results." The web research specialist's says "you search the web and extract factual claims." Don't use one system prompt for all agents — it's the equivalent of using one role for all your microservices.

Result summarization before upward propagation. When a specialist finishes and its result propagates up to the orchestrator, the result should be summarized first. The orchestrator doesn't need the raw tool-call trace of the specialist's work — it needs the conclusion and confidence level. Raw traces bloat the orchestrator's context.

Shared state in external memory, not in context. If multiple agents need the same data (a customer record, a configuration file, a previously computed result), store it in an external data store and let each agent retrieve it via tool call. Don't pass large shared data around in message payloads — it duplicates tokens across multiple agents' contexts.

Fan-out with aggregation. Orchestrator → parallel specialists → aggregation. The orchestrator fans out tasks to specialists concurrently (reducing total latency), then aggregates results. The aggregation step needs to handle partial failures (some specialists failed) and conflicting results (two specialists concluded different things). Design the aggregation logic carefully — "average the outputs" doesn't work for conflicting factual claims.

Capstone 2 — Begin: Multi-Agent Workflow

Start building Capstone 2 — a multi-agent workflow for a real task of your choice. Recommended scope: a research + synthesis task (e.g., competitive intelligence gathering, codebase analysis and documentation, or automated content pipeline).

The system must implement:

• An orchestrator + at least two specialists with distinct roles
• Explicit memory architecture with working memory budgeting and long-term memory writes
• At least one critic/refiner loop for quality validation
• Failure handling for each of the failure modes catalogued above
• Instrumentation: log every agent call with task, model used, tokens consumed, latency, and result status

You will harden this system with guardrails (Chapter 8) and production architecture (Chapter 9) in subsequent chapters.

Vocabulary Map

Practice Problems

Problem 1 — Failure Analysis

You've deployed a multi-agent content pipeline: an orchestrator, a web research specialist, a fact-checker specialist, and a writer specialist. Users report that the writer sometimes produces articles containing factual errors that the fact-checker should have caught. Identify three distinct failure modes (from the catalog in this chapter) that could each independently cause this symptom. For each, describe what the failure looks like in the agent trace and what the fix is.

Problem 2 — Design from Scratch

You're building a multi-agent system for software incident response. When an alert fires, the system must: (a) triage the alert, (b) gather metrics and logs from the affected service, (c) search the runbook database for relevant procedures, (d) propose a remediation action, and (e) execute the action only after human approval. Design the complete agent architecture: which agents are involved, what their roles and tool sets are, what the memory architecture looks like, how state flows between agents, and how failure handling works when the runbook search returns no results.

Problem 3 — Synthesis

Explain why these three architectural decisions are all consequences of the same underlying principle: (a) specialists should have narrow, role-specific system prompts rather than one generic system prompt; (b) results should be summarized before propagating upward to the orchestrator; (c) shared data should live in external memory, not in message payloads. What is the one principle they all instantiate, and why is it particularly important at the multi-agent level?

Problem 4 — Go Deeper

Multi-agent systems can theoretically route to the cheapest or fastest model for each task — a cheap model for simple routing decisions, a frontier model for complex synthesis. Design a model routing strategy for the orchestrator + specialist pattern: which decisions should use which tier of model (cheap/fast vs capable/expensive), how do you decide at runtime which tier to use, and what happens when the cheap model's output quality is insufficient and you need to escalate? Be specific about the routing logic and the quality signal that triggers escalation.

Problem 5 — Connect to Previous Chapter

Chapter 5 built a single agent with a filesystem-as-external-memory strategy for long runs. In a multi-agent system, multiple agents might need to access the same filesystem. Identify three specific failure modes that arise when multiple agents concurrently read and write shared filesystem state. For each, propose the appropriate mitigation from distributed systems — be explicit about whether you're applying optimistic locking, pessimistic locking, or event sourcing, and why that approach fits the specific failure mode.

What's Coming Next

You now have a multi-agent system that can coordinate, specialize, remember, and recover from failures. But you have no systematic way to know if it's actually producing good output. Each individual component — the orchestrator, the specialists, the memory layer — introduces its own failure modes, and the combination creates emergent failure modes that don't appear in unit testing. A system you can't measure is a system you can't trust, and a system you can't trust can't ship. Chapter 7 builds the measurement infrastructure — from error analysis to LLM-as-judge to the data flywheel that makes your system better with every production run.