Chapter 9 — From Demo to Production: Cost, Latency, and Scale

Your system works. It retrieves correctly, coordinates agents reliably, defends against injection, and passes its eval suite. You demo it to stakeholders. Everyone is impressed. You ship it. Within 48 hours you discover: it costs $4.20 per user session (unviable at scale), p95 latency is 14 seconds (users have left by 8), and at 50 concurrent users the model API starts rate-limiting you. The demo was a lie not because anything broke, but because a demo at 1 request per minute doesn't surface any of the problems that 500 requests per minute makes unavoidable. This chapter is the gap between the demo and the production system. It's engineering, not research — cost discipline, latency budgeting, and architectural patterns that scale.

Part 1: Prompt Caching — The First Cost Lever

Most LLM API calls send the same content over and over again. Your system prompt is the same for every user. The first few thousand tokens of every RAG call contain the same preamble, the same formatting instructions, the same role definition. In a customer support agent, the agent's personality, its tool schemas, its behavior rules — all identical across every call, every user, every session.

You're paying to process those tokens on every call. Prompt caching lets you pay once.

How It Works Physically

Model inference is compute in two phases: prefill (processing the input tokens to compute the key-value pairs for each layer of the transformer — the KV cache) and decode (autoregressive generation, one token at a time). Prefill dominates for long inputs; decode dominates for long outputs.

Prompt caching works by saving the KV cache for a prefix of your context. When a new request comes in with the same prefix, the model reloads the cached KV cache instead of recomputing it. The cached tokens cost a fraction of normal input tokens (typically 10-20% of the full input price, depending on the provider), and the time to process them is dramatically reduced.

Without caching:
Request 1: [system prompt (2000 tokens) + user query (50 tokens)] → full prefill cost
Request 2: [system prompt (2000 tokens) + user query (50 tokens)] → full prefill cost
Request N: same...

With caching:
Request 1: [system prompt (2000 tokens) + user query (50 tokens)] → full prefill cost
              [system prompt KV cache STORED]
Request 2: [system prompt cache hit + user query (50 tokens)] → 10% cost for cached tokens
Request N: same savings every time

The economics are significant. If your system prompt is 2,000 tokens, at $0.003/1K input tokens, that's $0.006 per call just for the system prompt. At 100,000 calls/day, that's $600/day for just the system prompt. With prompt caching at 80% reduction: $120/day. A $480/day savings.

Structuring Prompts for Maximum Cache Hits

For caching to work, the cached prefix must be byte-for-byte identical across requests. Any difference — even a single character — breaks the cache hit.

The principle: put static content first, dynamic content last.

# Bad cache structure — cache breaks on every request
[date: {today}]      ← changes every day
[user name: {name}]  ← different per user
[system instructions: ...]  ← same for all
[user query: {query}]

# Good cache structure — maximum cache hits
[system instructions: ...]  ← static, always first → cached
[tool schemas: ...]          ← static → cached
[user context: ...]          ← varies per user, injected after cached prefix
[user query: {query}]        ← fully dynamic, at the end

For RAG contexts, put the retrieved documents after the system prompt (which is cached) but before the question (which varies). This way the system prompt is always a cache hit, and the retrieval context is processed normally.

For multi-turn conversations, the cached portion is the system prompt plus the earlier conversation turns. Each new turn extends the context — the previous turns are cached, only the new turn is prefilled fresh.

Provider-specific notes: Anthropic's Claude supports cache control headers (you mark which sections to cache explicitly). OpenAI caches automatically for matching prefixes. Check your provider's documentation for how to signal cache-worthy content and how to measure cache hit rates.

Part 2: Streaming — Why It's a UX Architecture Decision

Without streaming, a user sends a message and sees nothing until the model has finished generating the complete response. For a 500-token response, that might be 5-8 seconds of blank screen. Users interpret blank screen as "broken" and abandon your product.

Streaming delivers tokens to the client as they're generated, one by one. The user sees words appearing progressively — exactly like watching someone type. The experience is dramatically better even though the total time to the complete response hasn't changed.

This is not just UX polish. For agent applications where the model narrates its reasoning ("I'll start by searching for X, then compare with Y..."), streaming makes the agent's behavior transparent and builds user trust. Without streaming, the agent works silently for 15 seconds and then produces a result. With streaming, users watch the agent think — which is both more engaging and more debuggable.

How Streaming Works on the Wire

The model API sends Server-Sent Events (SSE) — a stream of small JSON payloads, each containing one or a few tokens. Your server receives these events and forwards them to the client as they arrive.

# Backend: streaming a model response
async def stream_agent_response(request, user_message: str):
    messages = build_messages(user_message)
    
    async def generate():
        async with anthropic_client.messages.stream(
            model="claude-opus-4",
            max_tokens=2048,
            messages=messages,
        ) as stream:
            async for text in stream.text_stream:
                yield f"data: {json.dumps({'token': text})}\n\n"
        
        yield "data: [DONE]\n\n"
    
    return StreamingResponse(generate(), media_type="text/event-stream")

// Frontend: consuming the stream
const eventSource = new EventSource('/api/chat/stream');
let fullText = '';

eventSource.onmessage = (event) => {
    if (event.data === '[DONE]') {
        eventSource.close();
        return;
    }
    const { token } = JSON.parse(event.data);
    fullText += token;
    updateUI(fullText);  // Render progressively
};

What Streaming Changes About Your Architecture

Streaming complicates several things you might have assumed were simple:

You can't validate the output until it's complete. Output validation (Chapter 8) normally happens after the complete response is available. With streaming, you're delivering tokens to users before validation. Solutions: buffer and validate in parallel (deliver tokens while running validation on a rolling window), or accept that validation is post-hoc for streaming and implement compensation (show an error state after the response if validation fails).

Tool calls interrupt the stream. When a streaming model call encounters a tool call, the stream pauses while your code executes the tool. The user sees the model's "thinking" tokens stream, then a pause, then more tokens. Communicating this pause — "looking up your order..." — is important UX. Plan for it.

Agent loops need streaming state management. In a multi-step agent, you want to stream each step's reasoning, not just the final answer. This means managing multiple stream sources (one per iteration) and presenting them coherently to the user.

Part 3: Multi-Model Routing and Fallback

Not all tasks require the same model. A query classification step doesn't need a frontier model; a nuanced synthesis of conflicting evidence does. Using a $0.015/1K-token frontier model for everything is leaving money on the table.

Multi-model routing sends each sub-task to the cheapest model that can handle it reliably.

Task Complexity Classification

The first step is classifying task complexity. This can be done with a fast, cheap router model — a small, fast model that reads the task and outputs a complexity tier.

TASK_TIERS = {
    "simple": {
        "model": "claude-haiku-4-5",  # Fastest, cheapest
        "use_for": ["intent classification", "short reformatting", "lookup answer retrieval"]
    },
    "medium": {
        "model": "claude-sonnet-4-6",  # Balanced
        "use_for": ["summarization", "moderate reasoning", "structured extraction"]
    },
    "complex": {
        "model": "claude-opus-4-8",  # Most capable, most expensive
        "use_for": ["complex reasoning", "nuanced judgment", "multi-step planning"]
    }
}

def route_task(task: str) -> str:
    # Use the cheapest model to classify task complexity
    complexity = router_model.classify(
        f"Classify this task's complexity as 'simple', 'medium', or 'complex'.\n"
        f"Task: {task}\n"
        f"Simple: straightforward lookup, classification, or simple reformatting.\n"
        f"Medium: requires synthesis, moderate reasoning, or nuanced extraction.\n"
        f"Complex: requires deep reasoning, novel analysis, or complex judgment.",
        valid_outputs=["simple", "medium", "complex"]
    )
    return TASK_TIERS[complexity]["model"]

The routing model call itself should be fast and cheap. Use a small model for routing — the irony of using a frontier model to decide which model to use is real and should be avoided.

Cost and Quality Tradeoffs by Layer

In a RAG + agent system, different layers have different quality requirements:

Component             | Quality requirement | Recommended tier
---------------------|---------------------|------------------
Query intent parsing  | Low — simple enum   | Cheap (Haiku)
Query rewriting      | Medium              | Mid-tier (Sonnet)
Chunk relevance eval | Medium              | Mid-tier
Final synthesis      | High                | Frontier (Opus)
Tool call parsing    | Low — structured    | Cheap + constrained
Agent planning       | High                | Frontier
Critic evaluation    | High                | Frontier (same or better than generator)

Running this analysis and setting per-component model tiers can reduce costs by 40-60% with minimal quality degradation, because most components don't actually need the frontier model.

Fallback and Circuit Breaking

Your architecture should assume individual model APIs will have downtime, rate limit you, or degrade in quality. A fallback chain provides resilience:

MODEL_FALLBACK_CHAIN = [
    "claude-opus-4-8",      # Primary: best quality
    "claude-sonnet-4-6",    # Fallback 1: slight quality degradation, still good
    "gpt-4o",               # Fallback 2: different provider entirely
    "claude-haiku-4-5",     # Fallback 3: basic capability, minimal quality
]

async def call_with_fallback(messages, tools=None, target_tier="complex"):
    models = get_models_for_tier(target_tier)
    
    for model in models:
        try:
            return await call_model(model, messages, tools, timeout=30)
        except RateLimitError:
            log.warning(f"Rate limited on {model}, trying next")
            continue
        except ModelUnavailableError:
            log.warning(f"Model {model} unavailable, falling back")
            continue
        except TimeoutError:
            log.warning(f"Timeout on {model}, falling back")
            continue
    
    raise AllModelsFailedError("All fallback models exhausted")

Circuit breaking adds another layer: if a model has failed N times in the last M seconds, stop trying it and go directly to the next fallback. This prevents cascading timeouts from a degraded-but-not-down model.

class CircuitBreaker:
    def __init__(self, failure_threshold=5, window_seconds=60, recovery_seconds=120):
        self.failures = {}  # model -> list of failure timestamps
        self.open_until = {}  # model -> timestamp when circuit can be retried
    
    def is_open(self, model: str) -> bool:
        if model in self.open_until and time.time() < self.open_until[model]:
            return True  # Circuit is open (model is failing)
        return False
    
    def record_failure(self, model: str):
        self.failures.setdefault(model, []).append(time.time())
        recent_failures = [t for t in self.failures[model] 
                          if t > time.time() - self.window_seconds]
        if len(recent_failures) >= self.failure_threshold:
            self.open_until[model] = time.time() + self.recovery_seconds
            log.warning(f"Circuit opened for {model}")

Part 4: Cost Attribution — Tracking Every Dollar

At demo scale, you don't need cost attribution. At production scale, you absolutely do. Without it, you can't answer: Is this feature profitable? Which user persona consumes disproportionate costs? Is our agent running too many iterations for simple tasks? Where did the $8,000 model bill this month come from?

The Attribution Model

Every model call should be tagged with metadata that tells you why it was made:

@dataclass
class ModelCallMetadata:
    # Identity
    trace_id: str          # Unique per call chain
    session_id: str        # User session
    user_id: str           # User identifier
    
    # Attribution
    feature: str           # "document_qa", "support_chat", "code_review"
    component: str         # "retrieval_reranker", "final_synthesis", "agent_planner"
    agent_run_id: str | None  # If inside an agent run
    
    # Costs
    model: str
    input_tokens: int
    output_tokens: int
    cached_input_tokens: int  # Tokens served from cache
    cost_usd: float        # Computed: (input * price_in) + (output * price_out)
    
    # Performance
    latency_ms: int
    cache_hit: bool

Log this for every call, then aggregate:

-- Cost by feature per day
SELECT feature, DATE(created_at) as date, 
       SUM(cost_usd) as total_cost,
       COUNT(*) as calls,
       AVG(cost_usd) as avg_cost_per_call
FROM model_calls
GROUP BY feature, DATE(created_at)
ORDER BY date DESC, total_cost DESC;

-- Cost per user (identify high-cost users)
SELECT user_id, SUM(cost_usd) as total_cost, COUNT(*) as calls
FROM model_calls
WHERE created_at > NOW() - INTERVAL '7 days'
GROUP BY user_id
ORDER BY total_cost DESC
LIMIT 100;

-- Agent efficiency (detect runaway loops)
SELECT agent_run_id, COUNT(*) as steps, SUM(cost_usd) as total_cost
FROM model_calls
WHERE agent_run_id IS NOT NULL
GROUP BY agent_run_id
HAVING COUNT(*) > 10  -- Flag agents taking more than 10 steps
ORDER BY total_cost DESC;

Setting Cost Alerts

Set automated alerts on:

• Daily spend exceeding a threshold (catch runaway agents or traffic spikes)
• Per-user spend exceeding a threshold in a session (catch abuse or misconfigured agents)
• Average cost per call rising significantly (catch prompt changes that bloat token usage)
• Cache hit rate dropping (catch changes that broke prompt caching)

Part 5: Canaries and Shadow Deployments

You have a new model version, or a revised system prompt, or a retrieval config change. You want to ship it. But you don't want to discover that it degrades quality for 20% of edge cases after it's fully deployed.

Canary deployment: route a small fraction of traffic (1-5%) to the new configuration. Monitor quality metrics for both the canary and the baseline. If the canary shows degraded quality or elevated errors, roll back. If it's equal or better, gradually increase the canary percentage.

Shadow deployment: the new configuration runs in parallel with the production configuration on all traffic, but its responses are discarded — users always see the baseline response. This lets you measure the new configuration's quality and cost on real production traffic before any user sees its output.

def route_request(request: Request, new_config_pct: float = 0.05) -> ModelConfig:
    # Canary routing based on user_id hash (consistent per user)
    user_hash = hash(request.user_id) % 100
    
    if user_hash < new_config_pct * 100:
        return NEW_CONFIG  # Canary user — sees new behavior
    else:
        return BASELINE_CONFIG  # Majority — sees current behavior

# Shadow deployment: run both, discard new response
async def shadow_run(request: Request) -> Response:
    baseline_task = asyncio.create_task(call_with_config(BASELINE_CONFIG, request))
    shadow_task = asyncio.create_task(call_with_config(NEW_CONFIG, request))
    
    baseline_response = await baseline_task
    shadow_response = await asyncio.shield(shadow_task)  # Run in background
    
    # Log shadow response for comparison, return baseline
    asyncio.create_task(log_shadow_comparison(baseline_response, shadow_response))
    return baseline_response

For prompt changes, canaries are essential. A prompt change that looks good on your eval set might degrade for a specific user behavior pattern that isn't in your eval set. Canaries catch this before full rollout.

Part 6: Rate Limits, Retries, Timeouts, Idempotency

You know these patterns from distributed systems. Applied to LLM APIs, they work the same way with a few AI-specific nuances.

Rate Limits

LLM APIs rate limit on tokens per minute (TPM) and requests per minute (RPM). Token-based rate limits are the constraining factor for most applications because long-context calls use tokens faster than request count increases.

Pre-emptive rate limiting at your application layer: track your current TPM usage and queue or throttle incoming requests before hitting the API limit. This prevents the thundering herd — a burst of requests that instantly saturates your rate limit and causes all requests to fail simultaneously.

class TokenBucketRateLimiter:
    def __init__(self, max_tpm: int):
        self.max_tpm = max_tpm
        self.tokens = max_tpm
        self.last_refill = time.time()
        self.lock = asyncio.Lock()
    
    async def acquire(self, estimated_tokens: int) -> None:
        async with self.lock:
            # Refill tokens based on time elapsed
            now = time.time()
            elapsed = now - self.last_refill
            refill = elapsed * (self.max_tpm / 60)
            self.tokens = min(self.max_tpm, self.tokens + refill)
            self.last_refill = now
            
            # Wait if insufficient tokens
            while self.tokens < estimated_tokens:
                wait_time = (estimated_tokens - self.tokens) / (self.max_tpm / 60)
                await asyncio.sleep(wait_time)
                self.tokens = min(self.max_tpm, self.tokens + wait_time * (self.max_tpm / 60))
            
            self.tokens -= estimated_tokens

Retries

Retry with exponential backoff and jitter on rate limit errors (429) and transient server errors (500, 503). Do not retry on: context too long (400 with that specific error), invalid requests (400), authentication failures (401), or permanent failures.

async def call_model_with_retry(messages, max_retries=5):
    for attempt in range(max_retries):
        try:
            return await model_api.complete(messages)
        except RateLimitError:
            if attempt == max_retries - 1:
                raise
            wait = (2 ** attempt) + random.uniform(0, 1)
            await asyncio.sleep(wait)
        except TransientServerError:
            if attempt == max_retries - 1:
                raise
            wait = (2 ** attempt) + random.uniform(0, 1)
            await asyncio.sleep(wait)
        except ContextLengthError:
            raise  # Don't retry — you need to fix the context

Timeouts

Every model API call must have a timeout. Without a timeout, a hung API call will block indefinitely and consume a connection in your connection pool until it's eventually garbage collected. Under load, this causes connection pool exhaustion and cascading failures.

For streaming calls, the timeout is more nuanced: you need a timeout on the first token (time-to-first-token) and a timeout on the overall stream duration. A stream that starts delivering tokens but then hangs mid-generation needs a per-token timeout or a maximum total duration timeout.

Idempotency

This is important for agent loops that checkpoint and resume (Chapter 5). If a tool call succeeded but the checkpoint wasn't written before the crash, you'll retry the tool call on resume. If the tool call is not idempotent (write, send, charge), you'll execute it twice.

Two approaches: (a) use an idempotency key for each tool call — a unique identifier generated before the call that the tool server uses to deduplicate, (b) track completed tool calls in your checkpoint and skip already-executed ones on resume.

The idempotency key pattern is cleaner and more general:

def execute_tool_idempotently(tool_call: ToolCall, agent_run_id: str, step: int) -> ToolResult:
    idempotency_key = f"{agent_run_id}:step_{step}:{tool_call.name}"
    
    # Check if this call already succeeded
    cached_result = idempotency_store.get(idempotency_key)
    if cached_result:
        return cached_result
    
    result = execute_tool(tool_call)
    idempotency_store.set(idempotency_key, result, ttl=86400)  # Store for 24h
    return result

Part 7: Async and Queue-Based Architecture

A synchronous architecture for AI calls — user request → model API call → response — has a fundamental problem: model API calls are slow (2-15 seconds) and you're holding an HTTP connection open for the entire duration. Under load, this exhausts your web server's connection pool and causes timeouts.

Your distributed systems instincts are right here: don't block a web request on slow I/O. Use queues.

Sync architecture (breaks under load):
Client → Web Server → [blocked waiting 10s for model] → Response

Async architecture with queue:
Client → Web Server → Queue ──→ Worker → Model API → Result store
         │                                                    │
         └──→ [202 Accepted, job_id=abc]    Poll: GET /result/abc

The async architecture decouples request intake from model inference. Your web server accepts requests instantly (fast, can handle high concurrency) and returns a job ID. Workers pull from the queue and process requests with model API calls. Clients poll for results or receive results via WebSocket/SSE push.

For long-running agent tasks, this is mandatory. A 5-minute agent run cannot hold an HTTP connection open — it will time out and the client will assume failure. With a queue-backed architecture, the client gets a job ID immediately, the agent runs in the background, and the client polls (or is notified) when it's done.

# FastAPI async architecture example
@app.post("/agent/run")
async def start_agent_run(request: AgentRequest) -> AgentRunResponse:
    # Validate request
    task = request.task
    user = get_current_user(request)
    
    # Enqueue job (returns immediately)
    job_id = task_queue.enqueue(
        run_agent,
        task=task,
        user_id=user.id,
        job_id=generate_id()
    )
    
    return AgentRunResponse(job_id=job_id, status="queued")

@app.get("/agent/run/{job_id}")
async def get_agent_result(job_id: str) -> AgentRunStatus:
    job = task_queue.get_job(job_id)
    if job.is_complete:
        return AgentRunStatus(status="complete", result=job.result)
    elif job.is_failed:
        return AgentRunStatus(status="failed", error=job.exc_info)
    else:
        return AgentRunStatus(status="running", progress=job.meta.get("progress"))

For real-time streaming cases (where the user wants to watch the agent think), WebSockets or SSE can push incremental results as the agent runs, while the job still runs asynchronously in a worker.

Part 8: Production Observability

You have the per-call traces from Part 4. Production observability is those traces aggregated, visualized, and alertable.

The Latency Budget

For every user-facing AI feature, define where the time goes:

Latency budget for a RAG-powered support chatbot response:

Component               | Budget  | Measured | Status
------------------------|---------|----------|--------
Input validation        | 10ms    | 8ms      | OK
Query rewriting         | 500ms   | 450ms    | OK
Vector search           | 100ms   | 85ms     | OK  
BM25 search             | 50ms    | 60ms     | WARN
Re-ranking              | 200ms   | 180ms    | OK
Context construction    | 20ms    | 15ms     | OK
LLM call (first token)  | 1000ms  | 1200ms   | WARN
LLM streaming           | 3000ms  | 2800ms   | OK
Output validation       | 100ms   | 90ms     | OK
Total P50               | ~5000ms | 4888ms   | OK
Total P95               | ~8000ms | 11200ms  | FAIL

The P95 overage shows something is blowing up occasionally — probably the LLM call under load. That's the next thing to investigate (caching? model fallback? context too long on edge cases?).

Traces, Not Just Metrics

Aggregate metrics tell you that something is wrong. Distributed traces tell you what is wrong. Every user request should have a trace_id that is propagated through all downstream calls: vector DB query, LLM call, tool executions. When a request is slow or fails, you can pull the trace and see exactly which component took how long and where the failure occurred.

OpenTelemetry is the standard for structured trace export. Most observability platforms (Datadog, Honeycomb, Grafana Tempo) ingest OpenTelemetry traces. Add tracing to every async boundary in your application.

from opentelemetry import trace
tracer = trace.get_tracer("ai-service")

async def rag_pipeline(query: str, user_id: str) -> str:
    with tracer.start_as_current_span("rag_pipeline") as span:
        span.set_attribute("user.id", user_id)
        span.set_attribute("query.length", len(query))
        
        with tracer.start_as_current_span("query_rewriting"):
            rewritten = await rewrite_query(query)
        
        with tracer.start_as_current_span("retrieval") as retrieval_span:
            chunks = await hybrid_search(rewritten)
            retrieval_span.set_attribute("chunks.retrieved", len(chunks))
        
        with tracer.start_as_current_span("llm_call") as llm_span:
            response = await call_model(rewritten, chunks)
            llm_span.set_attribute("tokens.input", response.input_tokens)
            llm_span.set_attribute("tokens.output", response.output_tokens)
            llm_span.set_attribute("latency_ms", response.latency_ms)
        
        return response.text

AI-Specific Dashboards

Beyond standard metrics (request rate, error rate, latency), AI systems need:

• Quality metrics over time: faithfulness score trend, task completion rate trend. A quality regression shows up here before users complain.
• Cost per feature/day: did a recent change accidentally make a feature 30% more expensive?
• Cache hit rate: is caching working? A drop in cache hit rate signals prompt structure changed.
• Model usage distribution: which models are serving which percentage of requests? Are fallbacks triggering unexpectedly?
• Token distribution: histogram of input and output token lengths. Outliers (very long inputs) are often bugs or abuse.

Completing Capstone 2 — Production-Hardening the Multi-Agent Workflow

Take the multi-agent system from Chapter 6 and apply the production engineering from this chapter:

1. Add prompt caching. Identify the static portion of each agent's system prompt and structure it for cache hits. Measure before and after.

2. Implement streaming. At least the final synthesis agent should stream its response.

3. Add multi-model routing. Profile each component's task complexity and assign the appropriate model tier. Measure the cost reduction.

4. Implement cost attribution. Tag every model call with feature, agent_run_id, and component. Build a simple dashboard showing cost by component.

5. Add retry, timeout, and fallback. Wrap every model call in the retry + circuit breaker pattern. Configure a fallback chain.

6. Move to async architecture. If your system currently blocks on long agent runs, convert to queue-backed async with polling.

7. Add observability. Trace every request end-to-end. Set latency budget targets. Set a cost alert.

When this is done, you have a production-grade system, not a demo.

Vocabulary Map

Practice Problems

Problem 1 — Failure Analysis

Your AI writing assistant works perfectly in development but has terrible latency in production (p95 = 18 seconds). You look at traces and find: retrieval = 200ms, reranking = 150ms, context construction = 50ms, LLM first token = 14 seconds at p95. But p50 LLM latency is only 2 seconds. What causes the bimodal distribution between p50 and p95 for LLM latency, what would you look at in the traces to identify the root cause, and what are three distinct architectural changes that could each independently bring p95 below 6 seconds?

Problem 2 — Design from Scratch

You're building a code generation assistant that produces complete, production-ready functions from natural language descriptions. The expected workflow: user describes the function → agent thinks about edge cases → agent retrieves relevant code patterns from a codebase vector DB → agent generates the function → agent writes tests → agent runs tests and iterates if they fail. This can take 30-90 seconds. Design the complete production architecture: async vs sync, streaming approach, cost attribution tags, latency budget per component, rate limiting strategy, and failure handling when the test runner returns an error.

Problem 3 — Synthesis

Explain why these three architectural decisions are all expressions of the same underlying principle: (a) static content in prompts should come before dynamic content for caching to work; (b) async queue-based architecture decouples request rate from inference rate; (c) model routing sends simple tasks to cheap models and complex tasks to expensive ones. What is the common engineering principle, and why does it apply specifically to LLM-powered systems in a way that doesn't apply to, say, a REST API backed by a SQL database?

Problem 4 — Go Deeper

You have a multi-agent system where different agents run at different model tiers (cheap for routing, expensive for synthesis). Implement a cost control mechanism that: (a) sets a per-user daily budget, (b) automatically downgrades model tiers when a user approaches their budget, (c) gracefully communicates budget limitations to the user, and (d) handles the edge case where a user's request is complex enough that even the cheap model tier is too expensive relative to their remaining budget. Design both the data model and the routing logic.

Problem 5 — Connect to Previous Chapter

Chapter 8 requires that every tool call be validated before execution. In a streaming architecture where tokens arrive asynchronously and tool calls are emitted mid-stream, how does output validation change? Specifically: when in the streaming flow do you validate a tool call's arguments, how do you handle a validation failure mid-stream (the model has already partially responded), and how does this interact with the user experience of watching the model "think"?

What's Coming Next

Your system is now production-grade: it works correctly, defends against attacks, and performs under load. There's one final problem: no one knows it exists. Skills in your head and code in a private repository don't land interviews, don't attract users, and don't demonstrate your capabilities to the world. The difference between an engineer who is top-1% at AI engineering and one who merely has top-1% skills is evidence. Evidence is what you build in Chapter 10 — and it's the chapter that converts everything in this book into career outcomes.