Chapter 2 — Prompting Is Programming

You left Chapter 1 knowing what the model physically is: a frozen function that predicts the next token by scoring every token in its vocabulary based on the entire preceding context. The function never changes. It has no state between calls. It can't learn from you, remember you, or update itself. What it can do — the only thing it can do — is look at the tokens you put in front of it and produce a probability distribution over what comes next. Your only lever is what you put in the context window. That lever is the prompt.

This chapter is about learning to use that lever with precision. Not with intuition and vibes, but with a mechanical understanding of what each prompting technique physically does to the model's output distribution. The difference between an engineer who "writes good prompts" and one who understands prompting is the same as the difference between someone who knows which cables to pull and someone who knows what the cables do. You're here to understand what the cables do.

Part 1: Why the Same Prompt Produces Different Outputs

Let's start with something that should bother you if you haven't fully internalized Chapter 1: you send the exact same prompt twice, and you get two different answers. No code changed. No data changed. Same model. Same prompt. Different output.

If this were a SQL query, you'd file a bug. In a model, this is working as designed.

Here's why. After the model computes the probability distribution over its vocabulary for the next token, it doesn't just take the highest-probability token. It samples from the distribution. The parameter that controls how "peaky" versus "flat" that distribution is called temperature. At temperature 0, the model collapses to always picking the single most probable token — deterministic, but often flat and repetitive. At temperature 1, it samples proportionally from the distribution. At temperature 2, the distribution is flattened so much that low-probability tokens become competitive with high-probability ones, producing creative but often incoherent output.

Most production systems don't run at temperature 0. They run somewhere in the range of 0.2–0.8, depending on the task. That means there is inherent variance in every call.

On top of temperature, there's top-p (nucleus) sampling: the model restricts its sampling to only the smallest set of tokens whose cumulative probability exceeds p. At top-p = 0.9, the model effectively ignores any token that falls outside the top 90% of the probability mass. This cuts off the long tail of bizarre completions, but it still leaves significant room for variance within the retained distribution.

This has a practical consequence that most engineers don't fully internalize early: you cannot test a prompt by running it once. One successful run proves almost nothing. One failure proves almost nothing. The prompt has a distribution of outputs, and you need to understand the shape of that distribution — its mean quality, its variance, its failure modes — before you can trust it in production. You'll build the tooling to measure that distribution in Chapter 7. For now, internalize the implication: prompting is probabilistic programming.

The second source of variance isn't temperature — it's you. Small wording changes produce large output changes because the model is sensitive to the exact token sequence it sees. Changing "summarize this document" to "write a summary of this document" shifts the probability distribution because those token sequences appear in different contexts in the training data, and the model has learned different continuation patterns for each. This is not a bug you can work around; it's the nature of a system whose behavior was learned from human text rather than specified in code. Your job is to write prompts that anchor the model to the narrow region of the distribution where your desired output lives, and to keep them there with a regression harness.

Part 2: System Prompts as Behavior Contracts

Every major model API gives you two input channels: the system prompt and the user message. This distinction matters, but not for magic reasons. It matters because of how training data was structured.

During RLHF (reinforcement learning from human feedback, covered in Chapter 1), the model was trained on a huge corpus of interactions formatted with specific roles: a system message, a user message, and an assistant response. The model learned that "instructions in the system slot" and "requests in the user slot" have different authority patterns. Instructions in the system slot are more like the ambient context that shapes every response — the rules of the interaction. Instructions in the user slot are more like specific requests within those rules.

What does a system prompt physically do? It occupies the beginning of the context window — before the user message — and establishes the probability landscape for every token that follows. When you write You are a helpful assistant who always responds in JSON, you're not setting a configuration flag. You're prepending tokens to the context that shift the probability distribution toward JSON-formatted responses for all subsequent completions. The model has seen many sequences where assistant-role text followed system instructions about JSON, so the JSON continuation becomes high-probability given that context.

This has several direct implications for how you write system prompts.

Specificity beats generality. "Be helpful" does almost nothing — the model already outputs what it considers helpful. "Respond only with SQL queries, no prose, no explanation" is specific enough to actually constrain the distribution. The more narrowly you describe the desired behavior, the tighter the probability mass around your intended output.

Contradictions create undefined behavior. If your system prompt says "always respond in English" and your user message is in French, you've given the model two competing constraints. It will resolve them by sampling from a confused distribution — sometimes English, sometimes French, sometimes a mix. The model doesn't throw an error. It interpolates. If you haven't explicitly specified precedence, you don't know what you'll get.

System prompts have a recency disadvantage at scale. Chapter 1 covered the lost-in-the-middle problem: the model attends more reliably to the beginning and end of the context than to the middle. A system prompt at position 0 in a very long context is at the beginning — which is actually good. But if your system prompt is long and your conversation history grows to fill tens of thousands of tokens, instructions buried in the middle of the system prompt will receive less attention than instructions near the end of the last user message. Keep critical constraints at the beginning or end of your system prompt, and never bury them in the middle of a wall of text.

Think of the system prompt as a behavior contract — not a set of rules the model is "following" consciously, but a prior distribution you're establishing over the model's output space. Write it like a contract: clear, unambiguous, explicit about edge cases, and ordered with the most important constraints first.

Part 3: Chain-of-Thought — Why Forcing Intermediate Tokens Improves Final Token Quality

Here's a result that initially seems counterintuitive: asking the model to "think step by step" before answering actually produces better answers than asking for the answer directly, even when the intermediate reasoning is discarded. Why? This shouldn't make sense if you think of the model as a lookup table. It makes perfect sense when you understand the token prediction mechanism.

The model generates tokens one at a time. Each token is predicted based on all preceding tokens. The final answer token — say, 42 in response to a math problem — is predicted based on all tokens before it, including the intermediate reasoning tokens the model generated. When you force the model to generate intermediate reasoning steps, those reasoning tokens become part of the context that the final answer is predicted from.

Put another way: the model doesn't "hold" an intermediate state between tokens. Its entire "memory" at any point is the full token sequence up to that point. By generating intermediate reasoning explicitly, you're giving the model a richer, problem-specific context from which to predict the final answer. The reasoning tokens serve as a scaffold — they push the probability distribution for the answer tokens toward the region where correct answers live.

This is why chain-of-thought (CoT) works better for complex, multi-step problems than for simple ones. For "what's the capital of France," the answer token Paris is already the highest-probability continuation without any reasoning scaffold — the training data is saturated with that association. For "a train leaves Chicago at 8am at 60mph, another leaves Detroit at 9am at 80mph, when do they meet," the answer token depends on a sequence of intermediate calculations. Without the calculations in the context, the model is essentially guessing from a weak signal. With them, the final answer is much more constrained.

# Without CoT — model must predict the answer from the question alone
Q: A train leaves Chicago at 8am at 60mph...
A: [model must predict a number from weak signal]

# With CoT — each intermediate token narrows the distribution for the next
Q: A train leaves Chicago at 8am...
Think step by step:
A: Chicago train: 60mph. Detroit train: 80mph, 1hr delayed.
   Distance Chicago-Detroit: ~300 miles.
   Let t = hours after 8am when they meet.
   60t + 80(t-1) = 300
   140t = 380, t ≈ 2.71 hours
   Meet time: ~10:43am
   [final answer emerges from a context saturated with supporting calculations]

There are two common CoT invocation styles. Zero-shot CoT adds "think step by step" or "let's think through this" to the prompt — it works because those exact phrases appear in the training data in contexts where detailed reasoning follows, so they serve as a trigger. Few-shot CoT shows the model a complete example of the reasoning chain before asking the question — more reliable, higher cost, more tokens consumed.

A critical caveat: chain-of-thought does not guarantee correctness. It improves the probability of correct reasoning, but the model can generate plausible-looking intermediate steps that are wrong, and then arrive at a confident wrong answer via those wrong steps. This is one reason why evals matter so much — you'll instrument that in Chapter 7.

Part 4: Few-Shot Examples as Distribution Steering

When you include examples in your prompt — "here's an input, here's the output I want; here's another; now do this one" — you are not teaching the model anything. The weights don't update. What you're doing is narrowing the probability distribution.

The model has learned, from training, that when a context contains a pattern of [input] → [output] examples followed by a new [input], the most probable next tokens are a [output] that matches the pattern. Your few-shot examples shift the distribution by populating the context with tokens that activate the pattern-completion machinery the model learned during training.

This has several practical implications.

Example quality dominates example quantity. Two high-quality, representative examples typically outperform ten mediocre ones. A bad example teaches the model the wrong distribution. An inconsistent example set — where the examples don't follow a clear pattern — leaves the model uncertain and produces high variance.

Examples should cover the failure modes you care about. If your model keeps confusing two output formats, include an example that demonstrates the correct behavior at that exact decision point. Examples are targeted interventions in the distribution, not general goodwill.

The order matters. Because of recency bias (the model attends more to tokens near the end of the context), the last example before the test input has disproportionate influence on the output. Put your most canonical example last.

Label format is learned from examples, not from instruction. If you want the model to output {"sentiment": "positive"} rather than Sentiment: positive, showing one or two examples of the JSON format is more reliable than instructing it. Instructions describe; examples demonstrate. Demonstration is a stronger distribution signal.

# Example: Few-shot for sentiment classification
System: Classify the sentiment of customer feedback as positive, negative, or neutral.
         Output only valid JSON.

User: Here are some examples:
Input: "This product changed my life, incredibly fast and easy to use."
Output: {"sentiment": "positive", "confidence": "high"}

Input: "Doesn't work as described. Customer support was unhelpful."
Output: {"sentiment": "negative", "confidence": "high"}

Input: "It's fine, does what it says."
Output: {"sentiment": "neutral", "confidence": "medium"}

Now classify:
Input: "Took a while to set up but performs well once running."

Notice that the format, the keys, and even the confidence field are established entirely by the examples. The system prompt says JSON, but it's the examples that specify what valid JSON looks like in this context.

Part 5: The Prompt Failure Mode Catalog

Your prompts will fail. They fail in specific, recurring ways. The faster you can diagnose which failure mode you're in, the faster you fix it. Here is the complete catalog.

Under-specification

The prompt leaves too many degrees of freedom unspecified. The model fills the gaps with its prior, which may not match your requirements.

Example: "Write marketing copy for our product." What product? What audience? What tone? What length? What channel? The model will make plausible guesses, but they won't match your requirements because you didn't specify them.

Fix: Enumerate the constraints explicitly. Not as a bullet list of rules, but woven into the prompt: "Write a 3-sentence product description for [X], targeting software engineers who have tried competitor tools and been frustrated. Tone: direct, technically credible, no buzzwords."

Ambiguity

The prompt can be interpreted in more than one way, and different interpretations produce different outputs. The model will pick one interpretation stochastically, so you'll see variance that looks random but is actually the model sampling different parse trees.

Example: "Translate the letters" — does this mean translate the correspondence (documents), or translate the characters (OCR)?

Fix: Read your prompt as if you've never seen it before. Identify every noun, instruction, and constraint that could mean two things. Make the intended meaning explicit.

Conflicting Instructions

You've given the model instructions that can't simultaneously be satisfied. This happens most often when a long system prompt and a user message are fighting.

Example: System: "Never discuss pricing." User: "Compare the pricing tiers."

The model doesn't throw an error. It hedges, deflects, or partially complies depending on how it resolves the conflict — which is nondeterministic.

Fix: Explicitly specify how to handle conflicts. "If the user asks about pricing, politely redirect them to the sales page at [URL] and do not provide specific numbers." Now the conflict has a resolution rule, and the resolution is in the context.

Role Confusion

The model loses track of the persona or role you've given it, especially over long conversations or in complex prompts where the role definition is buried.

Example: You've set up a technical documentation assistant. Halfway through a long conversation, the user asks "what do you think about this?" and the model slips out of the documentation-assistant register and responds as a general conversational AI.

Fix: Restate the role constraint at the end of the system prompt (recency effect), and use a phrase that strongly anchors the distribution: "You are always in your role as [X]. If asked to step outside this role, politely decline and redirect."

The Jailbreak Surface

The model's training included exposure to adversarial prompts and was shaped to refuse certain requests. But that shaping is probabilistic, not deterministic — it affects the token distribution, it doesn't hardcode a refusal. A sufficiently unusual framing can push the distribution away from the refusal region.

This is not something you fully control on the production side of the API — the model provider is responsible for baseline safety training. But your system prompt can tighten the jailbreak surface for your application by being explicit: "You are a customer support assistant for [Company]. You only answer questions related to [Company's] products. If asked about anything outside this scope, respond with: 'I can only help with [Company] product questions.'"

The narrower the system prompt, the smaller the jailbreak surface, because there are fewer degrees of freedom for the model to be steered away from. Chapter 8 covers security in depth; this is the prompting layer of the defense.

Hallucination via Over-Confidence

You've asked the model a question it doesn't know the answer to with high probability, but you haven't given it permission to say "I don't know." So it generates the most probable continuation, which looks like a confident answer but is actually confabulation.

Fix: Explicitly include uncertainty as a valid output. "If you are not confident in the answer, say 'I don't have reliable information on this' rather than guessing." This shifts probability mass toward the uncertainty tokens for cases where the model's knowledge is weak, because you've made it a valid response pattern.

Verbose / Under-constrained Format

You asked a focused question and got a 1,200-word essay. You wanted JSON and got a JSON block wrapped in a paragraph of explanation.

Fix: Constrain format explicitly and early. "Respond only with a valid JSON object. No prose, no explanation, no markdown code fence." If you want a specific length, say it: "In exactly three sentences." Constraints that feel over-specified to a human are appropriate for a model.

Part 6: Structured Outputs — Why Constrained Decoding Works and When It Breaks

Getting a model to produce valid JSON is a solved problem at the API layer, but understanding why it's solved — and when it isn't — matters for building reliable systems.

The Naive Approach: Ask for JSON

You put "respond with JSON" in your prompt. The model has seen many JSON examples in training, so it tends toward JSON — but it also sometimes adds explanation text before the block, uses single quotes instead of double quotes, adds trailing commas (invalid JSON), or truncates the object mid-value if the context window runs out.

These failures are probabilistic. Most of the time the model produces valid JSON. Some of the time it doesn't. In a production system, "most of the time" is not good enough.

The Correct Approach: Constrained Decoding

Modern LLM APIs (OpenAI's response_format, Anthropic's tool use, Google's controlled generation) implement constrained decoding at the token level. Before sampling, the model builds a probability distribution over its vocabulary. Constrained decoding masks out — sets to zero probability — any token that would produce an invalid token sequence given the schema.

If your schema says the next field must be a number, all non-numeric tokens get masked. If the JSON must be closed with }, and the model is at the point where it should close, any token that isn't } gets masked. The model is physically prevented from generating an invalid token — not by post-processing, but by manipulating the distribution during sampling.

This means:

1. You always get valid JSON — not "usually valid," always. The constraint is enforced at the token level.
2. You still don't always get useful JSON — the schema says a field must be a string, but the model can produce any string. Structural validity and semantic correctness are separate problems. A field declared as status can be constrained to "open", "closed", or "pending" via an enum in the schema, but free-form string fields remain semantically unconstrained.
3. Schema complexity has a cost — extremely complex schemas with many conditional branches can slow sampling because the valid-token mask computation happens at each step. In practice this is rarely a bottleneck, but it's worth knowing.

# OpenAI structured output example
from openai import OpenAI
from pydantic import BaseModel

class TicketClassification(BaseModel):
    category: Literal["bug", "feature_request", "billing", "other"]
    priority: Literal["low", "medium", "high", "critical"]
    summary: str  # unconstrained — structural validity guaranteed, semantics are yours to eval

client = OpenAI()
response = client.beta.chat.completions.parse(
    model="gpt-4o",
    messages=[
        {"role": "system", "content": "Classify this support ticket."},
        {"role": "user", "content": ticket_text}
    ],
    response_format=TicketClassification,
)
classification = response.choices[0].message.parsed

When Constrained Decoding Breaks

Constrained decoding is robust for structured data, but it has limits:

Context window exhaustion. If the generated content fills the context window before the JSON is closed, you get a valid-but-truncated object. The model ran out of space and the decoding process terminated. Fix: model max-context vs. your expected output size must have enough headroom. Always.

Schema-content tension. If your schema requires a field value that the model has no good information about, the constrained decoding will force it to output something — it can't leave the field empty if the schema requires it. You'll get a structurally valid output with a hallucinated value. Schema enforcement doesn't eliminate hallucination; it just makes the hallucination fit into a box.

Provider parity. Not all model providers implement constrained decoding. Some provide "JSON mode" which is best-effort (strong bias toward JSON, but no hard guarantee). Know what your provider actually guarantees.

Part 7: Prompt Chaining — When One Call's Output Becomes the Next Call's Input

A single model call has a fixed context window, a fixed compute budget, and a fixed scope. Complex tasks that require multiple perspectives, iterative refinement, or sequential decisions can't fit in one call without degrading quality.

Prompt chaining is the technique of decomposing a task into a sequence of calls where each call's output becomes part of the next call's input. It's the simplest form of the agent concept you'll build in Chapter 5, but it's worth understanding on its own terms because it's often the right tool when an agent is overkill.

Task: Write a comprehensive market analysis for a new product

Chain:
Call 1: "List the 5 most important questions a market analysis must answer."
         → produces: [q1, q2, q3, q4, q5]

Call 2: "Research question: [q1]. Here is relevant context: [context]. Answer with evidence."
Call 3: "Research question: [q2]..."
...

Call 7: "Here are answers to 5 key market questions: [answers].
          Write a coherent 3-page market analysis synthesizing all of them."

Why does this work better than a single "write me a comprehensive market analysis" call?

First, each individual call gets a focused context — the model isn't trying to research five questions and synthesize them simultaneously in a single pass. The probability distribution for each output is anchored tightly to a single task.

Second, you can inspect and validate intermediate outputs. If Call 2 produces nonsense, you catch it before Call 7 synthesizes it into a polished document. You can add validation steps between calls, or fork the chain for different variants.

Third, you can cache intermediate outputs. If the market analysis needs to be regenerated with a different synthesis style, you can rerun only Call 7 using the cached research answers.

The cost is latency — sequential calls add up, and parallelizing requires dependency analysis (which calls are independent and can run concurrently). You'll revisit this tradeoff in Chapter 5 when building agents that manage this orchestration automatically.

Prompt chaining also naturally maps to the extract-transform-load pattern you likely already know. The model is doing ETL on text: extract structure, transform it, load it into the next stage. Your distributed systems instincts apply here — design the interfaces between stages carefully, validate at each boundary, and don't assume the upstream stage always produces clean output.

Part 8: The Regression Harness — Why It's Not Optional

Here is the scenario that will happen to you. You spend two hours crafting a prompt for a customer-facing feature. It works on your test cases. You ship it. A week later, a power user discovers a case where it produces wrong output. You tweak the prompt to fix that case. Two days later, a different edge case that was previously working is now broken. The prompt change that fixed one thing silently degraded another.

This is the classic regression problem — and in AI engineering it's worse than in traditional software because you can't read the prompt diff and reason about all its downstream effects. The prompt change affects a probability distribution, not a logical condition. There's no substitute for measuring it.

A regression harness for a prompt is simple in concept:

1. A dataset of (input, expected output) or (input, evaluation function) pairs
2. A runner that sends each input to the current prompt and collects outputs
3. A scorer that measures how well each output matches the expected output
4. A summary of pass rate, failure modes, and delta from the baseline

# Minimal prompt regression harness
def run_eval(prompt_version: str, cases: list[dict]) -> EvalResult:
    results = []
    for case in cases:
        output = call_model(prompt_version, case["input"])
        score = case["eval_fn"](output, case["expected"])
        results.append({"input": case["input"], "output": output, "score": score})
    
    pass_rate = sum(r["score"] for r in results) / len(results)
    return EvalResult(pass_rate=pass_rate, results=results)

# Compare before and after a prompt change
baseline = run_eval(old_prompt, eval_cases)
candidate = run_eval(new_prompt, eval_cases)

print(f"Baseline: {baseline.pass_rate:.1%}, Candidate: {candidate.pass_rate:.1%}")
for case in candidate.results:
    if not case["score"]:
        print(f"REGRESSION: {case['input']}")

The eval_fn varies by task. For structured outputs: exact match on fields or schema validation. For classification: label match. For free-form text: harder — you'll need an LLM-as-judge (Chapter 7) or a human-evaluated subset.

Building this harness takes a few hours. Shipping without it costs much more in debugging time, customer trust, and sleepless nights. The harness has two other benefits beyond catching regressions:

It forces you to define what "correct" means. Many prompting efforts stall because the author hasn't precisely specified what the desired output looks like. Writing eval cases forces that specification.

It builds your test corpus over time. Every failure in production is an eval case for tomorrow. Capture them, add them to the dataset, and your harness gets stronger with every bug you find. This is the beginning of the data flywheel concept you'll build out in Chapter 7.

Part 9: Context Engineering — What Goes In, In What Order

The context window is the model's entire working memory. Unlike a human who can prioritize and selectively attend, the model processes all tokens but attends unevenly based on position. You are responsible for deciding what goes in the context window and where.

The rule is simple to state and hard to apply: the model attends most reliably to the beginning and end of the context, and least reliably to the middle. This is the "lost-in-the-middle" phenomenon from the Liu et al. (2023) paper, and it has direct engineering consequences.

Position Matters

Critical instructions belong at the beginning (system prompt, first-line constraints) or end (immediately before the model begins generating). Supporting context — background information, examples, reference material — goes in the middle.

If you have a large document the model must analyze, your question or task should appear after the document, not before it. The question at the end of a long context is in the high-attention zone. A question buried at position 200 of 4,000 tokens will receive less attention than the same question at position 3,990.

# Less reliable:
"Answer this question: [question]

Background:
[3,000 tokens of reference material]"

# More reliable:
"Background:
[3,000 tokens of reference material]

Answer this question: [question]"

What to Include

Every token in the context window consumes cost (you're billed per token) and competes for the model's attention (denser context means each token gets relatively less weight). Include only what the model needs to complete the task.

Useful inclusions:

• System instructions and constraints (beginning)
• Relevant examples (middle, with most important example last)
• Specific context the model lacks (document sections, database results, user history)
• The task and output format (end)

Costly inclusions to audit:

• Entire documents when only a section is relevant (use retrieval — Chapter 4)
• Long conversation histories when only the last few turns matter
• Redundant examples after the model has clearly learned the pattern
• Metadata that doesn't affect the output

Recency Bias in Conversation

In a multi-turn conversation, the model tends to anchor on the most recent messages more than earlier ones. This is useful — it means the model responds to the current question rather than one asked four turns ago — but it also means important context established early in the conversation degrades over time. If your system prompt establishes a constraint (format, persona, restriction), and the conversation grows long, that constraint may erode.

Two mitigations: First, keep important constraints at the end of the system prompt, not the beginning (the system prompt is prepended to every context, so the end of the system prompt is adjacent to the beginning of the conversation history — proximity to the conversation helps). Second, for long-running applications, periodically inject a "context refresh" into the conversation — a brief restatement of the most critical constraints.

The Context Budget Mental Model

Think of your context window as a finite budget — say 128,000 tokens for a modern frontier model. Allocate it deliberately:

Total budget: 128,000 tokens

System prompt:           ~500  tokens  (instructions, constraints, persona)
Task context (RAG):    ~5,000  tokens  (retrieved docs — Chapter 4)
Conversation history: ~10,000  tokens  (recent turns, not full history)
Current request:         ~500  tokens
Output headroom:       ~4,000  tokens  (expected response length)
                      -------
Total allocated:      ~20,000  tokens  (leaving 108k of unused headroom)

For most applications, you'll use a fraction of the available context. The discipline is not running out when you need the extra space — which happens when users paste large documents, when conversation histories grow, or when retrieval returns larger-than-expected chunks. Always account for the worst case, not the average case.

Vocabulary Map

Practice Problems

Problem 1 — Failure Analysis

You deploy a prompt that classifies customer support tickets into five categories. In production, you observe that 8% of tickets are misclassified, and the misclassifications are concentrated on tickets that mention two categories at once (e.g., a billing complaint that's also a bug report). Walk through the prompt failure mode catalog: which failure mode is this, what is the physical explanation for why the model gets confused, and what are two distinct prompt changes you could test?

Problem 2 — Design from Scratch

You're building a code review assistant that reads a PR diff and outputs a JSON array of review comments, each with fields: file, line, severity (one of "info", "warning", "error"), and comment. The assistant must only flag real issues, not nitpick style. Design the complete prompting strategy: system prompt structure, whether to use few-shot examples (and if so, how many and what they should cover), output format enforcement approach, and how you'd handle a diff that's 50,000 tokens (which exceeds a single call's practical capacity for leaving output headroom).

Problem 3 — Synthesis

Explain why the following three facts are all expressions of the same underlying physical reality: (a) chain-of-thought improves accuracy on multi-step problems, (b) placing critical instructions at the end of a long system prompt is more reliable than placing them in the middle, (c) the last few-shot example has disproportionate influence on the output format. Use the token prediction mechanism from Chapter 1 to derive all three from first principles.

Problem 4 — Go Deeper

Suppose you need to build a prompt that is robust across model upgrades — i.e., when the model provider releases a new version, your prompt should continue working without modification. What properties of a prompt make it fragile to model upgrades? What properties make it robust? Given what you know about how models are trained and how prompts work, derive a set of design principles for model-upgrade-resilient prompts. (This requires reasoning beyond what was explicitly covered — derive it.)

Problem 5 — Connect to Chapter 1

Chapter 1 established four properties of a language model: Frozen, Stateless, Stochastic, and Predictive. For each property, identify the specific prompting technique in this chapter that exists precisely because of that property. The answer should be four one-to-one mappings with a physical explanation for each link.

What's Coming Next

You can now steer the model's output with precision — narrow its distribution, enforce its format, chain its calls, and catch its regressions. But everything in this chapter assumes the model's response is the end of the road. The model produces text. Your code reads it. That's where the interaction stops. The model is still sealed: it can't query your database, call your API, read a file, send an email, or do anything except generate tokens. The next chapter opens that seal. Tool use is how you give a frozen text predictor the ability to reach into the real world — and the architecture of how that happens on the wire is both simpler and stranger than you probably expect.