Chapter 5 — The Streaming ChatGPT Clone: Tokens Over the Wire, State in the Database

Replaces sessions: Streaming ChatGPT Clone, Parts 1 and 2

You will build: a full-stack chat app — streaming responses, Markdown and code rendering, conversations persisted to Postgres, history that survives a reload, and a context budget that keeps long chats from getting slow and expensive.

Theory prerequisites: AI Ch 1 — The Frozen Oracle (tokens, next-token prediction, context windows)

A chat app looks like the simplest possible AI product: a text box, a list of messages, one API call per turn. Building one that actually feels good is a different matter. The first version you write will wait ten seconds and then dump a wall of text. It will forget everything on reload. It will render half-written code blocks as garbage. It will get slower and more expensive with every message until a long conversation costs dollars per reply. And when the user closes the tab mid-answer, it will keep generating — and billing — tokens nobody will ever read.

Every one of those problems has a physical cause, and every fix follows directly from the cause. This chapter walks the whole path a single message takes — from the keystroke, through your server, to the model, back over the wire token by token, into the DOM, and into the database — and explains what can go wrong at each hop.

Part 1: Why Streaming Exists — The Physics of Generation

Recall from the Frozen Oracle chapter what the model physically does: it produces one token, appends it to the input, and runs again to produce the next. Generation is a loop, and each iteration of the loop is a full forward pass through the network.

That loop has two phases with very different costs:

• Prefill. The model processes your entire input — system prompt, history, the new message — in one parallel pass. This is fast per token because all input tokens are known up front and can be processed together. It determines the time to first token (TTFT), typically somewhere between a few hundred milliseconds and a couple of seconds.
• Decode. The model generates output tokens one at a time. Each token depends on the previous one, so this cannot be parallelised. Output arrives at a roughly fixed rate — tens to low hundreds of tokens per second depending on model and load.

Now do the arithmetic for a typical answer. A 700-token explanation at 70 tokens per second takes ten seconds to decode. Without streaming, the user stares at a spinner for ten seconds and then sees everything at once. With streaming, the user sees the first words after the TTFT — say 800 milliseconds — and reads along as the rest arrives. The total time is identical. The perceived latency drops by more than 90%.

That is the whole argument for streaming: the tokens exist long before the response is complete, because the model produces them in order. Holding them back until the end is a choice your code makes, not a property of the model. Streaming simply stops making that choice.

There is a second, less obvious benefit. A user reading the first paragraph can tell within two seconds that the model misunderstood the question — and hit Stop. Without streaming, they would have paid for the full ten seconds of generation before finding out. Streaming turns a sunk cost into an early exit, which only works if your server actually stops generating when the user stops reading. That is Part 4.

Part 2: The Shape of a Provider Stream

When you call a model API with streaming enabled, the response is not one JSON body. It is a long-lived HTTP response that the server writes to incrementally, as Server-Sent Events (SSE): a plain-text format where each event is a few field: value lines followed by a blank line.

Anthropic's Messages API emits a small, fixed vocabulary of events:

OpenAI's Responses API has the same structure with different names — text increments arrive as response.output_text.delta events, and the stream ends with response.completed. Gemini streams chunks that each contain a partial candidate. The names differ; the physics is identical: a sequence of small increments, followed by a terminal event carrying why generation stopped and how many tokens it cost.

The SDKs parse SSE for you. With the Anthropic TypeScript SDK, a stream is an async iterable of typed events:

import Anthropic from "@anthropic-ai/sdk";

const client = new Anthropic(); // reads ANTHROPIC_API_KEY from the environment

const stream = client.messages.stream({
  model: "claude-opus-5-5",
  max_tokens: 8000,
  messages: [{ role: "user", content: "Explain backpressure in two paragraphs." }],
});

for await (const event of stream) {
  if (event.type === "content_block_delta" && event.delta.type === "text_delta") {
    process.stdout.write(event.delta.text);
  }
}

const final = await stream.finalMessage(); // the assembled message, with usage
console.log("\nstop_reason:", final.stop_reason, "usage:", final.usage);

Notice the filter: you only forward text_delta. Current Claude models think before answering, and thinking arrives as its own block type. For a chat UI you usually don't stream raw reasoning to the user, and with the default display setting those thinking blocks carry no text anyway. Filtering by delta type is what keeps your UI correct when the model emits block types you didn't plan for.

Notice also stream.finalMessage(). You need the final message for two things the deltas don't give you cleanly: the stop_reason (did it finish, or hit max_tokens?) and the usage numbers you will store for cost tracking.

Part 3: The Server — A Stream Proxy That Owns the Key

The browser cannot call the model API directly. Your API key would ship to every user, and anyone could open DevTools, copy it, and spend your money. So the architecture has three hops:

Your server is a stream proxy: it reads increments from the provider and writes increments to the browser, while keeping the key, the system prompt, and the database on the server side.

Choosing your own wire format

You could forward the provider's SSE bytes untouched. Don't. Define your own small protocol instead, for three reasons:

1. Provider independence. If you switch from Claude to OpenAI or a local Ollama model for some conversations, the browser code doesn't change.
2. Smaller payloads. Provider events carry indices, types and metadata the browser doesn't need.
3. Server-only information stays server-only. Your protocol emits exactly what the UI needs and nothing else.

The simplest protocol that works is newline-delimited JSON (NDJSON): one JSON object per line. Three event types are enough:

{"type":"delta","text":"Backpressure is"}
{"type":"delta","text":" what happens when"}
{"type":"done","messageId":"cm3...","stopReason":"end_turn"}

Why not SSE to the browser too? The browser's built-in EventSource only supports GET requests with no body, and a chat message needs a POST with a body. Once you are reading the stream with fetch anyway, NDJSON is less parsing than SSE and just as robust.

The route handler

Here is the full Next.js route handler. Read it once top to bottom, then we'll go through the decisions in it.

// app/api/chat/route.ts
import Anthropic from "@anthropic-ai/sdk";
import { prisma } from "@/lib/db";
import { loadHistory } from "@/lib/history";

const client = new Anthropic();

const SYSTEM_PROMPT =
  "You are a helpful assistant inside a chat app. Use Markdown. " +
  "Put code in fenced code blocks with a language tag.";

export async function POST(req: Request) {
  const { conversationId, content } = await req.json();

  // 1. Persist the user's message before anything can fail.
  await prisma.message.create({
    data: { conversationId, role: "user", content, status: "complete" },
  });

  // 2. Build the context from stored history (Part 7 explains the budget).
  const history = await loadHistory(conversationId);

  // 3. Create the assistant row up front so a crash leaves a trace.
  const assistant = await prisma.message.create({
    data: { conversationId, role: "assistant", content: "", status: "streaming", model: "claude-opus-5-5" },
  });

  const stream = client.beta.messages.stream({
    model: "claude-opus-5-5",
    max_tokens: 8000,
    output_config: { effort: "low" }, // chat answers rarely need deep reasoning
    system: SYSTEM_PROMPT,
    cache_control: { type: "ephemeral" }, // cache the growing history prefix (Part 7)
    messages: history,
    // If the model declines on a safety classifier, retry on a fallback model in the same call.
    betas: ["server-side-fallback-2026-07-01"],
    fallbacks: "default",
  });

  const encoder = new TextEncoder();
  let full = "";

  const body = new ReadableStream({
    async start(controller) {
      const send = (event: object) =>
        controller.enqueue(encoder.encode(JSON.stringify(event) + "\n"));

      try {
        for await (const event of stream) {
          if (event.type === "content_block_delta" && event.delta.type === "text_delta") {
            full += event.delta.text;
            send({ type: "delta", text: event.delta.text });
          }
        }

        const final = await stream.finalMessage();
        const status =
          final.stop_reason === "max_tokens" ? "truncated" :
          final.stop_reason === "refusal" ? "refused" : "complete";

        await prisma.message.update({
          where: { id: assistant.id },
          data: {
            content: full,
            status,
            inputTokens: final.usage.input_tokens,
            outputTokens: final.usage.output_tokens,
          },
        });
        send({ type: "done", messageId: assistant.id, stopReason: final.stop_reason });
        controller.close();
      } catch (err) {
        // Either the client went away (we aborted) or the provider failed mid-stream.
        await prisma.message.update({
          where: { id: assistant.id },
          data: { content: full, status: stream.aborted ? "aborted" : "error" },
        });
        try {
          send({ type: "error", message: "The response was interrupted." });
          controller.close();
        } catch {
          // The browser is already gone; nothing to send.
        }
      }
    },
    cancel() {
      // The browser disconnected (tab closed, Stop pressed, navigation).
      // Stop generating — every further token is billed and unread.
      stream.abort();
    },
  });

  return new Response(body, {
    headers: {
      "Content-Type": "application/x-ndjson; charset=utf-8",
      "Cache-Control": "no-cache, no-transform",
      "X-Accel-Buffering": "no", // tell nginx-style proxies not to buffer
    },
  });
}

Four decisions in this handler are worth understanding rather than copying.

The user message is written first. If the model call fails, the user's message must still exist — otherwise a retry loses what they typed, and your history silently drops a turn. Write what you know before you do anything that can fail.

The assistant row is created before streaming starts, with status: "streaming". If your server process crashes mid-response, the row remains in streaming state forever. That is useful: a periodic job (or the next page load) can find rows stuck in streaming for more than a few minutes and mark them error. A row that never got created leaves no evidence that anything went wrong.

cancel() aborts the upstream stream. When the browser disconnects, the runtime calls cancel() on the response body. Without this line, your server keeps reading from the provider until the model finishes. On a long answer that is thousands of output tokens billed for a user who already left. This is the single most common cost leak in chat apps, and it is one line to fix.

The response headers fight buffering. Anything between your server and the browser — a reverse proxy, a CDN, compression middleware — may decide to collect the response into a buffer before sending it. Your stream still "works", but the user sees nothing until the end, which defeats the whole point. Cache-Control: no-transform asks intermediaries not to modify the body (compression requires buffering), and X-Accel-Buffering: no disables nginx's response buffering. If streaming works locally but not in production, buffering in the deployment path is the first suspect.

Using OpenAI instead? The shape of the handler is the same. Replace the stream with client.responses.create({ model, input, stream: true }) from the openai package, and forward event.delta from events whose type is response.output_text.delta. Your NDJSON protocol — and therefore your entire frontend — does not change. That is the payoff of owning the wire format.

Part 4: The Client — Reading a Stream Correctly

On the browser side, fetch gives you a response whose body is a ReadableStream of bytes. Reading it looks straightforward. It has two traps.

Trap 1: A chunk is not a line

The network delivers bytes in chunks whose boundaries have nothing to do with your protocol. One read might contain two and a half events. The next might contain the rest of the third event and half of a fourth. If you JSON.parse each chunk, you will fail intermittently — usually only in production, where the network is slower and chunks split differently than on localhost.

The fix is a line buffer: append each chunk to a string, extract every complete line, and keep the incomplete tail for the next read.

Trap 2: A chunk is not a character

UTF-8 encodes many characters as multiple bytes — every Hindi or Kannada character, every emoji, many symbols. A chunk boundary can fall in the middle of a character. If you decode each chunk independently with new TextDecoder().decode(chunk), the split character becomes two replacement characters (��). This bug only appears for users writing in non-Latin scripts, which is exactly why it ships unnoticed.

The fix is to decode the stream as a stream: TextDecoderStream (or decoder.decode(chunk, { stream: true })) holds partial byte sequences until the rest of the character arrives.

// lib/chat-client.ts
type ChatEvent =
  | { type: "delta"; text: string }
  | { type: "done"; messageId: string; stopReason: string }
  | { type: "error"; message: string };

export async function streamChat(
  conversationId: string,
  content: string,
  onDelta: (text: string) => void,
  signal: AbortSignal,
): Promise<Extract<ChatEvent, { type: "done" }>> {
  const res = await fetch("/api/chat", {
    method: "POST",
    headers: { "Content-Type": "application/json" },
    body: JSON.stringify({ conversationId, content }),
    signal, // aborting this closes the connection -> server's cancel() runs
  });
  if (!res.ok || !res.body) throw new Error(`Chat request failed: ${res.status}`);

  // Decode bytes as a stream so multi-byte characters survive chunk splits.
  const reader = res.body.pipeThrough(new TextDecoderStream()).getReader();
  let buffer = "";

  while (true) {
    const { value, done } = await reader.read();
    if (done) break;
    buffer += value;

    // Extract every complete line; keep the partial tail in the buffer.
    let newline: number;
    while ((newline = buffer.indexOf("\n")) !== -1) {
      const line = buffer.slice(0, newline);
      buffer = buffer.slice(newline + 1);
      if (!line.trim()) continue;

      const event = JSON.parse(line) as ChatEvent;
      if (event.type === "delta") onDelta(event.text);
      else if (event.type === "done") return event;
      else throw new Error(event.message);
    }
  }
  throw new Error("Stream ended without a done event");
}

The last line matters: if the connection closes without a done event, the response is incomplete — a proxy timeout, a server crash, a dropped mobile connection. Treating "stream ended" as "success" is how apps end up showing half an answer as if it were the whole one.

The Stop button

The signal parameter is your Stop button. Create an AbortController per request, pass its signal to fetch, and call controller.abort() when the user clicks Stop. Aborting the fetch closes the connection; closing the connection triggers cancel() on the server's response body; cancel() aborts the provider stream. One click propagates through three hops and stops the billing meter. The server saves whatever was generated with status: "aborted", so the partial answer is still there after a reload.

Rendering without melting the main thread

A fast model can deliver 50–100 deltas per second. If every delta calls setState, React re-renders the message list 100 times a second, and each render re-parses Markdown (Part 5). On a mid-range phone, the UI starts to stutter well before the stream ends.

The screen refreshes at 60 Hz, so anything faster than one update per frame is wasted work. Accumulate deltas in a ref and flush them to state at most once per animation frame:

// inside your chat component
const pending = useRef("");
const frame = useRef<number | null>(null);

function onDelta(text: string) {
  pending.current += text;
  if (frame.current === null) {
    frame.current = requestAnimationFrame(() => {
      const chunk = pending.current;
      pending.current = "";
      frame.current = null;
      setDraft(prev => prev + chunk);
    });
  }
}

This caps re-renders at the display's refresh rate no matter how fast tokens arrive. When the stream finishes, flush any remaining pending text before you mark the message complete.

Part 5: Rendering Markdown While It Is Still Being Written

Models write Markdown: headings, lists, tables and fenced code blocks. Rendering it is easy when the text is complete. Rendering it while it is half-written creates two problems.

Problem 1: Unfinished syntax

Midway through a response, the text might end like this:

Here is the function:

```typescript
export function debounce(fn, ms) {
  let t;

The opening fence has arrived; the closing fence hasn't. A Markdown parser sees an unclosed fence and either renders everything after it as code or renders the fence characters as literal text, depending on the parser. Either way the message flickers between layouts as tokens arrive.

The fix is to repair the tail before rendering: count the fence lines in the draft. If the count is odd, a fence is open — append a closing fence to the string you render (not the string you store). The code block then renders as a code block from its first line, and grows smoothly as more lines arrive. Apply the same idea to anything else that flickers in your parser, such as half-written table rows.

Problem 2: Quadratic re-parsing

If you re-render the whole message on every update, a response with n updates causes n full parses of text that grows each time. The total work is proportional to n². For a 3,000-token answer with code blocks and syntax highlighting, that quadratic curve is the stutter you see near the end of long answers.

The fix comes from noticing that Markdown is mostly block-structured: once a paragraph or code block is followed by a blank line and a new block has started, it will never change again. So:

1. Split the draft into top-level blocks (a Markdown lexer such as marked.lexer gives you this).
2. Render every block except the last as a memoised component keyed by its index and content — React skips them on every subsequent render.
3. Re-render only the last block, the one still being written.

Work per update becomes proportional to the size of the last block instead of the whole message, and the stutter disappears.

Syntax highlighting

Syntax highlighting is the most expensive part of rendering. Highlight a code block only once it is closed: show the live block as plain monospace text while it streams, then highlight it once when the closing fence arrives. Users don't notice the switch. They do notice a stutter.

Safety

The model's output is untrusted input to your page. If your Markdown renderer allows raw HTML, a model that has been fed a malicious document (or a user who asks it to) can produce <img src=x onerror=...> and run script in your origin. Keep raw HTML disabled — react-markdown disables it by default — and only allow the link protocols you expect (https:, mailto:) so a javascript: URL in a link can't run code. AI Ch 8 covers why model output must be treated as attacker-controlled.

Part 6: Persistence — What to Store and When

A chat app without persistence is a demo. Persistence turns it into a product: reload the page and the conversation is still there, open it on another device, search old chats, see what the app costs you per user.

The data model

model Conversation {
  id        String    @id @default(cuid())
  userId    String
  title     String?
  summary   String?   // rolling summary of turns that fell out of the context window (Part 7)
  createdAt DateTime  @default(now())
  updatedAt DateTime  @updatedAt
  messages  Message[]

  @@index([userId, updatedAt])
}

model Message {
  id             String       @id @default(cuid())
  conversationId String
  conversation   Conversation @relation(fields: [conversationId], references: [id], onDelete: Cascade)
  role           String       // "user" | "assistant"
  content        String
  status         String       // "complete" | "streaming" | "aborted" | "error" | "truncated" | "refused"
  model          String?
  inputTokens    Int?
  outputTokens   Int?
  createdAt      DateTime     @default(now())

  @@index([conversationId, createdAt])
}

Three fields here are easy to leave out and painful to add later:

• status. Without it, you cannot distinguish a complete answer from one that was cut off by a crash, a Stop click, or max_tokens. The UI needs this to show a "Continue" or "Retry" button. Your history builder needs it to decide what to send back to the model.
• model. You will change models. When a user reports that "answers got worse last week", you need to know which model produced which message.
• inputTokens / outputTokens. These are your cost ledger. Summing them per user per day is how you find the one user whose 400-message conversation is costing more than everyone else combined.

The two indexes match the two queries the app actually runs: "list this user's conversations, most recent first" and "load this conversation's messages in order."

When to write

The route handler in Part 3 writes three times per turn: the user message before the call, an empty assistant row before streaming, and the final content after streaming. You might be tempted to write the assistant content on every delta "so nothing is lost." Don't. That is 100 database writes per second per active stream, for no benefit — the partial text is already held in memory and is saved on abort or error by the catch block. Write at state transitions, not on every increment.

Titles

ChatGPT-style apps title each conversation automatically. Generate the title after the first response completes, never before or during it, so title generation never adds latency to the answer the user is waiting for. A small, cheap model is plenty for a five-word title:

// Fire-and-forget after the first exchange; never awaited by the chat request.
async function generateTitle(conversationId: string, firstUserMessage: string) {
  const res = await client.messages.create({
    model: "claude-haiku-4-5",
    max_tokens: 30,
    messages: [{
      role: "user",
      content: `Write a 3-6 word title for a chat that starts with this message. Reply with the title only.\n\n${firstUserMessage}`,
    }],
  });
  const block = res.content.find(b => b.type === "text");
  if (block?.type === "text") {
    await prisma.conversation.update({ where: { id: conversationId }, data: { title: block.text.trim() } });
  }
}

Part 7: History, Context Limits, and the Quadratic Cost of Chat

The model API is stateless. It does not remember your previous call. The only reason the model "remembers" the conversation is that you send the entire history with every request. This single fact drives almost every scaling problem in chat apps.

Why chat cost grows quadratically

Suppose each user message plus assistant reply averages 600 tokens. On turn k, you send roughly k × 600 input tokens. Over a 40-turn conversation:

• Turn 1 sends ~600 tokens. Turn 40 sends ~24,000.
• Total input across the conversation: 600 × (1 + 2 + … + 40) = 600 × 820 ≈ 492,000 tokens.

The cost of each turn grows linearly, so the cost of the whole conversation grows with the square of its length. Latency follows the same curve, because prefill time grows with input size. A conversation that felt instant at turn 3 feels sluggish at turn 60.

At $4 per million input tokens, that 40-turn conversation costs about $1.97 in input alone. Two techniques bring that down: a context budget, and prompt caching.

Technique 1: A context budget

Context windows today are huge — a million tokens on current Claude models — but the window is a ceiling, not a target. You choose how much history to resend, based on cost and latency, not on what fits. A budget-based history builder walks backwards from the newest message and keeps messages until the budget runs out:

// lib/history.ts
import type Anthropic from "@anthropic-ai/sdk";
import { prisma } from "@/lib/db";

const HISTORY_BUDGET = 60_000; // tokens of history to resend per turn; tune for cost vs. recall

// ~4 characters per token is a decent estimate for English; use the
// count-tokens API when you need an exact number (e.g. near a hard limit).
const estimateTokens = (text: string) => Math.ceil(text.length / 4);

export async function loadHistory(conversationId: string): Promise<Anthropic.Beta.BetaMessageParam[]> {
  const rows = await prisma.message.findMany({
    where: { conversationId, status: { in: ["complete", "truncated", "aborted"] }, content: { not: "" } },
    orderBy: { createdAt: "asc" },
  });

  const picked: typeof rows = [];
  let used = 0;
  for (let i = rows.length - 1; i >= 0; i--) {
    const cost = estimateTokens(rows[i].content);
    if (used + cost > HISTORY_BUDGET) break;
    picked.unshift(rows[i]);
    used += cost;
  }

  // The conversation sent to the API must start with a user turn.
  while (picked.length && picked[0].role !== "user") picked.shift();

  return picked.map(m => ({ role: m.role as "user" | "assistant", content: m.content }));
}

Notice which statuses are included. error rows and empty rows are excluded — they would confuse the model. aborted rows are included: the user saw that partial answer and may refer to it ("no, the second approach you started describing").

When old turns fall outside the budget, the model loses them entirely. For many chats that is fine; nobody refers back to message 3 at message 80. When it isn't fine, add a rolling summary: when messages fall out of the window, have a cheap model fold them into Conversation.summary, and send that summary at the start of the system prompt. The model keeps the gist of the early conversation for a few hundred tokens instead of tens of thousands. Chapter 8 generalises this idea into proper long-term memory.

For exact counts — for example, to warn the user before a pasted document pushes them past a hard limit — use the API's token counter rather than the estimate:

const { input_tokens } = await client.messages.countTokens({
  model: "claude-opus-5-5",
  system: SYSTEM_PROMPT,
  messages: history,
});

Technique 2: Prompt caching

Look again at what you send each turn: the system prompt, then the history, then one new message. Turn 41's request is turn 40's request plus two new messages. The prefix is identical. Prompt caching lets the provider keep the processed prefix and charge you a fraction of the normal price to reuse it.

With Claude, the cache_control: { type: "ephemeral" } field in the route handler enables automatic caching: each request caches everything up to its last block, and the next request reads that prefix from cache. On Claude Opus 5.5, cached reads cost $0.20 per million tokens instead of $4 — one twentieth. Redo the 40-turn arithmetic with caching: nearly all of the 492,000 input tokens are cached prefix reads, and only the ~24,000 new tokens are written to cache at a small premium. The input cost drops from about $1.97 to roughly $0.20 — and prefill latency drops too, because the cached prefix doesn't need to be processed again. (Very short prefixes don't cache, because each model has a minimum cacheable length, so the first few turns of a conversation pay full price.)

Where the two techniques collide

Here is the subtle part. Caching depends on the prefix staying byte-for-byte identical. A sliding window that drops the oldest message every turn changes the start of the prefix on every turn — which invalidates the entire cache, every turn. Combine a naive sliding window with caching and you get neither benefit: you pay for a full uncached prefill of 60,000 tokens on every message.

The fix is to slide in large jumps instead of one message at a time. Let the history grow until it exceeds the budget, then cut it down to half the budget in one step (folding the dropped messages into the rolling summary). The prefix then stays stable for many turns at a time and caches well, and you only pay one uncached prefill per jump. This is a recurring pattern in AI systems: a cost optimisation that changes the prompt can silently disable another cost optimisation that depends on the prompt not changing. Always check whether two optimisations fight before stacking them.

Part 8: The Failure Catalog

Every one of these will happen in production. Decide your behaviour for each before you ship.

Build Checkpoint

Build this in a fresh Next.js app with Postgres (local Docker or a hosted free tier) and Prisma. Each step has a condition that tells you it's done. Try each step without looking back at the chapter first.

1. One streaming call in a script. A Node script that streams a response to the terminal and prints stop_reason and usage at the end. Done when: text appears word by word, and you can point at the line that prints input and output token counts.

2. The stream proxy. A POST /api/chat route that streams NDJSON delta events and a final done event. No database yet. Done when: curl -N -X POST localhost:3000/api/chat -d '{"content":"hi"}' prints JSON lines as they arrive, not all at once.

3. The client reader. A chat page that renders deltas live, with the line buffer and TextDecoderStream. Done when: a response in Hindi or Kannada (ask for one) renders with no � characters, and throttling the network in DevTools to "Slow 3G" doesn't break parsing.

4. Stop and cancel. A Stop button wired to an AbortController, and cancel() on the server aborting the upstream stream. Done when: you log in cancel() and see the log line the moment you click Stop or close the tab, and the provider's usage dashboard confirms generation stopped.

5. Markdown that doesn't flicker. Fence repair for unclosed code blocks, memoised completed blocks, and highlighting only closed blocks. Done when: asking for "a 100-line TypeScript file" streams smoothly to the end, with the code block visible as code from its first line.

6. Persistence. The Prisma schema, the three writes per turn, a sidebar listing conversations, and automatic titles. Done when: you reload mid-conversation and everything is there, and an answer you stopped halfway reappears with an "aborted" marker.

7. Context budget and caching. The budget-based loadHistory, cache_control on the request, and usage logging. Done when: in a conversation longer than a few turns, the logged cache_read_input_tokens is non-zero and grows each turn, while the uncached input_tokens stays small.

8. Failure drills. Force each failure from Part 8 at least once: set max_tokens: 50, kill the dev server mid-stream, press Enter twice quickly. Done when: every case leaves the database and the UI in a state you would accept from a product you pay for.

Vocabulary Map

Practice Problems

Problem 1 — Failure analysis. Users on your production deployment report that answers "appear all at once after a long pause", but streaming works perfectly on localhost. List three distinct places between your route handler and the browser that could be buffering the response, how you would confirm each one, and the fix for each.

Problem 2 — Cost forensics. Your monthly bill doubles, but your number of messages only grew 20%. Using only the Message table from Part 6, write the queries (in SQL or Prisma) you would run to find the cause. Name two plausible root causes from this chapter that those queries would reveal.

Problem 3 — Design. Product wants a "regenerate" button that replaces the last assistant answer with a new one. Design it: what happens to the old row (delete, keep, version?), what history is sent for the regeneration, and how does it interact with prompt caching? Justify each choice.

Problem 4 — Go deeper. You add a feature that lets users edit any earlier message, ChatGPT-style, which branches the conversation from that point. Redesign the data model to support branches. How does your history builder change, and what happens to the cache hit rate when users edit a message near the start of a long conversation?

Problem 5 — Connect to earlier chapters. In the Frozen Oracle chapter, the model was described as stateless and non-deterministic. For each of those two properties, name one design decision in this chapter that exists because of it.

What's Coming Next

Your chat app now answers from what the model learned in training, plus whatever the user pastes in. The next step is grounding it in your documents — at scale, with retrieval quality you can measure. Chapter 6 takes the basic RAG pipeline from Session 4 to production: metadata filtering, reranking, hybrid search, and an ingestion pipeline that runs on a queue instead of blocking a web request.