Chapter 1 — LLM Calls in Node: One Interface, Four Providers

Replaces session: Introduction to GenAI

You will build: a TypeScript Node project with a thin provider-agnostic complete() / stream() wrapper over Claude, OpenAI, Gemini and a local Ollama model, returning one normalized { text, stopReason, usage } result, plus local embeddings and a cost calculator.

Theory prerequisites: AI Ch 1 — The Frozen Oracle (tokens, next-token prediction, context, sampling)

Calling a language model from code looks like the easiest integration you will ever write: install an SDK, paste a key, send a string, print the string that comes back. The first demo works, and that is the problem. The ten-line version prints the answer without checking whether the model finished or was cut off at a token limit. It throws away the token counts that make up your bill. It hard-codes a key that ends up in a Git commit. And it is welded to one provider's response shape, so trying a cheaper or local model means rewriting every call site.

Providers also describe the same physical events in different words. Claude says end_turn, OpenAI says completed, Gemini says STOP, Ollama says stop. Claude reports cached input separately from fresh input; OpenAI counts it inside input; Gemini leaves reasoning tokens out of its output count while the others include them. Sum these naively and your cost dashboard is wrong without anyone noticing.

This chapter starts with what physically happens to your message inside the model, because that explains every parameter and limit. Then it sets up a project that can't leak its keys, calls four providers, and builds a wrapper that turns four response shapes into one. Later chapters call models through some version of that wrapper.

Part 1: What Actually Happens When You Send a Message

The full theory is in The Frozen Oracle. This is the version that shows up as parameters, prices and errors in your code.

Step 1: Your conversation becomes one long string

The model has no concept of "messages". The provider flattens your system prompt and the user and assistant turns into one sequence with marker tokens between roles (a chat template). This is why the API is stateless: there is no conversation object on the server. Every request re-sends the entire history, and every token of it is billed again.

Step 2: Tokenization

A tokenizer splits text into tokens — chunks from a fixed vocabulary of tens to hundreds of thousands of entries — and maps each to an integer ID. Common English words are usually one token; rare words split into pieces. A workable rule of thumb for English is about 4 characters per token, so a 2,000-word document is roughly 2,600–2,700 tokens. Consequences:

• You pay per token. Every price in this chapter is dollars per million tokens (MTok).
• Tokenizers differ between providers and model generations. The same prompt is a different count on Claude, GPT and Gemini. Never budget one provider with another's count.
• Non-English text usually costs more. A sentence in Hindi or Kannada often takes several times the tokens of its English equivalent. Measure; don't assume the 4-characters rule.

Step 3: Embeddings — tokens become vectors

The network can't do arithmetic on an integer ID, so the first layer replaces each ID with an embedding: a vector of a few thousand numbers. Training moves these vectors so tokens used in similar contexts point in similar directions. That is the vector-space intuition: meaning becomes geometry, and "similar" becomes "close together". Embedding models (Part 5) produce one vector per piece of text instead of per token, so two paragraphs about the same topic land near each other even with no shared words. That is the basis of retrieval in Chapter 4.

Step 4: Attention — every token looks at every earlier token

Inside each transformer layer, attention lets each token gather information from the tokens before it. Each token emits a query ("what am I looking for?"); every earlier token offers a key ("what do I contain?") and a value ("my content"). Query-key scores become weights, and the token takes the weighted mix of values. That is how "it" ends up carrying information about the noun fifty tokens back. Why you care:

• Prefill (processing the input) grows faster than linearly with prompt length, because each token attends to all earlier ones. Long prompts cost latency as well as money.
• Generation reuses work through a KV cache: keys and values for tokens already processed are stored, not recomputed. That stored state is what prompt caching keeps between requests — which is why a cached prefix is both cheaper and faster.
• Attention is spread thin over long contexts. A fact buried mid-way through a 200,000-token prompt is found less reliably than in a 2,000-token one. A bigger window doesn't excuse you from selecting what you send.

Step 5: The context window

The context window is the maximum number of tokens per request — input and output combined. Current Claude Opus and Sonnet models offer 1M tokens; Claude Haiku 4.5 offers 200K; other providers and local models vary. Hitting it looks different depending on where:

The third row is the dangerous one (Part 5).

Step 6: Next-token prediction, in a loop

The last layer outputs a score (a logit) for every vocabulary token. A sampler picks one token, it is appended, and the whole stack runs again. Generation is a loop of one token per step, so output arrives at a roughly fixed rate and output tokens cost several times more than input. That loop is why streaming exists.

Part 2: Sampling — and Why the Knobs Are Disappearing

Always taking the highest-scoring token (greedy decoding) tends to produce flat, repetitive text. So providers sample from a probability distribution, and historically exposed two knobs to shape it.

Temperature divides every logit by a number T before converting scores to probabilities with softmax. T below 1 sharpens the distribution (the top token gets even more likely); T above 1 flattens it (unlikely tokens get a real chance).

Top-p (nucleus sampling) sorts tokens by probability and keeps only the smallest set whose probabilities add up to p, then samples from that set. It cuts off the long tail of absurd tokens no matter how flat the distribution is.

Here is the whole mechanism in a few lines of TypeScript:

// sampling.ts — run with: npx tsx sampling.ts
function softmax(logits: number[], temperature: number): number[] {
  const scaled = logits.map((l) => l / temperature);
  const max = Math.max(...scaled); // subtract max for numerical stability
  const exps = scaled.map((s) => Math.exp(s - max));
  const sum = exps.reduce((a, b) => a + b, 0);
  return exps.map((e) => e / sum);
}

function topP(probs: number[], p: number): number[] {
  const order = probs.map((prob, i) => ({ prob, i })).sort((a, b) => b.prob - a.prob);
  const keep = new Set<number>();
  let total = 0;
  for (const { prob, i } of order) {
    keep.add(i);
    total += prob;
    if (total >= p) break;
  }
  const kept = probs.map((prob, i) => (keep.has(i) ? prob : 0));
  const sum = kept.reduce((a, b) => a + b, 0);
  return kept.map((x) => x / sum);
}

const tokens = ["Paris", "Lyon", "France", "banana"];
const logits = [6.0, 3.5, 3.0, 0.5];
for (const t of [0.5, 1.0, 1.5]) {
  const probs = softmax(logits, t).map((x) => x.toFixed(3));
  console.log(`T=${t}`, Object.fromEntries(tokens.map((tok, i) => [tok, probs[i]])));
}
console.log("T=1.5, top-p 0.9", topP(softmax(logits, 1.5), 0.9).map((x) => x.toFixed(3)));

At T = 0.5, "Paris" gets about 99% of the probability; at T = 1.5, roughly 74%, and "banana" rises to about 2%. Top-p 0.9 then removes "banana" entirely. That is all these knobs ever did: reshape one distribution, one token at a time.

Why a provider might remove them

Current Claude Opus models reject temperature, top_p and top_k with a 400 error. That surprises anyone who learned "set temperature to 0 for deterministic output", until you look at what the knobs were used for and what changed underneath them.

1. Temperature 0 never gave determinism. Hosted inference batches your request with others, and GPU floating-point arithmetic isn't perfectly reproducible across batch shapes. Identical requests at temperature 0 could still diverge.
2. Reasoning models generate many tokens you don't see. Claude Opus 5.5 always thinks before answering, so its sampling settings shape thousands of reasoning tokens, not just the visible answer. A provider that has tuned sampling for good reasoning has a strong reason not to let each caller detune it: a "creative" temperature that livens prose can also make reasoning sloppier.
3. The real goals have better tools now:

Ollama (options) and Gemini (config) still accept sampling parameters. The wrapper in Part 6 leaves them out anyway: a parameter one of your providers rejects doesn't belong in a portable interface.

Part 3: A Node Project That Never Leaks a Key

The setup

mkdir llm-node && cd llm-node
npm init -y
npm pkg set type=module
npm install @anthropic-ai/sdk openai @google/genai ollama
npm install -D typescript tsx @types/node
mkdir src

tsx runs TypeScript directly (npx tsx src/main.ts) with no build step. Use a strict tsconfig.json:

{
  "compilerOptions": {
    "target": "ES2022",
    "module": "NodeNext",
    "moduleResolution": "NodeNext",
    "strict": true,
    "skipLibCheck": true,
    "noEmit": true
  },
  "include": ["src"]
}

With NodeNext, relative imports carry a .js extension even though the files are .ts ("./llm.js"): the path names the file Node loads at runtime.

Keys live in the environment, never in the code

Create .env in the project root:

ANTHROPIC_API_KEY=sk-ant-...
OPENAI_API_KEY=sk-...
OPENAI_MODEL=
GEMINI_API_KEY=...
GEMINI_MODEL=
OLLAMA_MODEL=llama3.2
OLLAMA_EMBED_MODEL=nomic-embed-text

Fill in OPENAI_MODEL and GEMINI_MODEL from each provider's current models page. Names change every few months; reading them from the environment makes an upgrade a config change. Then, before you write any other file:

printf ".env\nnode_modules\n" >> .gitignore
cp .env .env.example   # then blank out every value in .env.example

.env.example documents the variables without secrets and is committed; .env never is. Order matters: once a key is pushed, deleting the file doesn't help — it's in the history, and scanners crawl public repositories for keys within minutes. The only fix for a leaked key is to revoke it and create a new one. Node 20.12+ reads .env itself:

// src/env.ts
try {
  process.loadEnvFile(); // reads ./.env into process.env
} catch {
  // No .env file: in production the host injects real environment variables.
}

export function requireEnv(name: string): string {
  const value = process.env[name];
  if (!value) {
    throw new Error(`Missing env var ${name}. Add it to .env locally or to your host's secret settings.`);
  }
  return value;
}

The try exists because production has no .env file — the host injects variables — and the same code must run in both places. requireEnv fails with the variable's name instead of a confusing 401 from deep inside an SDK.

Never ship keys to the browser. Anything bundled for the browser — including variables your framework exposes publicly, like Next.js's NEXT_PUBLIC_ prefix — is readable in DevTools by any visitor. Every model call in this track goes through your server; Chapter 5 builds the proxy for a chat UI.

The dashboards, conceptually

The OpenAI platform dashboard, the Anthropic Console and Google AI Studio change layout often, but expose the same four concepts:

Part 4: Four Providers, Four Response Shapes

The same request — system prompt, one user message, output cap — against each provider's official SDK. The differences are the lesson.

Claude (Anthropic SDK)

// src/raw-claude.ts
import "./env.js";
import Anthropic from "@anthropic-ai/sdk";

const client = new Anthropic(); // reads ANTHROPIC_API_KEY

const msg = await client.messages.create({
  model: "claude-opus-5-5",
  max_tokens: 16000,
  system: "Answer in one short paragraph.",
  output_config: { effort: "low" },
  messages: [{ role: "user", content: "What is a context window?" }],
});

const text = msg.content.flatMap((b) => (b.type === "text" ? [b.text] : [])).join("");
console.log(text);
console.log(msg.stop_reason, msg.usage.input_tokens, msg.usage.output_tokens);

content is an array of typed blocks, not a string. On Opus 5.5 it can include a thinking block (empty by default — the reasoning is billed but not returned), and in Chapter 3 tool_use blocks. Filtering by type keeps your code correct as block types are added. max_tokens includes thinking tokens, so a low cap can stop generation before the visible answer starts.

OpenAI (Responses API)

// src/raw-openai.ts
import { requireEnv } from "./env.js";
import OpenAI from "openai";

const client = new OpenAI(); // reads OPENAI_API_KEY

const res = await client.responses.create({
  model: requireEnv("OPENAI_MODEL"),
  instructions: "Answer in one short paragraph.",
  input: [{ role: "user", content: "What is a context window?" }],
  max_output_tokens: 16000,
});

console.log(res.output_text);
console.log(res.status, res.incomplete_details?.reason, res.usage?.input_tokens, res.usage?.output_tokens);

output_text is an SDK helper that concatenates the text parts of the output array. There is no single stop-reason field: status is completed on success, or incomplete with incomplete_details.reason (max_output_tokens, content_filter) when cut short. A model refusal arrives as a refusal content part inside a completed response, so "completed" doesn't by itself mean "you got an answer".

Gemini (@google/genai, Google AI Studio key)

// src/raw-gemini.ts
import { requireEnv } from "./env.js";
import { GoogleGenAI } from "@google/genai";

const ai = new GoogleGenAI({ apiKey: requireEnv("GEMINI_API_KEY") });

const res = await ai.models.generateContent({
  model: requireEnv("GEMINI_MODEL"),
  contents: [{ role: "user", parts: [{ text: "What is a context window?" }] }],
  config: { systemInstruction: "Answer in one short paragraph.", maxOutputTokens: 16000 },
});

console.log(res.text);
console.log(res.candidates?.[0]?.finishReason, res.usageMetadata);

Gemini calls the assistant role model, wraps text in parts, and returns candidates with a finishReason (STOP, MAX_TOKENS, SAFETY and others). If the prompt is blocked there may be no candidates; the reason sits in promptFeedback.blockReason and res.text is undefined, not an error. Use @google/genai — Google's older JavaScript SDK is deprecated, though much code online still uses it.

Ollama (local)

ollama.chat({ model, messages, options }) from the ollama package returns message.content, done_reason, and token counts in prompt_eval_count / eval_count. There is no key — the server is on your machine — and the system prompt is just a system message. The output cap is options.num_predict and the context size is options.num_ctx, which you should always set (Part 5). The Ollama adapter in Part 6 shows the full call.

The comparison that motivates Part 6

Look at the last two rows. Log Claude's and OpenAI's input_tokens into one column and the Claude numbers under-count every cached request. Log Gemini's candidatesTokenCount as output and you under-count every thinking-heavy request — billed at the output rate. Normalizing usage is the difference between a correct cost ledger and a wrong one.

Part 5: Running Models and Embeddings on Your Own Machine

What Ollama is

Ollama is a local model server. You ollama pull open-weight models (Llama, Qwen, Gemma, Mistral families) and it serves an HTTP API on http://127.0.0.1:11434. The ollama npm package is a typed client for it; plain fetch to POST /api/chat or POST /api/embed returns the same JSON.

ollama pull llama3.2           # a small chat model, a couple of GB
ollama pull nomic-embed-text   # an embedding model
ollama ps                      # what is loaded in memory right now

Why local, and what it costs you instead

Local models cost nothing per token, keep data on your machine, and work offline. You pay in hardware and quality:

• Memory. Weights must fit in RAM or GPU memory. An 8-billion-parameter model at 4-bit quantization (weights stored at reduced precision) needs about 8 × 10⁹ × 0.5 bytes ≈ 4 GB, plus a KV cache that grows with num_ctx. A 70B model needs about 35 GB before any context — which is why laptop models are mostly 1–14B.
• Speed and quality. A laptop decodes tens of tokens per second from a small model. Small open models do well on narrow tasks (classification, extraction, short rewrites) and are noticeably weaker at long multi-step reasoning than frontier hosted models.
• Cold starts. Loading a model from disk can take seconds; Ollama unloads idle models after a few minutes unless you set keep_alive.

The silent truncation trap

Back to the third row of Part 1's context-window table. Ollama's default num_ctx has been a few thousand tokens in many versions — far below what the model supports — and a longer prompt is cut from the start, with an answer anyway. The symptom is a model that "ignores the instructions" or "can't find" something you clearly sent. The system prompt sits at the start, so it is cut first.

Always set num_ctx explicitly, and compare prompt_eval_count with your estimate of the prompt size. Much smaller means truncation. (It can also mean Ollama reused a cached prefix, so check with a fresh prompt.)

Embeddings on your machine

An embedding model turns text into one fixed-length vector; nomic-embed-text produces 768 numbers. Embed a batch in one call, then compare with cosine similarity — the cosine of the angle between two vectors, 1 for the same direction, near 0 for unrelated:

// src/embed.ts
import { requireEnv } from "./env.js";
import { Ollama } from "ollama";

const ollama = new Ollama({ host: process.env.OLLAMA_HOST ?? "http://127.0.0.1:11434" });

function cosine(a: number[], b: number[]): number {
  let dot = 0, na = 0, nb = 0;
  for (let i = 0; i < a.length; i++) {
    dot += a[i] * b[i];
    na += a[i] * a[i];
    nb += b[i] * b[i];
  }
  return dot / (Math.sqrt(na) * Math.sqrt(nb));
}

const docs = [
  "Refunds are processed within 5 business days.",
  "Our office is closed on public holidays.",
  "You can return an item within 30 days for a full refund.",
];
const query = "How long until I get my money back?";

const { embeddings } = await ollama.embed({
  model: requireEnv("OLLAMA_EMBED_MODEL"),
  input: [query, ...docs], // one request, one vector per input, same order
});

const [q, ...docVecs] = embeddings;
docVecs
  .map((v, i) => ({ doc: docs[i], score: cosine(q, v) }))
  .sort((a, b) => b.score - a.score)
  .forEach(({ doc, score }) => console.log(score.toFixed(3), doc));

The refund sentences should outscore the office-hours one although the query shares almost no words with them: the model has placed "money back" and "refund" close together. Three rules for Chapter 4:

• Vectors from different embedding models are not comparable, even at the same dimension. Store the model name with every vector and re-embed everything when you switch.
• Batch your inputs. One call with 100 inputs is far faster than 100 calls.
• Anthropic has no embeddings endpoint. A Claude app pairs Claude for generation with an embedding model from elsewhere — local like this, or hosted.

Part 6: The Core Build — One complete(), One stream()

The wrapper is deliberately thin: it normalizes the three things every caller needs — text, why it stopped, what it cost — and nothing else. It is not a framework; Chapter 9 covers agent SDKs that go further.

The design decisions

1. One normalized stop reason. Your app branches on it — Continue button on max_tokens, "declined" message on refused — and that logic should be written once, not per provider. The raw value is kept for logs, because "other" tells you nothing when debugging.
2. One normalized usage shape. inputTokens is everything the model read, cachedInputTokens the cached subset, outputTokens everything billed as output, reasoning included.
3. complete() is built on stream(). Adapters implement streaming only and complete() drains it: one code path to normalize, and no HTTP timeouts on long non-streaming requests.
4. Clients are created lazily. new OpenAI() throws without OPENAI_API_KEY; an Ollama-only user shouldn't crash over an unrelated key.

The code

// src/llm.ts
import { requireEnv } from "./env.js";
import Anthropic from "@anthropic-ai/sdk";
import OpenAI from "openai";
import { GoogleGenAI } from "@google/genai";
import { Ollama } from "ollama";

export type Provider = "claude" | "openai" | "gemini" | "ollama";
export type StopReason = "end" | "max_tokens" | "refused" | "context_exceeded" | "other";

export interface Usage {
  inputTokens: number;       // everything the model read, cached or not
  cachedInputTokens: number; // the part of inputTokens served from cache
  outputTokens: number;      // everything billed as output, including reasoning
}

export interface ChatMessage {
  role: "user" | "assistant";
  content: string;
}

export interface LLMRequest {
  provider: Provider;
  messages: ChatMessage[];
  system?: string;
  maxTokens?: number;
  effort?: "low" | "medium" | "high"; // honoured by Claude 5.x models; ignored for Haiku and other providers
  signal?: AbortSignal;
}

export interface LLMResult {
  text: string;
  stopReason: StopReason;
  rawStopReason: string;
  usage: Usage;
  provider: Provider;
  model: string;
}

export type StreamEvent = { type: "text"; text: string } | { type: "done"; result: LLMResult };

type Final = { rawStopReason: string; usage: Usage; model: string };
type AdapterEvent = { type: "text"; text: string } | { type: "final"; final: Final };

const STOP_MAP: Record<Provider, Record<string, StopReason>> = {
  claude: {
    end_turn: "end", stop_sequence: "end", max_tokens: "max_tokens",
    refusal: "refused", model_context_window_exceeded: "context_exceeded",
  },
  openai: { completed: "end", max_output_tokens: "max_tokens", content_filter: "refused" },
  gemini: {
    STOP: "end", MAX_TOKENS: "max_tokens", SAFETY: "refused", PROHIBITED_CONTENT: "refused",
    BLOCKLIST: "refused", SPII: "refused", RECITATION: "refused", PROMPT_BLOCKED: "refused",
  },
  ollama: { stop: "end", length: "max_tokens" },
};

// Lazy clients: a missing key only matters for the provider you actually call.
let anthropic: Anthropic | undefined;
let openai: OpenAI | undefined;
let gemini: GoogleGenAI | undefined;
let ollama: Ollama | undefined;

async function* claudeAdapter(req: LLMRequest): AsyncGenerator<AdapterEvent> {
  anthropic ??= new Anthropic();
  const model = process.env.CLAUDE_MODEL ?? "claude-opus-5-5";
  // Haiku 4.5 rejects `effort` and has no server-side fallbacks; the 5.x models take both.
  const isHaiku = model.startsWith("claude-haiku");
  const stream = anthropic.beta.messages.stream(
    {
      model,
      max_tokens: req.maxTokens ?? 16000,
      ...(req.system ? { system: req.system } : {}),
      messages: req.messages,
      ...(isHaiku
        ? {}
        : {
            output_config: { effort: req.effort ?? "medium" },
            // On a safety-classifier decline, the API retries on a fallback model in the same call.
            betas: ["server-side-fallback-2026-07-01"],
            fallbacks: "default" as const,
          }),
    },
    { signal: req.signal },
  );
  for await (const event of stream) {
    if (event.type === "content_block_delta" && event.delta.type === "text_delta") {
      yield { type: "text", text: event.delta.text };
    }
  }
  const msg = await stream.finalMessage();
  const u = msg.usage;
  const cacheRead = u.cache_read_input_tokens ?? 0;
  const cacheWrite = u.cache_creation_input_tokens ?? 0;
  yield {
    type: "final",
    final: {
      rawStopReason: msg.stop_reason ?? "unknown",
      model: msg.model,
      usage: {
        inputTokens: u.input_tokens + cacheRead + cacheWrite, // Claude reports these separately
        cachedInputTokens: cacheRead,
        outputTokens: u.output_tokens, // already includes thinking
      },
    },
  };
}

async function* openaiAdapter(req: LLMRequest): AsyncGenerator<AdapterEvent> {
  openai ??= new OpenAI();
  const events = await openai.responses.create(
    {
      model: requireEnv("OPENAI_MODEL"),
      ...(req.system ? { instructions: req.system } : {}),
      input: req.messages,
      max_output_tokens: req.maxTokens ?? 16000,
      stream: true,
    },
    { signal: req.signal },
  );
  for await (const event of events) {
    if (event.type === "response.output_text.delta") {
      yield { type: "text", text: event.delta };
    } else if (event.type === "response.completed" || event.type === "response.incomplete") {
      const r = event.response;
      yield {
        type: "final",
        final: {
          rawStopReason: r.status === "incomplete" ? (r.incomplete_details?.reason ?? "incomplete") : (r.status ?? "unknown"),
          model: r.model,
          usage: {
            inputTokens: r.usage?.input_tokens ?? 0, // already includes cached tokens
            cachedInputTokens: r.usage?.input_tokens_details.cached_tokens ?? 0,
            outputTokens: r.usage?.output_tokens ?? 0, // already includes reasoning
          },
        },
      };
    } else if (event.type === "response.failed") {
      throw new Error(`OpenAI response failed: ${event.response.error?.message ?? "unknown error"}`);
    }
  }
}

async function* geminiAdapter(req: LLMRequest): AsyncGenerator<AdapterEvent> {
  gemini ??= new GoogleGenAI({ apiKey: requireEnv("GEMINI_API_KEY") });
  const chunks = await gemini.models.generateContentStream({
    model: requireEnv("GEMINI_MODEL"),
    contents: req.messages.map((m) => ({
      role: m.role === "assistant" ? "model" : "user",
      parts: [{ text: m.content }],
    })),
    config: {
      ...(req.system ? { systemInstruction: req.system } : {}),
      maxOutputTokens: req.maxTokens ?? 16000,
      abortSignal: req.signal,
    },
  });
  let raw = "unknown";
  let model = requireEnv("GEMINI_MODEL");
  let usage: Usage = { inputTokens: 0, cachedInputTokens: 0, outputTokens: 0 };
  for await (const chunk of chunks) {
    if (chunk.text) yield { type: "text", text: chunk.text };
    if (chunk.promptFeedback?.blockReason) raw = "PROMPT_BLOCKED";
    const finish = chunk.candidates?.[0]?.finishReason;
    if (finish) raw = finish;
    if (chunk.modelVersion) model = chunk.modelVersion;
    const u = chunk.usageMetadata;
    if (u) {
      usage = {
        inputTokens: u.promptTokenCount ?? 0, // already includes cached content
        cachedInputTokens: u.cachedContentTokenCount ?? 0,
        outputTokens: (u.candidatesTokenCount ?? 0) + (u.thoughtsTokenCount ?? 0), // thoughts are billed as output
      };
    }
  }
  yield { type: "final", final: { rawStopReason: raw, model, usage } };
}

async function* ollamaAdapter(req: LLMRequest): AsyncGenerator<AdapterEvent> {
  ollama ??= new Ollama({ host: process.env.OLLAMA_HOST ?? "http://127.0.0.1:11434" });
  const parts = await ollama.chat({
    model: requireEnv("OLLAMA_MODEL"),
    messages: req.system ? [{ role: "system", content: req.system }, ...req.messages] : req.messages,
    stream: true,
    options: { num_predict: req.maxTokens ?? 4096, num_ctx: Number(process.env.OLLAMA_NUM_CTX ?? 8192) },
  });
  const onAbort = () => parts.abort();
  req.signal?.addEventListener("abort", onAbort, { once: true });
  try {
    for await (const part of parts) {
      if (part.message.content) yield { type: "text", text: part.message.content };
      if (part.done) {
        yield {
          type: "final",
          final: {
            rawStopReason: part.done_reason,
            model: part.model,
            usage: { inputTokens: part.prompt_eval_count ?? 0, cachedInputTokens: 0, outputTokens: part.eval_count ?? 0 },
          },
        };
      }
    }
  } finally {
    req.signal?.removeEventListener("abort", onAbort);
  }
}

const ADAPTERS: Record<Provider, (req: LLMRequest) => AsyncGenerator<AdapterEvent>> = {
  claude: claudeAdapter,
  openai: openaiAdapter,
  gemini: geminiAdapter,
  ollama: ollamaAdapter,
};

export async function* stream(req: LLMRequest): AsyncGenerator<StreamEvent> {
  let text = "";
  for await (const event of ADAPTERS[req.provider](req)) {
    if (event.type === "text") {
      text += event.text;
      yield event;
    } else {
      const { rawStopReason, usage, model } = event.final;
      const stopReason = STOP_MAP[req.provider][rawStopReason] ?? "other";
      yield { type: "done", result: { text, stopReason, rawStopReason, usage, model, provider: req.provider } };
      return;
    }
  }
  throw new Error(`${req.provider}: stream ended without a final event`);
}

export async function complete(req: LLMRequest): Promise<LLMResult> {
  for await (const event of stream(req)) {
    if (event.type === "done") return event.result;
  }
  throw new Error("unreachable: stream() always yields done or throws");
}

Walking through the decisions

Two adapter event types. Text increments, then exactly one final. stream() accumulates the text and turns final into the normalized done. If an adapter ends without a final — dropped connection, early close — stream() throws rather than returning half an answer as if complete. That is the most important rule in the file.

The usage arithmetic differs per adapter, by design. Claude: add cache reads and writes back into input. OpenAI: input already includes cache. Gemini: add thoughts to output. Ollama: no cache field. (Simplification: Claude bills cache writes at a premium and reads at a discount; this shape folds writes into ordinary input. Fine for a learning ledger; Chapter 5 stores the raw fields.)

Claude gets fallbacks: "default". Claude's safety classifiers occasionally decline benign requests (stop_reason: "refusal"). The server-side-fallback beta re-runs a declined request on a fallback model inside the same call, which is why the adapter uses client.beta.messages. If the whole chain declines, you still get refused.

effort is Claude-only here — it's what replaced the sampling knobs (Part 2). OpenAI and Gemini have their own reasoning controls; adding them is a good exercise once your models are pinned.

tool_use maps to other. This wrapper is text in, text out; Chapter 3 adds tools. Mapping unknown values to other — not to end — makes a new provider value show up in logs instead of passing as a silent success.

Using it

// src/main.ts — run with: npx tsx src/main.ts gemini
import { stream, type Provider } from "./llm.js";

const provider = (process.argv[2] ?? "claude") as Provider;
const controller = new AbortController();
process.on("SIGINT", () => controller.abort()); // Ctrl+C stops generation, and billing

for await (const event of stream({
  provider,
  system: "Answer in one short paragraph.",
  messages: [{ role: "user", content: "Why do LLM APIs charge per token?" }],
  maxTokens: 2000,
  signal: controller.signal,
})) {
  if (event.type === "text") process.stdout.write(event.text);
  else console.log("\n", event.result.stopReason, event.result.rawStopReason, event.result.usage);
}

Run it with four provider names: the text differs, the shape of the last line doesn't. The AbortController is the streaming idea you'll need most — when the reader leaves, stop generating, because every further token is billed and unread. Chapter 5 carries this signal from a browser Stop button through a server route to the provider, and covers stream events on the wire.

Part 7: Picking a Model — The Cost Arithmetic

Model choice trades quality, speed (time to first token, tokens per second) and cost. There is no best model, only a best model for a task at a volume.

Current Claude prices (per million tokens)

OpenAI and Gemini publish the same kind of table; prices change often, so read their pricing pages and use the same formula. Note that output costs 5 times input on all three: each output token is a sequential forward pass, while input is processed in parallel.

A worked example

A support-ticket classifier: each ticket is about 1,200 input tokens (instructions plus ticket) and returns a label with a one-line reason, about 150 output tokens. You process 50,000 tickets a day.

Daily volume: 50,000 × 1,200 = 60M input tokens, and 50,000 × 150 = 7.5M output tokens.

Now three corrections the naive calculation misses:

1. Thinking tokens are output tokens. Opus 5.5 always thinks. An extra 300 thinking tokens per ticket makes output 450 tokens and triples Opus output cost to $450 a day — $690 total. effort: "low" on a simple classification is a cost decision as much as a quality one. Normalized outputTokens captures this; an estimate from visible text doesn't.
2. Caching changes the input side. The 1,000 instruction tokens are identical for every ticket. Cached on Opus: 50M × $0.20 = $10, plus 10M uncached × $4 = $40 — input drops from $240 to about $50 a day.
3. Local isn't free at this volume. A laptop decoding 40 tokens per second spends 150 ÷ 40 ≈ 3.75 seconds per ticket on output alone. At roughly 5 seconds per ticket including prefill, 50,000 tickets take 250,000 seconds — about 69 hours. A day's tickets take three days.

The calculation as code, on normalized usage:

// src/cost.ts
import type { LLMResult } from "./llm.js";

// USD per million tokens. Add your OpenAI/Gemini models from their pricing pages.
const PRICES: Record<string, { input: number; output: number; cachedInput: number }> = {
  "claude-opus-5-5": { input: 4, output: 20, cachedInput: 0.2 },
  "claude-sonnet-5-5": { input: 2, output: 10, cachedInput: 0.2 },
  "claude-haiku-4-5": { input: 1, output: 5, cachedInput: 0.1 },
};

export function costUsd(result: LLMResult): number | null {
  if (result.provider === "ollama") return 0; // marginal cost; hardware is a fixed cost
  const price = PRICES[result.model];
  if (!price) return null; // unknown model: report "unknown", never guess 0
  const { inputTokens, cachedInputTokens, outputTokens } = result.usage;
  const uncached = inputTokens - cachedInputTokens;
  return (uncached * price.input + cachedInputTokens * price.cachedInput + outputTokens * price.output) / 1_000_000;
}

costUsd returns null, not zero, for an unknown model: a dashboard showing $0 looks like good news.

How to actually choose

Choose with an eval, not a vibe. Run 50 real inputs with known good outputs through two or three candidates via complete() and compare accuracy and costUsd. Because every provider returns the same shape, that script is twenty lines. AI Ch 7 covers building the eval set.

Part 8: The Failure Catalog

Build Checkpoint

Build this in a fresh folder. Try each step without looking back at the chapter first.

1. Project and secrets. The project, tsconfig.json, .env, .env.example, .gitignore and src/env.ts. Done when: git status lists .env.example but not .env, and removing OPENAI_MODEL makes the OpenAI script print your requireEnv error.

2. Four raw calls. One script per provider, printing text, raw stop reason and raw usage. Done when: an output cap of 20 tokens makes each script print its provider's "cut off" value (max_tokens, max_output_tokens, MAX_TOKENS, length).

3. Ollama and local embeddings. Pull a chat model and an embedding model; run embed.ts. Done when: both refund sentences outrank the office-hours one, and with num_ctx: 2048 a long document shows a prompt_eval_count far below its size.

4. The wrapper. Write src/llm.ts and run main.ts against each provider. Done when: all four runs end with the same result shape, and maxTokens: 20 yields stopReason: "max_tokens" everywhere.

5. Cancel. Press Ctrl+C halfway through a long answer. Done when: the process stops immediately and a log line in your SIGINT handler confirms the abort.

6. Cost. Print costUsd after every call. Done when: the printed cost of a Claude call matches your hand calculation from its usage.

7. A tiny model comparison. Ten questions with known answers through two providers via complete(). Done when: you have correct-answer count, total outputTokens and cost per provider, and can justify which you'd pick.

Vocabulary Map

Practice Problems

Problem 1 — Failure analysis. A teammate's Ollama prototype answers correctly for short documents but "ignores the system prompt" for long ones, with no error anywhere. Explain the mechanism, the response field that proves it, and two fixes with their trade-offs.

Problem 2 — Design. Add a retry policy to the wrapper: retry on 429 and 5xx, never on 400 or 401, never after the first text event. Where does the "has text been yielded?" check live, and why can't the adapters decide it themselves?

Problem 3 — Cost / Trade-off. A summarization feature sends 8,000-token documents and gets 400-token summaries, 20,000 times a day. Compute the daily cost on Opus 5.5, Sonnet 5.5 and Haiku 4.5. Then assume Opus thinks for an extra 600 tokens per request at medium effort and 150 at low. At what quality difference would you still pay for Opus at low?

Problem 4 — Go deeper. Add an optional schema field for structured outputs: Claude via output_config.format, OpenAI and Gemini via their JSON-schema options, Ollama via format. Read each SDK's types first. Which provider makes it hardest to guarantee the result parses, and what does your wrapper do when it doesn't?

Problem 5 — Connect to earlier chapters. The Frozen Oracle chapter describes the model as frozen, stateless and non-deterministic. For each property, name one line or design decision in this chapter's code that exists because of it.

What's Coming Next

You can now call four providers and get back text, a trustworthy stop reason and an honest token count. What you send is still a bare question. Chapter 2, Prompting Techniques as Code, treats the prompt as a program: system prompts, few-shot examples, chain-of-thought and structured outputs, written as typed TypeScript functions on top of complete(), and measured rather than guessed.