Chapter 1: The Frozen Oracle

What an LLM Actually Is and Why It Behaves the Way It Does

By the end of this chapter, you will be able to explain — from physical first principles — why a trillion-parameter model confidently states a wrong fact, why the same question gives a different answer on Tuesday than it did on Monday, why a model that aces a bar exam still stumbles on "how many r's are in strawberry," and why none of these are bugs. They are the inevitable, logical consequences of what these systems physically are. Get that model in your head once, and almost every AI failure you encounter in production will have an obvious root cause — and more importantly, an obvious class of fix.

Part 1: The Machine Doesn't Read Words — It Reads Numbers

Pick up any sentence — say, "the deployment failed at midnight." To you, those are words with meaning. To the model, they are none of those things. A neural network is a chain of matrix multiplications. It can only process numbers. So before a single computation happens, every piece of text you send to a model has to be converted into a sequence of integers.

The conversion works like this. There is a fixed vocabulary of chunks — not words, not characters, but chunks — built ahead of time by analyzing a massive corpus of text. Typical vocabularies contain between 32,000 and 128,000 of these chunks. Each chunk gets assigned an integer ID. When you send the string "deployment failed," the model's tokenizer looks up the IDs for each chunk it finds in that string, and what actually enters the model is something like [배포, ment, _fail, ed] mapped to [14235, 972, 8404, 292]. Those integers are what the model processes. The words, as linguistic objects, are gone.

These chunks are called tokens, and the process is called tokenization. The chunking algorithm that produces them is called Byte-Pair Encoding (BPE). Here is the short version of how it works: start by treating every individual byte as its own token. Then, repeatedly find the two adjacent tokens that appear most frequently together in the training corpus and merge them into a single new token. Repeat until you have reached your target vocabulary size. What you end up with is a vocabulary where common English words are usually single tokens ("failed" → one token), rare words get broken up ("tokenization" → maybe ["token", "ization"]), and very rare strings get broken down to individual bytes.

This has one consequence that will matter to you immediately: the model never sees individual characters unless they happen to be their own tokens. Consider the word "strawberry." Depending on the tokenizer, it might be split as ["straw", "berry"], or ["st", "rawberry"], or something else — but crucially, not as ten individual letters. When someone asks the model "how many r's are in strawberry?" — a question that you as a human answer by visually scanning the letters — the model has no letters to scan. It has to reconstruct the answer from the token representations it learned during training, which captured the word's meaning and typical context, not its spelling. It gets this wrong with surprising regularity. This is not a reasoning failure. It is a tokenization consequence.

The same physical fact explains why models struggle with arithmetic on numbers they haven't seen in their training data. The number 9.11 might be tokenized as ["9", ".", "1", "1"] or even as a single token ["9.11"]. Either way, the model did not learn arithmetic from first principles — it learned statistical patterns over sequences of tokens. The string "9.11" appeared in training data in contexts like software versioning, calendar dates, and event references — often in comparison to "9.9" — in ways that don't map cleanly to "which floating point number is larger." The model's "9.9 < 9.11" error is not stupidity; it's the model pattern-matching on the wrong distribution.

Once text is tokenized and converted to integers, each integer is used to look up a vector from the model's embedding table — a large matrix where each row corresponds to one token in the vocabulary. This vector is a list of hundreds or thousands of floating-point numbers. It encodes, in a geometric sense, what that token "means" relative to every other token. The model's input is not a sequence of words. It is a sequence of these embedding vectors — a matrix of floating point numbers. This is what actually gets processed.

The number of tokens the model can process in a single call is called the context window or context length. GPT-4 launched with an 8,000-token context. Models in 2024 and 2025 pushed this to 128,000, 200,000, and beyond. This number matters for two reasons you need to internalize now: first, it is a hard limit — send more tokens than the context window allows and the call fails. Second, it is the unit of cost. Every API you call charges you per token — input tokens, output tokens, or both. A long document, a long conversation, a detailed system prompt — they all cost tokens.

Part 2: The Whole Game Is Next-Token Prediction

You now know how text enters the model. Here is what the model actually does with it.

Given a sequence of token embeddings as input, the model outputs a probability distribution over every token in its vocabulary. That is it. The model says: "given everything I've seen so far in this sequence, here is the probability that each of the 100,000 tokens in my vocabulary is the right next one." The token "Paris" might have a 94% probability. The token "Lyon" might have 2%. Everything else gets the remaining 4% divided among 99,998 tokens.

A single token is sampled from this distribution. That sampled token gets appended to the input sequence. The whole thing runs again — now with a one-token-longer input — to produce the probability distribution for the token after that. And again, and again, until the model produces a special end-of-sequence token or hits a maximum length limit.

This is autoregressive generation: generating one token at a time, each token conditioned on all previous tokens. Everything a language model has ever said to you — every line of code, every essay, every hallucinated citation — was produced by this loop, running hundreds or thousands of times per response.

Read that again until the implication lands. The model is not "answering your question." It is not "looking things up." It is not "reasoning" in the way you reason when you solve a math problem step by step. It is, at every single step, asking one question: given this sequence of tokens, what token most plausibly comes next? The answer that emerges from thousands of iterations of that one question can look indistinguishable from genuine reasoning — and sometimes it is, in a functional sense. But it emerges from next-token prediction over patterns learned from an enormous corpus of human text. Nothing more, nothing less.

Why does this matter for you as an engineer? Because it tells you exactly what you can rely on and what you cannot. If the model was trained on millions of examples of text where "the capital of France is" is followed by "Paris", then the probability of "Paris" next is very high and the model will be right virtually every time. If you ask about the CEO of a company as of three months ago, the training data may not contain that fact, the probability distribution may be diffuse or wrong, and the model has no mechanism to detect its own uncertainty other than what it learned to say in uncertain contexts.

The model's knowledge is frozen at the point its training data was collected. Every fact it produces is an echo of its training corpus. It cannot look anything up. It cannot update its weights mid-conversation. It is, in this sense, an oracle: it knows a great deal, it was frozen at a point in time, and it will answer you with great confidence whether or not it has the information to do so accurately.

Part 3: Why It's Random — and Why That's a Design Choice

The model produces a probability distribution. You'd be forgiven for thinking: why not just always pick the most probable token? That would make it deterministic, predictable, consistent. The answer is that it would also make it terrible.

Here is what happens when you always pick the most probable next token: you get repetitive, boring, and eventually degenerate text. The probability distribution for the most common token is so dominant in many contexts that greedy selection collapses into loops — "the the the the" — or produces text that always takes the safest, most formulaic path. Sampling from the distribution rather than always picking the peak produces variety, novelty, and outputs that feel alive. This is not a hack. It is a fundamental design choice baked into how these systems are used.

The raw scores the model produces before converting to probabilities are called logits — one per vocabulary token, any real number, no constraint on range. To turn them into probabilities that sum to 1.0, the model applies the softmax function: exponentiate every logit, then divide each by the sum of all exponentiated logits. The largest logit wins the most probability, but every token gets some.

Now enter temperature, which is the single most important sampling parameter you will control. Before applying softmax, every logit is divided by the temperature value. At temperature 1.0, logits go in unchanged. At temperature 0.5, all logits are doubled in relative magnitude — the distribution sharpens, the highest-probability tokens dominate more, the output becomes more predictable. At temperature 2.0, all logits are halved — the distribution flattens, low-probability tokens get relatively more likely, the output becomes more creative and random. Temperature approaching 0 approaches the greedy decoding case — pick the top token every time.

Logits (raw):    [3.2,  1.1,  0.8,  -0.5,  ...]
                          ↓ divide by temperature
T=0.5:           [6.4,  2.2,  1.6,  -1.0,  ...]  → sharp, deterministic
T=1.0:           [3.2,  1.1,  0.8,  -0.5,  ...]  → default
T=2.0:           [1.6,  0.55, 0.4,  -0.25, ...]  → flat, creative
                          ↓ softmax
Probabilities:   [0.94, 0.03, 0.02, 0.01,  ...]  (approximate, T=0.5)

This is a tool you will control constantly. For code generation, JSON extraction, and any task where correctness matters more than creativity, lower temperatures (0.0–0.3) reduce variance. For brainstorming, story generation, and any task where novel outputs are the goal, higher temperatures (0.7–1.2) help. For evaluation and testing, you want low or zero temperature so outputs are reproducible.

There is one other sampling parameter you will use: top-p (nucleus sampling). Instead of sampling from the full vocabulary, restrict sampling to only the smallest set of tokens whose cumulative probability reaches threshold p. At top-p 0.9, you find the minimum set of tokens that together account for 90% of the probability mass, and only sample from those. This cuts the long tail of improbable tokens while still allowing variety among the plausible ones. Temperature and top-p are usually used together.

The practical consequence of all this: the same prompt, run twice, can produce different outputs. Not because the model is broken. Because it is sampling from a probability distribution. If you are building systems that need reliability — consistent structured outputs, deterministic behavior — you have to engineer for this stochasticity. You cannot assume repetition is safety. This will become the central problem of Chapter 2.

Part 4: Attention — How the Model Holds a Conversation

Before the transformer architecture arrived in 2017, sequence models processed text token by token, left to right, carrying a fixed-size "hidden state" that tried to encode everything seen so far. The problem: after 200 tokens, the hidden state had to somehow encode both "the user's name mentioned in token 3" and "the question asked in token 198." It couldn't do both well. Long-range dependencies — referring back to something said much earlier — broke these models.

The transformer solved this by abandoning the sequential hidden state entirely. Instead, at every layer of the network, every token looks at every other token and computes a relevance score: how much should this token's representation be influenced by that one? These scores are learned during training. A token representing "it" might learn to attend heavily to the nearest preceding noun. A token at the end of a long code block might learn to attend to the function definition at the beginning. This mechanism — computing pairwise relevance scores across the full input sequence — is called attention.

You do not need to understand the math of attention to use LLMs effectively. You need to understand two practical consequences.

First: attention is quadratic in the context length. To compute attention, every token attends to every other token. For a sequence of N tokens, that is N² attention computations. Double the context length, roughly quadruple the compute (and therefore the cost and latency). This is why long-context models are expensive to run and why there is active research into approximating attention more efficiently. It also sets a ceiling on how large a context window can get before the economics become untenable.

Second: the model does not attend equally to all positions. Research has consistently shown that language models exhibit what is called the lost-in-the-middle problem: information placed at the very beginning or very end of the context is retrieved more reliably than information placed in the middle. Put your most important instructions or context near the start or end of your prompt. If you have a 50-page document and a specific question about one section, putting the question after the document works better than burying it between pages 20 and 30. This is not a quirk — it is an architectural consequence you can design around once you know it exists.

Part 5: Why It Hallucinates — the Inevitable Consequence

You now have everything you need to understand hallucination from the ground up.

At every step of generation, the model samples from a probability distribution over its vocabulary. That distribution is the result of billions of parameters that were trained to make accurate, coherent text statistically likely — not to represent ground truth. The model has no internal truth-checker. It has no lookup table it queries before speaking. It has no mechanism that says "I am about to say something I don't have reliable training data on, so I'll stay quiet." What it has is a distribution over next tokens, produced by those billions of parameters, and that distribution is more or less peaked depending on how well-represented the relevant fact was in training.

For the question "what is the capital of France," the answer "Paris" was overwhelmingly common in training data. The probability of "Paris" next is near-certain. The model cannot get this wrong in normal conditions. For the question "what did the CEO of CompanyX say in their Q3 earnings call last month," no relevant training data exists (it may postdate the training cutoff, or the call may simply not have been in the corpus). The model produces a probability distribution regardless — it has to; that is its only output — and that distribution will be shaped by patterns about earnings calls in general, about CompanyX if it has any training data on it, and about what kinds of statements CEOs typically make. The output will sound like a plausible earnings call statement. It may be entirely fabricated. The model has no way to flag the difference.

This is not a bug that will be fixed in the next model version. It is a structural property of a system that produces text by sampling from learned distributions rather than by querying a ground-truth source. The fix is architectural — you bring ground truth into the context (which is what retrieval and RAG exist to do) or you verify outputs with a separate checking step (which is part of what evals exist to do). But the underlying model will always hallucinate under the right conditions.

One more piece of this that trips up engineers: hallucination is the price of fluency. A model that only ever said things it was certain about would be nearly useless — almost everything would be "I'm not sure." The model's fluency, its ability to produce coherent and contextually relevant text, comes from the same statistical machinery that causes hallucination. The two are inseparable. You engineer around hallucination with retrieval and validation. You don't eliminate it.

Part 6: The Context Window — Finite, Stateless, Expensive

Every API call to a language model is stateless. The model has no memory of your previous call. When you send a message, you are sending the entire relevant history of the conversation — not just your new message, but all previous turns — as a single chunk of tokens. The chat interface you use every day achieves its appearance of memory by prepending all previous messages to your new one every time you send a request. The model itself remembers nothing.

If you have built HTTP APIs, you understand this immediately: it is exactly like stateless REST. Every request must carry all the state it needs. The server holds nothing between requests. The difference is that in a language model, the "request" can be 100,000 tokens long, and every additional token in the conversation history makes the next call more expensive and slower.

The maximum number of tokens the model can receive in a single call is its context window. It is a hard limit imposed by the attention mechanism — extending it requires architectural changes and significant compute. Current frontier models (mid-2026) range from 128,000 to well over 1,000,000 tokens in context. Even so, this limit matters in practice because:

1. Very long inputs approach the limit and incur high cost.
2. Information placed deep in the middle of long contexts is attended to less reliably (the lost-in-the-middle problem from Part 4).
3. Prompt caching — a cost optimization you will use in production — only works on context that stays constant across calls.

The statelessness has another consequence that will drive major architectural decisions later: agents and multi-turn systems must explicitly manage what goes into context. If an agent runs a 50-step workflow, it cannot naively accumulate every intermediate result — the context fills up and the cost explodes. The decisions about what to remember, what to compress, and what to discard become engineering problems. They are solved with memory architectures and context management strategies, which you'll build in Chapter 6.

Think of the context window as the model's working memory — expensive, limited, and completely cleared between tasks unless you reload it explicitly. A human engineer solving a problem has a short-term working memory and access to long-term memory and external references. A language model has only its context window for working memory and only its trained parameters for "long-term knowledge." Everything else has to be injected at call time.

Part 7: Where the Personality Comes From — Pretraining and RLHF

A language model arrives at its usefulness in two phases, and the distinction matters for how you work with it.

Pretraining is the bulk of the work. Take a corpus of text — the public internet, books, code, papers, measured in terabytes — and train a neural network to predict the next token across all of it. This runs for weeks on thousands of GPUs at costs in the millions of dollars. What you get at the end is a base model that can generate coherent text in any domain it saw in training. It is also a base model that will complete any prompt in whatever style the training data suggests — including harmful, biased, or incoherent completions. It follows instructions poorly. It has no values. It is a raw next-token predictor of extraordinary breadth.

Post-training — primarily through a technique called RLHF (Reinforcement Learning from Human Feedback) — is what turns the base model into a useful assistant. Human raters evaluate model outputs and indicate which are better. A separate model (the reward model) is trained on these preferences. The base model is then fine-tuned using reinforcement learning to produce completions the reward model scores highly. The result: a model that follows instructions, refuses harmful requests, maintains consistent values, and responds in the characteristic style of the company that trained it.

This is where personality comes from. OpenAI, Anthropic, and Google trained different RLHF pipelines with different human raters operating under different guidelines and reward criteria. Claude, GPT-4, and Gemini are all transformer-based next-token predictors trained on overlapping corpora — yet they feel meaningfully different to interact with. That difference lives almost entirely in the post-training.

For you as an engineer, there are three practical takeaways. First, the model's "values" and behaviors are not fundamental properties — they are the result of a training process, which means they can be partially overridden or reinforced via the system prompt and careful prompting. You cannot fully override them, but you have meaningful leverage over tone, persona, and response style. Second, refusals are completions — the model was trained to refuse certain requests, meaning a refusal is just a very high-probability completion type for a certain class of inputs. There is nothing magical about it. Third, when the model says "as an AI, I cannot..." — that phrasing is itself a training artifact, a common completion pattern in the RLHF stage. The model is not reporting a physical limitation; it is completing a token sequence it was trained to produce.

Part 8: Four Properties — Your Complete Debugging Framework

You now have everything you need for a complete mental model. Every language model behavior you will encounter in production — every failure, every surprise, every design constraint — can be traced to one or more of these four properties:

Frozen. The model's knowledge was baked in during training. It cannot learn from your query. It cannot look things up. It has no access to events that occurred after its training cutoff. Anything it says about the current state of the world is a statistical best guess from frozen data.

Stateless. The model remembers nothing between API calls. Every call is independent. Memory is an illusion created by injecting conversation history into the context. This means that long-running interactions accumulate token cost, that agents must explicitly manage what goes into context, and that any "memory" you want to persist across sessions must be stored externally and re-injected.

Stochastic. The model samples from a probability distribution at every token. The same input can produce different outputs on different runs. Determinism is an approximation, not a guarantee. Systems that require reliability must be engineered for this — not by hoping for consistency, but by enforcing it through structured outputs, validation, and evals.

Predictive. The model is always doing one thing: predicting the most plausible next token. It is not "answering" in the way a database answers. It is not "reasoning" in the way you reason through a logic puzzle. It is completing a sequence. This completion machinery can produce things that look like reasoning, step-by-step problem solving, and sophisticated inference — and often those outputs are functionally correct. But the mechanism is completion, not deduction, and that distinction matters when you are diagnosing why it fails.

When your agent loops unexpectedly — frozen or stochastic, probably stochastic with insufficient enforcement. When the model ignores an instruction you gave it earlier in a long conversation — stateless, probably; the instruction got pushed to the middle of a long context and is underweighted. When the model gives you a confident but wrong answer about a recent event — frozen. When your structured-output parsing breaks intermittently — stochastic, not properly constrained.

Use these four as your first diagnostic pass. They will point you to the right class of fix in seconds.

Vocabulary Map

Practice Problems

Problem 1 — Failure Analysis

A user reports that your AI assistant confidently told them that Python's list.sort() method returns a sorted copy of the list. This is wrong — sort() sorts in place and returns None. Your assistant did not hedge. It stated this as fact.

Trace the failure to its root cause in the model's architecture. Use the vocabulary from this chapter. Then answer: is this a failure you could prevent by switching to a larger, more capable model? Why or why not?

Problem 2 — Design Under a Constraint

You are building an internal tool that answers questions about your company's live database schema. The schema changes monthly. The tool must never hallucinate a column name or table name — a wrong answer here causes engineers to write broken queries.

You cannot fine-tune the model. You cannot retrain it. You have only the model's API, a context window, and whatever engineering you can do around it.

What are the minimal architectural decisions you must make? Ground each decision in a specific property from Part 8.

(This problem is pointing you at something. You'll build the real version in Chapter 4.)

Problem 3 — Synthesis

Connect tokenization, the context window, and API cost into a single coherent mental model. Then apply it to this scenario: a user wants to build a customer support bot that handles incoming support tickets averaging 2,000 words each. Each ticket may require 3–5 back-and-forth turns before resolution. The bot must remember all previous turns in the conversation.

What specific risks does this product face, where do they come from physically, and what numbers would you need to estimate before committing to the architecture?

Problem 4 — Go Deeper

A language model is run with temperature set to 0 — greedy decoding: always pick the highest-probability next token. A developer claims this makes the output "deterministic" and uses it to skip testing on a fixed prompt, assuming the output will never change.

Is the output actually guaranteed to be identical on every run? Argue from the ground up. Do not look this up — reason through it. Consider what you know about floating-point arithmetic, hardware, and distributed systems. What assumptions would need to hold for true determinism? Are those assumptions realistic in a production cloud API?

(This goes beyond what the chapter explicitly stated. That is the point.)

Problem 5 — Looking Forward

You need to build a system where the model always outputs a valid, parseable JSON object — never raw prose, never a JSON object with a missing closing brace, never a Markdown code block around it. Based on what you now know about how the model generates text token by token, explain specifically why this is hard to guarantee. Then sketch, in one paragraph, what a solution would need to address at its core.

(This is setting up the central problem of Chapter 2.)

What's Coming Next

In Chapter 2, you will take everything you learned about why the model is stochastic, frozen, and predictive, and turn it into an engineering discipline: prompting as programming. The challenge is this — you now have a system that is fundamentally probabilistic, that can produce subtly different outputs on identical inputs, and that has no mechanism to know when it is wrong. You need to make it behave reliably in production. The techniques you'll learn are not tricks or heuristics; they are principled responses to the specific physical properties you learned in this chapter. Chain-of-thought, few-shot examples, structured outputs, and regression harnesses all exist because of the four properties you can now name. Understanding why they work — not just how to use them — is what you'll get in Chapter 2.