LLM System Design: The Method

Seven steps that turn "design an AI feature" into an answer interviewers can score.

The story

The prompt: "Design an AI assistant that answers customer questions from our help centre."

The candidate starts drawing right away: a vector database, an embedding model, a large model, a chat UI. It looks like every RAG diagram on the internet. Ten minutes in, the interviewer asks three questions:

• "What happens when the answer isn't in the help centre?"
• "How would you know whether version two is better than version one?"
• "What does this cost per conversation?"

The candidate doesn't have answers to any of them, because the design started from the components instead of the problem.

LLM system design rounds aren't testing whether you know what RAG is. Everyone does. They're testing whether you can define good, measure it, pay for it, and keep it safe when the core component is probabilistic. The method below makes you answer those questions in order, before and around the architecture.

Part 1: The seven steps

A 45-minute round splits roughly: 5 minutes on steps 1–2, 5 on step 3, 15 on step 4, then 5 each on steps 5–7, plus time for questions. Steps 5 and 6 are where most candidates are weakest, and where interviewers look hardest.

Step 1: Clarify — define "good" and the cost of "wrong"

Ask before you draw. The answers change the design:

Then state the quality bar as a number you can test: "At least 90% of answers on our eval set are correct and cite a source, and the assistant says 'I don't know' rather than guessing when the help centre doesn't cover it."

Step 2: Choose the simplest approach that could work

LLM designs fail more often from too much machinery than too little. Climb this ladder only as far as the task requires:

Say why you stopped where you did. "A workflow, not an agent: the steps are always retrieve, answer, check citations. Nothing here needs the model to decide what to do next, and a fixed workflow is cheaper, faster and easier to test." Agents earn their cost when the task is open-ended and hard to specify in advance, the value justifies more cost and latency, and errors can be caught and recovered from.

Step 3: Estimate — tokens, cost, latency

Back-of-the-envelope maths (SD 7 · Estimation) works the same way for LLMs, with tokens as the unit.

Tokens per request:

input  = system prompt + tool definitions + retrieved context + history + user message
output = the reply (plus any reasoning the model does, if billed)

Cost per day:

requests/day × (input tokens × input price + output tokens × output price)

A worked example, with assumed prices (check current prices for the models you'd actually use): 150,000 requests a day, 6,000 input and 300 output tokens each, on a model priced at $3 per million input tokens and $15 per million output tokens.

The estimate tells you where to optimise: here input dominates, so the levers are trimming retrieved context (better retrieval, a reranker, fewer chunks), caching the stable prompt prefix (providers charge a small fraction of the normal input price for cached tokens), and routing easy questions to a cheaper model.

Latency has two parts users feel: time to first token (the wait before anything appears) and generation speed (tokens per second). For chat, stream the response so the first words appear quickly. For long outputs, latency is dominated by output length, so asking for shorter answers is the cheapest speed-up there is. Measure both on your chosen model; they vary a lot by model and load.

Step 4: Architecture — the four decisions

Most LLM systems are the same shape. Your job is to make four decisions explicitly:

1. Context: what does the model need to see? Retrieval (vector, keyword or hybrid search, with tenant and permission filters), tool results, a bounded slice of history. Less, more relevant context beats more context: it's cheaper, faster and usually more accurate.
2. Model: which one, and how is it prompted? Start with a capable model to establish the quality bar, then see where a cheaper or faster one holds that bar (classification, routing, extraction often do). Keep the system prompt stable so it can be cached, and version it.
3. Output: how do you know it's usable? Structured output (a schema or a strict tool definition) instead of parsing prose; check the stop reason; validate fields; require citations for factual answers; a relevance threshold below which you say "I don't know".
4. Actions: what can it do, and who approves? Narrow tools that run with the user's permissions, rules enforced in code, confirmation for anything irreversible, idempotency keys, an audit log.

Step 5: Evals — how you know it works

This is the step that separates a production engineer from a demo builder. Have a concrete answer for each layer:

• The eval set: a few hundred real inputs (from logs, support tickets or experts), with reference answers or grading criteria, covering the hard cases: out-of-scope questions, other languages, attempts to manipulate the assistant. Never the examples in your prompt.
• Graders: exact or code checks where possible (valid JSON, cites a real source, refuses when it should); a model judge with a written rubric for open-ended quality, checked against human labels before you trust it; humans for a sample.
• Release gate: every prompt, model or retrieval change runs the evals; the change ships only if scores don't drop.
• Online: user feedback, escalation rate, "I don't know" rate, cost and latency per request. Every production failure becomes a new eval case.

Step 6: Failure modes — wrong, attacked, down, expensive

Walk through the four, briefly, with your mitigation for each:

Step 7: Operate — observe, version, improve

• Trace every request: each model call, retrieval and tool call, with inputs (redacted), outputs, tokens, cost and latency.
• Version everything that changes behaviour: prompt, model, retrieval settings, tool definitions. Store the versions with every output.
• Roll out gradually: shadow traffic or a small percentage first, compare against the current version on live metrics, keep a fast rollback.
• Close the loop: review failures weekly, add them to the eval set, fix, re-run, ship.

Cheat sheet

Say it in the interview

"Before the architecture: a wrong answer here means a customer acts on a policy we don't have, so I'd set the bar at grounded answers with citations and an explicit 'I don't know' below a relevance threshold. I'd build a workflow rather than an agent: retrieve, answer, verify citations. At 150k requests a day input tokens dominate the cost, so retrieval quality and prompt caching are the first levers. And nothing ships without a few hundred real questions in an eval set gating every prompt or model change."

Go deeper

• AI Engineering · Evals and From Demo to Production.
• GenAI with JS · Advanced RAG and Production Ingestion.
• SD 2 · The System Design Framework: the general version of this method.
• Next: LLM System Design Drills.