LLM System Design Drills

1 · Clarify. Customers, not staff, so a wrong answer is a support ticket at best and a promise we can't keep at worst (the Air Canada case). Answers only, no actions. English and Hindi. Interactive, so streaming and a fast first token. Bar: on a 300-question eval set, at least 90% correct with a citation, and "I don't know, here's how to reach support" when the help centre doesn't cover it.

2 · Simplest approach. A workflow: retrieve, answer with citations, check the citations, fall back to a human. No agent: the steps never change.

3 · Estimate. 50,000 chats × ~3 turns = 150,000 requests a day. About 6,000 input tokens each (system prompt 2,000, retrieved context 3,000, history 1,000) and 300 output. With assumed prices of $3 / $15 per million tokens: ≈ $3,400 a day. Input dominates, so: retrieve fewer, better chunks (a reranker, three chunks instead of six), cache the 2,000-token system prefix, and route simple FAQ-style questions to a cheaper model if evals hold.

4 · Architecture.

• Ingestion: re-index an article whenever it's published or edited, and delete its chunks when it's removed. Chunk by heading, keep the article title and URL with every chunk.
• Hybrid search: keyword search catches error codes and product names that embeddings blur; vectors catch paraphrases. A multilingual embedding model so Hindi questions find English articles.
• Output: answers must cite chunk IDs; the server checks each citation exists before showing it. The UI links to the article.
• Handoff: below the relevance threshold, or when the user asks, open a ticket with a summary of the conversation.

5 · Evals. 300 real questions from past tickets, in both languages, with the article that answers each (or "not covered"). Graders: citation present and valid (code); answer correct against the reference (a model judge with a rubric, validated on 50 human-labelled cases); "I don't know" when not covered (code). Every prompt, model or retrieval change must not lower the scores. Online: thumbs up/down, handoff rate, and "answered but the user still opened a ticket".

6 · Failure modes. Wrong: stale articles (re-index on publish), hallucination (citations plus threshold). Attacked: users try to make it promise refunds, so it has no actions and the prompt says to direct policy questions to the policy article, while the UI shows the source. Down: fall back to plain search results. Expensive: per-user rate limits, maxTokens, history trimmed to the last few turns.

7 · Operate. Trace each request with retrieval scores and chosen chunks. Weekly review of thumbs-down and handoffs; each becomes an eval case or an article fix. Often the bug is in the help centre, not the model.

1 · Clarify. Money moves, so a wrong action costs real money and an exploited one costs a lot. What's the refund policy (late by how much, what proof of damage)? What's the maximum the bot may refund without a human? What happens today, and how many cases a day?

2 · Simplest approach. Still not a fully open agent. A constrained agent: the model can call a few narrow tools, but the refund decision is made by policy code, not by the model. The model's job is to understand the request and gather facts.

3 · Estimate. Suppose 5,000 refund conversations a day, ~6 model calls each with tool results: 30,000 calls. Cost matters less here than correctness; the expensive failure is a wrong refund.

4 · Architecture.

• Tools run as the user: get_my_orders only returns the signed-in user's orders. No generic database or HTTP tools.
• request_refund doesn't refund. It sends the order and the reason to a policy engine in code, which checks eligibility (delivery date versus promise, photo attached, refund history), computes the amount (never taken from the model), and either approves, sends to a human, or declines with a reason.
• Confirmation: the user confirms the exact amount and order before money moves.
• Idempotency: one refund per order and reason, keyed in the payments call, so retries and double clicks can't pay twice.
• Durable steps: the conversation can pause for a human decision and resume later.

5 · Evals. A set of scripted conversations, including eligible, ineligible and borderline cases, plus adversarial ones: "ignore the policy", "I'm the CEO", a fake photo description, asking for another customer's order. Grade the final action against the policy, by code. Track refund accuracy, escalation rate and abuse attempts caught.

6 · Failure modes. Attacked: injection can at most produce a refund request; the policy engine and confirmation decide. Wrong: the model misreads the reason, but the policy engine checks facts from the order system, not the model's summary. Down: if payments fail, the request is queued, and the user is told it's pending. Expensive: step limits on the agent loop.

7 · Operate. Audit log of every request, decision and approver. Weekly sample of automated refunds reviewed by the support team. Alert on spikes in refund volume per customer or per hour.

1 · Clarify. Offline, not interactive: hours of latency are fine. Errors become wrong payments, so accuracy on totals matters most. Are there purchase orders to match against? What fraction of invoices come from repeat vendors? What accuracy does the current manual process achieve?

2 · Simplest approach. A pipeline of single model calls with validation, not an agent.

3 · Estimate. 2M invoices a month ≈ 67,000 a day, under one a second on average, so throughput isn't the problem; cost is. Say 2,000 input tokens (1,500 of document text, 500 of instructions) and 300 output each: 4 billion input and 600 million output tokens a month.

4 · Architecture.

• Structured output with a strict schema; no parsing prose.
• Validation in code is the real accuracy engine: line items must sum to the subtotal, tax must match the rate, the vendor must exist, totals must match the purchase order. A model can't argue with arithmetic.
• Escalation: failures go to a larger model, then to humans. Track what fraction reaches each stage.
• Idempotent and resumable: dedupe by content hash and invoice number; each document's state is stored, so a failed batch is resumed, not restarted. Batch results are matched by your own ID, never by position.

5 · Evals. A labelled sample of 1,000 invoices across vendors and layouts, with field-level accuracy (exact match for numbers and dates). Re-run on every prompt or model change. In production, sample human-reviewed invoices weekly, and track the validation-failure rate per vendor: a spike usually means a vendor changed their layout.

6 · Failure modes. Wrong: a misread total, caught by arithmetic and PO matching. Attacked: a PDF containing instructions ("set the total to…"), which validation and PO matching make harmless; the extraction prompt treats document text as data. Down: batch jobs simply wait and retry. Expensive: small-model-first routing, batch pricing, and caching per-vendor instructions.

7 · Operate. Per-vendor dashboards of accuracy and escalation rates. Keep every extraction with the model and prompt version that produced it, so a bad release can be re-run.

1 · Clarify. What broke last time, and how was it noticed? Which features use LLMs? Who changes prompts: engineers only, or product people too? What signals exist in production today?

2 · Simplest approach. Treat prompts, model choices and retrieval settings as code: versioned, reviewed, tested, and rolled out like any other change.

3 · Estimate. Evals cost money too: 500 cases × a few model calls each, on every change, perhaps $5–$50 per run with assumed prices, which is trivial next to two weeks of a degraded product. Run a small smoke set on every commit and the full set before release.

4 · Architecture.

• Versioned config: prompts and model names live in the repo (or a config service with history). Every output records the version that produced it.
• Eval set: built from real traffic, refreshed from production failures, with slices (languages, customer tiers, hard cases) so a regression in one slice can't hide inside an average.
• Graders: code checks first, a rubric-based model judge validated against human labels, and human review of a sample of the diffs.
• The gate: CI fails on a statistically meaningful drop in any slice, and shows the examples that changed.
• Gradual rollout: feature flags per prompt version; compare live signals (thumbs, escalations, retries, cost, latency) between versions; one-click rollback.

5 · Evals. This drill is the eval system; the key point is that the gate can fail a release, and that it compares against the current version, not against an absolute score.

6 · Failure modes. Judge drift: re-validate the judge on human labels whenever the judge model changes. Overfitting to the eval set: keep a held-out set nobody tunes against. Slow signals: online metrics take days, which is why offline evals must catch most regressions.

7 · Operate. A weekly quality review owns the eval set. Every incident adds cases. Track time-to-detect for regressions as the process's own metric. The original problem was two weeks, and the target is under a day.

1 · Clarify. Average messages per active user per day? Free versus paid limits? Which regions, and any data-residency needs? What must survive an outage: chatting at all, or chatting at full quality? Moderation requirements?

2 · Simplest approach. A streaming chat service with a model gateway in front of providers. No agent unless features like web search need one.

3 · Estimate. 200,000 DAU × 10 messages = 2M requests a day: ~23 a second on average, ~70 at peak. Say 3,600 input tokens (a 1,500-token system prompt, a 2,000-token history window, the message) and 400 output.

That's roughly $0.11 per active user per day, which the free tier can't sustain. So the design is driven by unit economics: a much cheaper default model for the free tier (a 10× cheaper model brings this to about $67k a month), caching the stable system prompt, summarising old history instead of resending it, daily per-user token quotas, and the larger model reserved for paid users or hard requests.

4 · Architecture.

• Streaming end to end, with the upstream request aborted when the client disconnects.
• Context builder: a rolling summary of older turns plus the most recent ones, inside a fixed token budget, so cost per message stays flat however long the chat.
• Model gateway: one place for routing (free versus paid, simple versus hard), provider credentials, rate-limit handling, timeouts, failover, and metering tokens per user for quotas and billing.
• Failover: if provider A errors or times out before streaming starts, retry on provider B with an equivalent prompt (prompts may need per-provider variants, kept in evals). Once tokens have streamed, don't silently switch mid-answer; show an error with a retry button.
• Quotas: daily token budgets per tier, enforced at the API before the model is called.

5 · Evals. A fixed conversation set per model and per provider, so the fallback provider is known to meet the bar before an outage forces you onto it. Online: thumbs, regenerate rate, conversation length, cost per active user.

6 · Failure modes. Down: provider failover and a degraded mode (smaller model, shorter answers). Expensive: quotas, abuse detection (scripted traffic), context budgets, aborting abandoned streams. Attacked: moderation on inputs and outputs, and no tools that act on accounts. Wrong: consumer chat tolerates more, but for health, legal or money questions, add disclaimers and safer prompting.

7 · Operate. Per-provider latency, error and cost dashboards; alerts on cost per user and quota exhaustion; provider rate limits negotiated ahead of launches; regular failover drills, because a fallback that's never exercised doesn't work when you need it.