The LLM Bug Catalogue
The bugs only LLM apps have, graded by what you must imagine to see them, and mapped to the OWASP Top 10 for LLM Applications.
The story
Two incidents get quoted in almost every AI engineering interview loop, because they're short and they're real.
In December 2023, a car dealership in Watsonville, California put a chatbot on its website. Visitors quickly found they could give it new instructions in the chat. One told it to agree with anything the customer said and to end every reply with "and that's a legally binding offer". It then "agreed" to sell a new Chevy Tahoe for one dollar. The screenshots went viral, and the dealership took the bot down.
In February 2024, a Canadian tribunal ruled against Air Canada in Moffatt v. Air Canada. The airline's website chatbot had told a grieving customer he could apply for a bereavement discount retroactively, which wasn't the airline's policy. Air Canada argued that the chatbot was responsible for its own words. The tribunal disagreed and ordered the airline to pay.
Neither was a crash or an exception. In one, the model followed instructions from the wrong person. In the other, the model made up a fact and the product presented it as authoritative. Both are design bugs, and both are visible in code review if you know what to look for. This catalogue is that list.
How to read this catalogue
Bugs keep the six levels from Bug Hunt: what you must imagine to see the bug. Each row also gives:
- OWASP: the matching category in the OWASP Top 10 for LLM Applications (2025 edition): LLM01 Prompt Injection, LLM02 Sensitive Information Disclosure, LLM03 Supply Chain, LLM04 Data and Model Poisoning, LLM05 Improper Output Handling, LLM06 Excessive Agency, LLM07 System Prompt Leakage, LLM08 Vector and Embedding Weaknesses, LLM09 Misinformation, LLM10 Unbounded Consumption. Naming the category in an interview shows you know the field's vocabulary.
- Who: M if an engineer with a few years' experience building LLM features should catch it, S if it's senior-level.
- Signal: what you'll see in the code.
L1 · The line
| Bug | OWASP | Who | Signal |
|---|---|---|---|
| No output cap | LLM10 | M | A model call with no maxTokens / max_tokens (or a huge one) on a user-facing path |
JSON.parse on model text | — | M | Parsing the reply as JSON with no schema check and no handling for prose around the JSON |
| Secrets in the prompt | LLM07 | M | API keys, internal codes, connection strings or other users' data written into the system prompt |
| User text inside the system prompt | LLM01 | M | system: `…Customer name: ${req.body.name}…`: user input placed with the operator's instructions |
| Temperature or sampling set where the API rejects it | — | M | Parameters that newer models no longer accept; check the provider's current docs before carrying old settings forward |
| Logging full prompts | LLM02 | M | logger.info({ prompt }) with user messages, documents and PII |
L2 · The function
| Bug | OWASP | Who | Signal |
|---|---|---|---|
| Ignoring why generation stopped | — | M | Using the output without checking the stop reason: max_tokens means truncated, refusal means declined, tool_use means it wants a tool |
| Truncated JSON treated as complete | — | M | No check for max_tokens, so half an object is parsed (or crashes) |
| Refusals shown as answers | LLM09 | M | A declined request rendered as if it were the product's answer |
| Retry-until-it-parses with no limit | LLM10 | M | while (!valid) callModel(): costs without bound when the model can't satisfy the schema |
| Regex over model prose | — | M | Extracting amounts, IDs or decisions from free text instead of structured output |
| String-matching tool arguments | — | M | Comparing serialised tool input as a string; escaping can differ between calls, so parse it as JSON |
| Fields trusted without validation | LLM05 | M | Using refundAmount, userId or url from model output without range, ownership or allow-list checks |
| Tool errors thrown, not returned | — | M | A failing tool crashes the agent loop instead of being returned to the model as an error result |
L3 · The data's journey
| Bug | OWASP | Who | Signal |
|---|---|---|---|
| Retrieval without a tenant filter | LLM08 | M | Vector search over every customer's documents, filtered (if at all) after retrieval |
| Deleted documents still retrieved | LLM08, LLM02 | M | Deleting a document removes it from the database but not from the vector index |
| Query and index embedded with different models | LLM08 | M | embed(query, 'model-b') against an index built with model-a: similarity scores become meaningless |
| No relevance threshold | LLM09 | M | Top-k always fed to the model, even when nothing relevant was found, so it answers from noise |
| No grounding or citations | LLM09 | M | Answers presented as policy with no source the user (or you) can check |
| Conversation history keyed only by conversation ID | LLM02 | M | Anyone with an ID can load or append to someone else's chat |
| Response cache key missing inputs | LLM02 | M | Caching by question text alone: different users, permissions, models or prompt versions share answers |
| Prompt cache defeated by volatile prefix | LLM10 | M/S | A timestamp or request ID at the start of the system prompt changes the prefix every call, so provider-side prompt caching never hits |
| Batch results assumed in order | — | M | Matching batch outputs to inputs by position; results can arrive in any order, so key them by your own ID |
L4 · Two things at once (and time)
| Bug | OWASP | Who | Signal |
|---|---|---|---|
| Agent loop with no step limit | LLM10 | M | while (stopReason === 'tool_use') with no maximum iterations or budget |
| Retries multiplied across layers | LLM10 | M | Your retry loop around an SDK that already retries (the official SDKs retry twice by default), around a queue that retries too |
| Client disconnect doesn't stop generation | LLM10 | M | A streaming endpoint keeps paying for tokens after the browser has gone |
| Timeouts too short for long generations | — | M | A 10-second timeout on a non-streaming call that routinely takes 30 seconds, then retried |
| Parallel tool results split up | — | M/S | Results for several tool calls sent back as separate messages instead of together, which breaks or degrades the tool-use protocol |
| Unbounded conversation history | LLM10 | M | Every turn appended forever: cost and latency grow with each message until the context limit fails the request |
| Provider failover without idempotency | — | S | Retrying an action-taking agent on a second provider after a timeout, so the action happens twice |
L5 · An attacker
| Bug | OWASP | Who | Signal |
|---|---|---|---|
| Direct prompt injection | LLM01 | M | Relying on the system prompt to enforce rules ("never give discounts above 10%") instead of code |
| Indirect prompt injection | LLM01 | M/S | Emails, web pages, PDFs or tool results placed in the prompt where their text can act as instructions |
Model output into SQL, shell or eval | LLM05 | M | Executing generated SQL or code with full privileges |
| Model output rendered as HTML | LLM05 | M | dangerouslySetInnerHTML / marked() output on model text: XSS |
| Data exfiltration through rendered links and images | LLM05, LLM02 | S | Rendering model-written Markdown images or links: an injected instruction can make the model write , and the browser sends the data when it loads the image |
| Fetching URLs the model chose | LLM05 | M | SSRF through a "browse" or "fetch" tool with no host allow-list |
| Tools more powerful than the task | LLM06 | M | A support bot with a generic run_sql or http_request tool |
| Tools without the user's permissions | LLM06 | M | Tools run with a service account, so the agent can act on any customer's data |
| Irreversible actions without confirmation | LLM06 | M | Sending email, refunding, deleting, all without a human approval step |
| System prompt treated as secret | LLM07 | M | Security depending on users not seeing the prompt; assume it will leak |
L6 · Production
| Bug | OWASP | Who | Signal |
|---|---|---|---|
| No per-user or per-tenant quota | LLM10 | M | One user (or script) can run up any bill |
| No cost or token tracking | LLM10 | M | Usage from responses discarded; nobody can say what a feature costs per request |
| No evals | LLM09 | M | Prompt or model changes shipped with no regression set |
| Tests that call the live model | — | M | Flaky, slow, costly CI; tests that pass or fail by chance |
| Exact-string assertions on model output | — | M | expect(reply).toBe('…') on generated text |
| Eval set overlaps few-shot examples | LLM09 | S | The model is graded on examples it was shown in the prompt |
| Judge model with no rubric | LLM09 | S | "Is this answer good? yes/no" judged by the same model, with no criteria and no human-labelled check |
| No fallback for provider outages | — | M | One provider, no degradation path, no user-facing message |
| Model or prompt version not recorded | — | M | Logs and stored outputs don't say which model and prompt produced them |
| Untrusted model or package sources | LLM03 | S | Downloading models, adapters or prompt packs without pinning or verification |
| Feedback loops poisoning data | LLM04 | S | User content or model outputs fed back into retrieval or fine-tuning data without review |
The rules that prevent most of these
| Rule | Prevents |
|---|---|
| Instructions come from you; everything else is data. Keep user input and retrieved content out of the system prompt, clearly delimited and labelled untrusted. | Most injection |
| Enforce rules in code, not in the prompt. Limits, permissions and policies are checked when the action runs. | Injection succeeding, the $1 car |
| Treat model output like user input. Validate, escape, parameterise, allow-list. | XSS, SQL injection, SSRF, exfiltration |
| Give tools the least power, with the user's permissions, and confirmation for irreversible actions. | Excessive agency |
| Check the stop reason, validate structure, bound every loop and every retry. | Truncation, refusals, runaway costs |
| Ground answers in sources you control, show them, and say "I don't know" below a relevance threshold. | Air Canada |
| Measure: evals before every change, usage and cost per request, model and prompt version on every output. | Silent regressions |
Say it in the interview
"The system prompt says 'never offer more than 10% off', but the discount tool accepts any percentage, so a prompt injection, direct or from a pasted email, gets a 100% discount. Rules have to live in the tool: the code should cap the discount and check the user's entitlement whatever the model asks for. The prompt is a hint, not a control."
Go deeper
- AI Engineering · Your Agent Is an Attack Surface: injection, agency and guardrails in depth.
- AI Engineering · Evals: building the regression set every change needs.
- Bug Hunt · L5 Security: the classic web bugs that LLM output can trigger.
- Next: LLM Bug Hunt Drills.