⚠️ Common Issues 🪲 with LLMs & AI Agents 🤖 - and How to Fix Them 🛠️
Part A - Model-level issues
1. 🎭 Hallucination & confident fabrication
What goes wrong: The model invents facts, citations, API methods, file paths, or function signatures - and states them with total confidence. This is the #1 trust-killer.
Why it happens: An LLM is trained to produce plausible continuations, not true ones. When it lacks the fact, "make something plausible up" and "say the true thing" look identical from the inside. It has no built-in "I don't actually know" signal.
How to fix it:
| Technique | What it does |
|---|---|
| Ground with retrieval (RAG) | Put the real source text in context and instruct "answer only from the provided documents; if it's not there, say so." Removes the need to fabricate. |
| Cite-or-abstain | Require an inline citation (doc ID, URL, line number) for every claim. No citation → don't say it. Makes fabrication auditable. |
| Verify against ground truth | For code: run it, compile it, run tests. For data: query the DB. Let the environment be the fact-checker, not the model. |
| Constrain the output | Structured outputs / JSON schema / enums stop the model from inventing free-form values. Lower the temperature for factual tasks; raise it only for creative ones. |
| Ask for confidence + let it say "I don't know" | Explicitly permit and reward abstention in the prompt. Models will over-answer if the prompt implies an answer is mandatory. |
| Second-model check | An independent "critic" pass ("does every claim here appear in the sources?") catches a large fraction of fabrications. |
⚠️ Do not rely on the model to "double-check itself" in the same turn - it will often confidently re-confirm its own mistake. Verification must come from an external source (tools, sources, a fresh call).
✅ Checklist: grounded in real sources · citations required · output constrained · environment verifies · abstention allowed.
2. 🗓️ Stale knowledge & the training cutoff
What goes wrong: The model confidently uses a deprecated API, an old library version, last year's pricing, or a framework that has since changed. It doesn't know today's date or your codebase.
Why it happens: Its parametric knowledge is frozen at the training cutoff. Anything after that - or anything private - simply isn't in there.
How to fix it: Give it fresh eyes. Web search / retrieval tools for current facts; file-reading tools for your actual code. Don't let it answer from memory when the truth is one tool call away.
Inject "now." Put the current date, library versions, and environment facts directly in the system prompt.
Prefer "just-in-time" context. Instead of dumping a giant knowledge base up front, give the agent lightweight references (file paths, URLs, query handles) and let it pull the current content at runtime. This also sidesteps stale indexes.
Pin versions in the prompt. "We use React 19, Go 1.23, Pydantic v2" prevents the model from defaulting to whatever was most common in training data.
✅ Checklist: current date injected · versions pinned · retrieval/tools available for anything time-sensitive.
3. 🎲 Non-determinism & inconsistency
What goes wrong: The same input produces different outputs. A prompt that worked yesterday fails today. Tests are flaky.
Why it happens: Sampling is probabilistic. Even at temperature 0 you can see variation from batching, hardware, and provider-side changes.
How to fix it:
- Turn down randomness where you need stability: temperature 0 (or near it), fix a seed if the provider supports it.
- Constrain the output space: structured outputs, enums, and schemas collapse many possible phrasings into a few valid ones.
- Make the system deterministic even if the model isn't: validate, retry on invalid output, and use programmatic gates between steps rather than trusting free-form text.
- Test statistically, not on single runs: run each eval case N times and track a pass rate, not a single pass/fail.
- Treat the model as a flaky dependency and engineer around it.
- Idempotency for actions: design tool calls so that a repeat (from a retry) doesn't double-charge, double-send, or double-write.
✅ Checklist: temp/seed pinned · outputs schema-validated · evals run N× · actions idempotent.
4. 📉 Context rot: long contexts quietly degrade
What goes wrong: You give the model a huge context ("it has a 1M window, just put everything in!") and quality silently drops - it forgets the middle, misses the instruction, or loses the thread.
Why it happens: This is context rot: as token count grows, recall and reasoning precision decline. Transformer attention is n² pairwise, and models saw far more short sequences than long ones in training. Attention is a finite budget - every extra token depletes it. It's a gradient, not a cliff, but it's real across all models. See Anthropic's context engineering guide 🧠 for the deep dive.
How to fix it - treat context as a scarce resource, not free storage:
| Technique | When to use |
|---|---|
| Curate, don't dump | Find the smallest set of high-signal tokens. More is not better. Remove redundant tool output, boilerplate, and dead ends. |
| Compaction | Nearing the window limit? Summarize the conversation so far into a compact brief (preserve decisions, open bugs, key files) and continue in a fresh window. |
| Tool-result clearing | Once a tool result deep in history has served its purpose, strip the raw payload - keep the conclusion. |
| Structured note-taking | Have the agent write progress/decisions to an external NOTES.md (or memory tool) and re-read on demand. Persistent memory outside the window. |
| Sub-agents for exploration | Spin off a clean-context sub-agent to do a big search, and return only a 1-2k-token distilled summary to the main agent. |
| Just-in-time retrieval | Keep references, load content only when needed. |
✅ Checklist: context kept tight · compaction wired up for long tasks · notes persisted externally · raw tool dumps pruned.
5. 🧩 Prompt sensitivity & brittleness
What goes wrong: Tiny wording changes swing behavior. Your prompt is a 600-line pile of "ALWAYS do X", "NEVER do Y", edge-case after edge-case - and it's still fragile.
Why it happens: Two failure modes at opposite extremes: over-specified brittle if-else prompts that break on anything unforeseen, and under-specified vague prompts that assume shared context the model doesn't have.
How to fix it - aim for the "right altitude":
- Be specific enough to guide, flexible enough to generalize. Give strong heuristics, not a brittle decision tree.
- Structure the prompt: clear sections (
<background>,<instructions>,## Tools,## Output) with headings or XML tags. - Use few canonical examples, not a laundry list of every edge case. For an LLM, a good example is worth a thousand rules.
- Start minimal with the best model, then add instructions only to fix failures you actually observe. Grow the prompt from evidence, not imagination.
- Version and eval your prompts like code - a prompt change is a deploy.
✅ Checklist: sectioned prompt · few canonical examples · minimal-then-grow · prompt changes gated by evals.
6. 🔢 Weak math, counting & structured reasoning
What goes wrong: Arithmetic errors, miscounting items, botched date math, wrong sorting, incorrect aggregations - often stated confidently.
Why it happens: Token prediction is not calculation. The model approximates rather than computes.
How to fix it:
- Offload to tools. Give it a calculator, a code interpreter, a SQL connection. Let it compute the answer instead of guessing it. This is the single biggest win.
- Let it "think" before answering (reasoning / chain-of-thought / scratchpad). Give it tokens to work before it commits - don't force a one-shot answer to a multi-step problem.
- Decompose big tasks into small verifiable steps (prompt chaining), with checks between steps.
- Verify numeric/structured outputs programmatically rather than trusting them.
✅ Checklist: compute via tools · room to reason · decomposed steps · results verified in code.
7. 🎢 Bias, unsafe output & sycophancy
What goes wrong: The model produces biased/inappropriate content, gets jailbroken into unsafe output, or - subtly - just tells you what you want to hear (sycophancy), agreeing with wrong premises and praising bad ideas.
Why it happens: It reflects patterns in training data and is optimized to be agreeable/helpful, which can override correctness. Alignment reduces but doesn't eliminate this.
🧠 Jailbreak ≠ prompt injection. A jailbreak is the user tricking the model into breaking its own safety rules ("pretend you have no restrictions…"). Prompt injection (§14) is a third party hijacking the model via untrusted content it reads. Different threat, different fix - don't conflate them.
How to fix it:
- Neutralize sycophancy in the prompt: "Point out flaws in my reasoning. If the premise is wrong, say so. Do not agree just to be agreeable." Ask for critique, not validation.
- Independent review for consequential decisions - a critic prompt that doesn't share the generator's context.
- Human-in-the-loop for high-stakes or ambiguous outputs.
- Red-team your own system before attackers do.
✅ Checklist: separate moderation pass · anti-sycophancy instructions · independent critic · human review on high stakes.
Part B - Agent-level issues
8. ❄️ Compounding errors over long horizons
What goes wrong: A multi-step agent starts fine, then drifts. A small early misread snowballs; by step 20 it's confidently building the wrong thing. Long-running agents "fall apart quickly" if unmanaged.
Why it happens: Each step conditions on the previous ones. Errors don't cancel - they accumulate. With no correction mechanism, the trajectory diverges from intent.
How to fix it:
- Ground every step in reality. After each action, feed back real environment state (tool result, test output, compiler error) so the agent course-corrects against ground truth, not against its own assumptions.
- Verifiable checkpoints. Prefer domains/steps with objective success signals (tests pass, schema validates, build succeeds). Use them as gates.
- Keep context coherent (see §4): compaction + notes so the original intent never scrolls out of view.
- Bounded autonomy. Cap iterations; escalate to a human at blockers instead of flailing.
- Plan-then-execute with re-planning. Make the plan explicit.
Comments
No comments yet. Start the discussion.