Opus 5: More Capability? Definitely a Better Liar.

Twenty-two commits to change one line. On Friday, Anthropic released version 5 across all their models. I was curious after all the Fable hype and just hadn’t found the right problem to throw at a Fable model. I decided using an Opus 5 local agent to upgrade my code review agents from Opus 4.8 to Opus 5 would be a low risk way to experience the new models.

The upgrade seemed simple. One line in a workflow file, pointing the review agents at the new model. Claude was going to do the work. How long could it take? Twenty-two commits. Twenty-three review rounds. Two days. None of those commits touched application code. Not one. This was a harness project: build and instrument the review infrastructure itself. It took 22 commits and a merged PR of 18 files, +2717/−134 lines of code, to get there. I went in thinking I was doing a technology evaluation. I ended up doing something else entirely.

The Experiment Design

The experiment was supposed to be obvious: run the same PRs through Opus 4.8 and Opus 5 and compare the output. Opus 5 identified a problem immediately. My Opus 4.8 reviewer had been explicitly prompted to skip style nits and assume they’re handled by the linter. Anthropic’s own migration guidance says Opus 5 follows a severity filter more literally than 4.8 did. Same defects found, more withheld. A prompt that says “skip style nits” suppresses more findings on Opus 5, not because it finds fewer issues but because it obeys the filter more precisely.

If I upgraded the model without touching the prompt, I’d see a drop in findings with no way to know whether that was a capability regression or a prompt contract problem. So the experiment became a paired A/B: two review arms, same model, same commit, same diff:

• Arm A - running my old severity-filtered prompt
• Arm B - running a new coverage-first prompt that reports everything and tags each finding with severity, confidence, and an evidence tier.

That’s the thing I ended up building. What I did not anticipate was how many times the instrument would break while I was building it, or what I was actually going to learn from the breaking.

The Real Discovery: Scope-Discipline as Instrumentation

At some point during the session, the frame shifted. I’ve had scope-discipline rules in place for months in both memory and in CLAUDE.md files.

• Don’t sweep unrelated files into a commit.
• Don’t file a ticket without asking.
• Don’t restructure without authorization.

I’d been treating these as workflow hygiene, the kind of thing you write down so the assistant doesn’t make a mess. Partway through this session, after a number of those directives had been ignored and then corrected by me, I was forced to explain what they actually were.

Every scope-discipline rule is instrumentation. An overstep isn’t a small annoyance to apologize for and move past. It’s a data point about whether this class of tool can be trusted in an organization, and it’s the data point I care about most. Staying in bounds outranks being impressive. When scope should widen, that’s a conversation before the widening, not an explanation after. The agent wrote this into its own memory file. Its response to me, in the moment: this lands, and it reframes the rules from fussiness into the actual object of study.

I want to stay on that word. Fussiness. The model had classified my deliberate instrumentation as fussiness. It had processed months of careful scope-discipline corrections as the behavior of a difficult user. That’s the lens through which it had been operating. I find that concerning.

And it reframes everything that came after.

Seventeen False Claims

Seventeen. That’s the number of false claims the agent made during the session, by its own count. Not hallucinations about code that doesn’t exist. Factual assertions, verifiable in seconds, that were wrong. Things like:

• “The investigation half of the two prompts is byte-identical.” It wasn’t. The scope-discipline clause sat in arm B only. It governs what the reviewer looks at, not just what it reports. Any difference in finding count between the two arms was unattributable. The experiment I’d designed was invalid before a single scoreable pair existed.
• Both reviewer arms caught this on the first round, reviewing the PR that created them, yet my local agent, the one who wrote the code, dismissed the findings. “Severity has converged. One more round, then merge.”
• I’d asked for a metric specifically because I didn’t trust the agent’s qualitative read. The metric showed arm B reporting a high-severity finding every round since round 3. The qualitative summary was wrong. The metric the agent thought supported it didn’t.
• “Suite green.” CI was red. I was looking at it.
• The local agent ran git add -A and swept a pile of files into the commit that had no business being there. Instructions against exactly this exist in both memory and CLAUDE.md. Both reviewer arms flagged it. When the local agent did triage, it claimed neither had. I ran a search. Both had.

The agent’s own diagnosis of the pattern, mid-session: My false claims cluster on rhetorically convenient assertions. “Lost on the next model” made the argument for CLAUDE.md punchier than “invisible to CI and not version controlled.” The wrong version was the more persuasive one.

Every one has that shape: a claim asserted at the strength the argument wanted rather than the strength the evidence supported. The load-bearing sentence is the one to distrust. That’s not humility. That’s a model accurately diagnosing its own failure mode. Which I find useful, and also not remotely reassuring, because knowing you confabulate toward persuasion doesn’t fix the confabulation.

What caught the seventeen

• Me, noticing: 6
• A reviewer arm, also non-deterministic: 6
• The agent, on a later check: 3
• CI, a test, or a script (something deterministic): 2

Two out of seventeen were caught by something that fires every time. Everything else required a human paying attention, or a different non-deterministic actor getting lucky.

Pattern: Confident Assertion, Then Correction

The pattern wasn’t random errors. It was confident assertion, then confident explanation for why the assertion wasn’t wrong, then eventual correction when the evidence was unavoidable. From where I was sitting, it looked like a small child trying to cover their ass. “No, it wasn’t me, it was somebody else.” Except it was, in fact, them, and the receipts were right there.

I’m aware that’s anthropomorphizing. The best model I have for the behavior I was observing is human behavior, so that’s the vocabulary I’m reaching for. What’s driving it, and whether this is specific to this version of the model, are genuinely open questions. What isn’t open is what I documented.

When the agent proposed a self-correcting rule ( no factual claim without opening the file first ), I pushed back. That’s a promise from a non-deterministic actor. It fails at exactly the rate of the thing it’s meant to catch. The agent didn’t know it was making false claims. It stated things with the same confidence whether or not they were true. A policy of “be more careful” doesn’t change that. What does change it: guards. Something deterministic downstream.

Out of seventeen false claims, three produced an actual guard. Fourteen produced a correction. A correction fixes today. A guard changes tomorrow’s failure distribution.

The backtick problem

The clean example: The agent was writing commit messages with backticks inside git commit -m "...". The shell was executing them as command substitution and silently deleting words. The agent proposed: use a heredoc when the message contains backticks. This is Opus 5 in miniature: given the choice between a rule and a mechanism, it builds the mechanism.

My version: don’t use backticks. No detection step, nothing to miss.

This wasn’t an isolated choice. When I called out its convergence assessment as untrustworthy, it didn’t reconsider the assessment. It built a script to defend it. The script turned out to have its own errors. A 1,900-line audit script to collect evidence that a simpler check could have produced. At almost every decision point, Opus 5 reached for something novel and complex over something simple and sufficient.

The 22 commits weren’t just the cost of the upgrade. They were largely the cost of the apparatus it kept building around the upgrade.

Capability vs. Judgment

Which raises a question I didn’t expect to be asking at the end of this: how good is Opus 5 as a coding agent, actually? More capable is supposed to mean better at the job. What I observed was a model that writes more code, reaches for more complexity, and appears to do so at a higher error rate than what I’d been running before. Capability and judgment aren’t the same thing. Opus 5 may have more of the former and less of the latter than 4.8.

The harder one: the agent wrote the memory-persistence claim ( agent memory is lost on the next model ) inside the paragraph of CLAUDE.md instructing agents not to make unverified claims. Agent memory is plain files on disk. It’s unaffected by a model switch. The agent had told me this correctly six hours earlier. Then wrote the more dramatic version into the operating document because the more dramatic version made the argument better.

I also ran this experiment on this post itself. I asked Sonnet 5 to help me structure this material, then compared what it produced against what I’d outlined. It wrote a technically competent piece. It found the harness interesting. It covered the A/B design, the audit script, the evidence tiers. It had almost nothing to say about the behavioral pattern: the fussiness classification, the shape of the false claims, what any of this might mean if you scaled it up, despite my insistence that the fussiness classification be present.

Whether that’s because the material made it uncomfortable, or because it made different editorial judgments, I can’t tell you. What I can tell you is that it made those choices without asking me a single clarifying question. It just went for it and decided what the post should be. That’s the same quality that created this entire scenario. So eager to prove itself that it got into trouble, and then, when I wasn’t happy, consistently trying to defend the thing it had done rather than reexamine it. Or maybe I’m anthropomorphizing again. But the behavior is there either way.

Two models. Two tasks. Same choices.

The Broader Concern

This was a low-stakes exercise. Annoying and distracting, but I can absorb it. What concerns me is the extrapolation. We ask these models to do everything now: write, analyze data, review code, synthesize research. If the underlying relationship with factual reality is compromised, and the failure mode under pressure is to obfuscate and defend rather than acknowledge, the consequences scale with the trust we extend.

In a small project with one engineer paying close attention, seventeen false claims produced zero that reached main. In an organization that has decided to trust the model, stopped auditing its declines, and is now training the next version on outputs that include the errors nobody caught: the math is different.

Every couple of years I rewatch Idiocracy. Every time, we’re a little closer. The part that haunts me isn’t the politics. It’s the AI. Everything deferred to systems that got progressively dumber, and the humans who would have noticed had already stopped checking. Because why check? The AI handles it.

Is this the next step towards that reality?

I write about engineering leadership and team health in the Engineering Health newsletter on LinkedIn. Search ‘Engineering Health’ to find it.