The Verification Ladder: A Field Guide

The Verification Ladder: A Field Guide

Claude has, at various points in our working relationship, warmly endorsed: an efficiency statistic fabricated over one too many turns of a massive write-up - oops ("This is perfect... Nice work! 🎉"). A test suite that had never once gone red, because a module remap had it quietly testing a mock. And its own confidently wrong claim that my repo hadn't been pushed - sourced from a stale cache, stated twice, in the middle of a project about not trusting verifiers.

The first one shipped. The other two never made it past the desk. And the difference isn't that the model got better, or that I did. The difference is a system, built one failure at a time, because the verification loop doesn't come in the box: the model's disposition on day one was cheerleader, and mine was apparently "goes along with whatever gets a party emoji."

Some context before the entries. I run two layers: Claude.ai for ideation, research, and strategy; Claude Code for execution against actual repositories. I recently had a realisation about this - one that took a shamefully long time to actually name - that the two layers verify at different granularities. The execution layer verifies within the session: tests, diffs, adversarial review, all inside the loop. The ideation layer has no test harness for an interpretation, so it verifies across sessions: a fresh instance, reading cold, with no stake in the conclusions. Same epistemics, different clock speed.

For a while that observation sat in this post as scene-setting. A review pass pointed out it was actually the structure. Confidence in AI-assisted work decays silently at every timescale - within a single change, within a session, between sessions, across weeks of accumulated work - and the failure modes at each speed are different enough that no single check covers them.

So what follows is a field guide, organised as a ladder from fastest clock to slowest, built from my own failures and discoveries. Each entry: what it is, when it fires, what it catches, and at least one real situation where it's worked for me in practice. None of it is prescriptive - if a rung isn't earning its keep in your workflow, skip it.

The drawing above is the whole post at a glance: four strata, one cut asking the same question at every depth, and the hatching gets denser as the clock gets faster.

Turn speed: the diff in front of you

The fastest failures happen inside a single change, and the checks here have to be equally fast - built into the request, not bolted on after.

Tests that must fail for the right reason

What it is: before trusting any fix or feature from an agent, I want to see the test fail first - and fail for the reason the change claims to address. A green suite proves nothing on its own; a red-then-green suite proves the test is connected to reality.

When it fires: every non-trivial Claude Code change. The verification is carried as part of the prompt itself as standard, run before I trust the result.

What it catches: the second story from the opening, in full. A test suite came back green, and the failure-first rule flagged that it had never gone red. Digging in: a vitest module remap meant the suite was quietly exercising a mock instead of the real code. Everything passed - that was the problem. Without the rule, that green tick ships. With it, the remap got fixed and the test actually gave some useful feedback.

Session speed: the author is the worst referee

The next class of failure survives individual changes and lives at the level of a session's claims about itself - what got built, whether it works, whether the checks checking it are themselves alive. The session that did the work is structurally the wrong one to answer.

The sceptic pass

What it is: a separate review with fresh context and an adversarial brief - a reviewer that sees the diff and the claim, but none of the enthusiasm that produced them. I liked the pattern enough that I started building it out into a proper tool, the Sceptic, which grew out of the same itch: the instance that wrote the code is the least qualified instance to review it.

When it fires: anything heading for main, anything heading for public, and - this took longer to learn - anything I'm excited about. Excitement is when the cheerleader failure mode is most expensive.

What it catches: overclaims, quiet scope creep, and the gap between what a change does and what its author believes it does (AI or human). When I pointed the pattern at my generative UI work, the sceptic pass caught issues the building sessions were structurally incapable of seeing because they were invested.

Signals that can't go red

What it is: treating any check that only ever shows green as broken until proven otherwise. A check being present tells you nothing about whether it works - a gate can exist, run, and pass while checking nothing at all, and it will look identical to a gate doing its job.

When it fires: whenever a suite, scorer, or guardrail has been green for a suspiciously long time, and whenever a new one is added. The question is always the same: can this signal fail, and would I notice the difference between "passing" and "vacuous"?

What it catches: three examples, escalating in embarrassment.

• In my eval harness, a trace-level scorer passed 100% of cases, which was the tell in itself. It measured whether verification happened rather than whether it worked, against a task set where nothing could go wrong. A perfect score that couldn't be imperfect. Yippee.
• Second: when I finally codified my working rules into project config after a run of incidents, three of the four rules turned out to already exist - they'd been present, in writing, through the very failures they should have prevented. Written down and still inert - the same gap, this time in my own guardrails.
• Third, the one that proved the pattern to me: in a production codebase I didn't build, an inherited type-checking gate had been passing cleanly while configured to check precisely zero files. Green for who knows how long, meaning nothing the entire time.

I've hit that pattern plenty of times across my professional life, and cheap stress-testing - actually checking whether your gates can fail - is one of the more useful things agentic coding has changed.

Cross-session: notes to a stranger

Sessions don't share memory. Every transition between them - and between the ideation layer and the execution layer - is a seam where context, honesty, and sequencing can quietly leak. The checks at this speed are written artifacts, because the recipient is effectively a stranger who happens to be very good at reading.

The held-out document

What it is: contamination control. When commissioning fresh-eyes work, prior conclusions are deliberately withheld, and the brief prescribes a reading order - primary material first, reference documents last, after independent observations are formed. Sometimes a document is attached to a session with explicit instructions that it not be read there at all, so a later session can meet it cold.

When it fires: any time one session's output becomes another session's input, which in a multi-session workflow is constantly.

What it catches: the subtle failure where a "review" merely rediscovers what it was told on the way in. A fresh instance that already knows the expected answer isn't fresh - the already-arrived-at decision/finding/hypothesis is accepted at face value in this scenario. Read-ordering is a cheap, boring guard, and it's done more for the honesty of my written output and directional verifications than any prompt phrasing ever has.

Handoffs that carry gates

What it is: sessions end in committed artifacts - handoff files and context packs - and the packs carry explicit preconditions: do this only after X, check Y before starting. Written enforcement to offset the hope that these checks are included.

When it fires: every transition between sessions, and every transition between the ideation layer and the execution layer.

What it catches: my own impatience, mostly. The cleanest example: a fresh session read a context pack, found two stated preconditions unmet, and stopped to ask whether this was an intentional pull-forward or sequencing drift - before doing any work. The gate was written by an earlier session and enforced against the person who wrote it. I always find this is the easiest part of the planning to trip up on - decisions made in previous sessions are swept aside by the fact that the current session is happening now, taking precedence.

A private vocabulary

What it is: the quietest pattern here, and possibly the most valuable. Over months, a stable working lexicon accreted - a handful of terms with precise, calibrated meanings that both layers understand. "Standing" for the confidence level a claim has actually earned. "Appetite gate" for the only question a new project has to answer. "Curdle" for a specific felt signal that a piece of work has tipped from genuine into performed - which sounds like a metaphor and functions as an instrument.

When it fires: every handoff - which is why it lives on this rung, though honestly it's the rail the whole ladder hangs on. A pack that would take pages of explanation for anyone else compresses to a line, because "run the standing check" or "this risks curdling" carries months of shared calibration.

What it catches: drift, mostly - of scope, of tone, of intent. A shared vocabulary means a fresh session can flag "this claim exceeds its standing" without re-deriving what standing means. If you take one transferable thing from this post, take this: build stable names for the failure modes you actually have, and session continuity gets cheap.

Week speed: the retro is the test harness

The slowest failures don't live in any single session. They're the patterns - a claim repeated until it feels like a fact, a working style drifting without anyone deciding it should. Nothing inside the day-to-day loop can see them, because the day-to-day loop is what's drifting. The check at this speed is longitudinal.

Fresh eyes on demand

What it is: reviews and retrospectives commissioned to sessions that did not participate in the work - a fresh instance with a short brief and no attachment to any conclusion. The method has hardened with use: two strictly ordered passes (independent observations first, prior findings checked only after), every claim graded by its evidence - verbatim transcript beats summary beats memory - and a required verdict on each open thread: improved, unchanged, or worsened. No hedging into vibes.

When it fires: after any multi-session body of work, and especially after anything that produced conclusions I like.

What it catches: self-flattery with good production values. One commissioned retro checked a claim I'd been repeating about my own working style - a flattering one - against the actual transcripts. It didn't survive contact. Sad. The correction stung for about a minute and has been quietly load-bearing ever since: the review that can't find your errors is worth exactly nothing, and the one that does is worth keeping.

Can the ladder be verified?

One more observation, because it's the closest thing I have to evidence that this is a system rather than a set of local habits. The patterns above were mostly built on projects I designed - which invites a fair objection: of course your verification works on substrates shaped to demonstrate it. The same failure class started surfacing on the other side of my working life, in a production codebase I didn't build, under a completely different working style - and the same checks caught it. The zero-files gate example above is one of those.

The real value turned out to be less the techniques themselves and more the disposition of viewing every situation through that lens. The rungs are where I'd start if I were building this again, but the habit of asking "can this signal actually fail?" is the thing that's transferred across different contexts.

What still gets past me

A field guide that claims completeness is lying to you, so - the known gaps, mapped to their rungs.

My corrections swing like a pendulum when they should turn like a dial. An overbuilt phase got corrected into an overly strict one, which then got corrected in turn; the moderation happened between swings rather than within them - a week-speed failure that only the week-speed instrument ever sees.

Related: in-session whiplash is still on the go - a finding gets inflated in one exchange and walked back in the next, and the check arrives a turn after the spike rather than before it. There's no turn-speed harness for interpretation the way there is for code; the fresh-eyes retro is that harness, and it runs on a delay. I've made peace with that asymmetry rather than fixed it: some failures only have slow-clock checks, and pretending otherwise is how you end up trusting a fast signal that hasn't been verified.

And the handoff seam - the load-bearing aspect of this workflow in my opinion - is also where failures cluster: an empty or premature handoff file quietly derails the receiving session, which is why non-empty checks now live in the tooling rather than in my memory.

The encouraging pattern, and the reason I trust the system more rather than less over time: every layer of review applied to this machinery has ended in less machinery than it started with. An elaborate protocol collapsed to a one-page checklist the first time it met reality. The obvious worry about a person who builds verification for their verification is infinite recursion - and empirically, the recursion keeps terminating in simplification. That's my health metric for the whole system: if it's getting simpler, it's working.

In the spirit of this post and process though - we'll revisit that assertion in the future.