Stop Leaving Findings in the Judge: The Ratchet That Turns Opinions Into Gates

When a model-as-judge flags something real, most teams do the worst possible thing: they leave it in the judge. The finding lives on forever as a slow, metered, non-deterministic Tier 3 opinion - re-litigated on every run, at cost, with a different verdict each time. That's not an eval strategy. That's paying rent on a bug you already found. The senior move is a ratchet: every recurring Tier 3 finding is a candidate to be promoted into a Tier 1 or Tier 2 check, where it becomes free, fast, deterministic, and able to block the run. This post is about how that ratchet works, and why it's the whole point of having a judge at all.

The independence axis

Briefly, before promotion makes sense, you have to rank evidence the right way. agent-eval ranks it on an independence axis - from independent to corruptible - not a cost axis:

• Tier 1 - proof the agent can't forge. Valid JSON, the file exists, it compiled, tests passed, it finished inside the timeout, the output is non-empty.
• Tier 2 - statistical signal vs a baseline the agent didn't author. Embedding similarity to the task spec, length and repetition profiles, whether the diff actually changed anything.
• Tier 3 - model-as-judge. A shared-substrate opinion. A signal, never a verdict.

Two properties fall out of this and they drive everything below. Tier 1+2 are the real-time gate: deterministic, ~$0, fast, so they can sit in the hot path and block a run. Tier 3 is offline-only: metered, slow, non-deterministic, so it can't. And Tier 1+2 can legitimately run over agent trajectories, while Tier 3 cannot - a model judging another model's reasoning is circular, because judge and judged share a substrate and there's no independent ground truth. So Tier 3 only gets to inspect artifacts the judged agent didn't write.

The ratchet: promote findings down the tiers

Here's the pattern. Your judge keeps flagging the same class of failure - say, "the summary references a section that doesn't exist in the source." That's a real defect. But it's expensive to catch this way, and the verdict wobbles. Ask: what unforgeable fact would have caught this? In this case, "every cited section header appears verbatim in the source" is a string-membership check. That's Tier 1. You just promoted a subjective opinion into a deterministic gate.

type Finding = {
  claim: string;          // the judge's recurring complaint
  sourceText: string;     // artifact the agent did NOT author
  citedSections: string[];
};

// Tier 1 promotion: the finding is now an unforgeable-proof check.
function citationsExist(f: Finding): { pass: boolean; missing: string[] } {
  const missing = f.citedSections.filter((s) => !f.sourceText.includes(s));
  return { pass: missing.length === 0, missing };
}

// Tier 2 promotion: when membership is too rigid, drop to a
// baseline-relative signal the agent didn't get to author.
function citationGroundedness(
  f: Finding,
  embed: (t: string) => number[],
  cos: (a: number[], b: number[]) => number
): number {
  const src = embed(f.sourceText);
  const scores = f.citedSections.map((s) => cos(embed(s), src));
  return scores.reduce((a, b) => a + b, 0) / (scores.length || 1);
}

citationsExist is Tier 1: it's a fact about text the agent can't forge. When exact membership is too brittle - paraphrased citations, translated sources - you don't jump back to the judge, you drop to Tier 2: citationGroundedness scores similarity against a baseline (the source) the agent didn't write. Still deterministic. Still ~$0. Still allowed in the hot path. Only if neither tier can express the property do you leave it with the judge.

Ship the 80%, meter the 20%

This is where the ratchet pays off. Most failures in production are not subtle: stale outputs, crashes, malformed JSON, hallucinated file paths, empty responses. Every one of those is catchable at Tier 1+2 alone - for free, in the hot path, blocking the run before a user ever sees it. The judge is for the ~20% subjective tail: tone, whether an explanation is actually helpful, whether a refactor is tasteful. That work is real, but you run it offline, metered, and you label its output honestly: opinion, not evidence. The ratchet steadily shrinks that 20% over time, because every recurring judge complaint that can be expressed as a fact eventually gets promoted out.

This is the line that separates a real eval layer from an "LLM-as-judge gives you a 7/10" tool. A 7/10 is not a gate and it's not reproducible. A promoted Tier 1 check is both.

You can't promote what you can't see

The ratchet has a hard dependency: to promote a finding, you need the raw material - the exact source text, the resolved tool inputs, the actual output the agent produced. This is where the two halves of the workflow lock together. agent-eval scores and gates the agent's output - the tiers, the drift checks, the hallucination checks above. But it can only score against unforgeable data if that data was captured faithfully. That's AgentLens: it captures the trace of how the agent got there - every model and tool step, the resolved inputs, the raw outputs.

Two things follow. First, when a Tier 3 opinion recurs, you open the trace, find the exact artifact the agent didn't author (the source document, the tool response), and that becomes the baseline your new Tier 1/2 check scores against. Second, the trace is itself agent-didn't-author data, which is exactly what Tier 1+2 need to run against without becoming circular. Without the trace, your judge is guessing and your gate has nothing trustworthy to grade. With it, every recurring opinion is one refactor away from becoming a free, deterministic check.

The takeaway

A model-as-judge is not the top of your eval stack - it's the intake queue for it. Its job is to surface recurring, real defects so you can promote them down the independence axis into checks that are cheaper, faster, and impossible for the agent to forge. agent-eval runs the tiers and the gate; AgentLens gives you the unforgeable trace to build the next gate from. Leave a finding in Tier 3 forever and you're not evaluating - you're just paying, slowly, to be reminded of a bug you could have gated on day one.