The Cold-Start Problem for Agent Evals: What to Gate on Day One With Zero Labeled Data

You just shipped an agent. It works in the demo. Now someone asks the reasonable question: “How do we know it keeps working?” And you reach for evals - and hit a wall. You have no labeled dataset. No golden outputs. No historical traces. Nothing to grade against. So the team stalls. “We’ll add evals once we collect data.” Meanwhile the agent runs in production, ungated, and the first time it silently breaks is the first time anyone notices. This is the cold-start problem, and the usual response - “just get an LLM to score the output 1-10” - is exactly the wrong instinct. You do not need labels to start gating. You need to understand which evidence you can trust on day one, and which you can’t trust ever.

Independence, not cost

Most eval discussions rank checks on a cost axis: cheap string matches at the bottom, expensive model-as-judge at the top, as if spending more buys you more truth. That’s backwards. The axis that matters is independence: can the agent forge this signal, or not? That reframing is the whole game for cold-start, because independent evidence needs zero labels. It’s true or false about the world regardless of what your agent intended.

Three tiers, ranked independent to corruptible

• Tier 1 - externally observable proof the agent can’t forge. Did it produce valid JSON? Does the file it claims to have written exist? Did the code compile? Did the tests pass? Did it finish inside the timeout? Is the output non-empty?
• Tier 2 - statistical signal against a baseline the agent didn’t author. Is the output embedding-similar to the task it was given? Is the length sane, or did it collapse into repetition? Did the diff actually change anything?
• Tier 3 - model-as-judge. A shared-substrate opinion. A signal, never a verdict.

On day one you have Tier 1 and Tier 2 completely for free. Neither needs a single labeled example, because neither asks “is this good?” - they ask “is this real?”

The day-one gate

Here’s a starter gate for an agent that’s supposed to produce a code patch. No dataset required:

type GateResult = { pass : boolean ; tier : 1 | 2 ; reason : string };

async function coldStartGate (
  task : string ,
  output : { patch : string ; targetFile : string },
  runtime : { durationMs : number ; timeoutMs : number },
  embed : ( s : string ) => Promise < number [] > ,
): Promise < GateResult [] > {
  const results : GateResult [] = [];
  // Tier 1 - unforgeable facts about the world
  results . push ({ tier : 1 , reason : " non-empty output " , pass : output . patch . trim (). length > 0 });
  results . push ({ tier : 1 , reason : " target file exists " , pass : await fileExists ( output . targetFile ) });
  results . push ({ tier : 1 , reason : " patch applies + compiles " , pass : await applyAndCompile ( output . patch , output . targetFile ) });
  results . push ({ tier : 1 , reason : " finished within timeout " , pass : runtime . durationMs < runtime . timeoutMs });
  // Tier 2 - statistical signal vs a baseline the agent didn't write
  const [ taskVec , patchVec ] = await Promise . all ([ embed ( task ), embed ( output . patch )]);
  results . push ({ tier : 2 , reason : " patch is on-topic for the task " , pass : cosine ( taskVec , patchVec ) > 0.35 });
  results . push ({ tier : 2 , reason : " diff changed something " , pass : ! /^ \s *$/ . test ( output . patch . replace ( / [ -+ ]{3} .*$/gm , "" )) });
  return results ;
}

Every check here is either true about the filesystem/compiler/clock, or it’s a distance against the task string - which the agent received but did not author. There is nothing to label. And this catches the overwhelming majority of real failures: the stale run, the crash, the malformed output, the hallucinated file path, the empty response, the patch that wandered off into an unrelated file.

Why Tier 3 stays out on day one (and after)

The temptation is to skip all this and let a judge model read the patch and give it a score. Resist it - and not just because you have no labels. A model judging another model’s work is circular. Judge and judged share a substrate: the same training distribution, the same blind spots, the same confident wrongness. There’s no independent ground truth in that loop. So Tier 3 is a signal about taste, never a verdict about correctness, and it may only inspect artifacts the judged agent didn’t get to write - never the agent’s own reasoning trace, which it can rationalize.

There are two more hard constraints. Tier 1+2 are the real-time gate: deterministic, effectively free, fast enough to block a run before a bad output escapes. Tier 3 is offline-only: metered, slow, non-deterministic - it cannot sit in the hot path. You run it later, in batch, over the ~20% subjective tail that Tier 1+2 can’t adjudicate, and you label its output “opinion, not evidence.” Ship the 80% you can gate deterministically today; don’t block your launch waiting for a judge you shouldn’t trust in the loop anyway.

The two halves you actually need

Gating output is only half the job. The other half is knowing what happened, and this is where cold-start teams quietly cheat: they gate on the agent’s self-report, which is exactly the forgeable thing Tier 1 is supposed to route around. This is why the eval layer and the trace layer ship as a unit. agent-eval scores and gates the output - the tier logic above: evals, drift, hallucination checks. AgentLens captures the trace of how the agent got there: every model call and tool step, the resolved inputs, the raw outputs. The two connect at a specific seam: Tier 1+2 need unforgeable data to score against, and the agent must not be the one who wrote it. AgentLens gives you exactly that - the real file that got written, the actual exit code, the true wall-clock duration - instead of the agent’s summary of what it thinks it did. Without the trace, your gate degrades into grading the agent’s own press release. With it, “finished within timeout” and “target file exists” become facts, not claims.

Start today

You don’t have a labeled dataset. You never will on day one. But you already have a filesystem, a compiler, a clock, and an embedding model - which means you already have a Tier 1+2 gate. Wire agent-eval to it, point AgentLens at your run to feed it unforgeable trace data, and gate the 80%. Collect the judge-tail labels while you’re already protected in production, not instead of protecting it. The cold-start problem was never about missing data. It was about asking the wrong tier for permission to launch.

Top comments (1)

I particularly appreciated the emphasis on independence as the key axis for evaluating agent performance, rather than cost. The tiered approach you outlined, with Tier 1 focusing on externally observable proof and Tier 2 leveraging statistical signals against a baseline, provides a robust framework for assessing agent behavior without relying on labeled data. The coldStartGate function you provided offers a concrete example of how this can be implemented in practice. I'm curious to know how you envision this approach evolving as the agent continues to operate and generate data - do you see the tiers shifting in importance or new tiers emerging as more data becomes available?