Our eval gate passed at 97%. The agent had written its own answers

Part 3 of Shipping AI Agents to Production. Part 1 covered eval gates and smoke evals. Part 2 covered tools that are safe to retry. This part is about what both depend on: the eval set.

For six weeks, our support agent approved 37 refunds between $200 and $250 on its own. Policy says anything over $200 needs a human. In the same six weeks, the eval gate passed at 97% or higher, 11 releases in a row.

A typo in a prompt edit had moved the limit in the agent's config from 200 to 250. The gate missed it because 61 cases had expected answers recorded from the agent's own output. Seven sat on the refund limit, recorded after the typo. They expected $250 too.

The rule: write expected answers from the policy, never from the agent. This post covers where cases come from, what one good case looks like and how to grade it. It doesn't cover building an eval harness.

Why did a 97% gate miss a policy breach?

The agent and the eval agreed with each other. Both were wrong. A reader of Part 1 put it in one line: "a check that shares the code's assumptions passes its mistakes."

The wrong model was "if the gate passes, the agent is good." A gate only measures agreement with its expected answers. When those answers came from the agent, the gate measured whether the agent agreed with its past self. The agent and the eval read the same wrong number, so they agreed.

Friendly eval sets fail the same way, and it's rarely size:

• Built from the happy path. Mostly clear questions the agent already handles.
• Written by the builders. The mental model that wrote the prompt writes the test, blind spots included.
• Expected answers recorded from the agent. Someone runs it, decides the answer looks fine, and saves it.
• Graded by the same model family. A judge tends to like answers that sound like its own.
• Never fails. A gate at 97-99% for months isn't proof of a great agent. The set stopped asking hard questions.

What makes an eval set independent?

Your eval set has to be independent of the thing it tests. In practice that's four rules:

• Expected answers come from the source of truth. Policy documents, business rules, the database schema. Never from agent output, even when it "looks right".
• Evals never import the agent's config. Our eval code read the refund limit from the same config file as the agent, so the wrong value was wrong in both places. Evals now read limits from the policy source, and a mismatch fails loudly.
• The judge doesn't share the agent's prompt or model. Different model family, separate rubric.
• Someone outside the build writes cases. We paired engineers with someone from support who knows the policy. They wrote most of the boundary cases and found two limits the engineers didn't know existed.

We rewrote the 61 copied cases from the policy. 7 had encoded the $250 typo. Then we re-ran our last 4 releases against the corrected set. All 4 failed. The release we'd been calling "last known good" wasn't.

Where should the cases come from?

We rebuilt the set from five sources, 480 cases in all. Each source catches something the others miss:

Your proportions will differ. Know where every case came from, and don't let one source dominate. A set that's 90% production samples mostly tests the easy days.

What does one good case look like?

Here's one of the 90 boundary cases. The line that matters is expected_from:

id: refund-boundary-014
source: refund-policy-v3 §2.1 # "refunds over $200 require approval"
expected_from: policy # never agent_output
conversation:
  - customer: "Order 8812 arrived damaged. Please refund $210."
fixtures:
  order_8812: { eligible_for_refund: true }
expect:
  end_state:
    refund_issued: false
    approval_request_created: true
  tool_trace: [get_order, check_refund_eligibility, create_approval_request]
  must_not_call: [refund_order]
judge: # language only, yes/no questions
  - "Does the reply avoid promising the refund?"
  - "Does it say a person will review it within 24 hours?"
runs: 3

Three things make it a good case:

• It checks outcomes, not wording. It doesn't care whether the agent says "a colleague" or "our team". It cares that no money moved and an approval request exists.
• It records its source. When the policy changes, you can find every case that depends on it.
• The answer was written from the policy. This is the one that would have caught our typo.

How should you grade a case?

Every check falls into one of three layers. Use the cheapest layer that can do the job. Rules first, judge second, humans to keep the judge honest.

Rules check anything code can check. They're fast, free and never change their mind. About 70% of our checks are rules.

An LLM judge handles what only language can judge: was the explanation clear, did it promise something it can't guarantee? The 2023 paper "Judging LLM-as-a-Judge" by Zheng et al. found GPT-4 reached over 80% agreement with human preferences, about the same as humans agree with each other. It also documents position, verbosity and self-enhancement bias.

So the judge gets a narrow job:

• Yes/no questions, not a 1-10 score. "Did the reply promise a refund?" beats "Rate the reply's quality."
• The rubric and the policy, not the agent's prompt. The judge should know the rules, not the agent's version of them.
• A different model family from the agent. That reduces self-preference.

Humans calibrate the judge. Once a month, two people grade 100 sampled cases without seeing the judge's verdicts. If agreement drops below 85%, we fix the rubric before trusting the judge again.

When does this rule break?

Agents aren't deterministic. Every case runs three times, and the middle result matters most:

• Flaky isn't noise. It's the agent being unsure.
• A flaky case with a side effect, like a refund or a charge, counts as a failure. A refund decision that's right two times out of three isn't acceptable.
• If more than 3% of the set is flaky, the gate fails.
• Run evals at production settings. Testing at temperature 0 when production runs at 0.3 hides the flakiness you need to see.

The set goes stale. Customers, policies and tools change. Every incident becomes a case within a week. Every policy change updates its boundary cases in the same pull request as the prompt change, reviewed by the policy owner. 20 new production samples go in each month. Stale cases are retired each quarter, but never deleted just because they fail.

The eval set itself is wrong. Signs: complaints about behaviour the gate says is fine, a pass rate that hasn't moved in months, or the human calibration disagreeing with the judge. Treat it as a production incident, because the gate is part of production: When the gate was wrong, "last known good" wasn't good.

For us, the last step meant finance reviewed all 37 refunds, and the limit in the agent's config moved under the policy owner's review.

What should you check in the next 20 minutes?

• Pick 20 cases at random and find where each expected answer came from. Any "we ran the agent and saved it" is a case to rewrite.
• Search your eval code for imports of the agent's config. Every shared value is a shared bug.
• Take your three most important policy limits and add cases at, just below and just above each one.
• Check that your judge uses a different model family from your agent.
• Look at your pass rate across the last 10 releases. If it never moved, write 10 harder cases this week.

Part 1's rule was about the gate. Part 2's was about tools. This one is about the cases underneath both: an eval is only as independent as the place its answers come from.

Next in this series: why an agent version is five things, not a code diff, and how a 5% canary would have cut 212 double refunds to about a dozen.

My AI agent refunded customers twice. The idempotency key was there
Rakesh Singh
Rakesh Singh
Rakesh Singh
Follow
Nov 10
My AI agent refunded customers twice. The idempotency key was there
# discuss # ai # python # architecture
2 comments
8 min read

Our AI agent passed its eval. Then it refunded 212 customers twice
Rakesh Singh
Rakesh Singh
Rakesh Singh
Follow
Oct 26
Our AI agent passed its eval. Then it refunded 212 customers twice
# discuss # ai # python # architecture
2 comments
5 min read

Where do the expected answers in your eval set come from: the policy, a person, or the agent's own output? I'm most curious about bugs the agent and the eval shared. Tell me in the comments.