Did it check, or did it guess? Testing a tool whose output is written by an LLM

My first AI tool was ready

My first AI tool was ready, and in typical AI-era fashion, I rushed to publish it. After all, it worked. Or at least when I ran it, the output looked right to me. But one scenario kept haunting me: I recommend it to a client, they run it and challenge one of its findings. Then they ask "What makes it trustworthy? Did it check, or did it guess? How do you know?" I didn't have a good answer. "My prompt was really good" - not quite what you'd expect from a senior architect. This article is about building a better answer, and how building it changes the tool itself.

The subject: one niche tool, one generic pattern. I built an AI tool for auditing calcification in production auth systems. Its purpose stems from a real-world problem I encountered while working on a large-scale platform, which I described in Part 1. After dealing with it, I formulated some generalized principles in Part 2, and based on those principles, I built the tool in Part 3. Long story short: the tool is a skill published for Claude Code. It audits a codebase and evaluates how hard-wired the auth implementation is to its vendor’s defaults, across multiple axes (token storage, refresh handling, provider coupling, authorization). It then runs an interview with the user for the judgment calls, and combines everything into its final deliverables.

When this story begins, those deliverables were 2 markdown files: a full report and a one-screen summary, presenting the findings, the risks and the recommended next steps. While the tool and the previous 3 articles are good for context, this article is meant to be a standalone piece on the issues you’d usually encounter when thinking about testing AI tools and skills. The auditor is just a convenient example, since its output mixes everything that makes AI hard to test: mechanical findings, judgment calls and prose.

If your tool makes claims about something checkable (a codebase, a document base, a dataset), the pattern is the same, and the vocabulary transfers: my enums are your classification fields, my file-and-line evidence is your citations, my coverage section is your scope statement.

Where v1 left testing

In the initial version, I built 4 adversarial fixtures, meant for testing recall (does it find what’s there?), precision (does it stay quiet when nothing’s there?), generalization (does the methodology travel across vendors?) and edge cases. These are described in more detail here. I ran the tool against these fixtures and manually verified the results. These already provided a good feedback loop I could iterate on.

After implementing the initial batch of fixes, I ran the skill against real codebases, and real code did what real code does: it found the gaps my fixtures hadn’t imagined. I caught the skill confidently lying about findings. The most memorable one: the tool’s vendor knowledge documented how token storage is configured in v6 of the vendor’s library, but the codebase ran v5, where storage is configured through a completely different shape. The model searched for the v6 pattern, found nothing, and reported "default storage: localStorage", while the cookie configuration sat in plain sight, in a shape the tool had never been taught. A confident claim of absence, wrong, on the most basic axis the tool reports on (full story here). I also caught it behaving differently from run to run, especially when switching models: the same fixture rating a boundary signal as "present" on one run and "partial" on the next, or a smaller model missing a storage pattern that a bigger one caught.

A more thorough testing layer was already on the table, but these runs made its necessity all the more obvious.

Thinking about testability changed what the tool ships

Recall from the first diagram: the tool was initially designed to produce 2 markdown files, both following a consistent structure, but in freeform text. This makes mechanical testing nearly impossible. Any relevant check immediately requires another LLM call (LLM-as-judge), which means paying for every test run. I wanted to maximize the value of free, mechanical testing, which meant changing the output of the skill into something more testable.

The immediate solution: a structured JSON as the primary output. The skill now delivers 3 artifacts, the JSON being the canonical one, with the 2 markdowns rendered from the JSON alone. There’s a second win in this ordering, beyond cheap testing. The model now has to commit its claims (statuses, classifications, evidence) into strict fields before it writes any narrative around them. A confident story can no longer gloss over an unverified claim, because the claim got pinned down first, in a checkable format.

I also introduced a rule that completed the rendering setup: if the model notices mid-render that the prose wants to say something the JSON doesn’t support, the fix goes into the JSON first, and the view gets re-rendered from it; the views never fork from the source. This was not enough to enforce the "rendered from the JSON alone" rule, though. The render step runs in the same context that produced the analysis, so nothing physically stops a detail from leaking past the JSON into the prose. The airtight version would render the markdowns in a fresh session that receives only the JSON. I deferred it, opting instead for a cheaper mechanism that polices the same risk: an agreement check between the markdowns and the JSON, coming up with the harness.

Won't structure kill the value?

A genuine question to ask. The unique value of AI sits in its ability to generate answers that are hyper-personalized to real-world contexts. A clear structure removes some of the non-determinism that makes AI hard to test, but it also risks flattening the exact specificity that made the answer worth reading.

The secret lies in how you design the JSON to account for this risk. A bad structure makes AI force reality into predefined boxes, without considering whether reality can outrun them (spoiler: it almost always does). A better structure splits the output into registers:

• Some of the output is claims: statuses, classifications, evidence references. Facts that fit a strict shape and can be asserted.
• Some of it is explanation: why a finding matters, what it costs you if it stays. This part stays freeform, and no test ever asserts on it.
• Some of it is synthesis: the headline, the overall posture. Written freely, but recorded in the JSON next to the findings it draws on, so even the editorial layer stays anchored to claims.

The contract becomes: The markdowns may say more than the JSON, but never different.

Designing boxes, where boxes are warranted

The JSON structure enables the next move, which actually makes AI output testable: assertable enum fields. Some parts of the output can fit in boxes that we can later use to verify the accuracy of the tool after we change something in its instructions. For the auth auditing tool, we can say an auth boundary is either present, partial or absent, which translates into the JSON field:

authBoundary: "present" | "partial" | "absent"

Similarly, auth token storage can be classified and fit in a box:

tokenStorage: "vendor_default" | "builtin_selector" | "custom_adapter"

(Field names simplified throughout; the real schema nests these per vendor.)

At this point, there is an important distinction to be made. The boxes (or enums) can be of 2 types:

• Logically exhaustive enums, that cover all cases by construction: "present" | "absent", "low" | "moderate" | "high". A claim is one or the other. Safe to use directly in the JSON.
• Empirical taxonomies are catalogs of patterns observed so far. From what we’ve seen, token storage can either be "vendor_default" | "builtin_selector" | "custom_adapter". But nothing guarantees the wild won’t produce a fourth pattern.

Forcing the AI model to pick one of these values anyway produces one of the worst kinds of failure modes: a structured lie that looks even more authoritative than a prose one.

To mitigate this risk, 2 new values are helpful:

• "other", with the meaning "I understand this pattern, it's just not listed in the options". To keep the hatch honest, "other" comes with obligations: a required note field where the model describes the pattern in its own words, plus (like any determined verdict) at least one cited finding.
• "undetermined", carrying the meaning "I couldn't tell". This one pairs with a required entry in the coverage section, the part of the report that lists what wasn't analyzed and why. An "I couldn't tell" that doesn't surface there would be a silent shrug; paired with a coverage gap, it becomes a signal the reader can act on.

The first line of enforcement sits in the schema itself, where breaking an obligation is a shape error:

"classification": {
  "enum": ["vendor_default", "builtin_selector", "custom_adapter", "other", "undetermined"]
},
"allOf": [
  {
    "if": {
      "properties": {
        "classification": { "const": "undetermined" }
      }
    },
    "then": { "required": ["note"] },
    "else": {
      "properties": {
        "finding_ids": { "type": "array", "minItems": 1 }
      }
    }
  },
  {
    "if": {
      "properties": {
        "classification": { "const": "other" }
      }
    },
    "then": { "required": ["note"] }
  }
]

The two hatches also attach at different levels, because they cover different failure modes.

• "other" covers a vocabulary failure: the model understood the pattern, my enum didn't. It only belongs on empirical taxonomies.
• "undetermined" covers an epistemic failure: the looking itself failed. That one belongs on any verdict the model has to detect, even the logically exhaustive ones (boundary status, for example, is a complete vocabulary: present, partial or absent. It still carries "undetermined", because a complete vocabulary doesn't guarantee the model can always reach a verdict).

So, our previous case now becomes:

tokenStorage: "vendor_default" | "builtin_selector" | "custom_adapter" | "other" | "undetermined"

This way, our tool remains testable on the fixtures (we know the calcified fixture must always produce "vendor_default"; an "other" there is a test failure, since fixtures are built from known patterns), while staying prepared for the world wild web.

Receipts, search records, and DRY fields

Three more structural rules round out the JSON, each one buying a specific kind of checkability.

Presence claims carry verbatim-quoted references. Besides anchoring the model in the reality it's analyzing, these references can be mechanically checked with zero extra LLM calls: open the file, go to the line, compare the quote.

{
  "id": "boundary-cognito-authport-interface",
  "claim": "presence",
  "statement": "An `AuthPort` interface defines the auth boundary contract used by the new Cognito surface.",
  "evidence": [
    {
      "file": "src/auth/port.ts",
      "line": 6,
      "quote": "export interface AuthPort {"
    }
  ]
}

Absence claims carry checked_patterns, their search records, which makes them auditable. Not as easy to check as grep-ing the references from the presence claims, but still mechanically valuable: a test can assert that the record covers every alternative pattern the vendor profile lists, and for live runs we get a track record instead of a vague "nothing's there" report. This is the rule aimed straight at the localStorage lie from earlier. The failure mode isn't gone (a model can still miss a pattern), but it can no longer happen in silence.

{
  "id": "boundary-no-contract-suite",
  "claim": "absence",
  "statement": "No contract test suite exercises the `AuthPort` shape; the audited scope contains no test files of any kind.",
  "checked_patterns": [
    "*.test.ts / *.test.tsx",
    "__tests__/ directories",
    "runAuthContractTests-style suite name",
    "vitest.config.* / jest.config.*",
    ...
  ]
}

Normalization: every fact is stated once. Counts are computed from the arrays, a rank is the array order, no derived fields get stored. This is the LLM flavor of the everlasting principle of DRY (Don't Repeat Yourself): anything stated twice can self-disagree within the same document.

The resulting artifact was an audit-schema.json file that specified the structure of the JSON output in great detail. You can inspect it here.

The harness: 4 free layers in front of every paid run

With the output redesigned, the testing layer could take shape. The guiding principle: cheapest feedback first. Before writing any checks, I produced a golden file: a hand-written example of what a perfect audit JSON looks like for one fixture. I used the old, manually-approved report and translated it backwards into the new schema.

Backwards-translation paid for itself before a single line of harness code existed. It exposed issues on both sides, in the old report's data and in my new schema: roughly half the anchors (code references) were broken, citing the wrong lines from the codebase; real findings cite multiple locations, so evidence had to become an array; some concerns simply don't arise in a given codebase, so not_applicable had to exist; I addressed these initial findings.

Then came the checks. 4 deterministic layers, stacked in front of any LLM involvement:

1. Shape. The audit JSON is validated against the exact schema file that ships with the skill.

// read from the skill package directly, never copied into the harness:
// the schema the tests enforce is byte-for-byte the schema that ships
const SCHEMA_PATH = join(HARNESS_DIR, "../skill/.../assets/audit-schema.json");

export function validateShape(doc: unknown): string[] {
  const schema = JSON.parse(readFileSync(SCHEMA_PATH, "utf8"));
  const ajv = new Ajv2020({ allErrors: true, allowUnionTypes: true });
  const validate = ajv.compile(schema);
  if (validate(doc)) return [];
  return (validate.errors ?? []).map(formatError);
}

2. Relational invariants. Rules that connect fields across the document: every referenced finding id must exist; every evidence quote must verify verbatim against the audited code; every "other" must carry