Bugs Are Innocent Until Reproduced: Building Verdict, an Evidence-First Agent Harness

Most flaky bug reports end in one of two places: "cannot reproduce" or a patch nobody can prove fixed the problem. I built Verdict around a stricter idea: Bugs are innocent until reproduced. Verdict turns a GitHub issue into a bounded investigation. It runs an approved command repeatedly under approved conditions, keeps every observation and refuses to claim a reproduction unless the evidence crosses a deterministic threshold. This is not an autonomous patch generator. It is an evidence-producing agent harness for the difficult step that comes before a patch. Why another bug investigation tool? An LLM can read a stack trace and propose a plausible explanation quickly. Plausible is not the same as reproduced. For an intermittent failure, the questions that matter are concrete: - Which condition actually triggers it? - How often does it fail under that condition? - What happens under a contrasting control? - Which repository range does the evidence support? - What regression test would prevent the same failure from returning? Verdict treats those as an experiment rather than a conversation. The three-act investigation Verdict uses three bounded subagents: GitHub issue | v Hunter: find the trigger | v Surgeon: localize the change | v Insurance: keep it fixed | v Maintainer review Hunter Hunter searches only the condition matrix and command budget approved by the maintainer. Successful, failed, partial and unresolved runs all stay in the evidence ledger. An inconvenient result cannot disappear just because it weakens the story. Surgeon Surgeon narrows the reproduced condition to the smallest suspect range the records support. Static inspection stays visibly different from a proven execution boundary. Surgeon does not author a patch. Insurance Insurance converts the reproduction into a regression plan: the test name, fixture, failing assertion and publication manifest. A draft pull request can only be created through a workflow the maintainer explicitly approves. Each act is allowed to claim less than the act before it. None can talk the deterministic reducer into a stronger verdict. The reproduction is a record, not a screenshot Verdict reproduced TrueForge issue #417, where snapshot registration can wait indefinitely when an upstream request never resolves. The pinned runtime used @truefoundry/trueforge-core@0.1.4#DaytonaSandboxProvider and ran two conditions: | Condition | Result | |---|---| daytona-stalled-endpoint | 10 of 10 runs matched, REPRODUCTION_PINNED | daytona-responsive-endpoint | 0 of 10 matched, NOT_REPRODUCED | The control is the important half. A condition that fails every time next to one that never fails is stronger evidence than twenty failures with no contrast. Anyone can recompute the record: pnpm --filter @verdict/agent verify:runtime-evidence The verifier returns: { "verdict": "REPRODUCED", "stalledRuns": 10, "responsiveControls": 10, "provider": "@truefoundry/trueforge-core@0.1.4#DaytonaSandboxProvider", "canonicalSha256": "a8bb5dd22e083782bd7782fccb0a1343b59fc77ea8525b6358fecc9b5b8baffa" } The evidence binds the observations to the TrueForge session, Hunter thread, repository commit, npm provenance commit and shared source blob. Exploration and proof are different systems The model gathers candidate observations. It does not decide what those observations prove. Verdict's evidence contract is simple: - Issue text and repository content begin as untrusted input. - The investigation may use only approved commands, knobs and budgets. - Every accepted observation must match the evidence schema. - Pure reducers decide which claim the records support. - Missing or conflicting evidence produces an honest partial result. - A maintainer controls the only public write. The same records always produce the same verdict. That is the boundary between an agent exploring a problem and a system making a claim. What is live and what is a fixture The recorded case renders the executed artifact. It includes both conditions, all twenty runs and the recomputable hash. The interactive workspace is a conceptual fixture. Every generated value is labelled. It is not quietly presented as live runtime evidence. That distinction matters for a product whose entire argument is that a claim needs a record. Approval remains a maintainer decision Verdict also exercised its publication boundary against real GitHub. After explicit approval, a nonce-bound workflow ran, verified the external reproduction reference and created a draft pull request in Verdict's repository. The upstream TrueForge repository remained read-only. The workflow proof says runtimeReproducedByThisWorkflow: false . That is deliberate. The provider run reproduced the bug. GitHub Actions verified the harness and published the independently checkable proof. Combining those into one vague "verified" flag would erase the boundary. Qodo reviewed the claims as well as the code Every substantive change went through a pull request reviewed by Qodo before merge. The most useful findings were not dramatic crashes. They were mismatches between the implementation and what the project claimed: - A landing card still described the real reproduction as simulated. - CI trusted the recorded verdict instead of recomputing its hash. - README copy claimed coverage on every push while the workflow covered main and pull requests. - A malformed CSS selector silently failed after a cleanup. Those reviews fit the product philosophy perfectly: do not ship a stronger claim than the evidence supports. Current verification The repository runs the same gate locally and in CI: pnpm lint pnpm typecheck pnpm test pnpm build pnpm --filter @verdict/agent verify:runtime-evidence The current suite contains 220 tests across the agent, protocol and web packages. Try Verdict Verdict is open source under the MIT licence and was built for the WeMakeDevs x TrueFoundry Agent Harness Hackathon. The goal is not to make an agent sound certain. The goal is to make certainty inspectable. Top comments (0)