My fact-checker said CONFIRMED about a group that doesn't exist

I built a tool that fact-checks crypto claims. You paste a sentence like "Whales are holding $XYZ" and it plans the on-chain calls that claim needs, runs them against Nansen's API, and answers CONFIRMED / OVERSTATED / CONTRADICTED / UNVERIFIABLE with the numbers that decided it.

Last week it answered CONFIRMED about a group of wallets that does not exist. Not "returned a weak signal." Not "was slightly off." It printed the word that means this claim is true, next to a cohort the data source tags exactly zero members of.

Here is how a tool whose entire purpose is refusing to overstate ended up overstating, and why the README had already warned me.

The promise I'd already written down

Three limitations ship in the repo's JUDGE.md. The second one says this, verbatim:

Nansen's Whale label is sparse. A post's "whale" is often a big wallet Nansen does not tag; when no Whale-labelled wallet exists in the token the tool says UNVERIFIABLE rather than pretending a CONTRADICTED.

I wrote that line early, and I wrote it because it is true of the data.

My API notes from the build (docs/DX-REPORT.md) record the measurement it came from:

• tgm/flow-intelligence 1d on WETH: whale_wallet_count = 0
• tgm/holders label_type=whale on the same token: 38 holders

Zero labelled whales on wrapped Ether. Not an edge case - the label is genuinely sparse, and the flow columns that report it are DEX-only. "No wallets of this class" is an ordinary, frequent answer from this API, not a rare one.

So I knew. I documented it. And then I enforced it in exactly one of the two places it mattered.

The choke point that wasn't

Claims come in three shapes: buying, selling, holding. The first two go down one branch, holding goes down another. Both need the same question answered first - does this class of wallet exist in this token at all?

That question has a function:

/** Does the subject class exist in this token at all (any window, any endpoint)? */
export function presence(e: Evidence, cls: "smart_trader" | "whale"): boolean {
  return (
    (e.flow1d?.[cls]?.wallets ?? 0) > 0 ||
    (e.flow7d?.[cls]?.wallets ?? 0) > 0 ||
    (e.holders?.count ?? 0) > 0 ||
    (cls === "smart_trader" && (e.table?.traders ?? 0) > 0) ||
    (e.named ? e.named.buyRows + e.named.sellRows > 0 : false)
  );
}

The flow branch calls it, and does the right thing:

if (!presence(e, cls)) return R("UNVERIFIABLE", "U-NOCLASS", [
  `Nansen tags no wallet as ${who} in this token (24 h, 7 d, holders) - ` +
  `the wallet in the post is not one Nansen labels`,
]);
return R("CONTRADICTED", "C-NOBODY", [ /* ... */ ]);

The holding branch never called it. Not once. Here is what it did instead - read it as a ladder, because that is how it fails:

function decideHolding(claim, e, rules, T7, R) {
  const who = subjectName(claim.subject); // "Whales" | "Smart Money"
  const cls = subjectClass(claim.subject); // "whale" | "smart_trader"
  const h = e.holders;
  const net7 = e.flow7d?.[cls]?.net ?? null;

  if (!h && net7 == null) return R("UNVERIFIABLE", "U-HOLD", [/* no data at all */]);

  // ↓ no presence() check here ↓
  if (h && h.count < rules.minHolders) return R("OVERSTATED", "O-HOLDERS", [/* ... */]);
  if (h && net7 != null && net7 <= -T7 && h.delta7d < 0) return R("CONTRADICTED", "C-EXIT", [/* ... */]);
  if ((h && h.delta7d < 0) || (net7 != null && net7 <= -T7)) return R("OVERSTATED", "O-TRIM", [/* ... */]);

  return R("CONFIRMED", "A-HOLD", [/* "balances are not shrinking" */]);
}

Two ways to be wrong, and the second one is much worse

Zero holders. tgm/holders answers with an empty page, so h.count === 0. The first rung catches it - 0 < 5 - and returns OVERSTATED, whose UI copy means partly true. Partly true about nobody. Bad, but at least it is hedging.

The holders call failed. Now h is null, and the seven-day flow answered with wallets: 0. Look at the ladder again: every remaining rung is guarded by h &&. O-HOLDERS needs h. C-EXIT needs h. O-TRIM needs h or a net flow below the negative threshold - and the net flow is 0, which is not below anything.

So the claim falls all the way through and lands on the last line:

{
  "label": "CONFIRMED",
  "rule": "A-HOLD",
  "presence": false,
  "reasons": [
    "7 d Whales net flow $0 (threshold $5K)",
    "Whales balances are not shrinking"
  ]
}

"Whales balances are not shrinking." Technically unfalsifiable and completely true, in the way that "all the unicorns in my garage are healthy" is true. There are no whales. Nothing is shrinking because there is nothing. And the tool rendered that as CONFIRMED, the strongest word it owns, on the one screen a reader actually looks at.

The path needs no exotic input. One failed HTTP call to a sparse endpoint, on a token whose class is empty - which, per my own notes, is WETH.

The fix is one line, which is the annoying part:

if (!presence(e, cls)) return R("UNVERIFIABLE", "U-NOCLASS", [/* same message the flow path uses */]);

One line, plus a comment, plus 45 lines of tests. The whole commit is +51.

I found the first case because I ran the source past an outside model, which spotted the zero-holders rung. Verifying its claim is what turned up the second: I wrote a throwaway test with holders: null to see what the other branch did, and got CONFIRMED back. The review found the hedge; checking the review found the lie.

All 13 recorded fixtures still reproduce with byte-identical sha256 evidence hashes after the change, which is the thing that let me ship a one-line edit to the verdict engine four days before a deadline without flinching.

What I actually take from this

A limitation stated in prose is not a constraint. I had the right belief, written in the right file, in public, in the artifact judges read. It did nothing. Prose cannot fail a build.

An invariant enforced in one branch is enforced in zero branches. presence() existed. It was correct. It was even called - just not on every path that needed it. A shared helper that callers must remember to call is a convention, and conventions decay the moment someone adds a fourth claim type.

The dangerous bug was in the fallback, not the happy path. Every test I had written covered tokens with data, because that is what the demo uses and what the fixtures recorded. The failure mode lived where the data was absent - h === null, wallets: 0, a call that didn't answer - and absence is exactly what nobody writes fixtures for.

If you have a decision function with a default return at the bottom, the question worth asking is not "is this the right default" but "what is the emptiest input that can reach this line?" Sparse-by-design data is a first-class input. I treated "no whales" as degenerate. It is not. It is Wednesday.

Honest limits, since that is the whole point of the tool

The fix makes the engine refuse more often, and refusing is the correct answer, but it is still a refusal - a user who pastes a claim about an unlabelled wallet gets UNVERIFIABLE, not an investigation of who that wallet is.

Two Nansen endpoints can report different numbers for the same 24 hours (flow-intelligence vs smart-money/netflow). I surface both rather than reconcile them; docs/DX-REPORT.md has the measurement. Net flows are priced at current rates and drift. The evidence hash covers the integers that decided the label, so a fixture replay always matches and a live re-run usually does.

For the numbers people ask for:

• 352 tests
• 100% statement/branch/function/line coverage on the engine
• 13 fixtures that replay offline with zero network calls
• cold p50 4.0 s / p95 5.0 s
• ~10 API credits per verdict
• 0 of 154 live calls failed in the benchmark

Details in docs/BENCH.md.

The tool is at https://rebuttal.edycu.dev and the code is at https://github.com/edycutjong/rebuttal - the commit in this post is fedb50b, and the test that pins both failure paths is packages/core/test/review4.test.ts.

If you have a verdict function with a return at the bottom of a ladder, go and check what the emptiest possible input does to it. That is the whole article.