50,000 property tests passed while my app crowned an impostor

You hear "buy PEPE", type PEPE into your wallet, and get fourteen tokens with the same name and the same frog. Which one do you buy? I built a tool to answer that. You type a ticker, and it asks Nansen's API for every token with that name across chains. Then it checks who actually holds and trades each one: labelled Smart Money, whales, top-PnL wallets, exchange flow, tagged top holders. Exactly one card turns green, or the tool abstains. Last week it crowned a token that its own scorer had flagged as an impostor. My property test ran 50,000 generated cases on that exact decision, and they all passed. While writing this post I found out why, and the answer was worse than I expected. Two rules in two files The engine judges each candidate twice. The impostor rule lives in score.ts . A candidate is an impostor when nothing labelled has touched it, it has no meaningful exchange flow, and it is either brand new or barely held: // packages/core/src/score.ts const impostor = f.labelledWallets === 0 && exchMag = ABSTAIN_THRESHOLD && !(winner.labelledWallets === 0 && (winner.recognisedHolders ?? 0) facts({ ...r, labelledWallets: r.smartTraderWallets + r.whaleWallets + r.topPnlWallets + r.publicFigureWallets })); For a candidate to have zero labelled wallets, all four independent draws have to land on 0. Across five seeded runs of 10,000 lists, about 44,000 candidates per run, the generator produced zero candidates with zero labelled wallets. So it produced zero impostors, and zero cases where the pre-fix crown() crowned one. Even with the right assertion, the old code passed. It is not just the impostor line. Every branch that matters for a zero-labelled token (the SHIB2 bar, the PEPEGA veto, the failed-holders abstain) was sitting in a corner of the input space the generator never visited. I changed one thing in the scratch copy, not in the repo. Each wallet count became fc.oneof(fc.constant(0), fc.nat(500)) , so zero comes up about half the time for each class. Then I ran it against the pre-fix crown() with the !winner.impostor property. fast-check found a counterexample on all five seeds, after 47, 53, 71, 109 and 244 cases. Two things went wrong, and fixing either one alone would not have caught the bug: - The invariant was the implementation restated. Write properties about what the output must never do, not about which branches the code takes. - The generator had no weight where the bugs were. A uniform draw almost never produces the edge case your domain revolves around. Here that case was "zero". If a value drives a branch, put it in the generator on purpose. As of this writing, the repo's generator still has that gap. The PEPEGA regression test pins the actual bug, so the crown rule is covered. The property test just doesn't cover as much of it as "50,000 cases" suggests. The rest of the system The score is plain arithmetic over four Nansen endpoints: - search/general : 0 credits - tgm/flow-intelligence andtgm/token-information : 1 credit each, for up to 8 candidates - tgm/holders : 5 credits, for the top two finalists only Market cap, volume and search rank are left out of the score, because those are what an impostor can buy. The repo includes a benchmark: 12 queries × 2 cold runs against the live API, on 2026-09-16. Cold p50 was 3.6 s, p95 7.2 s, warm p50 3 ms. Verdicts cost 18.6 credits on average and 26 at most. npm run verify replays twelve recorded verdicts offline with the same decision hash. It needs no API access and no network. The web page streams every Nansen call into a side rail as it fires, with the endpoint, credits, latency and a short hash of each response. Limitations - search/general decides the candidate set. The tool can't warn about an impostor Nansen hasn't indexed. - Label coverage is uneven across chains. A real token on a thinly labelled chain can lose to a bridged copy on a busier one, which is why there's a chain filter. - Flow data covers a 7-day window, so a real but dormant token can look quiet. - The zero-labelled escape hatch is still there. On 2026-09-16, AI16Z andPEPE UNCHAINED were crowned on wealth-tagged holders alone. The card says "0 labelled wallets" so you can see the weakness, but it's a weaker verdict than a green card backed by 90 labelled wallets. - DOGE crowns a Solana meme DOGE, because native DOGE has no contract to compare against. - USDC is the slow outlier, about 15 s cold. Nansen times out on some of its lookups, and the call drawer shows each timeout. Try it - Live: https://whichone.edycu.dev. Try PEPE , thenPEPEGA . - Code: https://github.com/edycutjong/whichone. npm run whichone -- PEPE --explain prints every term of the score (you need your own Nansen key). If you write property tests, count how often your generator actually produces the values your branches depend on. I hadn't, until this post. Top comments (0)