Distinguishing wrong from absent

The Sharp Edge of Exact-Match Grading

model-drift grades models weekly on a frozen suite with an exact-match grader - no LLM judge, so a score change is real. That design has a sharp edge: a call that returns no valid answer scores identically to a wrong one. A refusal, a max_tokens truncation, a timeout, a parser failure on a quietly-changed schema - all land as zero, indistinguishable from the capability collapse the board exists to catch.

A Real-World Example

I hit the catchable version: a model appeared to drop from 69% to 3% overnight. The run log showed a 429 on 34 of 35 calls - a rate limit scored as a regression. Rate limits are catchable only because they leave a status code; refusals and truncations don't.

The Board's Solution: Exclude by Reliability

The board excludes by aggregate reliability: it drops a run's accuracy point only when that run's reliability falls below 50% - a catastrophic-infra floor, never a single failed call. That restraint matters because failures aren't missing-at-random: the long, hard prompts are the ones that hit token caps and timeouts, so a blanket drop-every-failure rule would exclude failures that correlate with difficulty and inflate accuracy on the discriminating tasks.

An eval board has to tell a wrong answer from an absent one; miss that and you're scoring the provider's uptime, not the model. The design was already exclude-by-class, not blanket.

Code and Methodology

Code: github.com/egnaro9/model-drift

Writing the methodology in public is how it gets read this closely.