Moving beyond vibe-based evals

Everyone has shipped a "looks good to me" model. It never ends well. The 92% hallucination catch rate sounds impressive but what about the 8% that slip through? That is the difference between a demo and production. I want to know if these pipelines generalize across different use cases or just overfit to one set of test data. How many false positives did they trade for that 92% recall? Real evaluation is a cost-benefit problem not a single number. Curious if others are seeing similar tradeoffs in their eval pipelines.