Vibes Are Not a Benchmark: A 30-Minute Harness to Test Whether a Free Coding Model Can Touch Your Repo

Last month I wrote about the six questions I ask every new model. Those questions are good for a first impression, but they have a blind spot: they tell me how a model talks about code, not whether it can survive contact with my code. So I built a small, ugly, reproducible harness that runs any coding model against five tasks drawn from my own git history, scores the results with my real test suite, and writes the outcome to a CSV. This post is that harness, the reasoning behind it, and where free-tier tooling fits without falling apart.

Why repo-derived tasks beat benchmark tasks

Public benchmarks (HumanEval-style) have two problems for day-to-day tool selection. First, they're saturated and contaminated - models have seen them. Second, they don't look like your work. My work is: small refactors in a Python service, tests that fail for boring reasons, and occasional SQL migrations.

So instead of asking "is this model smart," I ask "does this model reduce the time I spend on the five task shapes I actually do." The trick that makes this cheap: your git history is a