A New Cheap Model Dropped. Here's the 2-Hour Canary Test I Run Before Touching It

Every few weeks a new coding model launches with a price tag that makes the incumbent look embarrassing, and my feed fills up with people rewriting their configs on day one. I've been burned by this twice: once a "drop-in replacement" silently stopped emitting valid unified diffs, and once a cheaper model passed all my prompts but tripled the retry rate on long files, which erased the savings. So now I don't evaluate new models on vibes or on leaderboard screenshots. I run a small canary suite built from my own repository history before the candidate model gets anywhere near real work. This post is that harness: the task extraction script, the runner, the decision table, and the honest limits of the approach. This builds on a personal scorecard I've written about before, but the goal here is different - not "rank models in general," but "answer one narrow question: is this specific cheap model safe to route my traffic to?" Step 1: Mine your own git history for tasks Benchmarks test what benchmark authors care about. Your git history tests what you care about. I pull completed tasks straight from commit metadata: #!/usr/bin/env bash # extract_tasks.sh - build canary tasks from your own repo history. # Each task = the state before a real commit + the human-written # commit message as the instruction. The real diff becomes the reference. set -euo pipefail REPO="$1" OUT="canary_tasks" N="${2:-30}" mkdir -p "$OUT" cd "$REPO" git log --format='%H' -n "$((N + 1))" -- 'src/**' | tac > /tmp/canary_commits.txt i=0 while read -r commit; do parent=$(git rev-parse "$commit^" 2>/dev/null) || continue # Skip merge commits and giant diffs - they make noisy tasks. [ "$(git cat-file -p "$commit" | grep -c '^parent')" -eq 1 ] || continue files_changed=$(git diff --name-only "$parent" "$commit" | wc -l) [ "$files_changed" -le 4 ] || continue msg=$(git log -1 --format='%s%n%n%b' "$commit") diff=$(git diff "$parent" "$commit") mkdir -p "../../$OUT/task_$i" echo "$msg" > "../../$OUT/task_$i/instruction.md" echo "$diff" > "../../$OUT/task_$i/reference.diff" git archive "$parent" | tar -x -C "../../$OUT/task_$i" 2>/dev/null || true i=$((i + 1)) done str: ref = (task_dir / "reference.diff").read_text() out = task_dir / "candidate.diff" if not out.exists() or not out.read_text().strip(): return "hard_fail" # empty / unparseable output apply = subprocess.run( ["git", "apply", "--check", str(out)], cwd=task_dir, capture_output=True) if apply.returncode != 0: return "hard_fail" # malformed diff ref_files = set(l.split()[1] for l in ref.splitlines() if l.startswith("+++ b/")) out_files = set(l.split()[1] for l in out.read_text().splitlines() if l.startswith("+++ b/")) if not out_files <= ref_files | {f.replace('b/', 'a/') for f in ref_files}: return "soft_fail" # scope creep return "pass" # shape is right; eyeball diffs after results = {p.name: grade(p) for p in Path(sys.argv[1]).iterdir() if p.is_dir()} print(json.dumps(results, indent=2)) This grading is intentionally shallow - it checks shape and scope, not semantic correctness. Semantic checking is step 3, and it's manual on purpose. Step 3: Read ten diffs, not a dashboard Aggregate scores hide the failures that matter. After the run, I open the candidate's diff next to the reference diff for ten randomly chosen tasks and ask one question: would I have caught this in code review, or would it have shipped? Silently-dropped error handling and "helpfully" reformatted files both look fine in a pass-rate number and are exactly what a cheaper model tends to introduce. If I find either pattern twice in ten samples, the model doesn't get routing traffic regardless of its score. Where the free tier fits This harness is cheap to run but not free - thirty tasks times N candidate models adds up, and re-running it every time a new model drops is precisely the cost that makes people skip evaluation and YOLO their configs instead. Disclosure: This article was prepared as part of MonkeyCode's product outreach. This is where I've been pointing the harness at MonkeyCode: its free model access covers the candidate side of the comparison for supported models, and the free server option means the runner itself - extraction, grading, result storage - runs without me paying for a box or tying up my laptop. The practical effect is that "evaluate the new hyped model before adopting" stops being a cost decision and becomes a habit. Two honest caveats. First, "free" tiers change - treat this as a way to run the evaluation now, not a permanent subsidy, and keep the harness provider-agnostic (mine just reads an env var for the endpoint) so you can move it in an afternoon. Second, free access shapes which models you can canary this way; if the model you actually want to test isn't available there, run a smaller task set against a paid endpoint rather than skipping evaluation entirely. The routing decision table Once a model passes the canary, I still don't route everything to it. The output of the whole exercise is a routing table, not a winner: | Task type | Route to | Why | |---|---|---| | Boilerplate, renames, mechanical refactors | Cheapest passing model | Hard fails are caught by git apply + CI | | Bug fixes with a failing test in hand | Cheapest passing model | Test is the oracle | | New feature code, no tests yet | Stronger model | No oracle → soft fails ship silently | | Anything touching auth, billing, migrations | Stronger model + mandatory review | Cost of a miss dwarfs token savings | | Long-context work (large files, many files) | Whatever won that specific canary subset | Long-context regression is the most common cheap-model failure I've seen | Limitations, and who shouldn't do this - Thirty commits is a vibe with structure, not statistics. It catches gross regressions and format breakage. It will not catch a 3% quality decline, and it can't compare two good models. - Your history is not your future. If the new work you're planning (a new language, a new service shape) isn't represented in the commit window, the canary tells you nothing about it. Extract tasks from the repos you're actually about to work in. - Reference diffs aren't ground truth. The human commit might itself have been mediocre. You're testing "can it do work shaped like my work," not "is it correct." - Don't bother if your volume is tiny. If you run a few agent tasks a week, the evaluation overhead exceeds any plausible savings - just pay for the model you trust. This pays off when you're routing real traffic or when your team is about to standardize on a model. - Never canary on a repo with secrets in its history, and keep the candidate model's sandbox as locked down as you'd keep any agent's. The harness above is the whole thing - extraction script, grader, table. If you want somewhere zero-cost to run your first pass, MonkeyCode's free server and free model access are a reasonable starting point; the scripts don't care where they run. The broader point: a model launch is a marketing event, and your git history is the only benchmark that knows what your work actually looks like. Two hours of canary testing is cheaper than one silent diff corruption. Top comments (0)