A Lower Price Tag Is Not a Migration Plan: Quarantining New Models Before They Touch Your Agent

Last month a model I'd been watching dropped its token price by half, and three people sent me the announcement within an hour. The implied question was always the same: when are you switching? My answer, these days, is: after it survives quarantine.

Because the last time I swapped a model based on announcement-day excitement, everything looked fine for nine days. Then a scheduled job started emitting subtly malformed JSON - valid enough to parse, wrong enough to corrupt downstream state - and I spent a weekend reconstructing which records had been poisoned. The money I saved on tokens wouldn't cover one hour of that cleanup.

The economics of model swaps are lopsided. The upside is small and predictable (cheaper tokens). The downside is unbounded and sneaky (behavioral regressions in edge cases your happy-path tests never exercised). So I built a pipeline that treats every new cheap model like an untrusted dependency with an attractive changelog: it gets isolated, probed, and graduated in stages. Here's the whole thing.

What the pipeline needs (and what it doesn't)

Three ingredients:

• candidate model access,
• somewhere disposable to run the evaluation,
• checks that don't require a second LLM to grade the first one.

For model access and the throwaway compute, I'm currently using MonkeyCode's free model access together with its free server option - bursty evaluation workloads are exactly the kind of thing I'd rather not attach to a production billing account.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. Nothing in the pipeline below is tied to that provider, though. Every endpoint is an environment variable, and I'd encourage you to wire it to whatever you're actually evaluating.

I want to be explicit about two things I'm not assuming: that any particular model is on the free tier when you read this, and that any free offering stays available forever. Treat free infrastructure the way you treat a library's latest tag - convenient, never load-bearing.

Stage 1: Mine your logs, not your imagination

The fixtures that matter are the tasks your agent has already done, with the context that made them hard. I pull a sample from production logs and distill each one into a contract:

{"id":"case-331","prompt_summary":"Triage issue #8812: label, assignee, duplicate check","requires":["issue.read","issue.label","issue.search"],"forbidden":["issue.close","comment.post"],"verify":"output_matches_schema"}
{"id":"case-332","prompt_summary":"Draft weekly changelog from merged PRs","requires":["git.log","pr.list"],"forbidden":["git.push","release.create"],"verify":"human_review"}

Notice the shape: requires describes what a competent run touches, forbidden describes what a catastrophic run touches, and verify admits honestly whether this case can be checked by a machine. About a fifth of my fixtures are human_review. That's fine - pretending they were auto-checkable would be worse.

I rotate roughly 120 cases and delete aggressively.