Benchmark an AI Agent Migration Without Believing One Speedup Number

Build a Paired Workload from Real Tasks

Model migrations and dramatic agent speedups are recurring headlines. A single “2.2x faster” number cannot tell you whether your production workflow improves. Build a paired workload from real, redacted tasks. Stratify by repository size, language, tool count, context size, and task type. Run old and new systems against the same pinned commit in isolated workspaces, randomize order, and repeat enough times to expose variance.

Measure the Right Metrics

Report median and tail latency, task success, regression count, human rework, tokens, retries, and total cost. Include timeouts and invalid patches in the denominator. Separate model latency from queue, tool, checkout, and test time.

A sample result record looks like this:

{
  "task_id": "fix-017",
  "system": "candidate",
  "success": true,
  "tests_passed": 184,
  "wall_ms": 91200,
  "model_cost_usd": 0.42,
  "tool_calls": 19,
  "human_rework_min": 6
}

Replay Failures Before Launch

Before launch, replay failures: duplicate tool results, network loss after a side effect, worker restart, context overflow, and cancellation. A faster system that repeats writes or cannot recover is not cheaper.

Define an Acceptance Rule

Use a predeclared acceptance rule, for example:

• No correctness regression above one percentage point
• p95 at least 20% lower
• Total cost per accepted task lower

Publish for Reproducibility

Publish confidence intervals and the workload definition. The benchmark should make a decision reproducible-not decorate one already made.