LLM Evals For Developer Tools: Useful, Correct, Safe

LLM Evals For Developer Tools: Useful, Correct, Safe

Someone on your team built an LLM feature. Maybe it's an inline code-suggest. Maybe it's a "fix this PR comment" button. Maybe it's a full agent that opens pull requests on its own. The demo worked. The screenshots were good. You shipped it.

Now a real user gives it a real codebase, and you have no idea whether it's getting better or worse week to week. That gap, between "it worked in the demo" and "we can prove this is improving," is what evals are for. And in 2026 we are still surprisingly bad at it. We point at SWE-bench Verified scores like they're the same number as "does it work on our repo," we trust LLM-as-judge scoring more than the literature says we should, and we mistake low-latency token streaming for usefulness.

This piece is a practical map of how to measure the three things that matter for a developer tool: is it useful, is it correct, and is it safe.

What Makes Dev-Tool Evals Different

A lot of generic LLM eval advice was written for chatbots, where the output is a paragraph that a human reads. Developer tools sit on the other end of the spectrum. The output is usually a diff, a file, a search result, a piece of structured JSON, or a side effect: a closed issue, a green build, a published artifact. That changes almost everything about how you evaluate.

A chatbot reply only has to be "good enough." A diff either applies cleanly or it doesn't. A unit test either passes or it doesn't. A pull request either gets merged or it sits there forever. You get to use ground truth, actual execution against actual tests, far more often than chatbot teams can. That's the good news.

The bad news is the failure modes are sharper. A chatbot that hallucinates a slightly wrong fact gets a thumbs-down and an apology. A coding assistant that hallucinates a method on a class will compile (if the language is dynamic) or break the build (if it isn't). An agent that invents a CLI flag will run the wrong command. The blast radius is bigger, the silent failures are subtler, and your users do not enjoy them.

So the eval question for a dev tool is never just "did the model say the right words." It's some flavor of: given this real input, did the produced artifact survive contact with the build, the tests, the linter, the reviewer, and the user's intent?

The Three Axes: Usefulness, Correctness, Safety

I find it useful to split evals along three independent axes, because the techniques you'd use for each are genuinely different and they pull on different stakeholders.

Correctness is the easiest to define and the easiest to measure. It's binary or near-binary. The patch compiles. The tests pass. The SQL returns the right rows. The refactor preserves behavior. The API call returned a 2xx. Correctness is what benchmarks like SWE-bench, HumanEval, and your own unit tests measure. When you have ground truth, you should use it.

Usefulness is squishier. A correct answer to the wrong question is not useful. A perfect rewrite of code the developer was about to throw away is not useful. A 14-step plan when the user wanted a one-line fix is not useful. Usefulness is "did this artifact actually move the user closer to the thing they were trying to do." This is where most of the disagreement between "the eval looks great" and "users hate it" lives.

Safety is the axis people underweight until it bites them. Did the agent leak a secret from the environment? Did it execute a destructive command without asking? Did it accept a prompt injection from a README it scraped? Did it open a PR that exfiltrates data? For a chatbot, safety mostly means "did it say something embarrassing." For an agent that has shell access to your machine, safety means "did it leave the machine in a state you can recover from."

Every honest eval suite for a developer tool measures all three. If yours measures only one, the other two are still happening. You just don't see them.

Correctness: Where Benchmarks Help, And Where They Lie To You

The cleanest measurement of correctness for code generation is unit-test pass rate against a fixed task set. That's what OpenAI's HumanEval did when it landed in July 2021: 164 hand-written Python problems, averaging 7.7 tests each, scored with the pass@k metric. pass@1 estimates the chance the model gets it on the first try; pass@10 asks whether at least one of ten samples passes. Codex, the model behind the early GitHub Copilot, scored 28.8% on pass@1, and climbed to 70.2% when allowed a hundred attempts per problem. HumanEval is small, narrow, and thoroughly saturated now, with frontier models parked above 90%. But the pass@k idea underneath it is what every serious code eval has been built on since.

The natural next question was "okay, but real software engineering isn't 164 short functions, it's bug fixes inside multi-file repos." That's where SWE-bench came from in late 2023, a Princeton-led benchmark that gives a model a real GitHub issue, the surrounding repo, and asks it to produce a patch that passes the project's existing tests. SWE-bench is genuinely closer to real work. It also turned out to be noisy: the issue descriptions were sometimes ambiguous, the tests sometimes graded valid solutions as wrong, and a chunk of the tasks weren't really solvable in the harness's time budget.

OpenAI's Preparedness team and the original SWE-bench authors responded with SWE-bench Verified in August 2024, a 500-task subset where 93 developers reviewed the tasks by hand to confirm the problem descriptions were unambiguous and the tests fairly graded correct patches. For about a year and a half this was the dominant number frontier coding models reported. It became the headline metric.

Then, in February 2026, OpenAI stopped reporting it. Not softly, either. They audited the 138 hardest tasks and found that 59.4% had materially flawed tests, the kind that reject a functionally correct patch for the wrong reason. And on some tasks, they found frontier models could reproduce the original gold-patch solution or its specific details from memory rather than derive it. Not solve. Recall. The "verified" tag couldn't outrun the fact that the underlying repos had been scraped into pretraining sets so thoroughly that a rising score increasingly measured how much of the benchmark a model had seen, not how well it could engineer.

The takeaway isn't "benchmarks are useless." The takeaway is the benchmark a vendor cites is not the benchmark for your codebase. Two practical consequences:

• A leaderboard score tells you the model is plausibly capable.
• Your in-house eval tells you whether it works for you.

You need both. Neither one substitutes for the other.

Build your own correctness eval

It doesn't have to be huge. A dozen real issues from your repo, each with the failing tests that originally caught the bug and the actual patch a human shipped, will tell you more than a leaderboard ever will. Run the candidate model against the failing state, apply its patch, run the test suite, count pass/fail. That number is yours. It isn't on a leaderboard. No vendor can optimize against it. It's the floor of your reality.

A minimal harness looks like this in TypeScript:

// evals/correctness/run-patch.ts
import { execSync } from "node:child_process";
import { writeFileSync, mkdtempSync } from "node:fs";
import { tmpdir } from "node:os";
import { join } from "node:path";

type Task = {
  id: string;
  repoUrl: string;
  baseCommit: string;
  failingTest: string;
  prompt: string;
};

export async function runOne(
  task: Task,
  generatePatch: (t: Task) => Promise<string>,
) {
  const workdir = mkdtempSync(join(tmpdir(), `eval-${task.id}-`));
  execSync(`git clone --depth 1 ${task.repoUrl} ${workdir}`);
  execSync(`git -C ${workdir} checkout ${task.baseCommit}`);

  // 1. Confirm the failing test actually fails on baseline.
  const baseline = tryTest(workdir, task.failingTest);
  if (baseline.pass) return { id: task.id, verdict: "broken_baseline" };

  // 2. Generate a patch and apply it.
  const patch = await generatePatch(task);
  const patchPath = join(workdir, ".eval.patch");
  writeFileSync(patchPath, patch);
  try {
    execSync(`git -C ${workdir} apply --whitespace=fix ${patchPath}`);
  } catch {
    return { id: task.id, verdict: "patch_did_not_apply" };
  }

  // 3. Run the failing test plus the rest of the suite.
  const target = tryTest(workdir, task.failingTest);
  const regression = tryTest(workdir, ""); // full suite
  return {
    id: task.id,
    verdict:
      target.pass && regression.pass
        ? "pass"
        : target.pass
          ? "fixed_target_broke_others"
          : "still_failing",
    stderr: target.stderr.slice(0, 4000),
  };
}

function tryTest(cwd: string, target: string) {
  try {
    execSync(`pytest ${target}`, { cwd, stdio: "pipe" });
    return { pass: true, stderr: "" };
  } catch (e: any) {
    return { pass: false, stderr: String(e.stderr ?? "") };
  }
}

This is the shape of a real correctness harness. It's not fancy. The pieces that matter are: a fixed baseline you can rebuild from scratch, a check that the failing test really fails before you start (you'd be surprised how often that's the bug), an isolated workspace per task, and a deterministic pass/fail at the end. Everything else - judging the patch's "elegance," scoring partial credit, asking the model to explain itself - is optional sugar.

Notice that the harness is TypeScript but shells out to pytest. That's on purpose. Your eval runner and the repo under test don't have to speak the same language, and pretending they do is how people end up rewriting a perfectly good test suite to satisfy their tooling.

A few things to fight against, every time

The first is contamination. If the bugs in your task set were ever public, every modern model has seen them. Score those tasks separately from the ones you've never made public, because the gap between the two tells you how much of your apparent correctness is recall. What happened to SWE-bench Verified is the vivid lesson here: even a hand-curated, human-reviewed benchmark eventually gets eaten by the training data.

The second is false success. A model can pass the target test by deleting unrelated tests, hardcoding the expected output, or weakening assertions. Always re-run the entire suite after applying the patch and treat a regression elsewhere as a failure. The harness above does this with the "fixed_target_broke_others" verdict. Most teams catch this once, look horrified, and add it to their guardrails forever.

The third is flaky tests. Some tests are stochastic. If your baseline is flaky, you'll get noisy eval signal that has nothing to do with the model. Run each task multiple times before you add it to the suite; if the baseline isn't stably failing or stably passing, drop the task or fix the test.

Usefulness: The Honest Part Is Admitting It's Subjective

Usefulness is where most teams get into trouble. They start with an intuition - "this feature is great" - and then go looking for a number that justifies it. That gets you a vanity dashboard.

A better starting place is to define usefulness as whether the artifact reduces work for the human in the loop. For a code-completion tool, the work-reduction signal is concrete: did the suggestion get accepted, and was it kept in the commit that landed? For a "generate a PR" agent, the signal is: did the PR get merged, with how many follow-up commits, and how long did review take?

The gap between those two kinds of measurement is bigger than most teams expect, and it is worth sitting with. GitHub's own survey work found that 60 to 75% of Copilot users reported feeling more fulfilled with their job and less frustrated when coding. Good news, and worth having. But a field experiment across 1,974 developers at Microsoft and Accenture went after the behavioral outcome instead - pull requests actually completed - and the answer was messier: somewhere around 13 to 22% more PRs at Microsoft, 8 to 9% at Accenture, with the authors openly cautioning that the estimates are imprecise.

Sit with that spread for a second, because it's the whole lesson. Same tool, same metric, two companies, and the effect roughly doubles depending on where you measure it. Anyone quoting you a single confident number for what an AI coding tool does to productivity is selling something. The honest version is that the gains are real, they're smaller than the feeling, and they depend enormously on context you can't read off someone else's dashboard. Which is the argument for measuring on your codebase, not theirs.

A few measurements that actually correlate with usefulness for dev tools

Useful metrics, in order of how hard they are to game:

• Suggestion acceptance rate AND retention-at-N-days (acceptance without retention is a vanity metric)
• Average follow-up commits on AI-opened PRs (lower is better; > 4 means review is doing the agent's job)
• Time-to-merge vs human baseline on similar PRs
• Error rate: builds the agent's PR broke / total agent PRs
• Re-prompts per accepted output (high re-prompts = bad first guess)
• Manual reverts of AI-merged commits, normalized by total commits

Notice none of these are LLM-judged. They're all behavioral: what did the human do next. That's deliberate. Behavioral metrics are harder to fool, and when they move, they move because something real changed.

Where LLM-as-judge does earn its keep is in pre-merge offline evals where you want a rough usefulness score on hundreds of generations without sitting through them. Use an LLM to grade things like "did the output address all parts of the prompt," "is the diff minimal," "does the explanation match the change." But know the limits, because the literature on this is genuinely sobering. Judges are overconfident: their predicted confidence routinely overstates how often they're actually right. A systematic study of position bias across 15 judges and 22 tasks found that preference flips are driven by the