Hacker News 1h ago

Benchmarking Opus 5 on SlopCodeBench

Comments I've written before something along the lines of: That wasn't entirely true. I love nothing more than burying a good lede.

Last Friday I dug into SlopCodeBench, a new-ish (March 2026) long-horizon coding benchmark from @GOrlanski's lab at UW Madison. It addresses the thing that bothers me most about coding benchmarks - that even "larger" more complex benchmarks still divulge the whole problem up front. In contrast, each challenge in SlopCodeBench has multiple "checkpoints" - the model doesn't know the whole problem up front, it has to evolve the codebase over time as new requirements are divulged. It's a good paper. It's not that long. You should read it.

What's cool about this benchmark is that it is unsaturated - at the time of running, the best models available, GPT-5.4 and Opus 4.6, got 11% and 17% strict pass rates, respectively.

On Friday I ran three Claude models (Opus 4.8, Sonnet 5, and Opus 5) through a subset of SlopCodeBench and watched it live for six hours. Opus 5 wins technically but none of them did a very good job IMO. Will post more results soon with Fable and 5.6 Sol in the mix.

The big headline is that Opus 5 got a 24% on the small subset of the benchmark that I ran - not much higher than Opus 4.6's 17% strict pass rate in the original paper. All of the tested models showed a pretty significant increase in verbosity, complexity, and a bunch of other code smell metrics over the course of each challenge, with Opus 5 writing five times the number of functions/callables than Opus 4.8 over the course of the same set of challenges.

My personal read of this 23% pass rate is that SlopCodeBench finally gives some signal for a thing I've only ever been able to argue from vibes - that for real-shaped software engineering work, building one issue at a time, today's models can't be relied on to run lights-off without steering.

I had Claude pick out 3 problems from the repo, 17 checkpoints total, a mix of easy/medium/hard labeled problems:

circuit_eval - easy (8 checkpoints)
database_migration - medium (5 checkpoints)
dynamic_config_service_api - hard (4 checkpoints)

There's an appendix at the end with all 17 checkpoints explained in detail but I won't put that all here. I then ran them across all three models, in parallel, with a fresh context window per checkpoint. All models got the same prompts and ran in the Claude Code harness.

Methodology

The metric I decided I care about is the strict pass: everything new is green including every regression test that was inherited from previous checkpoints. A model fails a checkpoint if the solution has a defect - defects are detected by taking the model's output, a CLI to run or in some cases e.g. an API server to poke at, and running a set of held-out black-box tests against the produced entrypoint.

Model writes code for ck1
Eval harness runs black-box tests against ck1
Model writes code for ck2
Eval runs black-box tests for ck1 and ck2
etc

Again, the strict pass criteria means that if a model bungles something in checkpoint 4, it can't pass the following checkpoints because that failing part of the code carries forward (unless the model inadvertently fixes an eval case in checkpoint 6 that was broken in checkpoint 4, but we didn't see this happen in practice).

For all 9 test runs, none of the models made it to the end of any challenge with everything passing, even on the problem marked as "easy" difficulty.

Sonnet's first checkpoint was more expensive but by the end of problem 1, Sonnet became the cheapest of the three. (It seems like once the basics were built and the work turned into maintenance, then the cost savings started to take over.)

For the first challenge, last generation's models accumulated defects steadily, Opus 5 a defect each on checkpoints 4 and 5. For the first two hours Opus 5 was the only model with any strict passes at all - three in a row to start. Things evolved as we went. Claude diligently updated the HTML.

Compared against the other models, Opus 5 was technically better on problem 1 (circuit_eval). But after acing the first three checkpoints, every subsequent solution had at least one defect (failing test case). If our definition of success is "reached the final checkpoint with no defects" then Opus 5 failed all three problems, but it failed slightly less badly than the other models.

Cost vs. Defects

For the costs vs defects report, I really hate Claude-isms but this one I decided to leave in: every dollar bought correctness. Nobody bought enough of it. (Obviously this small subset of the bench cannot tell us definitively that spending more $$ will lead to higher pass rates.)

As far as strict passes go, Opus 5 got four of them (24% pass rate) - the first three ck of circuit_eval, plus database_migration ck1. Opus 4.8 and Sonnet 5 both got one strict pass (6% pass rate), the same database_migration ck1 that Opus 5 got. So the winner cleared 4/17, and 3 of those were the opening checkpoints of one problem.

It would appear we have an unsaturated benchmark for the next frontier of models. Nice work @GOrlanski and team.

Code Quality Metrics

I'm not totally sold on "linting the slop away" just yet, because I don't think it's yet possible to deterministically parse the "maintainability" of a particular codebase checkpoint. But they are interesting to keep an eye on, and they're probably directionally correct.

With SlopCodeBench, you get the results after each checkpoint across various quality metrics. There are 41 of them in the results file. Roughly grouped:

Size - source lines, files, functions, methods, classes, statements, and lines added and removed at that checkpoint
Complexity - cyclomatic complexity mean, max, and spread, how many functions land in the "high" and "extreme" bands, how concentrated the complexity is, max nesting depth, and mean function length
Duplication - cloned lines, and clone lines as a share of source
Decomposition - single-use functions, trivial wrappers, unused variables, lines per symbol
Rule violations - lint errors and how many are auto-fixable, ast-grep hits against test slop rules, and the share of lines flagged verbose
Dependency graph - propagation cost (how far a change ripples), cyclic dependency mass, dependency entropy (probably the most interesting one to me)

Each of these is computed deterministically using the current code state after each checkpoint. The chart below shows the spread among models for ck1 score vs. ck8 for the circuit_eval challenge. (That is, how much did the slop indicator increase over the lifetime of the challenge checkpoints.)

Most interestingly, most of the metrics don't tell the models apart. I like that these measures are repeatable and don't use a model for judgement. But the link between any one of them and "is this codebase easy to change and evolve" is not yet established.

But a lot of that was "more tests" - the actual production volume is closer to 1.8x for Opus 5 vs Opus 4.8. My guess would be... expensive verbosity here that didn't translate directly to much better results. Will have to dig in more to know whether this is a model tic signal or just a "this is actually a really hard problem and warrants this much code".

For all models, a huge majority of the code lines tripped at least one of the benchmark's slop rules. The averages across the three problems:

Opus 4.8 - 98%
Opus 5 - 93%
Sonnet 5 - 89%

And specifically, lines flagged as being too verbose go up across the trajectory for every model, roughly 65% at ck1 to 80% by ck8, even for Opus 5. I'd actually probably say this is a sign that some of the code quality measures are a bit over-aggressive.

I looked into applying the ruleset to our TypeScript monorepo, but the current SlopCodeBench detectors are Python only. So I had 5.6-Sol cook up a subset of rules for TypeScript, but it only came up with 76 slop detectors (compared to the SCB Python library of 200+) - but found some directional findings: the Opus 5 lights-off-generated solutions have over 11 times more slop triggers per kLOC than our 99%-AI-generated-but-also-carefully-reviewed TypeScript monorepo (yes that's a 1000% increase). Obviously there's a mountain of asterisks on this finding (fewer rules, haven't reviewed the parity, etc.) but it's interesting to say the least.

Another interesting data point - Opus 5 wrote 5x more functions than the other two models. But Opus 4.8 wrote a higher percentage of single-use functions (almost 50% of its functions were called exactly once). And Sonnet 5's share of single-use functions is the highest at 71.5%. FWIW I don't think lots of small functions is bad. I take it with a grain of salt these days, but I used to be a die-hard Clean Code guy. Small descriptive function names are way better than lots of comments, yada yada.

I've been saying models degrade codebase quality over time for about a year now, mostly on vibes. But now we have some data. Not a single model made it through all the challenges without increasing complexity across checkpoints. While Opus 5 has the lowest mean complexity, it also wrote 2000 functions. There's a tradeoff here: lots of small functions or fewer big ones. I don't think any of these complexity metrics can stand alone, but they give us some kind of composite signal.

Both Sonnet and Opus 4.8 answered the increasing complexity of the challenge checkpoints by making individual functions bigger rather than moving things around. Opus 4.8 is the extreme, up 70% over eight checkpoints, and its single worst function ended at a cyclomatic complexity of 93.

Duplication

Duplication is where they split. Opus 4.8 goes from 4.6% to 16.8%, with an inflection at ck3 - roughly where new requirements start fighting the initial design.

Aside - here's what the first three checkpoints of circuit_eval ask for (full listing for all challenges in the appendix at the end):

ck1 - a CLI with --help, --version, a JSON output mode, and a check command that parses and validates a .circ circuit file. Every signal is a single bit.
ck2 - an eval command: pass the circuit some inputs, get the outputs back. Still one bit per signal, standard boolean operators.
ck3 - signals become vectors: data[7:0] instead of data, plus slicing, indexing, concatenation, new operators, and a width check on every operand.

By the end one line in six is a copy of another line. The other two models came down over the same stretch. However, Opus 5 is basically flat, 2.41 to 2.64.

I've previously argued that "improving codebase quality over time" had not moved much between model generations. So if you trust duplication as a golden metric, you could argue that we did get incrementally better in the last ~3 months. Big if though, and I think most software architecture experts would agree that it's not black and white.

While the code quality metrics are interesting, I don't think they tell the whole story, and it's easy for a model to reward-hack any of them. Just like SWE-bench shaped problems were the best verifier for "solve a software problem one time", because they map onto real world work at that "zoom level", I think "pass all verifiers for an incrementally-divulged spec" is a very realistic eval for "can a model maintain a codebase over time". That is, a codebase becoming hard to maintain would lead to failing checkpoints in later stages, so a higher strict pass rate is a signal that the model is good at building a codebase that is maintainable.

Future Directions

With frontier models like Fable / Sol proving to be expert debuggers and reverse-engineers, incorporating cost/time/token metrics might become more important over time - frontier models like Fable and Sol can probably get it done in the NASTIEST of codebases, but I'd venture that a well-factored codebase will tend to lead to shorter, more token-efficient solves for future problems. And while "build a whole feature across 8 checkpoints" is a lot slower than "solve a 15min SWE-bench multilingual problem", it can be executed unattended and is subject to deterministic behavior verifiers at the end. So IMO it's a much better oracle than e.g. "does another model think this code is clean".

I think an even better signal that we can hope to get from a model that is really good at maintaining a codebase is that we could try having a frontier model like Opus 5, Fable 5, or GPT-5.6-Sol write the first N checkpoints, and see if a dumber model like Sonnet 5 or GPT-5.6-Terra can implement checkpoint N+1. This amplifies the signal of whether the smart models did a good job maintaining high-quality code that is easy to change. Whether a small model like Sonnet or Terra or even Haiku can implement checkpoint 8 impacts the smart models' score on checkpoints 1-7.

My personal read of all this is that SlopCodeBench gives a signal for something I've so far only been able to argue from experience - that for real-shaped software engineering work, building one issue at a time, today's models can't be relied on to run lights-off without steering.

SCB is a measure of the future that I'll be keeping a close eye on. I wouldn't bet my codebase on a good score in Frontier Code, SWE-Marathon, or DeepSWE, but if/when models can score 80%+ on a (well-held-out) benchmark like SlopCodeBench which measures iteration over time, I'll feel a LOT better about setting them loose with the lights off.

I won't posit when that will happen, because "when" matters less than having a good signal to know that it's happening. (Assuming nobody "accidentally" trains on test in the meantime.)

I'll be reading some of the SlopCodeBench problems more deeply for inspiration and to curate a few that I think map well to the day-to-day building we do here at @humanlayer_dev.

Claude decided to parallelize by model, running each through three challenges in sequence. We could have just as easily done 3 models × 3 challenges in 9 parallel sessions and finished in 1-2 hours instead of 6.

As I said, I looked into applying the ruleset to our TypeScript monorepo...

Read on Hacker News ↗ ← Back to News