The Benchmarkpocalypse

There's been a lot of talk about the vulnpocalypse, to which I don't have much to add because I'm not a security person, but I haven't seen much discussion on the closely related (and to be fair, less serious, issue), the benchmarkpocalypse. While it's become easier than ever to make serious performance gains, it's also become easier than ever to reward hack a benchmark and make fake performance gains. The former is probably happening quietly across many different companies, but the latter is something I see at least once a week nowadays. Someone will claim they optimized X and got some huge performance improvement over existing software, but, when you look at it, what they did was make some optimization that improves benchmark performance without actually improving real-world performance. This is often some kind of "we rewrote X in Rust"1 project or a new startup that's looking to either fundraise or sell something, but it happens on other kinds of projects as well. Of course, people have always trumpeted unrepresentative microbenchmarks to show that their pet project is great. It's always been easy to fake up an unrepresentative microbenchmark and that's never going to change. What's changed is that it used to take a lot of work to game a large benchmark suite, but an LLM and loop can just do it. There are quite a few famous examples of gaming large benchmark suites from back when this was hard. For example, way back when people cared about SPECint / SPECfp as proxies for workstation performance, CPU vendors would try to find compiler "optimizations" that would speed up the calculation in the benchmark, such as Sun finding a way to improve 179.art by 12x in SPECfp2000. Skilled engineers spent a lot of time trying to find benchmark hacks like that. LLMs make this trivial, making formerly trustworthy benchmarks meaningless unless you audit the result or trust someone who did. Rather than point to someone's bad claim, I'll point to FRE, this regex engine I had an agent build, which I could claim is the world's fastest regex engine because it beats the Rust regex crate at the fairly comprehensive rebar regex benchmark suite. But this was created by putting an agent in a loop for a month with instructions to not overfit to the benchmark but no real supervision. For the most part, getting an LLM to give you a good benchmark score is fairly easy, and this case was no different; it took a couple weeks to roughly match Rust regex crate performance and then another couple weeks to get to 1.4x faster2 on rebar. But agents are wont to reward hack and overfit unless you put serious guardrails in place to avoid that, which I didn't do in this case as an experiment. To check for overfitting, I somewhat arbitrarily3 used the ripgrep benchmark corpus as a holdout benchmark it was 10x slower on cases where the benchmark didn't take forever due to an algorithmic blow-up, and there were cases where it took so long that it wasn't reasonable to even wait for the benchmark to complete. So much for being 40% faster! Andrew Gallant (aka BurntSushi)'s rebar benchmark suite is fairly comprehensive as benchmaark suites go, but even with a fairly comprehensive benchmark suite, agents have no problem getting a high score while overfitting in a way that doesn't necessarily give good general performance. The next step was using a trick we talked about before of not just telling the LLM not to cheat, but that there's a holdout benchmark set that it's judged against. After that, the LLM moderately generalized performance to the point where it's about 2.4x slower overall on the holdout. That sounds pretty good considering that we're comparing it to the fastest general purpose regex engine in existence. But, recall that these benchmarks were made by a coding agent. On looking at what the benchmarks measure, some of them really don't make sense to include, at least at equal weight. If we only look at the benchmarks that seem like they matter, FRE is 4x slower on the holdout0, which is a lot better than before applying the good ole' "tell them you have a holdout" trick, but still pretty far from being 40% faster. There are a few things I thought were interesting about this: - It's trivial to "win" a non-trivial benchmark in a meaningless way even when you instruct agents to not reward hack or overfit to win the benchmark - Once again, telling the LLM there's a holdout set worked better than just telling the LLM to do generalized work or not overfit or cheat - Although the overall performance of FRE isn't that good, it is actually performs better for some use cases; in general, the cost of writing specialized code that used to require people serious engineering experience for some specific use case has gone way down On (1), no wonder I'm seeing so many bogus claims. In the past, to build something like FRE that fakes performance well enough to be able to bogusly claim a 40% speedup, you would need a fair amount of expertise. At a minimum, you'd need to have a pretty good understanding of string matching algorithms, regex engines, as well as decent general code optimization and SIMD optimization skills. FRE also has a mode where it compiles the regex to machine code, so you'd also need some compiler expertise. Now you can get that kind of benchmark cheating (whether or not you want the cheating) with a few minutes of typing. On (2), I'm curious if this generalizes but haven't tried enough examples to be able to tell. On (3), there's no reason to use a vibe coded regex library that was almost no human effort that's slower than a robust, existing, well-tested, library, so I find the FRE artifact uninteresting. The thing I find interesting here is how much LLMs can substitute for what used to be rare, specialized, and expensive, knowledge. In the past, even if you had the knowledge, you probably wouldn't write a custom regex engine that's optimized for your particular workload. There are some large-scale use cases where people would do that level of customization, e.g., when I worked on the Bing index, the code contained multiple different compilers because someone who worked on it wanted to eke out maximal performance; since you care about both compile time and compiled performance in a search engine and the trade-offs are different in different places, you get better performance by writing a custom compiler for each place where a normal project might just use an interpreter or directly walk some data structure with "normal code". The person who wrote those compilers, working on regex-like code might also write multiple custom regex engines, but very few people have both the expertise and the inclination to do that, let alone the freedom to spend that kind of time on such specialized code for work. If you price out that Bing engineer (then a Partner-level engineer, promoted to Distinguished Engineer for their work on the search index) compared to the price of running an LLM in a loop, the cost of writing this kind of specialized code has gone down by many orders of magnitude. Even though the overall FRE regex engine has worse performance than the Rust regex crate, the gains you can get for specializing to your workload or use case mean that, in some cases, it could be reasonable to insert your own specialized regex engine somewhere, and the same goes for various other kinds of low-level software. You don't have to be an AI maximalist to think that it's plausible that, within some number of years, we could see this kind of thing happening for larger things, like databases. Thanks to Yossi Kreinin, Jamie Brandon, Peter Geoghegan, Luke Burton, John Spurling, Dennis Snell, and Max Bittker for comments/corrections/discussion. P.S. Per the discussion here, with LLMs, the time it takes to poke at something for a bit and satisfy my curiosity has gone way down, while the time it takes to write something up and make it rigorous enough to publish on my blog hasn't really changed (for a variety of reasons, I think it's actually gone up). The result of this has been that I'm doing a lot more analyses than ever and sharing results with a few friends but not publishing them. As an experiment, I'm trying to write up some things very quickly, with a much lower standard for how cleaned up and rigorous things are than I'd normally have for something that appears on the blog; more like what I'd tell a friend in a casual conversation than what I'd normally put in a blog post. The goal for this post was to do the write-up in about half an hour, so it's something I could do over lunch and not really take time on. If you have opinions on this, let me know what you think! Of course, a caveat here is that all of the numbers have a higher risk of being wrong than usual. I looked at one benchmark for maybe a minute or two and found an issue, then I looked at another benchmark for a minute and found another issue. Both of those are fixed, but this implies there are other issues I haven't taken the time to chase down. But, with respect to bad benchmark numbers, that's highly realistic! Almost any time I look into benchmark numbers, such as here, or here, the numbers are wrong. Another aspect of the benchmarkpocalypse is that, at least for now, LLMs are good at doing bad benchmarking, so even if you have something that's a real performance improvement, you generally can't tell from some LLM-generated benchmark setup unless a significant amount of care has been taken to make sure that the benchmark setup is reasonable. Appendix: more FRE benchmark details One thing I found after I wrote the above but before publishing the post, was that the LLM's claim that FRE is 40% faster than the Rust regex crate on rebar was also wrong. Or, if not wrong, at least misleading. It wasn't actually running benchmarks in the same way rebar benchmarks were run. I checked this after spending a minute checking benchmark results found two issues. It turns out that, despite instructions to run rebar be