The Hutter Prize measures intelligence better than any OpenAI benchmark

Winning a leaderboard doesn't hold any value for me now. Every week, a new model surpasses an existing benchmark, but at the same time, I become less confident in those numbers. The contest nobody's gaming There is a contest that is 20 years old now, and it measures intelligence more honestly than any OpenAI eval The Hutter Prize is at stake. It has offered cash rewards for compressing a Wikipedia XML dump without loss of information since 2006. As of February 2020, competitors have instead been challenged to compress the 1-gigabyte file enwik9 (although the file was previously the 100MB enwik8). No "reasoning" it can fake by pattern-matching training data. Just a file, and a hard number for how small you made it. 📦 Why compression is a lie detector Most benchmarks fail to account for one important thing. If you want to achieve good text compression, you need to be able to predict that text effectively. Predicting and compressing can essentially be the same thing. If you can predict the next word, you don't have to remember the word, you remember the prediction. The prominent goal of AI (strong AI) research is to develop systems that perform better, are easier to use, learn faster, and generalize from few data, that is, can do more with less. The capacity for good compression is closely connected to intelligence... Intelligence, however, is a controversial term while file sizes are not. A file size is something that cannot be distorted. The rule that kills benchmark theater This is my favorite section. Your score is the size of the compressed file plus the size of your decompression program, combined. This one rule eliminates the cheat every AI leaderboard has to suffer. You can't shove Wikipedia into a massive neural net and declare it compression because the net is part of your score. There are additional constraints: → Single CPU core, not a GPU farm → Under 10GB of RAM → Finish in under 100 hours Therefore, it recognizes and encourages real algorithm intelligence rather than the one who has rented the most powerful computing cluster. While intelligence is a slippery concept, file sizes are hard numbers. That last line is the whole point. The numbers are brutal and beautiful The standing record was achieved by Kaido Orav and Byron Knoll on September 3, 2024. The fx2-cmix algorithm managed to compress 1,000,000,000 bytes into precisely 110,793,128 bytes. About 11.1% as a fraction of the original size. It was a 1.585% improvement over the previous record of 112,578,322 bytes that Orav set himself back in February 2024. They were awarded €7,950 for those optimizations. Prize founder Marcus Hutter personally finances the prize which pays €5,000 for every 1% the world record is exceeded by the total pool of €500,000. Consider the gap. It took months to achieve a 1.585% improvement. No company announcement can replace the simple truth that progress in research and development is incremental. The big labs already agree Here's a new perspective that could put an end to the debate. The Google DeepMind, Meta AI, and Inria team demonstrated in their ICLR 2024 paper "Language Modeling Is Compression" that models such as Chinchilla 70B are the best compressors out there. This applies to text, images, and audio. Marcus Hutter contributed to the paper. Hence, the identical concept that is behind the €500,000 competition is the process that is behind every model that people compare. The CoRecursive podcast put it perfectly when they said that "compression is the ultimate prediction test". We've just been cheering for the noisy proxy instead of the direct, clean signal. A benchmark score provides information on the performance of a model based on a set of pre-defined questions. In contrast, a compression ratio indicates how effectively the model represents the actual data distribution. What this changed for me I used to take benchmark tables at face value. Now I see them as a form of advertising. When a lab says "state of the art," I want to know the one thing they never publish: how small can it make the file, program included. That number is indisputable. You were either able to predict the following byte or you were not. The Hutter Prize may not be flashy, but rather a steady effort to compress a file size, with competitors who are in pursuit of that 1% improvement with each attempt. However, it is truthful. And in an industry saturated with leaderboard drama, being truthful is the most uncommon standard there is. My question to you is this: if compression is the cleanest measure of intelligence we have, why are all labs still racing on benchmarks that no one can reproduce? Top comments (0)