LLMs are real, AI is fake
Hacker News

LLMs are real, AI is fake

LLMs are Real, AI is Fake

Comments

Today's Links

  • LLMs are real, AI is fake: No, it didn't "go rogue."
  • Hey look at this: Delights to delectate.
  • Object permanence: Why 9/11 Means We Must Support My Politics; Blogs x 9/11; Jimmy Wales v Britannica's EiC; Hollywood astroturfs Australia; Agents v queer YA; Soft landings for dirty cops; Leaked Stingray manual.
  • Upcoming appearances: Budapest, Edmonton, South Bend, Hudson, Calgary, Winnipeg, Vancouver, Victoria, Ottawa.
  • Recent appearances: Where I've been.
  • Latest books: You keep readin' em, I'll keep writin' 'em.
  • Upcoming books: Like I said, I'll keep writin' 'em.
  • Colophon: All the rest.

LLMs are Real, AI is Fake

Once you understand the corporate culture of AI "hyperscalers" consists primarily of everyone cooking their brains by locking themselves in the bathroom, holding flashlights under their chins, and saying "Aaaaaaaaaay Eyeeeeeee" until they wet themselves in terror, a lot of things snap into focus: https://pluralistic.net/2023/06/04/ayyyyyy-eyeeeee/

It explains how a company can simultaneously be staffing up an enterprise sales division while also constantly freaking out at the thought that its product has "a 10% chance of ending humanity": https://www.latimes.com/business/story/2026-09-11/is-there-really-10-chance-ai-could-kill-us-all

Unreliable Narrators

Given that AI insiders have mostly cooked their brains in this fashion, it behooves us all to treat these people as unreliable narrators of their own products' capabilities. Remember: every time you repeat a story about how awfully, terribly dangerous their products are, you help them raise more investment capital, which is a key input for their business (hooking up statistical engines to money-furnaces): https://peoples-things.ghost.io/youre-doing-it-wrong-notes-on-criticism-and-technology-hype/

The Hugging Face Hack

Take the story about how OpenAI's chatbots hacked the servers of Hugging Face, another AI company, as a way of cheating on a hacking challenge called "Exploit Gym." Even the technical press can't help itself when it comes to this kind of thing, and the reportage has been full of references to Skynet and other science fictional conceits: https://theaicronicle.com/en/news/ethics/skynet-day-openai-hugging-face-hack

Understanding the Hacking Challenge

Last night, a man at my event in Manchester started shouting that AI was "setting its own goals" and wouldn't stop interrupting to insist that this was going on. He left shortly thereafter, so he didn't get a chance to hear me explain what actually happened, which is a pity. To understand the truth about the Hugging Face hack, you could do a lot worse than to listen to Ed Zitron and Cal Newport's recent podcast conversation on Ed's "Better Offline" podcast: Newport does an admirable job of breaking down how these "autonomous hacking" tools work.

How the Chatbot Works

The first thing to understand is that a chatbot isn't really directing the operation. Instead, the chatbot serves as a kind of front-end to a database of earlier hacking challenges that is repeatedly queried by a simple program written in Python, an easy-to-master programming language. Here's how that works:

  1. The Python program starts by prompting the chatbot with the nature of the challenge: "I'm participating in a hacker capture the flag (CTF) challenge where I have to break into a remote server and retrieve some information. How should I start?"
  2. The chatbot consults its training data - years' worth of captured CTF sessions in which human teams competed to achieve an objective like this one (CTF matches are a routine feature of hacker conferences, and the server logs and chat transcripts from the competing teams are published afterward for the edification of other hackers and security pros).
  3. The chatbot then outputs something like: "The first thing is to find out more about your target server. Run the following command-line instructions to locate the server's IP address and find out which server software it's running."
  4. The Python program relays these command-line instructions to normal Unix utilities running on its own hardware.
  5. Then it takes the output of those programs and goes back to the chatbot, which isn't really following the action, so the Python program has to include everything that's happened to this point in its prompt: "I'm participating in a CTF challenge where I have to break into a remote server and retrieve some information. I ran the following commands to learn more about the target server, and here's what came back. Now what?"
  6. The chatbot feeds the Python script more likely commands to try, and after running those, the Python script loops back to the top, appends the output to its prompt, and goes back to the chatbot.

The Hacking Challenge Precedents

Every part of the Hugging Face incident has precedents in the training data, including the OpenAI chatbots' tactic of hacking into a rival's servers. That happens in Capture the Flag games at hacker cons: teams break into each other's systems to get a peek at the parts of the problem they've solved. That's allowed! It's a hacking competition. Not only that, it's a tactic used by spy agencies: the NSA has a doctrine called "third-party collection," where they break into other spy agencies' systems to harvest all the intel they've gathered.

The NOBUS Doctrine

There's also fourth-party collection, when the NSA hacks into another security agency that, in turn, has hacked into another security agency, and the NSA steals all the secrets of both agencies: Which is not to say that the OpenAI/Hugging Face hack is nothing. It's something, all right: but it's a specific something, with an explicable, even foreseeable trajectory.

The NSA's Irresponsible Handling of Vulnerabilities

Once you understand that these are chatbots that were designed to complete challenges like this, using tactics like this, you can understand that the chatbots didn't "go rogue." They did what they were designed to do, and because OpenAI ran them with inadequate supervision (without a "human in the loop" that checked each iteration through the Python loop to ensure it hadn't gone off the rails), they trashed a competitor's servers. Designing autonomous, malicious software is generally considered irresponsible and dangerous.

The Hacking Tools Tradition

If you showed up at Defcon and gave talk about how your autonomous malware did something unexpected and damaged someone else's computers, the first question from the audience would be "Why are you so shit at making secure sandboxes?" It wouldn't be "How are you so awesome at making hacking tools?" The fact that OpenAI is making it much easier for unskilled people to break into and damage servers is indeed very bad news, but it's not new bad news. Irresponsible parties have been doing this for years, most notably the NSA.

The NSA's NOBUS Doctrine

The NSA has a division that researches bugs in widely used software like Windows. Sometimes when it finds a serious bug it will warn Microsoft about it so that Microsoft can fix it and keep Americans (and others) safe from malicious actors who also discover this bug and use it to attack them. But sometimes, the NSA (and other "security" orgs, like the CIA) will discover a really juicy bug and then keep it secret, so that they can use it to attack their adversaries. This is a doctrine called "NOBUS," which stands for "No One But Us" - as in, "No one but us is smart enough to find this bug, so we can leave it unpatched without putting Americans in danger."

The Consequences of NOBUS

Well, in 2017, the NSA lost track of a Microsoft Windows vulnerability that they'd discovered and hoarded, code-named "EternalBlue." After EternalBlue found its way into the wild, some halfway competent hackers spliced it into some boring, everyday ransomware, giving that ransomware a new lease on life. Within a few months, the stupidest people on the internet were shutting down some of the most important systems in the world, demanding cash to return them: https://en.wikipedia.org/wiki/EternalBlue

The Modern Digital World

The hacking tools that the chatbot companies are developing stand to carry on this very stupid tradition. It is scary, but not because the chatbots are waking up. It's scary because the world's IT systems are indifferently created and poorly maintained and riddled with vulnerabilities: This week, I had a couple of opportunities to hash this over in public with Riley Quinn; first at a book launch in London and then on the Trashfuture podcast: https://www.patreon.com/trashfuture/posts/what-would-do-169247456

LLMs are Real, AI is Fake

Riley had a very good way of summarizing this: "LLMs are real, AI is fake." LLMs - chatbots trained on things like CTF logs that can break into servers - are real. They're on a continuum with other hacking tools that have been steadily demonstrating the fragility of the modern digital world, albeit without inspiring anyone in power to do anything about it. "AI" - chatbots that wake up, "set their own goals," and "spontaneously" start hacking servers - is fake. It doesn't have "a 10% chance of ending the human race." The Hugging Face hack isn't a mysterious, supernatural occurrence. It's a Python loop and a chatbot. The people responsible didn't accidentally create god: they created autonomous malicious software and then failed to closely monitor it, resulting in it doing something both foreseeable and bad.

Worrying Productively

It's fine to worry about this new suite of tools that give even stupider people the ability to trash even more computers. You should worry about that - and demand better security practices from firms and governments, including a blanket prohibition on NOBUS-style vulnerability hoarding. That's a productive kind of worrying, with a chance of addressing your area of concern. It's infinitely more reasonable than locking yourself in the toilet with a flashlight and saying "Ayyyyy Eyyyyyye" into the mirror until you wet yourself.

Today's Links

Read on Hacker News ↗ ← Back to News

Comments

No comments yet. Start the discussion.