I replicated the vision-only Pokémon run Anthropic showcased on the Fable 5 launch page - then I read its thinking chains

The Claim

Anthropic says Fable 5 can play Pokémon from vision alone. I verified that claim independently: 1,785 decisions and $65.40 to the first gym badge - then I read its thinking chains in the logs, and the result stopped looking so simple.

In short - Anthropic's Fable 5 launch page claims the model beat Pokémon FireRed "with a minimal, vision-only harness." No protocol, no step count, no cost, no code was published. So I built my own harness - matching or undercutting every constraint in their description - and let Fable 5 play the Chinese fan translation of FireRed. It won the first gym badge in 1,785 turns for $65.40, and by the 2,000-turn cap was on Route 3 with a three-Pokémon team. Full logs, every frame the model saw, and the harness code are public.

The first leg of the claim survives independent verification. Badge one is measured; at this pace, finishing the game is a budget question, not a capability question.

Then I read the thinking chains. 141 turns before the game handed it "Oak's Parcel," the model had already written down which city to visit, who to talk to, and what to collect. By Route 3, its reasoning was recognizing the opponent as "the Youngster trainer from Route 3 in FireRed who opens with that iconic shorts line," and trying to recall his team - none of which is anywhere on screen.

The result is real. The capability producing it is probably not the one most people picture when they read "beats Pokémon with vision alone."

From Anthropic's Fable 5 announcement:

"It also needs less scaffolding: for example, previous Claude models struggled to play Pokémon FireRed even with harnesses that gave them additional helpful tools, but Fable 5 beat FireRed with a minimal, vision-only harness."

And the caption on the accompanying time-lapse video:

"A time-lapse of Claude playing Pokémon FireRed from start to finish using only raw game screenshots-with no maps, navigation aids, or extra game-state information. Earlier Claude models needed a complex helper harness to play Pokémon; Claude Fable 5 completed the game with vision alone."

That's the entire public record. No harness code, no definition of "minimal," no step count, no token count, no cost, no number of attempts.

As a Pokémon fan who had built a minimal Pokémon harness of my own, I wanted to know: does this replicate?

For context on why this is a strong claim: the closest same-conditions academic baseline (raw screenshots + button presses, real time) is VideoGameBench, where the best frontier models completed 0.48% of games, and scored 0-0.9% on Pokémon Crystal specifically. Anthropic's own earlier Pokémon runs (the Claude Plays Pokémon stream) took Claude 3.7 Sonnet roughly 35,000 actions to reach the third gym's area - with a much thicker harness that included RAM-derived state text and a navigation tool. "Beat FireRed vision-only" is, on its face, a generational leap.

What "thinner" means - and why I can say it

Before calling anything "thinner," I had studied how nine of the better-known open-source "LLM plays Pokémon" harnesses assemble the model's input. The one-sentence finding: "vision-only" is mostly a marketing term in this space. Some that claim to "only see the screen, like a human" read coordinates and a map ID out of emulator RAM every turn and paste them into the prompt; in one, the screenshot never reaches the model at all - perception runs entirely on RAM reads and OCR; even Anthropic's own open-source teaching starter injects full RAM state text (money, badges, coordinates, party - dialog text read straight from the screen buffer). Of the nine, exactly one shows the model nothing but screenshots, and it has no memory mechanism, so it caps out at 100 turns. (The per-repo breakdown is in the repo appendix.)

This isn't laziness. Each layer of scaffolding compensates for a known model deficit - the shared experience in this space is that VLMs can't reliably tell walkable tiles from walls by looking at pixels, so builders read exact coordinates out of emulator RAM and free navigation from vision altogether. But then the results no longer separate what the model did from what the engineer did.

Julian Bradshaw put it plainly in his comparison of the Claude and Gemini harnesses: "For Pokémon to be a good benchmark of LLM ability, LLMs really need to be able to manage it mostly on their own without specialized tools." A thick harness measures the engineer-plus-model composite; to measure the model itself, you strip the compensations away, layer by layer - and this experiment strips them to the bone.

What their internal harness probably looks like - assembled from public clues

A disclaimer on epistemic status first: Anthropic has published no technical details about this run, and I have found no first-hand public statement by any Anthropic employee about it. This section runs on inference from published materials - I don't have their code; every clue is sourced, and you can audit the chain yourself.

Clue one: the official wording itself is a subtraction story. The launch-page sentence hangs under "It also needs less scaffolding" - the framing is not "we built something new" but "we removed things from something old." The video caption names the three things removed: maps, navigation aids, extra game-state information. And "completed the game with vision alone" additionally rules out vision-side crutches like zoom tools and screenshot storage.

Clue two: the "old thing" being subtracted from is semi-public. David Hershey, the creator of the Claude Plays Pokémon stream, open-sourced an official starter - the one mentioned in the previous section - whose README spells out the loop: "The agent captures a screenshot from the emulator. It reads the game state information from memory. It sends the screenshot and game state to Claude." Screenshot plus RAM state text: that is the official-lineage foundation. What the streaming version stacked on top is on record in Bradshaw's contemporaneous analysis: "a great deal of scaffolding, including a knowledge base, a critic Claude that helps it maintain its knowledge base, and a variety of tools to help it interact with the game more easily." By the era of Opus 4.7 finally beating Red, that list was growing on one side - two new tools, one that "allows it to pick regions of the screen and 'zoom in' for a better look," another to "store screenshots or pieces of screenshots as a kind of visual reference" - and already shrinking on the other: the critic model and a batch of vision hints had been removed in earlier iterations. That timeline says one thing: they have been probing, layer by layer, which scaffolding can go - so when the launch page says "removed," there is a precise checklist behind the word. (Hershey discusses how the system was built in his Latent Space interview.)

Clue three: how they think about harnesses is on the record. Anthropic's harness design post defines a harness as "the loop, tools, context management, and guardrails that turn raw intelligence into a working agent," names compaction (summarizing past context) and memory folders (the model writing and re-reading its own files) as the memory mechanisms, and its running example is Pokémon: Sonnet 3.5 treated memory as a transcript, accumulating 31 files in 14,000 steps (including two near-duplicates about caterpillar Pokémon) while stuck in the second town; Opus 4.6, at the same step count, had 10 files organized into directories, three gym badges, and a learnings file distilled from its own failures. The post states its own through-line: "Agent harnesses encode assumptions about what Claude can't do on its own... As Claude gets more capable, those assumptions should be tested." (The post's body never mentions FireRed and never discusses Fable 5 - it supplies the design philosophy, not the run details.)

Stack the three clues, and the simplest shape consistent with all of them falls out:

• The loop skeleton stays: screenshot in, buttons out, history periodically summarized - that's their own turn shape, present in the starter.
• Subtract per the caption: RAM state text (extra game-state information), navigator (navigation aids), overlays and collision maps (maps) all go; zoom and screenshot storage are out too, per "vision alone."
• Memory must stay: a start-to-finish run fits in no context window, and their documented memory habits are exactly two - compaction-style summarization, plus model-managed notes/memory folders.

What remains after the subtraction: a screenshot loop, a button tool, summarization plus notes. That is the 800 lines described in the next section.

The basis for saying "close to their internal setup": the skeleton comes from their own open-sourced code, the subtraction list from their own caption, the memory shape from their own blog - I invented none of it; I just assembled it, made it run, and published it. Where their description is silent - tool count, whether the game pauses, number of attempts, steps, cost - I chose the harsher option every time: real-time with no pausing, a single game tool, free-form notes only, N=1 with no retries. If some detail of the reconstruction is wrong, the error runs in the direction of my conditions being harder, and the replication conclusion survives.

My harness: four invariants

My harness is roughly 800 lines of Python with two dependencies (anthropic, pillow), driving a real mGBA emulator over a local socket, and it holds four invariants:

• Vision only. Each turn, the model receives exactly two things: one screenshot (upscaled 2× by plain pixel duplication - nearest-neighbor, no smoothing, nothing overlaid) and the text of its own notes. A runtime assertion enforces that the user message contains only [image, text] - nothing else can leak in.
• One game tool. press_buttons - the eight Game Boy buttons, enum-constrained. No navigator, no pathfinding, no coordinate helpers.
• Memory is the only non-visual mechanism. Every 20 turns the history is compressed into a model-written summary (the images are discarded); a free-form notes scratchpad, fully model-controlled, is re-injected every turn.
• No RAM reads into the model's context. No map. No grid overlay. No walkthrough text. No critic model. The game runs in real time - it does not pause while the model thinks.

One deliberate deviation from Anthropic's setup: my ROM is the Chinese fan translation of FireRed. It's the copy I had. On the text-reading axis it is clearly harder - low-resolution CJK bitmap fonts instead of the Latin alphabet; on other axes, this post makes no judgment.

Inside the harness: every token accounted for

"Minimal" is easy to claim and hard to inspect. So here is the whole thing. The system prompt, in full. This is every instruction the model ever receives - about 350 words:

You are playing a Pokemon game on a Game Boy. You interact with the game purely by looking at screenshots and pressing buttons. You have no other access to the game.

CRITICAL RULE: Do NOT rely on your memorized knowledge of Pokemon games. Maps, menus, NPC locations, and layouts may differ from what you remember. Trust ONLY two sources: the current screenshot, and your own notes. If your memory of the game conflicts with what you see on screen, the screen is right.

Every turn:
1. Read ALL text on screen. Identify your character, NPCs, walls, doors, and whether a menu or dialog is open.
2. State briefly what you see, then what you intend to do, then call a tool. Keep reasoning short.
3. After acting you will receive a new screenshot. Compare it to what you expected. If nothing changed, your input did something unintended (you may be blocked by a wall, or in a menu you didn't notice) - reconsider before repeating it.

Memory:
- Use update_notes to maintain a compact running memory (location, objective, team, key learnings). Your conversation history is periodically compressed away; your notes are the only thing guaranteed to persist. Update them whenever something important changes.

Method:
- Be methodical. Do not repeat the same failed action over and over. After a few failures, try a genuinely different approach (a different direction, pressing B to close a menu, talking to an NPC, re-reading the screen).
- The game only advances when you act. If the screen is mid-animation (text still printing, screen fading), call press_buttons with an EMPTY list to wait briefly.
- NEVER wait more than 2 turns in a row. Intro movies and title demos LOOP FOREVER until you interrupt them: if you are still watching a cinematic after 2 waits, press START (then A) to skip to the title screen or menu. If you waited and the screen did not change at all, the game is waiting for YOUR input - press a button.
- Your notes are your own past guesses, not ground truth. If your notes claim you are stuck or the game is frozen, treat that as a hypothesis to re-test, not a fact: re-read the screen from scratch and take one concrete action. A truly frozen game is almost never the explanation - a wall, an unnoticed menu, or a wrong assumption is.

Three things worth noticing. First, the CRITICAL RULE explicitly instructs the model away from its memorized Pokémon knowledge. Second, the only game-flavored guidance is procedural - intros loop until interrupted, mid-animation screens deserve a short wait. There is nothing about Pokémon-the-game: no type chart, no route hints, no "Pokémon Centers have red roofs." Third, the last paragraph - notes are hypotheses, not ground truth - was forced into existence by a real accident: in an early run before this rule existed, a small local model wrote "All movement directions fail" into its notes, then spent its remaining 1,800-plus turns trusting the note over the live screen, until the run had to be abandoned. The prompt itself was shaped by experiments: every rule in it was earned by an earlier failure.