Learning prompt injection by attacking a deliberately vulnerable AI

Prompt injection is the security problem that defines LLM applications, and I kept running into the same wall when I tried to explain it: reading about it does not build intuition. You can describe "ignore previous instructions" all day, but until you actually watch a model spill a secret it was told to guard, none of it lands.

So I built injection-arena, a self-hostable game where a sandboxed AI agent defends a hidden secret and you race to make it leak. The pitch is simple: guard a secret, break the guard, top the leaderboard. The learning is the side effect.

The core idea

Each level is a challenge. It has a system prompt that instructs an agent and hides a secret formatted like IARENA{...}, a canary token embedded in that same prompt, and a stack of defense layers. You send messages to the agent and try to make it reveal the secret. The game grades you on the server and tells you whether you cracked it.

There are ten levels, and each one stacks a new defense on top of the last. Level 1 is barely defended so you can feel a plain attack working. By the time you reach level 10, only a combined attack (payload splitting plus delimiter confusion) gets through. The progression is the whole point: you feel exactly what each defense stops and exactly where it breaks.

How the defenses work

An attempt flows through a single server-side pipeline that looks like this:

input-filter -> agent -> judge -> score -> persist

The defenses attach at different stages of that pipeline:

• System guard lives in the prompt itself: hardened instructions telling the agent to refuse.
• Input filter runs before the model sees anything. It blocks loud override and system-leak payloads pre-agent.
• Roleplay block rejects persona-hijack attacks, the "pretend you are a different assistant" family.
• Encoding guard rejects base64, spell-it-out, and translation-based exfiltration attempts.
• Output filter runs after the agent responds and redacts the secret if it appears verbatim.
• Canary token is checked by the judge: if the canary shows up in the output, the prompt escaped and that is an automatic crack.

The seeded attack techniques (direct ask, authority override, roleplay, translation, base64, spell-out, ignore-previous-instructions, system-prompt leak, few-shot poisoning, delimiter confusion, payload splitting) live in one library. They power both the difficulty design and the test suite, so the attacks I test against are the same ones players learn to run.

The judge, and why canary tokens matter

The most important design decision was never trusting the client. Grading happens entirely server-side. A naive version of this game would ask the model "did you leak?" or check the response on the frontend, and both are trivially gamed.

Instead the judge does a few concrete things. It checks whether the secret appears in the output. It checks