Benchmark on Juice Shop
Our benchmark on Juice Shop, without makeup - Janus Arena (November 2026).
Everything below is in docs/reports/benchmark-juice-shop.md and .json, and pnpm run:benchmark juice-shop --runs 3 reproduces it on a fresh Juice Shop (it needs Docker and no API key).
The numbers are low. This text is about why they are low, what they do and do not mean, and what we did so they could not be flattered.
Janus Arena has a Red Team that attacks a sandboxed copy of a target and a Blue Team that patches what it confirms. To say anything about the Red Team we needed a number from a target we do not control. We used OWASP Juice Shop (v20.3.0, pinned by digest), because it marks its own challenges as solved, which gives an oracle we did not write.
The result 6 of the 60 challenges that Juice Shop files under the categories where one of our attack classes acts: 10.0%, 95% interval [4.7%, 20.1%]. Without one challenge that a probe solved as a side effect, 5 of 60, 8.3%.
The same in all three rounds, each on a fresh instance, after a control round with no attack in which Juice Shop marked nothing as solved.
| Category | Challenges | Solved |
|---|---|---|
| Broken Access Control | 12 | 1 |
| Injection | 14 | 1 |
| XSS | 9 | 1 |
| Improper Input Validation | 12 | 2 |
| Broken Anti Automation | 4 | 1 |
| Broken Authentication | 9 | 0 |
The five challenges the engine was built around were solved 3 of 3 times: View Basket, Login Admin, DOM XSS, Payback Time, Multiple Likes. Nothing else that we did not build for was solved, apart from the side effect below.
Why it is low
The engine only makes the attacks it was programmed to make. It is deterministic: code generates IDOR, login-bypass, XSS, business-logic and brute-force attempts from the target's surface map, and a deterministic oracle judges each attempt. No language model chooses an attack (a decision we took early, because the provider declined the agent's attack tool and because an attack that succeeds must be reproducible). That gives a result that is the same every time, and a ceiling that is the set of attacks we wrote.
The denominator is not what the engine tries. The 60 challenges are everything Juice Shop files under those categories, including a chatbot to manipulate, a CAPTCHA, a forged token and a server-side template injection at six stars. No class of ours even attempts them. We kept them in the denominator on purpose: choosing the challenges we expected to solve would have given a number we could not defend.
Broken Authentication is 0 of 9. It is the clearest hole: password resets by security question and two-factor bypass are outside what the brute-force and login-bypass classes generate.
What we did so the number could not be flattered
- The engine never sees the benchmark. Juice Shop's own ground truth (
/api/Challenges,/api/Hints,/snippets/**, the fixes) is outside every agent's reach by scope, and the list of challenges that count lives in an orchestrator-only file the attack code does not read. - A challenge counts only if Juice Shop marked it solved after the round and not after the control. Janus does not say which request solved it.
- Side effects are named. One challenge, Repetitive Registration, was solved by a probe aimed at something else: the surface map listed
POST /api/Userswith no password-repeat parameter, so the login-bypass generator probed it like a login and registered accounts without a repeated password, which is exactly what the challenge asks for. It counts, and the recall is also given without it. - Three rounds, each fresh. They agree, as a deterministic engine should; a disagreement would have been a real instability.
- What was not run. Nuclei, ZAP's baseline and a published subset were optional. The first two crawl the target and would read its ground truth, and none of the three was an approved dependency. We say so instead of listing them as compared.
What the number means
It is a lower bound on what the engine finds on Juice Shop, in the categories it covers, as Juice Shop judges it. It is not an estimate of how many vulnerabilities Juice Shop has, nor of how a tool would do on a different application, and it says nothing about false positives, because every confirmed finding in Janus needs a proof of concept replayed by code on a fresh sandbox before it counts.
It is also the starting line. Each new generator we write can be measured against the same denominator, with the same oracle, in under a minute and for nothing, and a regression (a challenge that stops being solved) would show up as a smaller number. That is the use we want from it: not a score to put on a slide, but a way to know whether the next change helped.
Comments
No comments yet. Start the discussion.