UX cost of a security patch
Janus Arena (November 2026). Every figure comes from real runs recorded in the project's event log: a brute-force fix on OWASP Juice Shop, and five Launch Day runs on ShopLab, a demo shop.
The reports are in the repository (docs/reports), and the console's demo build replays two of the runs in your browser. The users are synthetic: language-model agents that play a person in a real browser. Nothing here is a conversion figure.
A security fix is judged by one question: does the attack still work? It is a good question and it is not enough. A fix can stop the attack and also stop the people it was meant to protect, and the second half only shows up if you measure it.
This is what happened when we measured both, in two experiments, and what we could not conclude from them.
The rule that keeps it honest
Attacks are generated by code and judged by code. An attack succeeds only when a deterministic oracle says so (for brute force: every wrong password was refused fast, nothing throttled, and the right password still signs in afterwards). A model proposes the patch; it never decides whether the patch works.
The shoppers are agents, but whether one bought what it was asked to is read from the shop's orders, not from what the agent says. Rates carry 95% Wilson intervals.
Experiment 1: two ways to stop a password guesser
The finding was a brute-force on POST /rest/user/login. The Blue Team wrote alternatives and three independent verifiers judged each: the original proof of concept must stop working (V1), the regression tests must pass (V2), and a paired run of synthetic shoppers must stay within tolerance (V3).
Two alternatives got to the end:
| Alternative | Still exploitable? | Shoppers who bought, original β fix | p95 of the login |
|---|---|---|---|
| Per-account limit, 10 tries in 5 minutes | no, throttled at guess 11 | 4/6 β 5/6 | 50 β 90 ms |
| Account lockout | no, throttled at guess 6 | 4/6 β 3/6 | 46 β 41 ms |
Both close the attack. They differ in who else they stop. One of the personas, InΓ©s, is forgetful: before the right password she tries five old ones, as someone who does not remember which password this shop has.
Five is the usual threshold of a lockout, and on the lockout variant InΓ©s was refused with a 429 in both of her sessions and never completed a purchase. The limit of ten tries is above the five she tries.
What this shows is a mechanism, not a rate. Six shoppers per variant is too few for the difference between 4/6 and 3/6 to mean anything: it is one session, and the intervals overlap completely. What is solid is the cause, because it is in the events: a specific persona, a specific threshold, a 429.
A third alternative (a progressive delay) is not in the table because the API credit ran out before it was written; we report it as not measured rather than as passed. The whole comparison cost 0.98 β¬.
Experiment 2: Launch Day
The cost of a defense changes with how many people arrive at once and where they come from, so we put a launch on ShopLab: 18 synthetic shoppers and two saboteurs per variant. One saboteur sends requests at a steady rate; the other guesses at an account's password.
Variant A runs without the defense, B with it, and every shopper plays the same task on both sides. Each defense was one run.
| Defense in B | Attacks that got through, A β B | Shoppers who bought, A β B | Shoppers blocked, A β B |
|---|---|---|---|
| Rate limit per client, 20/min | 100% β 50% | 86% β 88% | 0% β 0% |
| Rate limit per account, 3/min | 100% β 50% | 80% β 79% | 0% β 0% |
| Checkout step-up (type a code) | 100% β 100% | 75% β 73% | 0% β 0% |
| Rate limit per client, 6/min, nine shoppers at once behind three shared addresses | 100% β 0% | 75% β 73% | 0% β 18% |
Four things stand out.
Each defense stops what it was built for, and nothing else. The per-client limit stopped the saturation saboteur after 20 answered requests and left the password guesser alone, and the per-account limit did the reverse. The two 50% are two defenses, each covering one of the two saboteurs.
The step-up stopped neither attack. It is a defense for another problem (an account that was just created buying with a stolen card), and the run says so: attacks 100% β 100%. What it costs shoppers shows in the median number of steps, 15 β 16, not in anyone being blocked.
A defense that costs nothing can become one that costs shoppers. At 20 requests per minute no shopper was blocked. At six, with nine of them arriving together behind three shared addresses (an office, a campus), both saboteurs were stopped and 2 of 11 shoppers were blocked too, where none had been without the limit. The interval is wide (5% to 48%).
The same limit with only three shoppers at a time (an earlier run that was interrupted, so it is not in the report) blocked nobody, and neither did 20 per minute with shared addresses and low concurrency: the burst is what makes the limit bite. The same number written differently, "this limit is safe", would have been true in one run and false in the other.
Contamination was zero. Every shopper marks its shipping address with an invented code, and after the task we check that no order it can read carries another session's code. None did, so the numbers are not an artefact of sessions leaking into each other.
What we do not claim
- Not that a limit of 20 is the right one. Those are the values we tried, on a demo shop, with one task.
- Not a measure of attack success with any precision. There are two saboteurs per variant, so "100% β 0%" is 2/2 against 0/2, an interval from 0% to 66%. It says the defense stops them in these runs, not how often.
- Not conversion, and not what real people do. Synthetic shoppers say where a flow is hard for an agent that behaves like a person; they do not predict revenue.
- Not that the defenses combine. Each was measured alone.
What to take from it
Measure three things before you merge a security fix: whether the attack still works, how many of the people you protect it stops, and what it adds for those it does not stop. The second number is the one nobody has by default, and in these runs it was zero for one setting and 18% for another that differed in a single parameter.
A gate that shows both on the same pull request, and never merges on its own, is what Janus Arena is for.
To reproduce the Launch Day numbers:
pnpm run:launch-day <runId>...
writes the report from the recorded events, with no model involved; the five runs cost between 0.24 β¬ and 0.29 β¬ each.
Comments
No comments yet. Start the discussion.