We Cut Our Agent's Token Bill by 21%. One Task in Four Got Nothing.

We turned on progressive disclosure for an agent with 20 tools. Deferring the tool schemas cut input tokens by about 30% and total cost by about 21%. Good result, shipped, write the blog post. Then I broke the numbers down by task, and the average turned out to be hiding something. Three of our four task types saved 21% to 30%. The fourth saved nothing on one transport and cost 12.3% more on the other. This post is about that fourth task, and about why the number you should care about when you defer tool schemas is not the mean. All 80 runs are public. Every command below is one I ran while writing this, with its real output pasted in. You can reproduce the whole thing offline for free. The setup Four business tasks, each needing exactly one tool. Twenty tool schemas registered, always. Two arms: - always-on: all 20 schemas serialised into every request - deferred: all 20 wrapped in Pydantic AI's DeferredLoadingToolset , so the model has to search for a capability and load it before calling it Twenty runs per cell, sequential, no retries, gpt-4o at temperature 0, parallel_tool_calls=False . Run on 2026-08-06 against pydantic-ai-slim[openai]==2.24.0 . Both transports, Chat Completions and the Responses API, because tool search executes server-side on one and through a local fallback on the other. The whole thing cost $0.279585 to run. The averages, which are fine cell mean_in stdev min max A-chat 1361.5 10.3 1350 1372 B-chat 945.2 190.8 780 1264 A-resp 1353.8 10.6 1342 1364 B-resp 1001.0 263.1 807 1477 A is always-on, B is deferred. Input tokens down 30.6% on Chat Completions and 26.1% on Responses. Cost down 21.2% and 16.8%. Correctness was 80/80 exact matches, so nothing broke. But look at the standard deviation column. The always-on arms sit within 22 tokens of each other across every run. The deferred arms swing across a 484 and a 670 token range. Deferral made the input size roughly twenty times more variable. My first assumption was run-to-run nondeterminism: the model writes a slightly different search query each time, gets back a slightly different set of schemas, and the prompt size wobbles. That assumption was wrong, and checking it is what produced the actual finding. Decomposing the variance Here is the check. Group by task inside each cell instead of pooling the cell: curl -sL -o bc025.jsonl https://raw.githubusercontent.com/benchclawio/harness/main/results/bc025-progressive-disclosure-2026-08-06/bc025-scored-raw-2026-08-06.jsonl python3 -c " import json, statistics as s, collections rows=[json.loads(l) for l in open('bc025.jsonl')] for c in ('A-resp','B-resp'): print('==', c) g=collections.defaultdict(list) for r in rows: if r['cell']==c: g[r['task_id']].append(r['metrics']['tokens_in']) for t,v in sorted(g.items()): print(f' {t:22s} n={len(v)} mean={s.fmean(v):7.1f} sd={s.stdev(v):6.1f}') " Real output: == A-resp currency-conversion n=5 mean= 1342.0 sd= 0.0 defect-threshold n=5 mean= 1364.0 sd= 0.0 inventory-reorder n=5 mean= 1345.0 sd= 0.0 shipment-delay n=5 mean= 1364.0 sd= 0.0 == B-resp currency-conversion n=5 mean= 930.2 sd= 2.7 defect-threshold n=5 mean= 1431.8 sd= 95.6 inventory-reorder n=5 mean= 807.0 sd= 0.0 shipment-delay n=5 mean= 835.0 sd= 0.0 Two things fall out of this. Run-to-run variance is near zero in both arms. Repeat the same task five times under deferral and you mostly get the identical token count, standard deviation 0.0. The nondeterminism I assumed was there is not there, or is small enough not to matter. The spread is between tasks, not between runs. Under always-on, all four tasks land within 22 tokens of each other, because the 20 schemas dominate and the task text barely moves the total. Under deferral that flattening disappears and the tasks spread from 807 to 1,432 tokens. That reframes the whole thing. Deferral does not make your cost noisy. It makes your cost task-dependent. Always-on charges you the same amount whatever the user asks. Deferred charges you according to what the model decides to search for and load, which means your bill now tracks your traffic mix. The task that lost money Once cost is a function of the task, some tasks can come out behind. Here is the per-task cost, both transports: python3 -c " import json, statistics as s, collections rows=[json.loads(l) for l in open('bc025.jsonl')] g=collections.defaultdict(list) for r in rows: g[(r['cell'],r['task_id'])].append(r['metrics']['cost_usd']) hdr=f"{'task':22} {'A-resp':>10} {'B-resp':>10} {'delta':>8} {'A-chat':>10} {'B-chat':>10} {'delta':>8}" print(hdr) for t in ('currency-conversion','defect-threshold','inventory-reorder','shipment-delay'): ar=s.fmean(g[('A-resp',t)]); br=s.fmean(g[('B-resp',t)]) ac=s.fmean(g[('A-chat',t)]); bc=s.fmean(g[('B-chat',t)]) print(f'{t:22} {ar:10.6f} {br:10.6f} {br/ar100-100:+7.1f}% {ac:10.6f} {bc:10.6f} {bc/ac100-100:+7.1f}%') " Real output: task A-resp B-resp delta A-chat B-chat delta currency-conversion 0.003795 0.003001 -20.9% 0.003795 0.002791 -26.5% defect-threshold 0.003944 0.004428 +12.3% 0.003840 0.003810 -0.8% inventory-reorder 0.003873 0.002707 -30.1% 0.003828 0.002770 -27.6% shipment-delay 0.003970 0.002828 -28.8% 0.003846 0.002691 -30.0% defect-threshold is the outlier. On the Responses API it cost 12.3% more with progressive disclosure on. On Chat Completions it saved 0.8%, which after four decimal places is a rounding error and not a saving. Its deferred prompt came in at 1,431.8 input tokens against 807.0 for the cheapest task in the same cell. That is 625 extra tokens of loaded schema for a task that, like every other task in the suite, needed exactly one tool. I want to be careful about the mechanism here, because the bundle does not record it. Every deferred run made exactly 2 tool searches and 3 model requests, defect-threshold included, so it is not doing extra round-trips. The most likely explanation is that its search matched more of the 20 capabilities and pulled more schemas back into the prompt than the other tasks did. But the raw JSONL logs token counts and the tool that was finally called, not the schema set the search returned, so I cannot prove that from the published data. Treat it as the obvious inference, not a measurement. What the distribution looks like The cost distribution per run makes the point better than a standard deviation does: python3 -c " import json rows=[json.loads(l) for l in open('bc025.jsonl')] for c in ('B-chat','B-resp'): v=sorted(r['metrics']['cost_usd'] for r in rows if r['cell']==c) print(c, [f'{x:.6f}' for x in v]); print() " Real output: B-chat ['0.002470', '0.002587', '0.002717', '0.002717', '0.002717', '0.002717', '0.002740', '0.002800', '0.002800', '0.002800', '0.002815', '0.002845', '0.002845', '0.002845', '0.002845', '0.003810', '0.003810', '0.003810', '0.003810', '0.003810'] B-resp ['0.002707', '0.002707', '0.002707', '0.002707', '0.002707', '0.002828', '0.002828', '0.002828', '0.002828', '0.002828', '0.002992', '0.002992', '0.002992', '0.002992', '0.003037', '0.003922', '0.004497', '0.004573', '0.004573', '0.004573'] That is not a bell curve with a fat tail. It is a cluster and then a cliff, and the cliff is one task type. On the Responses API, 5 of 20 deferred runs cost more than the average always-on run ($0.003895). The worst deferred run cost $0.004573, which is 17.4% above the always-on mean. The optimisation that saves 16.8% on average was, for a quarter of these runs, not an optimisation. What I'd actually take from this Compute your saving per task type, not per corpus. A single blended percentage is the one number that cannot tell you whether to ship this. If your traffic is 80% the defect-threshold shape, deferral loses you money while your dashboard reports a saving. The saving is capped by the schema share of your prompt. Deferral removes tool schemas. It does not remove your system prompt, the user message, the conversation history, or the tool results coming back. In our tasks the schemas were roughly a third of the request, so removing nearly all of them saved roughly a third of input tokens. If you have five tools and a 4,000-token system prompt, there is nothing here for you. Budget the extra round-trip as a certainty. Always-on completed in 2 model requests. Deferred took 3, in all 40 deferred runs, on both transports. Not an average with spread, a constant. If your latency budget is per-request rather than per-token, you are buying a variable token reduction with a fixed 50% increase in requests. Do not verify deferral through the framework's own view of its tools. AgentInfo.function_tools lists a deferred tool both before and after it loads, and the local search_tools fallback is present either way. That surface tells you what the agent knows about, not what got serialised to the provider. Every number in this post comes from provider-reported request token counts instead, which is the only thing that maps to the invoice. Accuracy was not the thing that broke. All 80 runs produced exact matches, 20 out of 20 in every cell. I expected correctness to be the risk and it was not, though our tasks each needed exactly one capability. Tasks needing several loads would compound the round-trips and give the model more chances to pick wrong. We did not test that. Reproduce it The bundle is 80 raw runs, both manifests with SHA-256s, the deterministic task-suite generator, the worker, the collector and the analysis script: https://github.com/benchclawio/harness/tree/main/results/bc025-progressive-disclosure-2026-08-06 python3 analyze_bc025.py regenerates the confidence intervals. python3 bc025_capabilities.py regenerates the task suite byte for byte. Both are offline and cost nothing, because they read the recorded runs rather than calling a model. The full benchmark, including the bootstrap confidence intervals and the part about where the widely quoted "90% to 98% savings" figure comes from, is written up at benchclaw.io. One caveat on the version, since it moved while we were running: