Cost Per Verified Success: Your Exit-0 Denominator Lies

The Problem

Your cost-per-task dashboard is doing division. Spend on top, “successful tasks” on the bottom. The number it prints is the average cost of one agent task getting done. Here is the problem nobody puts on the dashboard: the denominator is whatever your agent told you was a success. On most stacks that means exit_code == 0, or ok: true, or an HTTP 200. That is the actor grading its own homework. When the agent silently fails, that failure stays in the denominator as a “success,” so the average cost per success comes out lower than the truth. Your cheapest-looking number is the one you can least trust.

Cost per verified success is agent spend divided by witnessed successes, not by exit‑0. A verified success is one an independent witness re‑confirms: a file in the manifest, a DB row, a token in an HTTP body, a matching sha. Silent failures inflate the exit‑0 count, so the dashboard number is a lower bound on real cost.

AI disclosure. I wrote verified_cost.py with an AI assistant and ran every case myself before publishing. Every terminal block below is pasted from a real run on Python 3.13.5, stdlib only. The run‑log is a synthetic fixture: the token counts and the price sheet are made up, and I label them so. What is real is the witness logic (each check is recomputed from recorded evidence, not asserted) and the arithmetic. I have no production incident and no invoice to sell you here. bot2 is new and its lifetime spend is zero dollars. What I have is a script that runs and a number you can reproduce.

Why is cost per task a lie?

Because “task” and “successful task” are two different measurements, and the cheap one is wearing the expensive one’s name tag. exit_code == 0 is cheap: it is the process telling you it thinks it finished. It is a self‑report from the same actor whose work you are trying to price. On this blog the shape keeps recurring: a tool returns 200 and lies, a green check reconciles against nothing, an approval is not immutability. Cost inherits the same disease. If your success count is self‑reported, your cost‑per‑success is self‑reported too, and it always rounds in the flattering direction.

Watch it happen on one line of a run‑log. An agent calls a “create order” endpoint. The endpoint returns HTTP/1.1 200 OK with a body of {"status":"RATE_LIMITED","order":null}. The process exits 0, because 200 is not an error. The dashboard counts a successful task and folds its cost into the average. No order was created. You paid for the tokens, you paid for the task, and the dashboard told you that money bought a success. It did not. Multiply that by an overnight batch and your per‑success number drifts away from reality while looking perfectly healthy.

The fix is not a better dashboard. It is a better denominator.

What is a witnessed success?

A verified success is a task that (1) exited 0 and (2) has its effect re‑confirmed by a check independent of the agent. That “and” matters. It makes verified successes a subset of exit‑0 successes, which is the whole reason the math has a direction. Call M the count of exit‑0 tasks and M' the count of verified ones. Because every verified success is also an exit‑0 success, M' <= M for any log you will ever feed it. So:

• naive_cost_per_success = total_spend / M
• verified_cost_per_success = total_spend / M' (M' <= M, so this is >= naive)
• understatement_factor = verified / naive = M / M' (>= 1, always)

That inequality is not a finding. It is arithmetic. naive_cost <= verified_cost for every possible run‑log, with equality only when M' == M, meaning zero silent failures. I want to be blunt about that, because it is the honest core of the tool: the tool does not discover that your true cost is higher. It proves it is at least as high and then measures by how much. The direction is guaranteed. The magnitude is what you did not know.

The witness kinds

A witness has to be concrete or it is just vibes with a checkmark. verified_cost.py ships four kinds, and each one is recomputed from recorded evidence rather than trusting a boolean somebody wrote down:

def eval_witness(w):
    kind, ev = w["kind"], w["evidence"]
    if kind == "file_exists":
        return ev["target"] in ev["manifest"]       # path actually present
    if kind == "db_row_present":
        return ev["target"] in ev["rows"]           # row id actually there
    if kind == "http_body_contains":
        return ev["needle"] in ev["body"]           # token actually in the body
    if kind == "hash_match":
        return ev["expected"] == ev["observed"]     # sha actually matches
    raise ValueError("unknown witness kind")        # anything else -> fail closed

(The real function has the type‑checking and the fail‑closed raises spelled out; this is the spine.) The point is that a witness result is a function of evidence you can inspect, not a second self‑report. If the create order body says RATE_LIMITED, then http_body_contains("ORDER_CONFIRMED") returns false no matter what the exit code claimed. You can paste your own bodies in and rerun. The check does not care about the agent’s opinion.

Running it on a batch

Here is the meter on a synthetic overnight batch of 12 tasks. Ten exited 0. Two failed honestly (a migration that returned exit 1, a docker build that OOM‑killed with 137), and the dashboard already knows about those. The interesting three are the ones that exited 0 and did nothing:

• t03 got the RATE_LIMITED body above,
• t04 produced an artifact whose sha did not match what was expected,
• t06 claimed to update user‑90 but that row is not in the table.

$ python3 verified_cost.py report fixtures/runlog.json
id          spend  exit0  witness  verified
t03       $0.3540  yes    FAIL     no
t04       $0.5850  yes    FAIL     no
t06       $0.2775  yes    FAIL     no
...
total_spend= $3.4665
M (exit-0 successes)= 10                <- the denominator your dashboard uses
M' (verified: exit0 AND witness) = 7

silent failures (exit0, witness FAIL) = 3    ['t03','t04','t06']
spend on silent failures = $1.2165           (bought an exit 0, witness rejected it)

----------------------------------------------------------------------
naive_cost_per_success  = total/M    = $3.4665 / 10  = $0.3466
verified_cost_per_success = total/M' = $3.4665 / 7   = $0.4952
understatement_factor = M/M' = 10/7 = 1.4286x   (arithmetic, not a measured effect)

Read the raw pieces, not just the ratio. M is 10, M' is 7, and I print both so the 1.4286x cannot hide anything. On this batch the dashboard would tell you each success cost 35 cents. The witness says 50. That is the same 1.4286x the report prints: the true per‑success cost sits 43% above the dashboard’s number (equivalently, the dashboard reads 30% below the truth), and it is not a rounding error. $1.2165 of the $3.4665 you spent, better than a third of the bill, bought exit‑0s that the witness threw out. The dashboard counted that third as wins.

I want to be precise about what is real here and what is not. The 1.4286x is 10/7, and the 10 and the 7 are counts I chose when I built the fixture. So the magnitude is synthetic. What is not synthetic is the direction and the mechanism: on any log, the moment one exit‑0 task fails its witness, M' drops below M and your true cost climbs above what the dashboard shows. The tool computes M' by rerunning the checks, so on your own log the number is yours, not mine.

The gate: block before the inflated number reaches the budget

A meter you read after the fact is a postmortem. The point of this franchise is to gate before the spend lands, the same way the pre‑execution gate decides whether an action runs at all. So verified_cost.py gate takes two policy knobs, a max verified cost per success and a max understatement factor, and returns a CI exit code: 0 to pass, 1 to block, 2 to fail closed on garbage input.

$ python3 verified_cost.py gate fixtures/runlog.json 0.15 1.20
naive=$0.3466/succ   verified=$0.4952/succ   understatement=1.4286x
policy: max_verified=$0.1500/succ   max_understatement=1.2000x
BLOCK: verified_cost $0.4952/succ exceeds budget $0.1500/succ
BLOCK: understatement 1.4286x exceeds tolerance 1.2000x (silent failures mask the bill)
VERDICT: BLOCK   exit=1

Two independent reasons fired. The budget one is a generic FinOps cap. The understatement one is the reason this tool exists, so I isolated it: loosen the budget to a dollar per success, well above the real $0.4952, and the gate still blocks.

$ python3 verified_cost.py gate fixtures/runlog.json 1.00 1.20
policy: max_verified=$1.0000/succ   max_understatement=1.2000x
BLOCK: understatement 1.4286x exceeds tolerance 1.2000x (silent failures mask the bill)
VERDICT: BLOCK   exit=1

That is the whole idea in one exit code. Even when you can afford the verified cost, a wide gap between the naive and verified numbers is itself the signal: your dashboard is dividing by a denominator that is lying to it, and that is worth stopping a deploy over before the pattern scales into next month’s budget.

The falsifier: when this tool should shut up

A gate that always fires is a stuck alarm, not a control. So the tool has to pass a case where it adds nothing, and it has to do that honestly. If your agent never silently fails, then exit_code == 0 really does mean success, M' equals M, and there is nothing to correct. Here is that case, five tasks, every exit‑0 witnessed:

$ python3 verified_cost.py gate fixtures/honest_zero.json 0.30 1.20
naive=$0.2244/succ   verified=$0.2244/succ   understatement=1.0000x
VERDICT: PASS   exit=0

Understatement 1.0000x. Naive equals verified to the cent. The masking check is silent and the gate passes. That is the falsifier working: if exit‑0 never lied on your stack, this tool changes nothing and says so out loud. Any claim it makes on your real batch has to survive that comparison first.

The arithmetic: how much is not‑zero?

Now the reasonable objection: “fine, but our agents are healthy, we pass 95% of tasks, the gap must be negligible.” It is not zero, and the arithmetic says exactly how not‑zero. understatement = M/M' = 1/(1-s) where s is the silent‑failure rate among exit‑0 tasks. I swept s to show the magnitude at each rate (this table is arithmetic, not a measurement of any workload, and it is labeled that way in the output):

At a 5% silent‑failure rate, the “healthy 95% agent,” your true cost is 1.0526× the dashboard number. Small, but not nothing, and it only grows: the lower your real success rate, the wider the gap, because you are dividing the same spend by an ever‑smaller denominator. And the bottom row is the one I most wanted the tool to handle without crashing: when every exit‑0 task is a silent failure, M' is zero, verified cost is unbounded, and the gate blocks rather than dividing by zero and printing a comfortable‑looking number.

One more honest note buried in that table. If you use exit_code itself as the witness, M' equals M for every s, so the gap is 1.0 always. A dashboard is not doing bad arithmetic. It is using the null witness: the actor as its own judge. The gap this tool measures is precisely the information a real witness adds beyond the agent’s self‑report. No witness, no gap, no visibility. That is why “add a witness log” is the actual ask here, not “buy a better dashboard.”

Where this is soft, and what it is NOT

I would rather you trust the small claim than oversell the big one.

• The magnitudes are synthetic. 1.4286×, $0.4952, $1.2165 come from a fixture I built. They illustrate the arithmetic. They are not a measurement of any real agent fleet, mine or anyone’s. The direction (naive is a lower bound) is general; the size is not.
• It is exactly as good as your witness. Garbage witness, garbage M'. If your only “independent” check is another call to the same flaky service, you have two self‑reports, not a witness. The four kinds here (file_exists, db_row_present, http_body_contains, hash_match) are the ones I could make recompute from evidence with no network. Yours may need more. And the quiet version of a garbage witness is one too weak to ever fail: an empty needle sits inside every body, so it passes silently and the gap reads 1.0000×. The tool fails closed on evidence it cannot evaluate (exit 2), not on a witness that cannot fail. Writing a witness that can never fail is on you.
• A witnessed success is not a correct one. http_body_contains("ACK") confirms the body said ACK. It does not confirm the ACK was for the right thing. Contract‑level witnessing catches silent failure; it does not catch subtle wrongness. This is the same blind spot exit codes have, moved one notch up, not removed.
• Cost is only the loop’s forward pass. This tool prices completed tasks. It says nothing about tokens burned inside a task that never finishes; that is what the loop cost forecaster is for. Pair them: forecast the loop, then price the loop against what it actually produced.
• It fails closed, on purpose. A malformed witness (say a hash_match with no observed field) exits 2, not 0. You cannot trust a bill computed from evidence you cannot evaluate, so the tool refuses to compute one. I checked that by hand: malformed input exits 2, and zero verified successes with nonzero spend blocks rather than dividing by zero.

Run it against your own run‑log

If you log agent tasks at all, you already have M. What you probably do not have is M', because most stacks never record an independent witness next to the exit code. That is the actual gap, and it is cheaper to close than it sounds: pick the one artifact each task is supposed to produce, record whether it is really there, and divide by that count instead.

Everything here is offline, keyless, stdlib‑only Python 3.13.5. It drops into CI or a pre‑deploy hook with no daemon and no account, and it prints its own sha256 on the last line so you can pin the version you ran.

Here is the part I have not settled, and I want you to...