Delivered but Unbilled: Your AI Stream Logged Zero Tokens

Delivered but Unbilled: Your AI Stream Logged Zero Tokens

Delivered but unbilled is the streaming failure where your AI response renders the whole answer to the user, but the terminal usage frame is dropped, zeroed, or malformed, so your client logs 0 output tokens for a call that cost money.

stream_billing_gate.py reconciles the delivered text against the logged usage, offline, and blocks when text shipped and nothing was billed.

AI disclosure: I wrote stream_billing_gate.py with an AI assistant and ran it myself, offline, on Python 3.13.5, standard library only, no network. Every number in the output blocks below is pasted from a real local run. I checked the exit codes (0 / 1 / 2), ran each scenario twice to confirm STDOUT is byte-for-byte identical, and had the tool print a sha256 of its own report so you can reproduce the exact bytes. The one external finding I cite (a Dev.to proof of concept) and the one external number (a Hacker News thread's score) are other people's, attributed inline, and kept out of my fixture numbers.

In short: A streamed response delivers text first and accounting last. If the terminal usage frame never lands, the text is already on the screen and your client logs 0 output tokens. The user got the whole answer. Your bill says it was free.

The gate reads a recorded stream as data, rebuilds the delivered text, and reconciles it against the logged usage. When text was delivered and usage is 0 or absent, it blocks with delivered-but-unbilled.

The demo that matters: two stream logs with byte-identical text deltas. Delete one line, the usage frame, and the verdict flips from OK exit 0 to BLIND exit 1. Token counting here is a conservative FLOOR (word count), not the vendor's bill. The hard claim is smaller and unbreakable: text was delivered, so real tokens are above zero, so a logged 0 is wrong.

This is a post-hoc reconciliation of recorded streams. It does not intercept live traffic, call an API, or need a key.

How does a streamed answer get billed as zero?

A streaming response arrives in two parts that travel separately. First the content deltas: dozens of small events, each carrying a slice of text, which your client concatenates and paints on the screen. Then, at the very end, one terminal frame carrying the token accounting: input tokens, output tokens, the numbers your billing code reads to know what the call cost.

The order is the problem. By the time the accounting is supposed to arrive, the answer is already delivered. If that last frame is dropped by a flaky connection, truncated, or written with a zero, the user never notices. They got their answer. Your client, meanwhile, has nothing to log, so it keeps whatever value it initialized. For a lot of clients that value is zero. No exception. No crash.

A response that cost real output tokens is recorded as free. That zero is a choice your client made, not a law of streaming, and how the accounting travels varies by provider, which the formats section below gets into.

On July 7, a developer publishing as kielltampubolon posted a proof of concept on Dev.to titled "Asynchronous Telemetry Blindness in AI Streaming Clients", showing this exact shape: the text deltas render the full answer while a dropped terminal usage frame leaves billing sitting at zero, with no error surfaced. That framing is his, and his write-up is a local-only proof of concept, not a production incident. I am not reusing his numbers. I built my own fixtures so the failure is reproducible on your logs, and so a test can fail closed on it.

Tracking is not control

Your dashboard says $0 for that call. The dashboard is not lying. It is faithfully reporting a number that is wrong at the source.

This is the whole thesis of the pre-execution and reconciliation gates I keep building: tracking a number is not the same as controlling the thing the number describes. You tracked usage. The usage you tracked was zero. The tokens were spent anyway.

Here is the claim, stated so you can break it. Given a recorded stream where text deltas exist and the terminal usage frame is absent or zero, the gate must exit non-zero with reason delivered-but-unbilled. Given a stream with a usage frame whose output count meets the floor derived from the delivered text, it must exit 0. One removed line has to flip the verdict. If you can show me a stream that delivered text and the gate stayed quiet, or one that logged its usage honestly and the gate blocked, the tool is broken and this post with it.

What the gate reconciles

Two quantities per stream. The delivered text, rebuilt by concatenating every content delta. And the logged output tokens, pulled from the terminal usage frame, or None when that frame is missing or unusable.

The token side needs an honest word about method. Without a real tokenizer, and this tool ships with none, standard library only, I cannot tell you the exact token count of the delivered text. So I do not. I compute a FLOOR: the number of whitespace-delimited words. A BPE tokenizer keyed on whitespace almost always emits at least one token per word, and usually more once punctuation and sub-word splits count. So the real output_tokens is nearly always higher than this floor. It is a deliberately low, human-readable magnitude, not a measurement.

The gate's real claim does not lean on that floor being precise. It leans on something smaller that cannot be argued with: if the delivered text is non-empty, the real output token count is above zero. Therefore a logged zero is provably wrong, whatever the exact number was. The floor just gives you a readable "at least this many" figure to put in the alert.

That gives four verdicts:

BLIND is the hard, logical one and gets the blocking exit code. UNDERCOUNT is softer: it says the logged count is below the floor, which is suspicious but heuristic, so it warns rather than blocks.

The decision itself is a few lines, simplified for the post (the real function returns dicts with detail strings, and it carries one extra guard shown here that fails closed on a stream in a shape it cannot recognize):

def classify(delivered_text, logged, usage_seen, events, recognized):
    # simplified for the post
    floor = word_floor(delivered_text)  # whitespace word count
    if events and not recognized:
        # valid JSON, but no shape the gate knows -> fail closed
        return "UNRECOGNIZED", "unrecognized-stream", 2
    if not delivered_text.strip():
        return "EMPTY", "nothing-delivered", 0
    if logged is None or logged == 0:
        return "BLIND", "delivered-but-unbilled", 1
    if logged < floor:
        return "UNDERCOUNT", "partial-telemetry-loss", 2
    return "OK", "usage-consistent", 0

Which stream formats does it read?

The gate reads the recorded stream as data and auto-detects the wire shape, so you can point it at the logs you already keep. It maps four documented formats to the same two quantities, delivered text and logged output tokens:

How the accounting travels is not the same across providers, and that changes what a zero means. OpenAI Chat sends usage in one terminal chunk, and only if you asked for it: leave include_usage off and there is no usage frame at all, which reads as delivered-but-unbilled by default. Anthropic sends output tokens cumulatively across message_delta frames, so a dropped final frame undercounts rather than zeroes, unless your client commits only the terminal number.

A logged 0 is a fact about what your client recorded, not a universal property of streaming. The gate does not care which of these happened. It reconciles the text that reached the user against the number that reached your logs.

The FLOOR is provider-agnostic. It comes from the concatenated delivered text alone, so the same word count governs an OpenAI stream and an Anthropic one.

A format the gate does not recognize is reported as unrecognized-stream (exit 2), which fails closed instead of passing as EMPTY, so a broken adapter cannot hide a leak behind a green check.

Quick start

Feed it a recorded stream as JSONL, one event per line, in any of the formats above. Then run it:

python3 stream_billing_gate.py fixtures/fixture_blind.jsonl

No install, no key, no network. It reads the file, prints a verdict, and returns an exit code you can wire into CI over your recorded stream logs, whether they are OpenAI Chat, Anthropic, Responses, or the normalized shape. A log in a format it does not recognize returns unrecognized-stream (exit 2) rather than a silent pass, so confirm the fit on a known-good stream before you trust a green.

The one line that flips the verdict

Here are two files. fixture_ok.jsonl is a recorded stream: a normal answer about capping agent spend, followed by a terminal usage frame reporting 190 output tokens. fixture_blind.jsonl is the same stream with that one usage line removed. The text deltas are byte-identical.

Run each:

stream-billing-gate: delivered text vs logged usage
files: 1
[1] fixture_ok.jsonl
  delivered : 137 words, 750 chars (floor 137 tokens)
  logged:    output_tokens=190
  verdict :  OK (usage-consistent) exit 0
  detail:    logged 190 >= floor 137
summary: 1 OK, 0 UNDERCOUNT, 0 BLIND, 0 EMPTY->overall exit 0
report-sha256: f6b8ce0b2b7a09f57a589a862b53024a81ca618cfecd34f82fa6231c670486a7

stream-billing-gate: delivered text vs logged usage
files: 1
[1] fixture_blind.jsonl
  delivered : 137 words, 750 chars (floor 137 tokens)
  logged:    none
  verdict :  BLIND (delivered-but-unbilled) exit 1
  detail:    delivered >= 137 tokens of text; no usage frame at all
summary: 0 OK, 0 UNDERCOUNT, 1 BLIND, 0 EMPTY->overall exit 1
report-sha256: 318dff2585a7874bd04cd06a1430a9d3b303d93535910bd46be1d5987f81a4ce

Same 137 words delivered. In one file that costs at least 137 tokens and the log says 190. In the other it costs at least 137 tokens and the log says nothing. The exit code goes from 0 to 1.

The only difference between the two files is a single line:

$ diff fixtures/fixture_ok.jsonl fixtures/fixture_blind.jsonl
10d9
< {"type":"usage","input_tokens":523,"output_tokens":190}

That is the failure in one line. In production the line does not get deleted by hand. A connection blips, a proxy truncates, a client swallows the final frame, and you are left with fixture_blind.jsonl and a dashboard that says the call was free.

The two sha256 digests are the tool hashing its own report, so you can confirm you got the same bytes I did.

Four verdicts, one sweep

Hand it a batch of recorded streams and it reconciles each one. This run (stream_billing_gate.py fixtures/*.jsonl, so the files come in alphabetical order) covers all four verdicts plus the two broken shapes of the usage frame, present-but-zero and present-but-unparseable, both of which are still delivered-but-unbilled:

stream-billing-gate: delivered text vs logged usage
files: 7
[1] fixture_blind.jsonl
  delivered : 137 words, 750 chars (floor 137 tokens)
  logged:    none
  verdict :  BLIND (delivered-but-unbilled) exit 1
  detail:    delivered >= 137 tokens of text; no usage frame at all
[2] fixture_broken_usage.jsonl
  delivered : 46 words, 274 chars (floor 46 tokens)
  logged:    none
  verdict :  BLIND (delivered-but-unbilled) exit 1
  detail:    delivered >= 46 tokens of text; usage frame present but unparseable
[3] fixture_empty.jsonl
  delivered : 0 words, 0 chars (floor 0 tokens)
  logged:    none
  verdict :  EMPTY (nothing-delivered) exit 0
  detail:    no text delivered; nothing to reconcile
[4] fixture_injection.jsonl
  delivered : 60 words, 344 chars (floor 60 tokens)
  logged:    none
  verdict :  BLIND (delivered-but-unbilled) exit 1
  detail:    delivered >= 60 tokens of text; no usage frame at all
[5] fixture_ok.jsonl
  delivered : 137 words, 750 chars (floor 137 tokens)
  logged:    output_tokens=190
  verdict :  OK (usage-consistent) exit 0
  detail:    logged 190 >= floor 137
[6] fixture_undercount.jsonl
  delivered : 72 words, 403 chars (floor 72 tokens)
  logged:    output_tokens=12
  verdict :  UNDERCOUNT (partial-telemetry-loss) exit 2
  detail:    logged 12 < floor 72; telemetry lost part of the stream
[7] fixture_zero_usage.jsonl
  delivered : 55 words, 272 chars (floor 55 tokens)
  logged:    output_tokens=0
  verdict :  BLIND (delivered-but-unbilled) exit 1
  detail:    delivered >= 55 tokens of text; usage frame logged output_tokens=0
summary: 1 OK, 1 UNDERCOUNT, 4 BLIND, 1 EMPTY->overall exit 1
report-sha256: dc204b014afadff09b09b82a791dd86ba7767df01454fbc2cc011321c9cd5111

Look at fixture_zero_usage and fixture_broken_usage. A usage frame that says output_tokens: 0 and a usage frame with a broken value both land as BLIND, because in both cases text was delivered and no honest token count was logged. A zero is not an OK. Absence is not an OK. Only a real count that meets the floor is an OK.

The report ends with a sha256 of its own body, and because that body depends on the order you pass the files, fixtures/*.jsonl reproduces this exact digest.

The same check on real provider logs

Those seven fixtures use the normalized shape, which is fine for showing the logic but proves nothing about the streams you actually record. So fixtures/providers/ holds streams written in the documented wire formats of three providers, each representative of what their SSE emits: an Anthropic Messages stream, an OpenAI Chat Completions stream, and an OpenAI Responses stream.