The Answer Looked Fine. The AI Bill Didn’t.

A Friday release adds a friendly touch to a support chatbot. Greet customers by loyalty tier. Give a Gold member a more personal welcome before answering their question.

The answer still looks reasonable. The order status is correct. Nobody changed the model. But the request costs more.

That was the fictional case at the centre of Beyond the Prompt: The Engineering of AI Cost, the session I presented with Balaji Saravanan at Presidio India Exchange 2026 on 4 November. We used an application called AI Cost Engineering Lab to follow a support request, inspect its receipt, investigate a regression and evaluate a release gate.

I wanted the engineering work to be visible. A good answer on a chat screen hides a lot of decisions about context, retrieval, tools, retries and pricing. So we gave the audience a chatbot to start with and a detective game to investigate what happened behind it.

Presidio’s published schedule listed our 50-minute talk in the Day 1 Tech Talk programme. Event announcement. This article follows that investigation, then goes deeper into the part I would build around a production application: a CI/CD gate that can refuse a cost regression without rewarding a worse answer.

The game uses synthetic execution and teaching rates. The chatbot has a separate live-provider path. The production pipeline described later is a reference design that extends the demo’s ideas. The screenshots and numbers below show the teaching application, with their evidence source identified.

Start with the request the customer actually made

Our fictional customer, Asha, asks about an order. ShopBot has customer-scoped, read-only tools for orders, shipments, profiles, policies and support cases. The visible experience is a support conversation. Underneath it, the application records the model invocations, tool work and evidence needed to explain the answer.

The final deployed ShopBot interface, shown with viewer access. The customer and records are fictional. Chat supports a separately selected live or synthetic execution source.

A single customer message can lead to several model calls. A tool result can become input to the next call. The next call can carry previous messages, tool definitions and policy passages again. A failed attempt may have already consumed billable usage before the application retries.

That is why the first useful question is more specific than “which model did we use?” What did the complete request do, and what did that work cost?

Read the receipt before choosing an optimization

For a provider that bills these categories separately, a normalized model charge can be written as:

model charge = fresh input tokens × fresh input rate + cache-write tokens × cache-write rate + cache-read tokens × cache-read rate + output tokens × output rate

Apply the rate’s unit conversion, such as dividing by one million for a per-million-token price. Then sum across every charged invocation and attempt in the request.

The normalization matters. Providers can expose different usage fields and token inclusion rules. An adapter must map those fields into mutually exclusive billing categories before applying a dated price table. Adding cached tokens to an input total that already includes them would charge for the same work twice.

Amazon Bedrock’s Converse API exposes usage, stop reasons and latency metrics. Its cache-read and cache-write fields give an application a way to retain the provider’s observations alongside its own trace. Bedrock Converse response documentation

I keep four distinctions in the receipt:

• Provider usage records the counters returned by the inference API.
• Estimated composition explains how much context appears to come from instructions, retrieval or tools. A character-based token estimate belongs here.
• Calculated charge applies a versioned price table to usage.
• Invoice reconciliation accounts for what the provider actually billed, including applicable commercial terms.

Our lab calculates execution charges. A production cost model also needs to define how it treats embeddings, retrieval infrastructure, tool APIs, orchestration and human escalation. Keep that boundary attached to every number.

If a required usage counter or price is unavailable, record an unknown value. Replacing it with zero makes the next comparison look cheaper without establishing any saving.

Cost per resolved conversation changes the question

Suppose two versions each process 100 conversations. This is an illustrative calculation:

• Version A / Version B
• Total measured execution charge: $1.00 / $0.80
• Conversations resolved under the rubric: 90 / 60
• Charge per attempted conversation: $0.0100 / $0.0080
• Charge per resolved conversation: $0.0111 / $0.0133

Version B spends less in total and more per successful outcome. The calculation is:

cost per resolved conversation = total in-scope cost of all attempted conversations / number of conversations successfully resolved

The numerator includes work spent on unsuccessful attempts. The resolution definition needs a versioned rubric. If nothing was resolved, the ratio is undefined and the release decision must handle that explicitly.

For monthly planning, multiply expected conversation volume by expected cost per conversation. For a resolution target, account for the resolution rate and follow-up work. Report traffic volume separately so a larger workload does not masquerade as a per-request regression.

Five places to investigate

We organised the investigation around five layers. They are useful places to look for a cause. Their effects can overlap, so adding five independent “savings” estimates is unreliable.

• Model: What I would inspect - Exact model identity, routing, input/output rates, answer quality. A possible experiment - Route a bounded task to a smaller model and score the same cases.
• Context: What I would inspect - Stable instructions, message history, dynamic prefixes, cache observations. A possible experiment - Move changing data after the cache boundary.
• Retrieval / RAG: What I would inspect - Retrieved passages, relevance, duplication, grounding. A possible experiment - Reduce redundant passages while checking answer coverage.
• Tools: What I would inspect - Fields returned, payload size, repeated reads, failure behaviour. A possible experiment - Return the fields needed for this task.
• Agent behaviour: What I would inspect - Calls, attempts, retries, termination and escalation. A possible experiment - Bound the loop and inspect failed cases.

The Journey view animates recorded events from a completed synthetic run. Playing the trace does not make another model call.

The journey connects those layers. A broad tool response can increase context on several subsequent calls. A retrieval change can reduce input tokens and also remove the evidence needed to answer correctly. A smaller model can require more attempts. The full workflow decides whether the change helped.

Case #1 - The Expensive Greeting

The game takes the investigator through Case file → Crime scene → Line-up → Lab → Verdict. Each location has a job in the investigation. The map makes the investigation sequence visible. The clues come from stored run evidence.

The crime scene establishes what changed. CR-2041 is the fictional festival-weekend release. It introduces loyalty-tier greetings. We reproduce the healthy and changed versions on the same questions, then compare cost, quality and test volume. This two-question reproduction is a diagnostic sample. The displayed charges use synthetic usage and teaching rates. It cannot establish a production regression rate.

The investigator opens evidence records, reads the before-and-after summaries and labels each contact. Those labels are notes. An accusation in the Line-up still has to survive a one-change lab test.

The final game gives the player three Lives for wrong accusations, followed by unlimited attempts in Free Play. Those mechanics keep the investigation moving. They do not alter the evidence required to close the case.

The first clue is where the changing text appears. Prompt caching depends on the content before the cache boundary remaining eligible for reuse. In this case, customer-specific material changes that prefix. The pattern looks like this:

• Before CR-2041: Stable instructions and tool definitions / Cache boundary / Customer-specific context and current question
• After CR-2041: Changing personalization / Stable instructions and tool definitions / Cache boundary / Current question

The teaching trace changes from a cache write followed by reads to repeated writes. Moving the varying material after the stable prefix is the Context experiment. Claude’s caching documentation describes exact prefix matching and explains why placing a breakpoint on changing content can keep producing fresh writes. Claude prompt caching documentation

A production test also needs the provider’s minimum cacheable length, TTL, supported model and request format. A stable hash is evidence about prompt construction. Cache counters establish what the provider observed. An unknown cache observation stays unknown.

The second clue is what the profile tool returns. The greeting needs a small projection of customer information. Returning the full profile can send unrelated fields into the prompt and carry them through later calls. A minimal response might look like:

{ "display_name" : "Asha" , "loyalty_tier" : "Gold" }

The actual allowlist follows the task and policy. If answering the question requires an order record, fetch that scoped record through the appropriate tool. Our Tools experiment changes the profile projection and measures the result.

The lab also keeps a Tools-only variant. That lets the investigator compare the effect of fixing the payload while the context problem is still present. The lesson is to measure the combined change. Fixing one contributor does not prove the remaining request is within budget.

An alibi needs a boundary. The same model identity before and after rules out a model switch in that comparison. It does not establish that every generation will behave identically. Similarly, a familiar successful-call count does not explain rejected attempts or retries. The game can leave Agent as Alibi unclear when the evidence does not justify clearing it. A production investigation needs that same ability to withhold a conclusion.

What the gate actually demonstrated

The game’s larger gate trial uses 60 matched synthetic questions per comparison. This is separate from the two-question crime-scene reproduction. Teaching verdicts from the final app. The values are calculated using synthetic execution and invented fixture rates.

Teaching gate metric / Healthy reference / CR-2041 / Both fixes:

• Calculated charge per resolved chat: $0.007263 / $0.011300 / $0.007504
• Eligible-call cache hit rate: 66.7% / 0.0% / 66.7%
• Profile output, p95 estimated tokens: 20 / 1,616 / 20
• Agent steps, p95 successful model calls: 3 / 3 / 3
• Resolution under the fixture rubric: 100% / 100% / 100%

The as-shipped version is BLOCK. The version with both fixes is PASS under the teaching policy. Its charge remains slightly above the reference but within the allowed increase.

The implementation also carries an evidence-admissibility flag. A passing synthetic teaching verdict does not authorize a production release. The repository’s hosted cost-quality workflow exercises synthetic evaluations and gate plumbing. Production model replay, deployment approval and canary automation need their own implementation. That separation is where I would begin the production work.

Building a production cost-and-quality gate

A gate needs an evidence-producing pipeline, a policy evaluator and an enforcement point that prevents release. Printing a red or green report only implements the middle part. The delivery path I would build is:

PR and candidate artifact ↓ Gate 0 - static contracts and budget checks ↓ Gate 1 - bounded, matched real-model evaluation ↓ Gate 2 - evidence validation and policy decision ↓ Required check for the evaluated artifact ↓ Canary → observe → promote or roll back

1. Decide what success means before running the comparison

Use a versioned evaluation set with known expectations. Include common support paths, ambiguous requests, tool failures, policy boundaries, long histories and cases that must escalate. Add slices for material production segments such as language or request category.

Define resolution in observable terms. For an order-status question, that might require the correct order, the recorded shipment state and an estimate presented with the proper qualification. For a refund question, check eligibility, policy grounding and the permitted next action. A fluent answer alone does not satisfy those checks.

Use deterministic checks for facts that the application can verify. If a model judge is useful for a broader rubric, version its model and prompt, calibrate it against human labels, and track disagreement. The judge’s output is another measurement with an error rate.

Keep evaluation cases held out from prompt tuning. When a production incident becomes a regression case, version the new dataset and retain the previous one for comparison history.

2. Freeze the inputs and identify both artifacts

A meaningful comparison records more than a branch name:

{ "candidate_digest" : "sha256:<candidate artifact digest>" , "baseline_digest" : "sha256:<approved baseline digest>" , "dataset_sha256" : "<versioned evaluation set hash>" , "policy_sha256" : "<approved policy hash>" , "rubric_version" : "support-resolution-v3" , "price_table_version" : "provider-region-date-version" , "evidence_source" : "live_provider_evaluation" , "cache_protocol" : "cold-and-warm-measured-separately" }

These are example field names, not a record of a production deployment.

Freeze the customer question, conversation history, tool fixtures and policy corpus for both sides. Pin model identifiers and inference parameters where the provider supports it. If model routing is the variable being tested, record that difference explicitly.

A baseline must come from an approved registry of release artifacts. Allowing a candidate to select an unusually expensive reference or edit the acceptance policy would undermine the