The Hidden Math: Why "Cheapest Model" Doesn't Mean Cheapest Execution

You're building an AI agent. Smart cost strategy: route to GPT-4o when you need reasoning, Haiku for simple classification, Groq when it's available and fast. In your head, the math is simple: pick the cheapest model per task. In practice, you ship code and never really know if it worked.

Here's the gap: cheapest model ≠ cheapest execution. A single agent run might span multiple model calls. One task uses Anthropic's Sonnet ($3 / 1M input, $15 / 1M output). Another uses OpenAI's GPT-4o mini ($0.15 / 1M input, $0.60 / 1M output). A third falls back to Groq Llama 3.1 when latency spikes. By the time the execution finishes, you have no idea which model was actually called, how many tokens each burned, or whether that "cheap" fallback actually saved money - or whether a long output summary from one call inflated the token count beyond what you budgeted. The assumption breaks immediately.

Why the math matters (and why it's invisible)

Token costs compound differently by model. Take a real scenario: your agent needs to summarize a customer support transcript (2,000 input tokens). You budgeted for Haiku: 2,000 tokens × $0.80 / 1M = $0.0016 input cost. Cheap. But Haiku hits a rate limit. Code routes to Sonnet as fallback. Same 2,000 input tokens, now $0.006 - nearly 4× more. Multiply that across 100 daily runs, and you've overrun your mental budget without knowing.

Output lengths are invisible. Worse: you don't even know the output token count. An agent that "completes" a task could've produced 500 output tokens (brief response) or 5,000 (verbose reasoning). With Sonnet outputting at $15 / 1M, that's the difference between $0.0075 and $0.075 per run. Neither is huge. But if your agent runs 1,000 times a month and half your executions unexpectedly verbose? You're now $37 over budget without seeing why.

Multi-model routing compounds the blind spot. When you route across three providers with six fallback models, you lose the ability to reason about cost at a glance. Did GPT-4o actually run, or did it fail and drop to Sonnet? How many times did Groq get chosen? Which model produced the longest outputs? You shipped it all, but you're flying blind on the arithmetic that actually happened.

The unsexy solution: measure and account

Here's what works (and it's not flashy):

1. Log every model call with input + output token counts. Before you route, log which model was selected. After the call returns, log the actual token counts from the response metadata. Not a guess - the real numbers. Why: your mental model of "mostly cheap, sometimes fallback" gets replaced with actual data. You can group by model, see fallback rates, and measure whether your routing actually favors the cheap path.

2. Compute cost inline using per-model rates. Every model has a published price per 1M tokens (input and output separate). Compute cost at execution time using actual tokens. For example:

   • Input: 2,000 tokens
   • Output: 1,500 tokens
   • Model: Anthropic Sonnet ($3 / 1M input, $15 / 1M output)
   • Cost: (2,000 / 1M × $3) + (1,500 / 1M × $15) = $0.006 + $0.0225 = $0.0285 per run

   Sixty runs a day? That's $1.71 / day, $51 / month. You now know the actual stake.

3. Bucket by model + routing path. At the end of each day (or run batch), group executions by which model actually ran and whether it was your first choice or a fallback. Count, sum costs, measure fallback frequency. If Groq was supposed to be your primary but it failed 20% of the time, you now have evidence to either increase fallback tolerance, switch providers, or accept that Sonnet is your real cost center.

4. Alert on outliers. A single execution shouldn't surprise you. If one run's cost is 3σ above the 30-day average for that agent, you want to know why: did it produce an unexpectedly long output? Did it hit a fallback you didn't expect? Was there a retry loop? The math stops being invisible.

What this buys you

You move from "I think this is cheap" to "I know what this costs, per execution, per model, per routing path." That's the difference between budgeting in hope and budgeting in fact. The math isn't exciting. But it's urgent, because runaway cost is usually a routing or retry problem hiding in production, and it stays hidden until the bill lands.