CARDIAC-PURR - I built an LLM cost-router. Here's what 100 questions per provider, across 9 providers, taught me.

The Problem: Unmanaged Model Selection

You ship a feature, it calls GPT-4, it works beautifully, you move on. Six months later, someone checks the logs and - surprise - half your traffic is "what's my account number" going through a model smart enough to pass the bar exam. This isn't a one-off mistake, either - it's the default outcome of not routing at all. Support-queue traffic is mostly simple, but "mostly simple" tends to get shipped through whatever model handled the hard case that made you reach for a big model in the first place. The model's not the problem. The lack of a router is.

CARDIAC-PURR: A Cost-Router Architecture

CARDIAC-PURR sits between your app and your LLM providers, monitoring each query and deciding whether it actually needs a large model or would do fine with a cheaper one. I tested it against 9 providers, each with its own small→medium→large split:

DeepSeek's small and medium tiers point at the identical model (deepseek-v4-flash) - there's no true 3-way split for DeepSeek, just flash vs. pro. That sounds like a limitation, and I originally treated it as one, but it turns out to be the reason DeepSeek posts the highest savings of any provider tested: flash is dramatically cheaper than pro, and 92% of traffic never touches pro at all.

Testing Methodology

I've been running this in production since March. For this post, I wanted a clean, controlled comparison: same 100 questions, asked to nine providers, so the numbers below are apples-to-apples. I ran 100 queries per provider - Anthropic, OpenAI, Google, Azure OpenAI, Mistral, DeepSeek, Cohere, Grok, and Qwen. Real money spent. Real API calls. Same 100 questions across all nine providers.

The queries were intentionally unglamorous: 75% factual definitions ("What does NDA stand for?"), 17% explanatory ("Describe the main steps in..."), 8% complex reasoning. This is what actual support queues look like.

Measuring Success

I measured two things separately because mixing them is how benchmarks lie:

• Routing accuracy: Did the router pick the right tier?
• Vertical accuracy: Did accuracy hold up across domains (legal, healthcare, finance, IT)?

With n=100 per provider, 100% accuracy doesn't mean zero future misclassifications - it means the 95% confidence interval is [96.4%, 100%].

Run date: July 12, 2026 - router commit 57175b7, which fixed a warmup-accounting bug that had previously forced the first ~47 post-warmup queries to the LARGE tier regardless of actual complexity.

Performance Results

Notes:

• † DeepSeek is the best performer in this benchmark, and the reason flips an earlier, wrong estimate: a prior pass reported 46.0% from a coarse price-ratio table, not real dollars. The actual number, computed from real per-query costs against DeepSeek's LARGE-tier price, is 1 - $0.0059/$0.0505 = 88.3%.
• ‡ Cohere's medium and large tiers are priced identically ($12.50/MTok combined), so there's no discount available between those two tiers by design - that structural fact caps Cohere's best-case savings below the other providers' ceiling regardless of errors.
• § Grok is the lowest-savings provider in this run - structural, not a routing weakness.
• ⚠ Cohere is the only provider with a nonzero error rate in this run (2/100 - real timed-out API calls, distinct from cascades, which were zero for every provider this run).

DeepSeek Analysis

DeepSeek's shared small/medium model is so much cheaper than the large tier that it posts 88.3% savings, the highest of any provider in this benchmark, once measured from real per-query dollars rather than an earlier coarse price-ratio estimate.

Anthropic's savings (78.8%) are the lowest among the clean-run providers, though, and that's worth explaining on its own: Claude's small tier (Haiku) has a 256-token output cap. When you ask medical or legal questions that need longer answers, Haiku hits that cap. The router detects this on the response and escalates to a larger tier.

Grok Limitations

Grok doesn't have that option right now: all three of its tiers - small, medium, and large - call the same model, grok-4.3. The only lever the router has is a reasoning_effort parameter (none/low/high), which is a compute-budget knob on one model, not a routing decision between differently-priced models.

The measured per-tier cost ratios (relative to LARGE = 1.0) make this concrete:

Anthropic's small tier (Haiku) costs 5.2% of the large (Opus) because it's a fundamentally smaller model. Grok's small tier costs 60.5% of the large because it's the same model just told to think less - fewer reasoning tokens burned before answering, same per-token price. That gap in ratios is the entire explanation for why Grok tops out at 30.9% savings while Anthropic gets 78.8%+.

Two real bugs were found and fixed during investigation. One was a _GROK_EFFORT_MAP definition at function scope instead of module scope in universal_http_client_v20.py - it was intermittently unreachable. The router's LARGE-tier gate used a hardcoded c_current < 0.65 threshold instead of the calibrated self.c_target (0.600), which made LARGE unreachable for queries scoring between 0.600 and 0.649. Both are fixed and spot-checked - 4/4 verification queries now route correctly with real dollar separation between tiers ($0.00037 → $0.00175 → $0.0045).

Latency Characteristics

Routing decision time (pre-inference): measured in isolation over 3,000 samples - p50 0.42ms, p95 0.68ms, p99 0.97ms, max 1.85ms. Pure in-process computation, no I/O, so it stays sub-millisecond in the typical case.

End-to-end latency (routing + provider response):

Cohere's 622.7ms is the outlier, and this run I can point to exactly why: it's not steady-state routing cost; it's the retry overhead from the same two timeout events described above. Two calls that had to time out and retry pull the aggregate proxy-overhead number up dramatically even though 98 of Cohere's 100 calls behaved normally.

Anthropic's P99 is the ugliest number in the table - 46.2 seconds. Worth explaining rather than hand-waving: router overhead for Anthropic was 16.7ms this run, same as every other clean provider, so the router itself isn't the delay. Anthropic's IT-vertical queries are the bottleneck.

Outlook

The numbers below supersede any earlier pass. With a cleaner dataset and corrected pricing, the picture remains clear: routing accuracy is the fair cross-provider comparison, and on that metric 8 of 9 providers hit 100%. The real challenge moving forward is expanding sample size and validating on different workload distributions. Run date: July 12, 2026 - router commit 57175b7, which fixed a warmup-accounting bug that had previously forced the first ~47 post-warmup queries to the LARGE tier regardless of actual complexity.