The most expensive outages return HTTP 200

The most expensive outages return HTTP 200

A 500 is an unexpected jolt from your sleep, while a 200 silently drains your savings. That's the bug you never get trained for. All is functioning. Every output is pristine. And your cloud bill just tripled.

Downtime is the easy mode. You are immediately alerted when a service is down. Pagers start beeping loudly. Dashboards show red alerts. Customers mention you in tweets. An entire ecosystem exists for this. Uptime monitors, error budgets, on-call rotations. We excel at discovering anything that becomes broken.

What kind of bug causes everything to work as usual, but the cost suddenly triples? The bug that returns 200.

That liteLLM prompt cache invalidation incident

On July 13, 2026. It's really the stuff of nightmares. In theory, prefix-based prompt caching seems easy. If you reuse the same prefix, the generator can serve the remainder at a lower cost from the cache. But, the moment you mutter a different prefix, you pay the full high price.

The catch here is that the cache keys are based on the exact sequence of the message. If you move a line, reorder a cue, the prefix doesn't match and cache is a miss. 💸

An essential system message is moved by someone. Or a few roles are reordered in the request payload. The responses are perfect. The latency hardly changes. Almost every test is successful. And the cache quietly stops working for you. Now, every call costs you the complete token amount. You don't get alerted because nothing is wrong.

Why correctness monitoring misses it

All of our testing efforts are geared towards answering one single question: Did you get the right output? We should not ask that question for cost bugs. The result is always right here. That's the pitfall.

Consider what our assertions are actually testing:

→ Did we get a 200? Yes.
→ Is the response shape valid? Yes.
→ Does the content match expectations? Yes.
→ Did it cost what it should have? ...nobody asked.

Cost is handled as an accounting exercise rather than an engineering indicator. It's contained in a monthly PDF that somebody from finance department goes through, instead of a test failing a pull request. This gap is where the money drains out. An unnoticed expense leak can go on for weeks before somebody looks at the bill and says "hey, what's this."

Cost is a correctness property

Here's the way I like to put it. If a code change triples your spend for identical output, that's a bug. A regressive step, not a "cost optimization opportunity." This should be as obvious and annoying as a broken test.

At our startup, we began treating cost in the same way you treat latency. It's something you insist on, something that causes a build to crash. The cache hit rate isn't a metric that's good to have - it's a commitment.

Even simple changes can lead to unexpected consequences. It could be a rearranged payload, a retry mechanism that bypasses caching, or a logging modification that increases the size of each call.

A 500 error exposes and publicly embarrasses you, so it's no wonder that we do everything we can to eliminate those. On the other hand, a 200 response that fails silently and costs you income is a private embarrassment, potentially months down the road, and more easily ignored.

The takeaway

Instead of cheering for the green dashboard, remember that green simply indicates that there have been no crashes, but not that nothing has failed. In fact, the most costly failure may be up and running at the moment, happily serving 200s, and slowly leaking you tokens that are not cached.

What was the most costly "It's fine" bug you've encountered, and how long did it persist before it was discovered? 👀