One Cache Miss, Fifty Database Calls

Your cache hit rate can be 99% and your database can still fall over. I got reminded of this during a load test recently: the dashboard showed a wall of cache hits, and yet the database briefly lit up with a burst of identical queries. Same key, same query, fifty times, all inside a couple hundred milliseconds.

That burst has a name, cache stampede, and the annoying part is that the standard IMemoryCache pattern we all write does nothing to stop it. So I built a small demo to catch it in the act.

The demo

One minimal API, two endpoints, same fake database. The fake database counts every call and takes ~200 ms, like a real query under load. The app starts itself on a random port, then fires 50 concurrent requests at each endpoint with a cold cache. My project targets net10.0, but nothing here is new to .NET 10 - HybridCache has been stable since .NET 9.

Here's the endpoint most of us have written a hundred times:

app.MapGet("/products/memory/{id}", async (string id, IMemoryCache cache, FakeDb db) =>
    await cache.GetOrCreateAsync($"mem:product:{id}", entry =>
    {
        entry.AbsoluteExpirationRelativeToNow = TimeSpan.FromMinutes(5);
        return db.LoadProductAsync(id, flavor: "memory");
    }));

It reads like it should be safe. GetOrCreateAsync sounds atomic: get it, or create it, surely one or the other. In my opinion it's one of the most misleadingly cozy method names in the whole framework. There's no lock in there. Every request that misses starts its own factory, they all run the query, and they all write the result. Last one wins. Fifty requests arriving before the first query finishes means fifty queries.

The fix is embarrassingly small. Register builder.Services.AddHybridCache(); and write this instead:

app.MapGet("/products/hybrid/{id}", async (string id, HybridCache cache, FakeDb db, CancellationToken ct) =>
    await cache.GetOrCreateAsync(
        $"hyb:product:{id}",
        async token => await db.LoadProductAsync(id, flavor: "hybrid"),
        new HybridCacheEntryOptions { Expiration = TimeSpan.FromMinutes(5) },
        cancellationToken: ct));

HybridCache dedupes concurrent callers per key. The first request runs the factory; the other forty-nine wait for that same task instead of starting their own.

What actually happened

Here's the output from my run, after a warmup pass so nobody's paying JIT tax:

Firing 50 concurrent requests per endpoint (cold cache), fake query takes ~200 ms

IMemoryCache cold burst:  50 db calls, 215 ms wall
HybridCache cold burst:    1 db calls, 208 ms wall
IMemoryCache warm burst:   0 db calls,   9 ms wall
HybridCache warm burst:    0 db calls,  11 ms wall

Usual disclaimer: this is a cloud container, the "database" is Task.Delay(200), and the app is hammering itself over localhost. Not a lab. The number I trust here is the call count, not the milliseconds. And the call count is the whole story: 50 versus 1.

Notice what didn't change, though. Wall time was a tie. Both bursts finished in roughly 210 ms, because HybridCache's waiting requests still have to wait for the one real query. Stampede protection doesn't make the cold burst faster for your users. What it does is send your database 2% of the work.

My fake database has a fixed delay, so I can't show the compounding effect in numbers, but a real one doesn't behave like that - fifty identical concurrent queries make each other slower, which stretches the miss window, which lets more requests pile in. That's the spiral this kills.

The warm rows are there for honesty too. Once the key is cached, both approaches are fine. Caching was never the problem. The 200 milliseconds after expiry were the problem.

The footgun I hit on the way

My first version of the demo crashed, and the reason is worth knowing. I used the same key string, product:42, in both endpoints. Turns out HybridCache's L1 is the same MemoryCache instance that DI hands to IMemoryCache, and it stores its own wrapper objects in there. So my plain IMemoryCache read found HybridCache's entry and died with:

System.InvalidCastException: Unable to cast object of type 'MutableCacheItem`1[Product]' to type 'Product'.

If you're migrating an app cache-by-cache and both APIs are alive at once, prefix your keys. Ask me how I know.

When I wouldn't bother

If the factory is cheap, a dictionary lookup or a sub-millisecond primary-key fetch, fifty duplicate calls cost less than the time you'll spend thinking about them. IMemoryCache is also still the right tool when you're caching things that can't or shouldn't be serialized, since HybridCache wants serializable values the moment you add an L2. And the protection is per process: three pods still means three cold queries, one each, unless you wire up a distributed backplane behind it. One query per pod is usually a fine place to stop worrying.

But for anything that's expensive to build and shared across requests, the hot product page, the tenant config, the dashboard aggregate, this is about as cheap as insurance gets. One service registration, one method swap.

Full runnable sample: https://github.com/ssukhpinder/dev-to-code-samples/tree/main/002-hybridcache-stampede

Clone it, crank Clients up to 500, watch the counter.

Have you caught a stampede in production, or did you only find out from the database bill? Tell me in the comments.

• still counting database calls nobody asked me to count