The Thundering Herd Problem (Cache Stampede) - and How to Tame It

1. What Is the Thundering Herd Problem?

The thundering herd problem (also called a cache stampede or dogpile effect) happens when a popular cache key expires or is missing, and a large number of concurrent requests all miss the cache at the same instant. Instead of one request hitting the database, hundreds or thousands do - simultaneously.

Picture a hot product page on an e-commerce site. The cache entry for product:12345 expires at exactly midnight. If that product gets 2,000 requests/second, all 2,000 requests in that window see a cache miss and stampede toward the database at once. The DB, which was only ever meant to serve the occasional cache-refill query, gets hammered with duplicate work - often enough to tip over under load, which then causes more timeouts, more retries, and a feedback loop that can take down the whole service.

The core issue isn't that the cache missed - misses are normal. The issue is redundant, uncoordinated work: every request tries to regenerate the same value independently, when only one regeneration was ever necessary.

2. The Solution Strategy: Locking (a.k.a. "Cache Mutex" / Request Coalescing)

The code below solves this with a distributed lock pattern built on Redis's SETNX (set-if-not-exists). The idea: On a cache miss, only the first goroutine to notice gets to regenerate the value. Everyone else waits for that one regeneration to finish, then reads the freshly-populated cache. This turns an N-way stampede into effectively one database query, no matter how many concurrent requests miss at once.

3. Walking Through the Code

Step 1 - Try the cache first

This is the fast path. Most requests, most of the time, exit right here. No locking overhead is paid unless there's actually a miss. Notice the key cache:product:<id> as there is another key for lock lock:product:<id>, as the cache key will delete once expires.

func getProduct(id string) (Product, error) {
    val, err := redis.Get(ctx, "cache:product:"+id)
    if err == nil {
        return val, nil // cache hit, done
    }
}

Step 2 - Race for the lock

SETNX is atomic in Redis: only one caller among all the concurrent racers will get acquired == true. That caller becomes the designated "regenerator":

• It queries the DB (the expensive operation).
• It repopulates the cache with a fresh TTL (300*time.Second).
• It deletes the lock, signaling "I'm done, cache is fresh."

acquired, _ := redis.SetNX(ctx, "lock:product:"+id, "1", 5*time.Second)
if acquired {
    val := fetchFromDB(id)
    redis.Set(ctx, "cache:product:"+id, val, 300*time.Second)
    redis.Del(ctx, "lock:product:"+id)
    return val, nil
}

The lock itself has its own TTL (5*time.Second). This is a safety net: if the regenerating goroutine crashes, panics, or the process dies before it can Del the lock, the lock self-expires instead of being held forever - preventing a permanent deadlock where nobody can ever regenerate that key again.

Step 3 - Everyone else waits and polls

All the losing goroutines (those that didn't get the lock) don't go to the DB themselves. Instead they poll the cache a handful of times with a short sleep in between, waiting for the winner to finish and populate cache:product:*. As soon as they see the key exists, they return it - they get a fresh value without ever touching the database.

for i := 0; i < 5; i++ {
    time.Sleep(50 * time.Millisecond)
    val, err := redis.Get(ctx, "cache:product:"+id)
    if err == nil {
        return val, nil
    }
}

Step 4 - Bounded fallback

If the winner is unusually slow (5 × 50ms = 250ms of waiting with no result), the losers give up polling and fetch from the DB directly. This is a deliberate trade-off: it sacrifices some stampede protection in the worst case in exchange for bounded latency - no request waits forever, no matter what happens to the winner.

return fetchFromDB(id), nil

4. Why This Works

The lock converts a thundering herd into a queue of one worker, many waiters - a classic instance of the broader technique known as request coalescing or singleflight (Go's golang.org/x/sync/singleflight package implements the same idea in-process, without Redis, for a single-node case).

5. Trade-offs and Failure Modes to Be Aware Of

• Polling adds latency for losers. In the worst case, a loser waits up to 250ms before falling back to the DB itself. If your DB call is itself slow, tune the sleep/retry count accordingly, or switch to a pub/sub notification instead of polling (Redis Pub/Sub or a condition-variable-like mechanism) so losers are woken up immediately rather than polling blindly.

• Lock TTL vs. DB latency. If fetchFromDB can legitimately take longer than 5 seconds, the lock could expire while the winner is still working, letting a second goroutine "acquire" the lock and duplicate the fetch. Size the lock TTL comfortably above your p99 DB latency.

• Fallback defeats the purpose under sustained slow DB. If the DB is consistently slow (not just a one-off), many losers will time out their poll loop and hit the DB anyway, partially reintroducing the herd. This pattern protects best against short stampedes (cache expiry moments), not sustained DB degradation.

• Stale reads during regeneration. Everyone except the winner is blocked or reading old-then-new data; if you need to avoid ever serving a stale value while it's mid-regeneration, this pattern needs pairing with logic that serves the old value during the lock window (stale-while-revalidate) rather than making losers wait.

6. Related/Alternative Strategies

• Probabilistic early expiration (XFetch): recompute the value slightly before it actually expires, with a probability that increases as expiry nears - spreads regeneration out over time instead of concentrating it at a single instant.

• Stale-while-revalidate: serve the old (slightly stale) value immediately while one background goroutine refreshes it, so no request ever blocks.

• In-process singleflight + distributed lock combo: dedupe requests within a single node using singleflight first (cheap, no Redis round-trip), and only use the Redis lock across nodes - reduces Redis load further in high-QPS services.

The pattern in your snippet - SETNX lock + bounded poll + DB fallback - is a solid, production-common baseline. The main dial to tune is the poll interval/count vs. lock TTL vs. your DB's actual latency profile.