Why an old caching trick is your secret to lower LLM costs

Why an old caching trick is your secret to lower LLM costs An LLM can answer the same question a thousand times and charge you each time. Before paying for another answer, check whether anything that could change it has changed: the request, its context, the model settings, or the underlying data. I fingerprint those inputs and dependencies to create an exact-match cache key. If that key points to an answer that’s still valid and safe to reuse, I return it without calling the model. The savings start with a simple decision: knowing when the work is already done. I didn’t learn this lesson from an LLM job. In production data pipelines, I’ve encountered a recurring pattern: a nightly job recalculates aggregations that haven’t changed since the previous run. It passes all its checks and moves the results into production successfully, all while burning compute that could have been used elsewhere. The waste hides in plain sight because nothing appears broken. It often surfaces during a cost review, when someone notices that a significant portion of upstream compute is re-answering a question whose inputs never changed. The fix is change detection: hash the upstream inputs that could change between runs, fingerprint the job’s dependencies, and skip recomputation when the fingerprints match. Done well, this significantly reduces the compute that job consumes. The lesson is common, and it’s the same one we keep trying to drive home in LLM workloads. There, repeated requests can also produce repeated charges, since billing is by token. The problem is simple enough to state, but the more you look into it, the more you need a framework to engineer a good answer. For most of the LLM calls in our codebase and infrastructure, we’re billed by tokens, and many APIs treat duplicate requests as new ones anyway. Duplicate sources are almost as inevitable as rain. Upstream users converge on similar questions to answer with their LLM tools. Batch jobs dutifully repeat boring boilerplate every time they run. Prompt-engineering experiments in development and CI runs invoke the same prompt repeatedly. And tool-calling agents may hit the same knowledge-base tool many times in a single work day. Native prompt caching is a different thing from the response caching I’m describing. In prompt caching, providers reuse cached prompt computation and charge eligible cache reads at reduced rates; output generation remains billable. In response caching, we try to skip the call entirely when an answer already exists in our own infrastructure. Tier 1: exact match The simplest approach is to normalize the model request body, run it through a cryptographic hash like SHA-256, then look up the hash in an in-memory store like Redis. If we find a match, we return the answer without waiting for model inference. An exact-match cache works best when we can expect our model requests to be bounded and predictable. That doesn’t sound exciting, but for most of our batch pipelines, CI runs, and boilerplate summarization tasks, it’s exactly what we need. Tier 2: semantic match For many workloads, exact match isn’t enough. We’d like to look up a response for a query that’s close but not identical. So we take the user’s query, run it through an embedding model, and store the resulting vector in a vector database. When a new query arrives, we run it through the same model and search for close matches by cosine similarity. Close enough by what measure? A common starting point is a cosine-similarity threshold in the [0.90, 0.95] range, but treat that as a number to tune, not a default - the right value depends on your embedding model and your data, and you should test it against real queries. Note that vector stores differ in what they return: cosine similarity rises toward 1 for closer matches. At the same time, some engines report a distance that falls toward 0, so confirm which your threshold is comparing against. Either way, a looser threshold raises the risk of wrong matches, where the system answers one query while the user was asking about another. (“What’s the weather in my town?” can’t be safely conflated with the same question about a different town just because the cosine similarity is high.) Tier 3: hybrid A common approach runs both tiers in sequence: check the exact-match store first, and run semantic search only on a miss. When semantic search returns a close-enough match, the result is promoted back into the exact-match store under the hash of the new query that triggered it, so the paraphrase and its answer are an exact hit next time. This favors cheap exact matches on repeat traffic. The pseudocode below shows the full flow: normalization and SHA-256 for exact match; a Redis get followed by a set on a miss; embedding the query and searching the vector DB with top_k=1; checking cosine similarity against the per-category threshold; and setting the TTL before writing the response back into the exact store. Both tiers key on more than the query text alone: the context and documents in the prompt, the model and its settings, the version of any retrieved source, and the caller’s access scope. Two identical questions asked against different documents, or by users with different permissions, must not share a cache entry. def cached_completion(query, ctx): # ctx bundles everything that changes what the correct answer is: # the context/documents in the prompt, the model and its settings, # the source-version of any retrieved content, and the caller's access scope. key = sha256(normalize(query, ctx)) # Tier 1: exact-key lookup on Redis (O(1)). # Correctness still depends on cache contents, request scope, and freshness. if (hit := redis.get(key)): return hit # Tier 2: semantic search, restricted to the same scope as the request. emb = embed(query) match = vector_db.search(emb, top_k=1, filter=scope_of(ctx)) if match and same_scope(match, ctx) \ and match.score >= threshold_for(category(query)): # Promote, but preserve the original freshness deadline. remaining = match.expires_at - now() if remaining > 0: redis.set(key, match.response, ttl=remaining) return match.response # Miss on both tiers: call the model, validate before writing back. resp = llm(query, ctx) if is_valid(resp): # no errors, no empty payloads, no malformed JSON ttl = ttl_for(category(query)) redis.set(key, resp, ttl=ttl) vector_db.insert(emb, resp, ttl=ttl, scope=scope_of(ctx)) return resp One threshold does not fit all. Code-like queries often need stricter thresholds, around 0.95 or higher, because small wording changes can produce entirely different results. Conversational queries can tolerate looser thresholds, in the 0.85 to 0.90 range. These numbers are starting points, not settled values - validate them for your own workload and embedding model before relying on them. Cache freshness works the same way, and the right TTL follows from how much staleness the use case can tolerate, not from the data type alone. A cached market-data answer might be acceptable for only a minute or two, because a stale price can be actively misleading. An internal HR policy answer can often be reused for weeks, because the underlying document rarely changes and a slightly old answer is usually still correct. The interval is a judgment about acceptable staleness, not a fixed property of the content. The math For illustration, suppose a workload of 1,000,000 calls per month at $0.006 per call, roughly $6,000 with no caching. Say a hybrid cache gives about a 60% hit rate, whichever tier hits first, avoiding 600,000 calls to the model, and that embedding and vector-store costs come to about $150. That brings monthly spend closer to $2,550, a 57.5% reduction, plus the latency win of answering many questions without waiting on the model. One caveat worth shouting: measure your hit rate before you project any savings. The decisions There’s more to this than the tiered framework. Tune your TTLs to the freshness each data type actually needs, and invalidate entries when you update the content behind them. A fine-grained approach assigns a per-category TTL based on how quickly each answer goes stale: a news summary might hold up for an hour, while a live sports score is worthless within seconds and shouldn’t be cached at all during a game. Live scores require a freshness policy matched to the application. Verified final scores can support much longer caching, with invalidation for corrections. The distinction here is whether the underlying value is still moving. A blunter approach skips per-category tuning entirely and purges the whole cache whenever the source content changes. Either way, run the cache in shadow mode first, logging what you would have returned without changing behavior. Evaluate cached answers against verified reference answers or expert review. A fresh model response can help identify differences, but it is not ground truth. Warm the cache from a historical set of common queries before you rely on it, and validate answers before writing them back, so you don’t poison the cache with errors, empty responses, or malformed content. When not to cache? Skip it for requests with personal or account-specific data, to avoid leaking one user’s cached output into another’s request. Skip it for creative tasks, where you want a different answer each run. And skip it for genuinely real-time data like stock prices and live inventory, where an answer even a minute old may be too stale for the application. The takeaway The principle predates the web: Donald Michie described memo functions in 1968. When you can, fingerprint the question and store the hashed exact form alongside the semantic-variant form, so you avoid repeated model calls while a valid cached answer remains available.