Stop Trusting Your Cache Hit Rate. 7 Tests Show Why
This article was originally published on the Actian blog.
In this post, you'll run seven tests against a semantic cache built on Actian VectorAI DB and measure how often a cache hit returns the wrong answer. You'll need Docker, Python 3.11 or 3.12, and uv. All the code is in the companion repo.
A semantic cache reuses an LLM response when a new query is similar enough to a stored one. Hit rate tells you how often that happens. It doesn't tell you whether the reused answer fits the new question.
At a 0.80 similarity threshold, the test cache hit 79.2% of lookups, and 6.4% of those hits were wrong. Some of the wrong hits scored above 0.95.
The results at a glance
| Test | Failure mode | Mitigation | Result |
|---|---|---|---|
| 1. Threshold sweep | One threshold accepts too many wrong matches | Calibrate against a 5% FPR target | At 0.83, FPR was 4.8% and hit rate was 70.1% on held-out queries |
| 2. Negation | Negated questions score like paraphrases | Check for negation differences | Negated queries served fell from 98.7% to 4.0%, and the paraphrase rate stayed at 72.1% |
| 3. Numbers and dates | Changed values still match | Require exact matches on numbers and date words | Changed-value queries served fell from 100% to 0%, and the paraphrase rate stayed at 95.5% |
| 4. Tenant and intent boundaries | Matches cross partitions or intents | MUST filters on tenant and predicted intent | FPR fell to 4.7%, with a hit rate of 58.4% |
| 5. Local verification | Wrong hits pass the threshold and filters | Check sampled hits with decider-0.8b | FPR fell from 6.0% to 2.8% in a 100-hit sample, and the results are provisional |
| 6. Query length | Short queries produce more wrong hits | Calibrate thresholds by query length | FPR fell to 2.0%, with a hit rate of 64.2% on held-out queries |
| 7. Stale answers | Cached responses outlive their source data | Mark stale entries and exclude them from lookup | FPR fell from 24.9% to 11.4%, and the remainder needs refill or intent filtering |
How the test cache works
The cache embeds each query, finds the closest stored question in VectorAI DB, and returns the stored response when the cosine similarity meets the threshold. Otherwise, the query goes to the model.
- The cache stores 3,075 questions from Banking77, a public dataset of banking support queries labeled with 77 intents. That fits under the Community Edition's 5,000-vector limit.
- Lookups use another 770 questions from the same dataset, embedded with
sentence-transformers/all-MiniLM-L6-v2. - A wrong hit is one where the cached question has a different intent, belongs to another simulated tenant, or refers to a source that has changed.
- The false-positive rate (FPR) is wrong hits divided by all hits, and the target is 5% or lower.
- Thresholds are calibrated on one half of the lookup set and evaluated on the other half.
Tenants and source changes are simulated, and the negation and number pairs come from templates. The results show how each failure mode behaves under controlled conditions.
1. Set up the project
Clone the repo and move into the project folder.
git clone https://github.com/actian-devs/developer-content.git
cd developer-content/blog/semantic-cache-fp
Start VectorAI DB and install the dependencies.
docker-compose up -d
uv sync
Check that the container is up with docker ps before you run the tests. The first run downloads the dataset and embedding model, so it takes longer than later runs. Every script writes its measurements to the results/ directory.
2. Sweep the threshold
The sweep looks up all 770 questions at every threshold from 0.50 to 0.99 and records the hit rate and FPR at each step.
uv run python test1_threshold_sweep.py --threshold 0.80
At 0.80, the cache returned 39 wrong answers across 610 hits, an FPR of 6.4%. "Why was I unable to do a transfer?" matched "Why was I not able to complete my transfer?" at 0.921, even though Banking77 labels them declined_transfer and failed_transfer.
Raising the threshold to 0.83 brought the FPR to 4.8% on held-out queries, and the hit rate fell to 70.1%. Use 0.80 for the remaining tests to reproduce the article's numbers.
3. Test negation, numbers, and dates
Each test caches one question from a pair, asks the other, and checks whether the cache reuses the answer. Reworded questions with the same meaning check that good reuse still happens.
uv run python test2_negation.py --threshold 0.80
uv run python test3_numbers_dates.py --threshold 0.80
The cache served 74 of 75 negated questions (98.7%). "Should I not cancel my card?" matched "Should I cancel my card?" at 0.990. Negated pairs had a median similarity of 0.946, higher than the 0.936 median for true paraphrases, so no threshold up to 0.99 blocked 95% of negation errors.
A check that rejects a match when only one query contains negation cut the served share to 4.0% and kept a paraphrase rate of 72.1%.
The cache also served all 38 changed-value queries. "Set my daily card limit to $2500" matched "Set my daily card limit to $250" at 0.962, and a charge on 21 June matched one on 21 July at 0.964. A threshold of 0.97 was needed to block 95% of them, which dropped the paraphrase rate from 95.5% to 13.6%.
An exact-match check on numbers and date words blocked every changed value and kept the paraphrase rate at 95.5%. Both checks are simple. The negation check can miss indirect changes in meaning, and the value check doesn't treat "fifty dollars" and "$50" as equal.
4. Enforce tenant and intent boundaries
This test spreads the cache across three simulated tenants, then repeats the lookups with VectorAI DB filters.
uv run python test4_partitions.py --threshold 0.80
| Configuration | FPR | Hit rate |
|---|---|---|
| No filters | 69.7% | 79.2% |
| MUST filter on tenant | 9.1% | 61.4% |
| MUST filters on tenant and predicted intent | 4.7% | 58.4% |
| Tenant MUST plus top-two intents as SHOULD | 7.9% | 60.6% |
Without filters, 409 of the 610 hits crossed tenant boundaries. A MUST filter on tenant removed every cross-tenant hit. The intent filter used a lightweight classifier that predicted the correct intent 88.4% of the time. Per-intent thresholds reached a 9.6% FPR, with only about five calibration questions per intent in this sample.
5. Verify hits with a local model
This test sends sampled hits to decider-0.8b, a small model that reads both questions and judges whether one answer serves both. It runs on a laptop, while larger models such as decider-4b need a GPU.
Export the lookups first, then run the model on them.
uv run python test5_decider_verify.py \
--threshold 0.80 \
--export-hits data/test5_lookups.json
uv run --extra decider python test5_decider_verify.py \
--threshold 0.80 \
--model Mapika/decider-0.8b \
--max-hits 100 \
--hits-file data/test5_lookups.json
On an Apple Silicon M3 MacBook with a yes-probability cut-off of 0.5, the verifier caught four of the six wrong hits in a 100-hit sample. It also rejected 25 of 94 good hits (26.6%). Median latency was 106 ms per hit, with a 116 ms 95th percentile at batch size one.
A cut-off of 0.3 rejected only two good hits and caught one wrong hit. With only six wrong hits in the sample, treat this as a preliminary result.
6. Compare short and long queries
uv run python test6_query_length.py --threshold 0.80
Queries of eight words or fewer made up 43% of lookups and 27 of the 39 wrong hits. Their FPR was 9.4%, against 2.9% for queries of nine to eleven words and 4.6% for longer ones.
Calibrated thresholds of 0.87 for short, 0.80 for medium, and 0.83 for long queries brought the FPR to 2.0% on held-out queries, with a hit rate of 64.2%.
7. Catch stale answers
Each Banking77 intent acts as a source, and the test changes 15 of the 77 sources after caching.
uv run python test7_stale_answers.py --threshold 0.80
Before invalidation, the FPR rose to 24.9%, with 113 hits returning the right intent and an outdated answer. One filtered set_payload call marked all 600 entries from the changed sources as stale, and a MUST NOT filter excluded them from lookups.
The FPR fell to 11.4% with a 67.4% hit rate. The remaining errors matched a neighboring intent, so pair invalidation with a cache refill or intent filtering.
8. Build the results table
uv run python make_table.py
These results depend on the embedding model, the Banking77 sample, and the experiment's definition of a wrong answer. Intent labels are an imperfect proxy for whether two questions can share a response.
Running a cache you can trust
Measure FPR on a labeled sample of your own traffic, sweep the threshold, and pick the lowest setting that meets your correctness target. Report FPR beside hit rate on every dashboard, and use a stricter threshold for short queries.
Store tenant, intent, and source with every entry, and filter on them during the search. Use a MUST NOT filter on a stale flag after a source changes. Compare negation, amounts, and dates before serving a hit, and add local verification for high-stakes intents such as refunds and cancellations.
Wrapping up
Wrong hits concentrated in negated requests, changed amounts and dates, another tenant's data, and outdated policies. A threshold alone couldn't catch them. Set the threshold against a correctness target, partition the cache with VectorAI DB filters, and verify the hits you can't afford to get wrong.
What's next
Pull VectorAI DB Community Edition and run the same checks on a labeled sample of your own queries. Join the Actian Discord community for help and discussion.
docker pull actian/vectorai:latest
Comments
No comments yet. Start the discussion.