Should our LLM wiki use vector search? We benchmarked it before deciding

Every "add vector search" proposal comes with the same promise: users phrase questions their own way, keyword search misses, and embeddings fix it. A feature request on Synthadoc made that case for us (issue #411): Synthadoc uses BM25 exclusively.

For wikis under ~200 pages, BM25 is fast and precise; at 500+ pages, paraphrase and concept queries increasingly miss without a semantic layer. It also asked us to turn embeddings on by default.

That's a big change for a pip-installed tool (more on the cost below), so before writing any retrieval code we built a benchmark to answer three questions:

1. Does BM25 really get worse at paraphrased questions as a wiki grows?
2. Does the vector search we already ship (off by default) fix it?
3. If not, what design would and by how much?

Yes, keyword search degrades with wiki size - but only for paraphrases. On a 2,000-page wiki, BM25 put the right page in the top 5 for 48% of paraphrased questions, down from 71% at 200 pages. Questions that use the page's own words or an exact name stayed at ~99-100% at every size.

Our existing "vector re-rank" mode would have made search worse overall. It lifted paraphrases (48% → 60%) but wrecked exact-name lookups (99% → 63%), for a net drop from 84% to 74%.

Fusing the two rankings fixed paraphrases (48% → 69%) and lifted recall overall, but exact-name lookups still slipped to 94%. One more rule closed that gap: when BM25 has a clear winner (its best score at least 5% ahead of the next), keep BM25's top hits on top.

Final result on 2,000 pages: paraphrases 48% → 68%, exact names 99% → 98%, page's-own-words 100% → 100%, overall 84% → 90% - and the right page reaches the top 20 for 88% of paraphrases instead of 65%.

Bonus: the benchmark exposed that BM25 rebuilt its index on every query. Caching it made keyword search 18× faster (P95 362 ms → 20 ms) for every user, embeddings or not.

It held up on data we never tuned on. On two English corpora with human-written queries and human relevance judgements (science claims, medical questions), semantic search was better on every measure. On Chinese it was worse - our embedding model is English-only - so Chinese, Japanese and Korean pages and questions now stay on keyword search, automatically.

So it's now on by default for new wikis (vector = "auto"): semantic search whenever the add-on is installed, which the Docker image is. And faster: running the query embedding alongside BM25 instead of after it cut semantic search from 166 ms to 62 ms (P95, 2,000 pages).

The same scale-up broke our knowledge graph. At 5,000 pages the graph view took 34 seconds to appear and then froze the browser. It now opens on a cluster overview in about 6 seconds and runs at a smooth 60 fps.

Background: how Synthadoc finds pages today

Synthadoc is an open-source tool that compiles your sources - PDFs, web pages, notes - into a Markdown wiki (it works nicely as an Obsidian vault), then answers questions from that wiki with citations. Answer quality depends almost entirely on one step: finding the right pages to hand to the LLM.

Today that step is BM25, the classic keyword-ranking function behind most search engines before neural search. It scores pages by how many of the query's words they contain, weighting rare words more. It's fast, needs no model, and is excellent when the user uses the same words as the page.

We also ship an optional [search] vector = true mode. It takes BM25's top 20 results and re-sorts them by embedding similarity - how close the query and the page are in meaning, according to a small embedding model (BAAI/bge-small-en-v1.5, run locally via fastembed).

Keyword vs. vector, in one example

• Query BM25 Embeddings
• "ENIAC vacuum tubes" ✅ exact words on the page ✅
• "the first electronic machine built to compute artillery tables" ❌ few shared words ✅ same meaning
• "Moore School 1946" ✅ rare exact tokens ⚠️ names and numbers carry little "meaning"

Each approach fails where the other is strong. That's the whole story of this post in one table - but we wanted numbers, not intuition.

Why measure first

Turning embeddings on by default isn't free for a pip-installed tool:

• +~175 MB of dependencies (onnxruntime alone is 80 MB) and a 65 MB model download on first start - awkward for offline machines and slim Docker images.
• An English-only model, in a project that also serves Chinese, Japanese and Korean users.
• A retrieval change touches every answer the product gives.

If the gain is small, or only shows up on wikis far bigger than our users run, that cost isn't worth paying by default.

How we built the benchmark

We wanted something realistic, reproducible, cheap to run, and small enough to keep in the main repo.

A wiki-like corpus. We pinned 1,985 English Wikipedia articles from 24 overlapping computing categories - history of computing, AI, cryptography, databases, networking, and so on - each at a fixed revision ID, so the text never changes under us. Overlap matters: paraphrase misses happen when many similar pages compete. Each article is converted to a Synthadoc wiki page (lead plus sections, capped at 1,500 words, roughly the size of a compiled wiki page).

Three kinds of question per page. For 500 randomly sampled pages, an LLM (Claude Sonnet 5.5) wrote three questions that page answers:

• lexical - uses the page's own key terms;
• paraphrase - the same need, worded by someone who doesn't know the terminology: no words from the title, and at most two of the page's distinctive terms (checked mechanically, not trusted to the LLM);
• identifier - a short lookup of an exact name, acronym, number or date.

That gave 1,430 questions (some paraphrases failed the check twice and were dropped rather than mislabelled).

A human-written cross-check. LLM-written paraphrases might suit embedding models unfairly well, so we also ran 300 real questions from HotpotQA over its ~3,000 Wikipedia paragraphs.

Wiki sizes. We scored the same questions on nested 200, 500, 1,000 and 1,985-page wikis to test the "it only matters past ~200 pages" claim.

Retrieval only, no LLM in the loop. Each setup is the real Synthadoc search code, called the way the query agent calls it. Scoring a run needs no API key and gives the same numbers every time.

Small in git. Only titles + revision IDs, the frozen questions, HotpotQA question IDs and the results are committed - about 0.5 MB. A script downloads and caches the text (both sources are CC BY-SA).

Metrics. Recall@5 - how often the right page is in the top 5 results we hand to the LLM. We also tracked Recall@10/20, MRR and nDCG (how high it ranks) and P95 latency.

Two bugs the benchmark caught in itself

Worth sharing, because both would have silently produced a wrong conclusion:

• Our "distinctive term" check thought "the" was distinctive. The first generation run threw away 74% of paraphrases. The checker reused our stopword list, which only covers question words ("what", "how") - so "the", "and" and "for" counted as giveaway terms. We switched to corpus-wide rarity: a word only counts if it appears on ≤10% of pages. Accepted paraphrases went from 26% to 87%.
• Two random samples with the same seed picked disjoint pages. Python's random.sample and random.shuffle walk the same random stream, so our "which pages get questions" sample and our "which pages form the smaller wiki" shuffle, both seeded with 411, never overlapped. Every smaller wiki scored zero questions. Now they use different seeds, with a test that checks the overlap is proportional.

What we found

1. BM25 is near-perfect on its own words, and steadily worse on paraphrases

Recall@5, BM25 only:

• Wiki size lexical identifier paraphrase
• 200 pages 1.000 1.000 0.711
• 500 pages 1.000 1.000 0.647
• 1,000 pages 1.000 1.000 0.573
• 1,985 pages 0.998 0.994 0.479

The issue was right, and now we know by how much: every doubling of the wiki costs roughly 7-8 points of paraphrase recall. Even Recall@20 for paraphrases falls from 0.80 to 0.65 - a third of those questions never get the right page into the candidate list at all.

2. Our existing vector mode helps paraphrases and breaks exact lookups

Recall@5 on the full 1,985-page wiki:

• Question type BM25 BM25 + vector re-rank (today's vector = true)
• lexical 0.998 0.986
• identifier 0.994 0.625
• paraphrase 0.479 0.595
• all 0.840 0.742

Three reasons, all visible in the code once the benchmark pointed at them:

• It throws away BM25's order. Re-ranking sorts purely by embedding similarity, so a page that matched "Moore School 1946" exactly can drop below pages that are merely about early computers.
• It only embeds each page's first 512 characters. A name or number further down the page is invisible to the embedding.
• It can only reorder what BM25 already found. Paraphrase Recall@20 is 0.646 with or without it - re-ranking can't rescue a page BM25 missed.

On HotpotQA's human-written questions the re-rank looked great (Recall@5 0.717 → 0.842), but HotpotQA's distractor paragraphs were chosen for word overlap with the question - it's built to be hard for keyword search. On our own kind of content, the picture is very different. This is exactly why we benchmark on our own use case.

It's also the slowest setup: every query reloads every stored embedding from SQLite.

3. Fusion: let each method vote

If BM25 and embeddings each fail where the other is strong, don't make one the judge of the other. Our prototype runs both over the whole wiki, takes the top 20 from each, and merges them with reciprocal rank fusion (RRF): every page scores 1 / (60 + rank) in each list it appears in, and the scores are added. No score calibration - a page near the top of either list surfaces, and a page high in both wins.

That fixed paraphrases (Recall@5 0.479 → 0.694) but still cost exact-name lookups (0.994 → 0.936): an exact BM25 hit now shared the vote with pages that were merely similar in meaning.

4. Knowing when to trust the keywords

Rather than guess at weights, we asked the data: when is BM25's top result right? One signal stood out - how far BM25's best score is ahead of its second. When the top score is at least 1.2× the next:

• Question type how often that happens BM25's top result is then correct
• identifier 88% 97%
• lexical 86% 100%
• paraphrase 12% 84%

A dominant keyword match almost always means an exact name or term hit, and it rarely happens for paraphrases. So the final rule is simple: when BM25's best score is at least 1.05× its second, keep BM25's top 3 on top; otherwise let RRF decide. (We also tried simply weighting BM25's vote higher. It backfired: BM25's entries then always outrank the embedding ones, and fusion collapses back into keyword search.)

Recall@5 on the full 1,985-page wiki:

• Question type BM25 Vector re-rank Plain fusion Fusion + keyword pin
• lexical 0.998 0.986 0.994 0.998
• identifier 0.994 0.625 0.936 0.984
• paraphrase 0.479 0.595 0.694 0.676
• all 0.840 0.742 0.883 0.896
• MRR (all) 0.791 0.618 0.718 0.828

It's the only setup that beats BM25 on every overall measure, including how high the right page ranks (MRR). And the part re-ranking could never do - paraphrase recall within the top 20, i.e. whether the right page reaches the LLM's candidate list at all:

• Wiki size BM25 Vector re-rank Fusion + keyword pin
• 200 pages 0.800 0.800 0.956
• 500 pages 0.765 0.765 0.966
• 1,000 pages 0.720 0.720 0.920
• 1,985 pages 0.646 0.646 0.875

The gain grows with the wiki - paraphrase Recall@5 improves by 9 points at 200 pages, 11 at 500, 14 at 1,000 and 20 at 1,985 - which is exactly the shape the original issue predicted. On HotpotQA's human-written questions, fusion scores 0.815 at top 5 (BM25: 0.717) and 0.970 at top 20 (BM25: 0.920). The handful of exact-name questions it still misses are mostly ambiguous labels: "A Mathematical Theory of Communication 1948" is cited on several pages about information theory, but only one is labelled correct.

5. The bug we found in plain sight: BM25 rebuilt its index on every query

BM25 latency grew from 77 ms at 200 pages to 362 ms at 1,985. Profiling one query on the full wiki:

• Step Time
• Build the BM25 index 356 ms
• Score the query 3 ms

Synthadoc cached the tokenised pages but rebuilt the index on every search. Caching the built index (per search scope, cleared whenever a page changes) took BM25's P95 from 362 ms to 20 ms at 1,985 pages with identical results - a speed-up every user gets, whether or not they turn on embeddings.

Fusion's P95 was 122-166 ms, and profiling showed why: the search ran BM25, then embedded the query, so the two costs added up. Embedding the query is about 20 ms on a laptop CPU; comparing it with every page embedding in memory takes under 2 ms. Both steps are independent, and the embedding model releases Python's GIL while it runs, so we now start the query embedding in a worker thread before BM25 runs. Same results, and P95 at 1,985 pages dropped from 166 ms to 62 ms (BM25 alone: 15 ms). We also keep the last 256 query embeddings - an embedding depends only on the text and the model, so a cache entry can never go stale.

Either way, that is well under 1% of an answer that spends seconds in the LLM.

6. The "better" embeddings that made things worse

Our prototype embedded only each page's title, tags and first 512 characters, so the obvious next step was section-level embeddings: one per heading, so content deep in a long page is searchable. On its own, the embedding search got clearly better - paraphrase Recall@5 0.616 → 0.669, and exact-name lookups 0.305 → 0.659. Fused with BM25, it got worse:

Recall@5, full wiki One embedding per page One per section paraphrase 0.676 0.613 all 0.896 0.878 time to embed the wiki 3 minutes 82 minutes

The more precise section embeddings mostly found the pages BM25 already found. Fusion gains from the two methods disagreeing - each catching what the other misses - and the coarser page embeddings disagreed with BM25 in more useful places. Twenty times the compute for a worse result was an easy call: we ship one embedding per page.

7. Checking it on data we didn't tune on