Why Pure Vector Search Fails on Kannada Literature - And How Hybrid RRF Fixed It

A field note on why your RAG app doesn't have a model problem - it has a retrieval problem. Built on a 346-page scanned Kannada novel: OCR, hybrid retrieval, reranking, deterministic routing - and the numbers that proved it worked. The moment I stopped trusting my own system was the moment it confidently answered a question about page 120 with a passage from an entirely different chapter. I was building a RAG agent for Heli Hogu Kaarana, a Kannada novel by Ravi Belagere, digitized from a scanned 346-page PDF. The v1 was the standard recipe: chunk the text, embed with a multilingual model, store in ChromaDB, retrieve top-5, prompt Gemini. It demoed well. It evaluated terribly. This post is everything I learned rebuilding that naive pipeline into a grounded, RAGAS-validated retrieval system - and why the LLM was never the bottleneck. Retrieval was. TL;DR - Multilingual embeddings underperform on agglutinative Kannada - hybrid BM25 + dense retrieval fused with Reciprocal Rank Fusion fixed the retrieval failures. - A deterministic regex router bypasses semantic search entirely for page-level queries: 100% precision, zero hallucination surface. - Final system: 0.92 RAGAS faithfulness / 0.89 context recall on a 50-query golden set, at 2.8s P50 end-to-end on serverless. The Problem Space Nobody Warned Me About Three things make this a hard problem: 1. The book doesn't exist digitally. It's a scanned physical book. Kannada OCR is genuinely hard - ligatures, conjunct characters, noisy print. Your retrieval is only as good as your ingestion. 2. Kannada is agglutinative. A character's name like Himavant appears as ಹಿಮವಂತ, ಹಿಮವಂತನ, or ಹಿಮವಂತನಿಗೆ depending on grammatical case. Multilingual embeddings trained on mostly high-resource data compress these into a weak, inconsistent semantic space. 3. Literary text is sparse and specific. Rare colloquialisms, proper nouns, and page-level references are exactly the queries where cosine similarity goes to die. Generic LLMs have essentially never seen this book. Without perfect retrieval, they don't retrieve - they invent. The Naive Approach, and How It Failed V1 architecture: chunk → embed (multilingual MiniLM) → ChromaDB → top-5 → Gemini . Failure modes, in order of severity: - Proper nouns vanished. Queries using rare or inflected forms returned thematically similar but factually useless chunks. - Page queries became vibes queries. "What happens on page 42?" went through semantic search and returned whatever felt close. - No self-awareness. The system had no mechanism to know when it was wrong, so it answered anyway. Exhibit A: One Query, Two Systems (answers paraphrased for brevity) Query: "What does Himavant say on page 120?" V1 (dense-only): A fluent paragraph about Himavant's conflict - drawn from a different chapter. No citation. Wrong page. Full confidence. V2: The regex router classifies it as an exact-page query. A metadata lookup returns the page-120 chunk in ~milliseconds. The answer quotes it, cited [Page 120] . Same LLM behind both. The difference was entirely in retrieval architecture. The Architecture Five phases, one principle: never let a single retrieval signal make the final decision. ┌──────────────────────────┐ │ User Query │ └────────────┬─────────────┘ ▼ ┌──────────────────────────┐ │ Regex Query Router │ └───────┬──────────┬───────┘ exact page │ │ semantic ▼ ▼ ┌───────────────┐ ┌───────────────────────────┐ │ Page Lookup │ │ Dense (ChromaDB) + BM25 │ │ (metadata) │ └─────────────┬─────────────┘ └───────┬───────┘ ▼ │ ┌───────────────────────────┐ │ │ RRF Fusion (k = 60) │ │ └─────────────┬─────────────┘ │ ▼ │ ┌───────────────────────────┐ │ │ Cross-Encoder Rerank │ │ │ + confidence guardrail │ │ └─────────────┬─────────────┘ ▼ ▼ │ ┌───────────────────────────┐ └───────────►│ Context Builder │ └─────────────┬─────────────┘ ▼ ┌───────────────────────────┐ │ Gemini → Groq fallback │ └─────────────┬─────────────┘ ▼ ┌───────────────────────────┐ │ Sarvam TTS / gTTS │ └───────────────────────────┘ Deep Dive 1: Ingestion Is Half the Battle Garbage in, hallucination out. The ingestion pipeline: - pdf2image at 300 DPI - OpenCV denoising + thresholding (scanned pages are filthy) - Surya OCR for Kannada extraction - indic-nlp normalization for Unicode and ligature cleanup - Semantic chunking with page metadata attached to every chunk That last point is the quiet hero. Page metadata on every chunk is what makes citations possible and what powers the deterministic router below. If your chunks don't carry provenance, your RAG system can't be held accountable. Deep Dive 2: What BM25 Actually Fixes (and What It Doesn't) Here's the uncomfortable truth about multilingual embeddings on Indic languages: the semantic space is undertrained. For exact lexical items - names, places, rare colloquialisms - BM25 catches what embeddings miss, because it matches the surface forms that actually appear in the text. To be precise about what that means: BM25 rescues exact lexical overlap. It does not solve cross-inflection matching - ಹಿಮವಂತನ and ಹಿಮವಂತ are different tokens unless you stem, and my pipeline normalizes Unicode and ligatures, not morphology. Cross-inflection matching remains a partially open problem here; hybrid redundancy and the reranker are what compensate for it. Exhibit B (paraphrased): The query uses a rare colloquialism from chapter 7. Dense-only returns a thematically similar passage from chapter 3 - cosine loved the vibe. Hybrid returns chapter 7, because BM25 matched the exact surface form. That's the whole thesis in one retrieval. But BM25 alone fails at paraphrase. "Explain the protagonist's internal conflict" has zero lexical overlap with the passage that answers it. So neither wins. Both run, in parallel, on every semantic query. The problem then is merging. Cosine similarity lives in roughly [-1, 1] . BM25 scores are unbounded [0, ∞) . You cannot add them. You cannot even meaningfully normalize them. Enter Reciprocal Rank Fusion - fuse the ranks, not the scores: RRF_score(d) = Σ_m 1 / (k + rank_m(d)) with k = 60 . Scale-invariant, dead simple, brutally effective: def rrf_merge(dense_ranks, sparse_ranks, k=60): scores = {} for rank, doc in enumerate(dense_ranks, 1): scores[doc] = scores.get(doc, 0) + 1 / (k + rank) for rank, doc in enumerate(sparse_ranks, 1): scores[doc] = scores.get(doc, 0) + 1 / (k + rank) return sorted(scores, key=scores.get, reverse=True) Deep Dive 3: Cross-Encoder Reranking on a Latency Budget Bi-encoders encode query and passage independently - fast, but shallow. A cross-encoder reads them together with full cross-attention. Much more accurate, much more expensive. The compromise: RRF narrows the field to a small candidate set; the cross-encoder (mmarco-mMiniLMv2-L12-H384-v1 ) reranks only those. Cross-encoder precision at bi-encoder cost. One more guardrail: if the top reranked score falls below a confidence threshold - deliberately conservative, tuned against the golden dataset - the system refuses to answer. A graceful "I can't ground this in the text" beats a fluent fabrication, every time. Deep Dive 4: The Deterministic Page Router Some queries should never touch semantic search. If the user asks about page 42, the correct answer is a metadata lookup - full stop. import re def route(query: str): m = re.search(r"\b(?:page|ಪುಟ)\s*(\d{1,3})", query) if m: return ("exact_page", int(m.group(1))) return ("semantic", None) A regex intercepts page-intent queries and fetches chunks by page metadata directly: 100% precision, ~5-12ms routing latency, zero hallucination surface for that entire class of queries. This is the cheapest "AI win" in the whole system. Not every query needs a neural network. Some need a hash lookup. What Each Layer Actually Earns I'm not going to pretend each component contributed equally. Here's the honest failure taxonomy that drove each addition: - Dense-only: strong on thematic/paraphrase queries; weak on exact names, rare colloquialisms, and inflected proper nouns. - BM25-only: the mirror image - exact on names and rare terms; blind to paraphrase. - RRF fusion: covers both failure taxonomies. The redundancy is the point. - + Cross-encoder: kills the near-misses - passages that mention the same names in unrelated scenes. This is where context precision visibly improved. I've deliberately kept the component-level benchmarks (per-retriever recall@k, reranker MRR lift) in the repo's eval suite - scripts/eval/eval_hybrid.py and eval_reranking.py - so you can reproduce them on your corpus rather than trust mine. The end-to-end numbers below are what the full pipeline scores. The Proof: RAGAS or It Didn't Happen I built a 50-query golden dataset from the novel - exact-fact, thematic, multi-hop, and rare-colloquialism queries - and evaluated with RAGAS. I treat it as a regression suite, not a benchmark - it catches regressions across query classes; it makes no claim to statistical power. | Metric | Score | Target | |---|---|---| | Faithfulness | 0.92 | > 0.85 | | Answer Relevancy | 0.88 | > 0.80 | | Context Recall | 0.89 | > 0.80 | | Context Precision | 0.85 | > 0.75 | One honesty note: these scores reflect the primary Gemini path. Fallback tiers trade marginal quality for availability and were spot-checked, not benchmarked. Latency profile (warm path; serverless cold start adds ~1-2s on first invocation): | Stage | P50 | P95 | |---|---|---| | Query routing | 5ms | 12ms | | BM25 search | 120ms | 250ms | | Dense search | 300ms | 450ms | | RRF + rerank | 450ms | 800ms | | LLM generation | 1.2s | 2.5s | | TTS first byte | 0.8s | 2.0s | | End-to-end | 2.8s | 5.0s | The jump from v1 to v2 wasn't a better model. It was better retrieval, measured. Production Notes - Resilience: generation runs a strict fallback chain - Gemini Flash → Groq Llama-3.3-70B → Llama-3.1-8B → Llama-4 Scout - with automatic 429 handling. Voice mirrors it: Sarvam AI TTS (Kannada-native, ~450-char chunking + WAV stitching) with gTTS as keyless fallback. - S