Phase 4: Retrieval Quality & Grounded Answers
The Story Starts: "Why Did It Confidently Lie to Me?"
👦 Nephew: Uncle, I tested our RAG system this week. I asked about a leave policy that doesn't exist in any of our documents. Instead of saying "I don't know," it made up a completely fake answer. Confidently. With a fake number of days!
👨🦳 Uncle: Welcome to the most common failure in RAG systems - and the exact reason Phase 4 exists. Tell me - in Phase 3, what did Top-K retrieval actually guarantee ?
👦 Nephew: It... returns the 5 closest vectors?
👨🦳 Uncle: Say that sentence again, slowly, and notice what it does NOT say.
👦 Nephew: It doesn't say those 5 are actually... relevant. Just that they're the closest of whatever exists.
👨🦳 Uncle: Exactly the gap. Even if your database has zero chunks about leave policy, Top-K will still confidently hand back 5 chunks - probably about something completely unrelated, like office timings or dress code - because "closest" is a relative ranking , not a quality guarantee . The LLM then does what LLMs do: it takes whatever context it's given and tries its best to answer from it, even when that context is garbage. That's not the LLM's fault. That's a retrieval design flaw. Today we fix it.
Phase 4 Overview: Four Key Concepts
Phase 3: Storage, Indexing & Token Economics ✅
↓
Phase 4: Retrieval Quality & Grounded Answers ← WE ARE HERE
│
├─ Step 1: Reranking
│ (Retrieve cheap & wide, then score properly)
│
├─ Step 2: Relevance Threshold / Abstention
│ (Knowing when to say "I don't know")
│
├─ Step 3: Grounded Prompt Assembly with Citations
│ (Every claim traceable to a source)
│
└─ Step 4: Hybrid Search as a Safety Net
(Catching what vector search misses)
↓
Phase 5: Evaluation & Guardrails at Scale
Step 1: Reranking - Retrieve Wide, Then Score Properly
Why "Closest" Isn't "Most Relevant"
👨🦳 Uncle: Let's go back to your resume example from Phase 2. Vector similarity is fast because it's approximate - that's the whole point of the ANN index we built in Phase 3. But "fast and approximate" means the ranking among your Top-20 candidates can genuinely be a bit sloppy at the edges.
Query: "What is the notice period for resignation?"
Vector search Top-5 (by cosine similarity):
- "Notice period starts from submission date" (0.89)
- "Employees must inform HR before leaving" (0.85)
- "Resignation letters must be signed by manager" (0.84)
- "Office holiday calendar for 2025" (0.81)← WRONG, but close enough to sneak in
- "Exit interview process overview" (0.79)
👦 Nephew: Wait - the holiday calendar chunk scored 0.81? That's completely unrelated!
👨🦳 Uncle: This happens more than beginners expect. Embedding models compress meaning into 1536 numbers - some unrelated chunks accidentally land "nearby" in that space due to shared vocabulary, formatting, or document structure, even when a human would instantly see they don't answer the question. Vector search is a wide net , not a precise judge .
The Fix: Two-Stage Retrieval
👨🦳 Uncle: The production pattern is always two stages, never one:
STAGE 1 - Retrieve (cheap, wide, approximate)
Vector search → Top-20 candidates
Fast (uses HNSW index from Phase 3)
Casts a wide net, some noise expected
STAGE 2 - Rerank (expensive, narrow, accurate)
A RERANKER model looks at the actual QUERY + each CANDIDATE CHUNK together, and scores relevance directly → Keep only Top-5 after reranking
Slower per-item, but far more accurate
👦 Nephew: What makes a reranker more accurate than the embedding similarity we already had?
👨🦳 Uncle: Here's the key architectural difference, and it's worth understanding properly:
Embedding model (bi-encoder):
Query→ [vector A]─┐
├─ compare AFTER, separately
Chunk→ [vector B]─┘
The model NEVER sees query and chunk together.
Reranker (cross-encoder):
[Query + Chunk] → fed into model TOGETHER → model directly outputs a relevance score
The model sees BOTH at once, so it can reason about how they actually relate - not just how "nearby" their independently-computed vectors happen to be.
👨🦳 Uncle: A cross-encoder is slower - you can't pre-compute anything, because it needs the query at scoring time - which is exactly why you never run it against millions of chunks directly. You run it only against the small Top-20 that vector search already narrowed down for you. Cheap net first, expensive judge second.
Code: Reranking with Cohere's Rerank API
npm install cohere-ai
const { CohereClient } = require ( " cohere-ai " );
const cohere = new CohereClient ({ token : process . env . COHERE_API_KEY });
async function rerankChunks ( query , candidates , topN = 5 ) {
// candidates = [{ chunk_text, metadata, similarity }, ...](Top-20 from vector search)
const response = await cohere . rerank ({
model : " rerank-english-v3.0 " ,
query : query ,
documents : candidates . map ( c => c . chunk_text ),
topN : topN ,
});
// response.results gives back indices INTO our original array,
// re-sorted by true relevance score
return response . results . map ( result => ({
... candidates [ result . index ],
rerank_score : result . relevanceScore ,
}));
}
// Usage:
// const top20 = await vectorSearch(queryEmbedding, { limit: 20 });
// const top5 = await rerankChunks(userQuestion, top20, 5);
👦 Nephew: Why 20 candidates and not, say, 5 straight from vector search?
👨🦳 Uncle: Because if the true best chunk was sitting at position 8 in the approximate vector ranking (remember - approximate!), and you only ever pulled Top-5, the reranker never even gets a chance to see it. Retrieving wider (Top-20 or Top-30) before reranking gives the accurate judge enough candidates to actually find the right answer, even when the fast-but-approximate first pass ranked it imperfectly.
Step 2: Relevance Threshold - Knowing When to Say "I Don't Know"
👨🦳 Uncle: Now the fix for your original problem - the fake leave policy answer. Reranking alone doesn't solve it, because even after reranking, the system will still confidently hand back its "Top-5 best of what exists" - even if none of them are actually good.
👦 Nephew: So we need... a minimum bar? Like, "if nothing scores above X, don't even bother answering"?
👨🦳 Uncle: Precisely. This is called a relevance threshold , and it's one of the highest-leverage, most-skipped guardrails in RAG systems.
const RELEVANCE_THRESHOLD = 0.5 ; // tune per your reranker's score distribution
async function retrieveWithAbstention ( query ) {
const top20 = await vectorSearch ( query , { limit : 20 });
const reranked = await rerankChunks ( query , top20 , 5 );
const bestScore = reranked [ 0 ]?. rerank_score ?? 0 ;
if ( bestScore < RELEVANCE_THRESHOLD ) {
return { hasRelevantContext : false , chunks : [] };
}
return { hasRelevantContext : true , chunks : reranked };
}
Then, at prompt-assembly time:
async function answerQuestion ( query ) {
const { hasRelevantContext , chunks } = await retrieveWithAbstention ( query );
if ( ! hasRelevantContext ) {
return " I don't have information about that in the available documents. Could you check with HR directly, or rephrase your question? " ;
}
// proceed to Step 3 - build a grounded prompt from `chunks`
}
👦 Nephew: How do I pick the actual threshold number? 0.5 feels arbitrary.
👨🦳 Uncle: It is somewhat empirical, and that's honest to say - you tune it against your own data. A practical way: run a batch of known "should answer" questions and known "should NOT be answerable" questions through your reranker, look at the score distributions for each group, and pick a threshold that sits in the gap between them. Revisit it periodically - as your document set grows, the distribution can shift.
Mental model - the bouncer at the door: Vector search invites 20 people who "sort of look like" they belong. The reranker checks IDs properly. The threshold is the bouncer's rule: "if literally nobody in this line actually qualifies, don't let anyone in - just close the door and say so." Without that bouncer, your system politely waves in whoever's closest to the front of the line, qualified or not.
Step 3: Grounded Prompt Assembly with Citations
The Problem with a Plain Context Dump
👨🦳 Uncle: Even with great chunks, how you hand them to the LLM matters. A lazy prompt looks like this:
Context: Employees receive 30 days notice period. Notice period starts from submission date. Manager approval required for resignation.
Question: What is the notice period?
👦 Nephew: What's wrong with that? It has the right chunks.
👨🦳 Uncle: Nothing technically wrong - but if the LLM's answer is later questioned ("where did you get 30 days from?"), you have no way to trace the claim back to a specific source. In a compliance-sensitive domain - HR policy, legal, medical, financial - "trust me" isn't good enough. You need traceability.
The Fix: Numbered, Attributable Context
function buildGroundedPrompt ( query , chunks ) {
const contextBlock = chunks . map (( c , i ) =>
`[ ${ i + 1 } ] (Source: ${ c . metadata . source_file } , ${ c . metadata . department } )\n ${ c . chunk_text } `
) . join ( " \n\n " );
return `You are answering questions using ONLY the numbered sources below.
${ contextBlock }
Instructions:
- Answer using ONLY the information in the sources above.
- After each claim, cite the source number in brackets, like [1].
- If the sources don't fully answer the question, say so explicitly - do not guess or use outside knowledge.
Question: ${ query }
Answer:`;
}
Example LLM output with this prompt:
"The notice period is 30 days, starting from the date of submission [1][2]. Resignation also requires manager approval before it is finalized [3]."
👦 Nephew: Now I can map [1] , [2] , [3] straight back to the actual source_file and department in metadata!
👨🦳 Uncle: Exactly - and this is where Phase 3's metadata design finally pays off in the user-facing product. You can render those citations as clickable links back to the original document, which is the difference between "an AI said so" and "here's exactly where this comes from, go verify it yourself."
Code: Parsing Citations Back to Sources
function attachCitationSources ( llmAnswer , chunks ) {
const citationPattern = / \[(\d + )\] /g ;
const usedIndices = new Set ();
let match ;
while (( match = citationPattern . exec ( llmAnswer )) !== null ) {
usedIndices . add ( parseInt ( match [ 1 ], 10 ) - 1 );
}
const sources = [... usedIndices ]. map ( i => ({
text : chunks [ i ]?. chunk_text ,
file : chunks [ i ]?. metadata ?. source_file ,
department : chunks [ i ]?. metadata ?. department ,
}));
return { answer : llmAnswer , sources };
}
Step 4: Hybrid Search - Catching What Vector Search Misses
The Blind Spot: Exact Terms
👨🦳 Uncle: One more honest weakness. Say someone asks: "What does error code ERR_4521 mean?"
👦 Nephew: Vector search should handle that fine, right? It's just text.
👨🦳 Uncle: Try it in your head using what you learned in Phase 2. Embedding models are trained to understand meaning and concepts - "leadership" relating to "team management." But "ERR_4521" isn't a concept - it's an exact, arbitrary identifier. The embedding model has no real semantic understanding of a specific error code string; it might embed it as "some kind of technical identifier" and miss the one chunk that specifically documents ERR_4521, especially if that chunk's surrounding language doesn't semantically resemble the question.
👦 Nephew: So the same weakness applies to product codes, order numbers, exact names...
👨🦳 Uncle: Exactly - anything where exact match matters more than meaning . This is precisely where old-fashioned keyword search still wins, and it's why production systems don't choose one or the other - they run both , and combine the results. That's hybrid search.
Architecture: Running Both Searches Together
User Query: "What does error code ERR_4521 mean?"
│
┌─────┴─────┐
↓ ↓
Vector Search Keyword Search (BM25 / full-text)
(semantic) (exact term matching)
↓ ↓
Top-10 Top-10
└─────┬─────┘
↓
Merge & Deduplicate
↓
Rerank the COMBINED set (Step 1)
↓
Final Top-5
Code: Postgres Full-Text Search Alongside pgvector
-- Add a full-text search column (once, at table setup time)
ALTER TABLE document_chunks ADD COLUMN chunk_tsv tsvector
GENERATED ALWAYS AS ( to_tsvector ( 'english' , chunk_text )) STORED ;
CREATE INDEX idx_chunk_tsv ON document_chunks USING GIN ( chunk_tsv );
-- Keyword search query
SELECT chunk_text , metadata ,
ts_rank ( chunk_tsv , plainto_tsquery ( 'english' , $ 1 )) AS keyword_score
FROM document_chunks
WHERE chunk_tsv @@ plainto_tsquery ( 'english' , $ 1 )
ORDER BY keyword_score DESC
LIMIT 10 ;
Node.js: Combining Both Result Sets
async function hybridSearch ( query , queryEmbedding ) {
const [ vectorResults , keywordResults ] = await Promise . all ([
db . query (
`SELECT chunk_text, metadata, 1 - (embedding <=> $1) AS score
FROM document_chunks
ORDER BY embedding <=> $1
LIMIT 10`,
[ queryEmbedding ]
),
db . query (
`SELECT chunk_text, metadata,
ts_rank(chunk_tsv, plainto_tsquery('english', $1)) AS score
FROM document_chunks
WHERE chunk_tsv @@ plainto_tsquery('english', $1)
ORDER BY score DESC
LIMIT 10`,
[ query ]
),
]);
// Merge, de-duplicating by chunk_text, keeping the highest score seen
const merged = new Map ();
for ( const row of [... vectorResults . rows , ... keywordResults . rows ]) {
const key = row . chunk_text ;
if ( ! merged . has ( key ) || merged . get ( key ). score < row . score ) {
merged . set ( key , row );
}
}
const combined = [... merged . values ()];
return rerankChunks ( query , combined , 5 ); // Step 1's reranker makes final sense of it
}
👦 Nephew: So hybrid search doesn't replace reranking - it feeds reranking a better, more complete set of candidates first?
👨🦳 Uncle: Exactly right. All four steps today form one pipeline , not four separate options: hybrid search casts the widest, smartest net (semantic and exact); reranking judges that combined set accurately; the relevance thresho
Comments
No comments yet. Start the discussion.