Beyond Vector Search: Building Better RAG Retrieval with Hybrid Search and Reranking

The first two parts of this series covered why production RAG systems fail and how the quality of the data foundation directly affects everything that comes after it. We looked at document ingestion, parsing, chunking, and metadata design-the layers responsible for turning raw information into something a retrieval system can actually work with. But even perfectly processed documents are useless if retrieval cannot find the right information. In this third part, we'll move into the retrieval layer itself. We'll look at why vector search alone is often insufficient, how semantic and lexical search complement each other, and how reranking can turn a large set of possible matches into a small set of highly relevant documents. We'll also cover query optimization, metadata filtering, and context compression-key techniques for building retrieval pipelines that perform reliably on real-world queries. Production RAG Architecture Series ✅ Why Most RAG Systems Fail in Production: The Hidden Architecture Problems Behind AI Search ✅ Building a Production RAG Pipeline: Document Processing, Chunking, and Metadata Design Beyond Vector Search: Building Better RAG Retrieval with Hybrid Search and Reranking (you are here) Scaling RAG Systems: Production Architecture, Performance, and Cost Optimization Evaluating Production RAG Systems: Metrics, Monitoring, and Common Failure Patterns Chapter 7 - Embeddings Embeddings are not magic. They are coordinates. That is the whole trick. A piece of text goes in, a vector comes out, and now similar meanings sit close to each other in space. If chunking decides what the system sees, embeddings decide how it remembers it. That sounds abstract until you try to build retrieval on top of it. Then it becomes the center of the whole system. What an embedding really is Imagine a map. On that map: “dog” sits near “wolf”. “invoice” sits near “payment”. “upgrade” sits near “billing policy”. “password reset” sits somewhere else. The model is not understanding meaning the way a human does. It is learning a geometry where related things end up near each other. That geometry is what retrieval uses later. And that is why embeddings matter so much. If the geometry is good, retrieval feels smart. If the geometry is bad, everything downstream starts guessing. Why this is not enough This is where people usually make the first mistake. They think: “If I use a good embedding model, retrieval will work.” It won’t. A good embedding model can only work with the text you give it. If the chunk is messy, too broad, too short, or stuffed with unrelated ideas, the vector will still be messy. Just in a more expensive way. A bad chunk becomes a bad vector. A bad vector becomes a bad candidate. A bad candidate becomes a confident wrong answer. A concrete example Take these chunks: 1. Active invoices must be closed before upgrading. 2. Customers can upgrade from Professional to Enterprise. 3. How to reset your password. 4. Downgrading is allowed only if no active trials exist. A decent embedding model should understand that 1 and 2 belong near upgrade-related questions, while 3 is clearly off in another part of the world. That sounds obvious, but in real systems it gets messy fast. Because now you have: legal docs, support docs, product policies, release notes, tables, code snippets, and old versions of the same document all mixed together. At that point embeddings are not a detail anymore. They are the shape of the search space. What makes a good embedding model A good model for production should: understand your language, behave well on short queries, not collapse technical terms into generic similarity, and work on your actual domain, not just “general text.” A model that is decent for blog posts may be weak for: policy documents, multilingual corpora, technical manuals, product docs with version numbers, or support data full of exact identifiers. So the real question is not “which embedding model is popular?” The real question is “which model gives me the right geometry for my corpus?” Example code python from sentence_transformers import SentenceTransformer import numpy as np model = SentenceTransformer("all-MiniLM-L6-v2") chunks = [ "Active invoices must be closed before upgrading.", "Customers can upgrade from Professional to Enterprise.", "How to reset your password.", "Downgrading is allowed only if no active trials exist." ] vectors = model.encode(chunks, normalize_embeddings=True) def cosine(a, b): return float(np.dot(a, b)) query = "Can Enterprise customers upgrade directly from Professional while keeping active invoices?" query_vector = model.encode([query], normalize_embeddings=True)[0] ranked = [] for chunk, vector in zip(chunks, vectors): score = cosine(query_vector, vector) ranked.append((chunk, score)) ranked.sort(key=lambda x: x[1], reverse=True) for chunk, score in ranked: print(f"{score:.4f} | {chunk}") This is the smallest possible version of the idea. Query becomes a vector. Chunk becomes a vector. Similarity becomes a number. The number is not truth. It is only a signal. But in a good system, that signal is useful enough to move the right chunk to the top. Why chunk shape changes embedding quality A short query and a long chunk do not behave the same way. A query like: “Enterprise upgrade active invoices” is compact and vague. A chunk like: “Customers can upgrade from Professional to Enterprise. Active invoices must be closed before upgrading. Contact billing if invoices remain open.” contains multiple ideas. The embedding becomes a compressed summary of all of that. If the chunk contains too many unrelated ideas, the vector turns into an average of everything, which is another way of saying it gets blurrier. That is why embeddings and chunking are inseparable. You cannot fix one without thinking about the other. The domain problem General embeddings are often good enough to impress in demos. Production is where they start revealing their limits. A support system might need to understand: plan names, billing states, status codes, product tiers, policy phrases, internal jargon. A generic model may know the words, but not the importance of those words in your system. That is why evaluation on real queries matters. Not benchmark queries. Your queries. The real lesson Embeddings are not magic meaning detectors. They are a way to build a space where retrieval can do its job. If the space is designed well, the system can find the right things. If the space is noisy, the retriever will still return something plausible, and plausible is often the most dangerous kind of wrong. That is the entire game. Chapter 8 - Hybrid Search Vector search is good at meaning. Keyword search is good at precision. Production needs both. That is the whole chapter. If you only use embeddings, the system understands the idea of the query but can miss the exact phrase that actually matters. If you only use keywords, the system catches exact matches but misses the intent behind the question. Hybrid search exists because real users do both things at once. Why vector search is not enough Vector search is great when a person asks naturally. “How do I upgrade my plan?” That kind of question has room for interpretation. The model can infer the intent even if the wording is loose. But then the user asks something like: “Can Enterprise customers upgrade directly from the Professional plan while keeping active invoices?” Now exact words matter. Professional. Enterprise. active invoices. A vector model may understand the general billing theme, but it can still miss the exact policy sentence because the answer depends on precise terms, not just conceptual similarity. That is where pure semantic retrieval starts lying politely. Why keyword search is not enough Now flip the problem. A keyword system is brilliant when the query contains exact tokens. If the question includes: product names, version numbers, error codes, clause IDs, policy names, exact phrases, then BM25 or another lexical retriever often finds the right passage instantly. But if the user says: “Can a customer move to the top tier if they still owe money?” a pure keyword search may fail because the document says: “Active invoices must be closed before upgrading.” That is the same idea, but not the same wording. So keyword search is precise, but not smart. Vector search is smart, but not precise enough. The answer is both Hybrid search is not some fancy optimization. It is the basic admission that no single retrieval signal is enough. The flow usually looks like this: Query ↓ Vector Search ↓ Keyword Search ↓ Fuse Results ↓ Rerank ↓ Send to LLM The idea is simple: semantic retrieval finds the concept, keyword retrieval finds the exact phrase, fusion combines the strengths, reranking picks the best final candidates. A real example Take this query: “Can Enterprise customers upgrade directly from the Professional plan while keeping active invoices?” Vector search might return: billing policy chunks, plan upgrade chunks, invoice-related chunks. Keyword search might return: exact mention of “Professional”, exact mention of “Enterprise”, exact mention of “active invoices”. If you merge both lists, suddenly the system has a much better chance of building the full answer instead of just a vaguely related one. That is the difference between “sounds right” and “is right.” The fusion problem The tricky part is that vector scores and BM25 scores do not live on the same scale. You cannot just add them blindly and hope the universe respects your optimism. That is why production systems use score fusion methods like: weighted sum, rank-based fusion, Reciprocal Rank Fusion. The exact method matters less than the principle: do not force two different ranking systems to pretend they are the same thing. RRF in plain English Reciprocal Rank Fusion is popular because it rewards documents that rank well in both systems without caring too much about score scale. A simple version looks like this: def