I Rebuilt My RAG Pipeline Without LangChain - What Got Better and What Got Worse

The first time I seriously doubted the framework was not because the model hallucinated. It was because the answer looked plausible, contained a citation, and was still wrong. The assistant had retrieved a chunk from a deprecated help page because one part of the pipeline applied a metadata filter, another part did not, and the final prompt assembly made the whole thing look coherent. Debugging it meant stepping through wrappers, runnable compositions, and framework-specific assumptions instead of asking the real question: why did retrieval favor the wrong document? That was the point where I stopped treating LangChain as the core of the RAG system and started treating it as an optional integration layer. This is not an anti-framework article. LangChain solved a real problem: it gave developers a fast way to compose LLM applications when the ecosystem was young and everyone was still figuring out the basics. But once RAG moved from demo to production, the problems changed. The hard parts stopped being “call the model” and became: - permission-aware retrieval - stable chunking - hybrid search - reranking - evaluation - document ingestion failures - embedding migrations - traceability when an answer goes wrong Rebuilding the pipeline without LangChain made some things dramatically better. It also made some things more annoying, more expensive, and more time-consuming than I expected. This article is about both. TL;DR If you are deciding whether to keep, adopt, or remove LangChain from a production RAG system: - Removing LangChain improved debugging, retrieval control, evaluation, observability, and cost discipline. - Removing LangChain made harder document loading, integration maintenance, and the long tail of “small” pipeline decisions. - The biggest win was not performance. It was that the pipeline became explicit. - The biggest downside was that I became responsible for a lot of boring glue code that frameworks usually hide. - My current rule: prototype with high-level tools, but own the retrieval core when the product depends on answer quality. 📋 Table of Contents - 1. The abstraction stopped being a shortcut and became a boundary - 2. Chunking stopped being “split by 800 characters” - 3. Retrieval became a small query planner - 4. Hybrid search was the unglamorous fix for exact identifiers - 5. Reranking became the highest-leverage quality gate - 6. Embedding generation became a data-engineering job - 7. Evaluation got easier once the pipeline had seams - 8. Observability changed from “the answer looks weird” to “chunk 7f2a was dropped” - 9. What got worse: the long tail of boring integration work - 10. Where I draw the line now 1. The abstraction stopped being a shortcut and became a boundary Scenario: A user asks, “What changed in webhook authentication?” The system retrieves something that mentions authentication, but not the correct product version. The final answer sounds confident. The problem is not the model. The problem is that the retrieval request did not carry the right filters, and the abstraction made that hard to see. Why it matters: In early RAG projects, abstractions help you move quickly. You connect a loader, a splitter, an embedding model, a vector store, and a prompt template. But in production, the interesting failures happen in the spaces between those components. When those spaces are hidden behind generic chain-like composition, you end up debugging the composition layer instead of the retrieval behavior. Solution: I rebuilt the pipeline around explicit stages with small interfaces. Not a huge framework. Just enough structure to make each stage testable. from dataclasses import dataclass from typing import Protocol @dataclass(frozen=True) class RetrievedChunk: chunk_id: str doc_id: str text: str score: float metadata: dict class Retriever(Protocol): def retrieve( self, query: str, *, filters: dict | None = None, limit: int = 20, ) -> list[RetrievedChunk]: ... The pipeline then became a sequence of ordinary functions: def answer_question(user_query: str, user_context: UserContext) -> FinalAnswer: plan = plan_query(user_query, user_context) candidates = retriever.retrieve( plan.retrieval_query, filters=plan.filters, limit=40, ) evidence = select_evidence(plan.raw_query, candidates) prompt = build_prompt(plan.raw_query, evidence) return generate(prompt, request_id=plan.request_id) Why this works: The important part is not that this code is “framework-free.” The important part is that the seams are visible. If retrieval is bad, I look at plan_query and retriever.retrieve . If the prompt is bad, I look at build_prompt . If the answer is unfaithful, I inspect evidence . There is no chain abstraction sitting between me and the failure. 💡 Practical note: If your LangChain usage already has clear boundaries around retrieval, parsing, and prompt construction, removing the framework may not help much. The problem is not the library itself. It is whether the library hides the decisions you now need to debug. 2. Chunking stopped being “split by 800 characters” Scenario: A support article contains a table of error codes. The user asks about one specific code. The retriever returns a chunk that includes the correct code, but not the header row explaining what the columns mean. The model guesses. Sometimes it guesses wrong. Why it matters: A lot of early RAG advice treated chunking as a text-length problem: Pick a chunk size, add overlap, repeat. That works for simple prose. It falls apart for real documents: - tables - code blocks - numbered steps - headings with nested context - FAQs - legal clauses - API reference docs - product changelogs In production, chunking is not a text problem. It is a document-structure problem. Solution: I stopped thinking of chunks as “pieces of text” and started treating them as evidence units. An evidence unit should carry enough context to be interpreted without its surrounding document. For Markdown-like documents, that usually means: - preserve heading hierarchy - keep tables intact when possible - attach column/header context to table rows - keep code blocks with their immediately preceding explanation - avoid splitting a numbered step away from its introductory sentence A simplified version of the chunk model: @dataclass(frozen=True) class Chunk: chunk_id: str doc_id: str heading_path: tuple[str, ...] text: str block_type: str # paragraph, table, code, list token_estimate: int source_url: str updated_at: str For tables, I do not only store the raw row. I store enough surrounding structure to make the row meaningful: row_text = ( "Error Code: E1042\n" "Meaning: Webhook signature expired\n" "Resolution: Regenerate signing key and replay event\n" "From table: Error reference / Webhooks / Common failures" ) Why this works: The model does not only need the right passage. It needs the right passage in a form where the meaning is self-contained. A chunk like: E1042 | Webhook signature expired | Regenerate signing key is much weaker than: Error Code: E1042 Meaning: Webhook signature expired Resolution: Regenerate signing key and replay event From table: Error reference / Webhooks / Common failures The second one is easier to retrieve, easier to rerank, and easier for the model to use faithfully. ⚠️ Gotcha: Overlap is not a substitute for context. Overlap helps at sentence boundaries, but it does not recover lost table headers, section titles, or document-level metadata. 3. Retrieval became a small query planner Scenario: A customer asks, “How do I rotate API keys?” The system retrieves a generic security page instead of the tenant-specific admin guide. Another user asks, “What changed in v2?” The retriever returns v1 and v3 documentation because the query does not carry version intent. Why it matters: A raw user query is rarely the best retrieval query. Users are terse. They use pronouns. They assume context. They mix product names, abbreviations, and incomplete descriptions. If you pass the raw query straight into a vector store, you are asking semantic search to solve problems that often belong to query planning. Solution: I introduced a lightweight planning stage before retrieval. It does three things: - rewrite the query for retrieval - extract filters - choose the retrieval mode @dataclass(frozen=True) class QueryPlan: request_id: str raw_query: str retrieval_query: str filters: dict mode: str # "hybrid", "keyword", "semantic" A simplified planner: def plan_query(raw_query: str, user_context: UserContext) -> QueryPlan: filters = { "tenant_id": user_context.tenant_id, "allowed_doc_types": user_context.allowed_doc_types, } if user_context.product_version: filters["product_version"] = user_context.product_version retrieval_query = rewrite_for_search(raw_query) mode = choose_retrieval_mode(raw_query) return QueryPlan( request_id=user_context.request_id, raw_query=raw_query, retrieval_query=retrieval_query, filters=filters, mode=mode, ) The important part is not the exact implementation. The important part is that retrieval now receives structured intent. For example: { "tenant_id": "acme", "product_version": "v2", "doc_type": ["admin_guide", "api_reference"] } That is much better than hoping the embedding model figures it out. Why this works: Most production RAG failures are not “the model is dumb.” They are retrieval-context failures. A query planner lets you separate: - what the user asked - what should be searched - what documents are allowed - whether exact matching matters more than semantic matching That separation becomes critical once you add multi-tenancy, permissions, or versioned documentation. 🔍 Why this matters: Permission filtering should be part of retrieval, not a post-processing idea. If you retrieve first and filter later, you often lose the best candidates to inaccessible documents. 4. Hybrid search was the unglamorous fix for exact identifiers Scenario: A user searches for SKU-8842 or error E1042 or endpoint /v2/webhooks/signatures . Vector search