RAG for developers who aren't AI engineers: what actually matters

Most non-AI developers have a mental model of RAG that is either wrong or dangerously incomplete. Not because they're bad engineers - because almost everything written about RAG is either a 10-minute framework tutorial that hides every real decision, or a research paper aimed at people who already do this for a living. There's very little in between. This article is the in-between.

I recently built a production RAG system over 62 ancient-history books (~46,000 chunks) and measured every design decision against a fixed test set - including the decisions that didn't work. I'm going to skip the framework marketing and tell you the small number of things that actually determine whether your RAG system is trustworthy, with the numbers to back them up. You don't need ML knowledge to follow this. If you can build a REST API and query Postgres, you can build everything described here.

The 60-second mental model

RAG (Retrieval-Augmented Generation) is just this: question → find relevant text in your corpus → paste it into the LLM prompt → "answer ONLY from this text". That's the whole idea. A search engine bolted onto an LLM, with instructions to answer from the search results instead of from its training data.

The only genuinely new component for a typical backend developer is embeddings. An embedding model turns text into a vector (an array of ~1,000 floats) where similar meaning produces nearby vectors. So "find relevant text" becomes nearest-neighbor search over vectors: embed all your document chunks once at ingest time, embed the user's question at query time, fetch the k closest chunks. If you know Elasticsearch: it's like that, but matching on meaning instead of keywords. "Why did the ship sink" will match a paragraph about "the vessel foundered in the storm" even though they share zero words.

Everything else - chunking, retrieval, prompting - is ordinary engineering. Which is exactly why regular developers can and should build these systems. What's missing from most tutorials isn't skill, it's knowing where the traps are.

The trap: naive RAG works in the demo and lies in production

Here's the part that shocks people. The standard tutorial setup - split documents into fixed 500-token chunks, embed with whatever the tutorial used, retrieve top-5, feed to the model - appears to work immediately. You ask a question, you get a fluent, confident, well-structured answer with a citation. Demo done, ship it, right?

I measured that exact baseline on my corpus with a 135-question test set where I knew which passages contained each answer. Result: recall@5 = 35.2%. Read that again. For roughly two out of three questions, the correct passage never reached the model at all. And here's the dangerous part: the model answered anyway. Fluently. With the same confident tone as when the retrieval worked.

This is the core thing non-AI developers get wrong about RAG. The failure mode is not "no results found" like a search engine, and not an exception like an API call. The failure mode is a confident, plausible fabrication that is indistinguishable from a correct answer unless you already know the answer. A system that's right 65% of the time and silently makes things up the other 35% is worse than useless - because you can't tell which answer you're looking at.

Nothing about the demo tells you this is happening. The only way to know is to measure. Which brings us to the single most important piece of advice in this article.

Before you improve anything: build a test set

This is the step every beginner tutorial skips, and it's worth more than any technique you'll ever add. Before touching chunk sizes, embedders, rerankers, or agent frameworks, do this:

• Write 30-50 real questions about your corpus - the kind your actual users will ask, not the kind your documents happen to answer neatly.
• For each question, record where the answer lives (which document, roughly which passage).
• Crucially, include 5-10 questions your corpus cannot answer. These are your hallucination traps. The correct behavior is a refusal; anything else is fabrication you'd otherwise never see.
• Measure two things separately:
  • Retrieval: is the right passage in the top-k results? (Pure code, no LLM judging needed - this is the number to optimize first.)
  • Generation: given the right passages, is the answer grounded in them? Does the system refuse when it should?

This costs one or two days and it changes everything, because RAG advice on the internet is wildly corpus-dependent. Things that transformed one system do nothing on another. Without your own test set, every blog post - including this one - is a coin flip. With it, you can check any technique against your data in an hour.

Concrete example of why this matters: the standard 2024-era advice says "hybrid search (keyword BM25 + vector) always beats pure vector search at scale." I believed it. I expected it to win on my corpus and pre-registered that prediction. When I actually ran it, hybrid retrieval was byte-identical to plain dense retrieval - same recall, category by category, not one result changed. I only know that because I measured. (The full write-up of that negative result is its own story.)

Your test set is not test infrastructure you write once and forget. It's the steering wheel.

What actually moved the needle (ranked)

With a test set in place, here's what mattered on my corpus, in order of impact.

1. The embedding model - the biggest single lever

Swapping the default embedder for a strong one (qwen3-embedding-8b, via a hosted API) took recall@5 from 35% to 53% - an 18-point jump from changing one line of configuration. The hardest-hit category improved by +41.7 points: questions phrased in modern English against text written in formal Victorian translation. The weak embedder simply couldn't bridge the vocabulary gap; the strong one could. The lesson: don't inherit the tutorial's embedder. It's the component doing the actual "understanding" in your retrieval, models differ enormously, and swapping it is trivial. Check the MTEB retrieval leaderboard, pick a few strong candidates, and measure them on your test set.

2. What you put inside a chunk

Chunking discussions usually obsess over token counts. The bigger win for me was what the chunk contains. A raw 500-token slice from the middle of chapter 12 loses all context - the embedder sees "he then marched north" with no idea who "he" is or which war this is. The fix: at ingest time, prepend context to each chunk before embedding it - the book title, the chapter heading path, a one-line note about what this section covers. On my worst-performing question category (questions requiring synthesis across a section), this transformed results: +18.2 recall points. Chunking is about meaning-per-chunk, not arithmetic.

3. Things that did NOT help (measure before adding)

Two techniques that every "advanced RAG" listicle recommends:

• Hybrid BM25 + vector search: as described above - zero change on my corpus. A strong embedder already found everything keyword search could find.
• Cross-encoder reranking: I measured five rerankers. Only the most expensive hosted one beat the no-reranker baseline at all, by ~1.7 points - for a permanent paid dependency on every query. Dropped.

The point is not "never use these." On your corpus they might win. The point is that both add cost, latency, and complexity, and the only way to know whether they pay for themselves is your test set. "Fashionable" is not evidence.

Making it refuse instead of hallucinate

Retrieval quality gets the right text in front of the model. The second half of trustworthiness is what the model does when the text isn't there - and this you have to engineer deliberately. Three parts:

• Prompt for grounded answering. Instruct the model to answer only from the provided sources, cite which source supports each claim, and explicitly refuse when the sources don't contain the answer. This is necessary but nowhere near sufficient.
• Test refusal like a feature. This is what the out-of-scope traps in your test set are for. My favorite trap type: a question about a named work the corpus doesn't contain but does mention. For example, my corpus includes a footnote about the poet Sappho but none of her actual poetry. Ask "what does Sappho's poetry express?" and a weak setup confidently answers from the footnote, sounding completely authoritative. The correct behavior is: "the corpus mentions this work but does not contain it," with the source of the mention. That distinction - between having information and having a reference to information - is exactly the kind of failure you'll never notice without deliberately testing for it.
• Know that model choice matters here too. In my measurements, honest-refusal ability varied dramatically between LLMs - swapping the generator model took out-of-scope refusal from 73% to 100% with the same prompt. Prompting alone had a ceiling; I iterated the prompt through five versions and watched edits slide along a fixed trade-off curve without moving it.

End state on my system, measured on the full test set: 0% false refusals (it answers every answerable question) and 96% honest refusals on the trap questions. Those two numbers together are what "reliable" means for RAG - and both are ordinary integration-test numbers you can put in a CI run, not vibes.

A concrete starting path for a TypeScript developer

You do not need an AI framework, a vector-database vendor, or Python to start. Here's the stack I'd hand a TS developer today:

• Postgres + pgvector - vector search inside the database you already run and know how to operate. At hundreds of thousands of chunks it's more than enough.
• A hosted embedding API - one HTTP call per chunk/query.
• Any decent LLM API for generation.
• ~200 lines of TypeScript before you even consider a framework. Writing the pipeline by hand first is the best way to actually understand it - frameworks make far more sense once you know what they're abstracting.

The five steps:

1. Ingest: split documents into chunks along natural boundaries (sections, paragraphs), prepending title/heading context to each chunk's text.
2. Embed and store: one API call per chunk, insert into a table with a vector column, add an index.
3. Retrieve: embed the incoming question, SELECT ... ORDER BY embedding <=> $1 LIMIT 10. That operator is a cosine-distance nearest-neighbor search - that's the whole "vector database."
4. Generate: build a prompt with the retrieved chunks (labeled with their sources), the grounding-and-refusal instructions, and the question.
5. Build the test set and iterate. Measure retrieval first. Fix retrieval before blaming the model. Re-run on every change.

Steps 1-4 are a weekend. Step 5 is what separates a demo from a system.

If you want a reference implementation of the full production version - the eval harness, the measured ablations including the failed ones, cost controls, prompt-injection defenses - the entire project this article is based on is open:

• Live demo: historian.loroplanner.com (try asking it something the corpus can't answer - that's the interesting part)
• Code + case study: github.com/LevRiabov/antic-historian

It's Python, but nothing about the architecture is Python-specific - the structure maps one-to-one onto the TypeScript stack above.

I build RAG and LLM-evaluation systems, and I'm available for contract work. If your team is trying to make an LLM answer reliably from your own data - or trying to figure out whether the one you built already can be trusted - my case study above shows how I approach it, with published numbers. Reach out: le*******@zohomail.eu.