RAG or fine-tuning? How to tell which one your model needs

When the model inside your product gets something wrong, there are two usual causes. Either the model does not know something it needs, such as your pricing, your policies or last week's release. Or it knows enough and still gets the format, the tone or the call wrong.

Retrieval-augmented generation (RAG) fixes the first. Fine-tuning fixes the second. Originally published at overmindlab.ai.

The two work on different parts of the system. RAG changes what the model reads. Fine-tuning changes the model itself.

RAG is like an open-book exam, where the answer can only be as good as the page you turn to. Fine-tuning is like practice, which changes what you do without looking anything up.

RAG vs fine-tuning, side by side

When should you use RAG?

RAG adds a lookup step before the model answers. A retriever finds passages related to the request, often with embeddings (a way of matching text by meaning), and puts them in the prompt next to the question. The model answers from what it was handed and does not change.

RAG is the right tool in four situations.

• The facts change. Prices, policies and docs move faster than any training schedule. Update the document, re-index it, and the next answer uses it.
• The knowledge is private or too large to fit. The base model never saw your help center, and no prompt holds all of it. Retrieval picks the few passages that matter.
• Answers need a source. The model answers from specific passages, so you can show users where each answer came from.
• Access differs by user. You can filter retrieval by who is asking, whereas anything trained into the weights can surface for any user.

RAG has limits too. Answers are only as good as the passages the retriever returns, and every passage adds tokens and latency. Retrieving facts does little for behavior. If the model has the right facts and still gets the format wrong, more retrieval will not help.

When should you fine-tune?

Fine-tuning continues training a model on examples of the job, each pairing an input with the output you want. After enough of them, the model produces that output without being told.

It pays off in four situations.

• The task is narrow and repeats. Classification, extraction, routing and fixed-format replies are the classic cases.
• The model has the knowledge but not the behavior. It knows enough to answer, yet keeps breaking the output format or picking the wrong tool.
• You have real examples, ideally hundreds to a few thousand labeled runs from production traffic. How to turn traces into a training dataset shows how to build them.
• Cost or latency matters. A small fine-tuned model with a short prompt can replace a frontier model carrying a long one. Open-weight vs frontier models covers when that switch pays.

Fine-tuning is a poor way to add facts. Models learn new facts slowly this way, and the facts go stale the moment they change. Keep knowledge in retrieval and behavior in training.

Where does prompt engineering fit?

Prompt engineering changes the instructions. RAG changes the facts. Fine-tuning changes the model. The usual order follows the cost.

Start with the prompt, because it takes minutes and you can undo it instantly. Add RAG when failures trace back to missing knowledge. Fine-tune when failures are about behavior on a narrow task and better prompts have stopped improving your test scores.

Read the failures to tell the gaps apart. Quoting an old price is a knowledge gap. Formatting the right price wrongly is a behavior gap. Prompt engineering vs fine-tuning covers how to tell when prompting has hit its ceiling.

Can you combine RAG and fine-tuning?

Yes, and agents that need both should use both. The two fix different gaps, so they stack. A support agent might retrieve the customer's plan and the right help article, then hand both to a model trained on thousands of resolved tickets.

Train on what the model will see in production. If your agent answers from retrieved passages, keep those passages in the input of each training example. The model then learns to answer from context the way your agent's best runs did. This is the idea behind RAFT, short for retrieval-augmented fine-tuning.

Keep the split clean. Facts stay in retrieval, so a changed policy needs a document edit and no training run. Behavior lives in the weights, so the prompt can shrink to the request and its passages.

RAG, fine-tuning or both in five worked examples

What training adds when the document is already in the prompt

RAG decides what the model reads. Training changes how faithfully it reads. Overmind's benchmark, When bigger isn't better, measured that second part on three reading tasks. Each gives the model a document to answer from, the same step every RAG pipeline ends with.

The clearest case is contract review. Across 4,000+ questions on 100+ contracts, the model had to say whether a clause was in the contract and quote its exact wording. The benchmark scored the base model before and after training on real examples of the task, alongside a frontier flagship model.

Contract clause detection, 4,000+ questions across 100+ contracts (from When bigger isn't better)

Training took exact quotes from 4.2% to 59.6% and fabricated quotes from 3.36% to 0.12%. Retrieval played no part in that gain, because every model was given the same contract text.

The other two tests compared trained models with the frontier flagship, and the results pointed the same way. On biomedical papers (BioRED), 48.2% of the relationships the frontier flagship claimed were not there, more than 7x the trained model's 6.6%. On NASA aviation incident reports, it added a detail that was not in the report to more than 1 in 3 synopses (33.4%), against 13.4% for the smallest model Overmind trained.

Because these tests hand the model the document, they measure the reading step only. They say nothing about retrieval quality.

Fine-tuning a RAG agent with Overmind

Keep your retriever. Overmind trains the model that reads what it returns. The Overmind Python SDK can record each retrieval as its own span, so a trace shows what was fetched next to what the model answered. You build a dataset from those production traces, and every row links back to the trace it came from.

Then you fine-tune an open-weight base model with LoRA or full supervised fine-tuning. Before you switch traffic, benchmark the trained model with your evaluators on a separate eval set. By default Overmind compares the base model before training with the trained model after. Comparing against the model your agent runs today is opt-in.

The trained model is served through an OpenAI-compatible endpoint, and you can download its weights once the deployment's archive is ready. The training docs cover each step.

If you have an agent in production, its own past work is the training data, and your coding agent can set Overmind up for you. Open the repo in Claude Code, Cursor, Codex or OpenCode and paste Set me up on Overmind https://console.overmindlab.ai/setup.md into the chat. If you prefer the browser, start for free and copy the prompt from the console.

FAQ

What is the difference between RAG and fine-tuning?

RAG retrieves relevant documents when a request arrives and adds them to the prompt, so the model answers from current facts. Fine-tuning trains the model on examples, which changes its weights. RAG changes what the model knows in the moment. Fine-tuning changes how it behaves every time.

Should you try RAG, fine-tuning or prompt engineering first?

Prompt engineering, because it takes minutes and you can undo it instantly. Add RAG when failures come from missing or outdated knowledge. Fine-tune when failures are about behavior on a narrow, repeated task and prompt changes have stopped improving your test scores.

Is RAG cheaper than fine-tuning?

RAG is cheaper to start, because there is no training run. Fine-tuning can be cheaper to run, mostly because a fine-tuned model can be a smaller one. In Overmind's contract benchmark, where both models read the same contract, the small trained model cost about 20x less per question than the frontier flagship, before counting the one-off training run.

Does fine-tuning add knowledge to a model?

Only slowly and unreliably. A 2024 EMNLP paper found that models learn new facts from fine-tuning more slowly than facts they already know, and that their tendency to hallucinate rises as those new facts are learned. Its authors say this supports the view that models get most of their knowledge from pre-training, while fine-tuning mainly teaches them to use it.

What is RAFT?

RAFT, short for retrieval-augmented fine-tuning, is a training recipe from a 2024 UC Berkeley paper. Each training example pairs a question with retrieved documents and a step-by-step answer that quotes the relevant passage. Usually the relevant document sits among distractors, and in some examples only distractors are given. The model learns to find the right passage and ignore the rest.

Does RAG stop hallucinations?

No. The model can still misread the retrieved text or add details that are not in it. In Overmind's aviation benchmark, a frontier flagship given the incident report still added a detail the report did not contain to 33.4% of its synopses. Retrieval addresses missing knowledge. Training on real examples of the task improves how faithfully the model uses it.