RAG vs Fine-Tuning: Choosing the Right Approach

Sooner or later, every team adding AI to a product asks: should we train the model on our own data? It sounds like the obvious move. Often it isn’t.

There are two different problems hiding inside that question. One is knowledge: the model doesn’t know your products, your policies or last week’s price change. The other is behaviour: the model knows enough, but it doesn’t answer in the format, tone or level of detail you need. Retrieval-augmented generation (RAG) is built for the first. Fine-tuning is built for the second. Mixing them up is an easy way to spend months on the wrong thing.

How RAG works

RAG leaves the model alone. When someone asks a question, your system finds the relevant pieces of your own content and hands them to the model with the question. The model’s job becomes reading and summarising, not remembering. A RAG system has two halves.

Preparing your content

First, your documents are split into chunks: passages small enough to be specific, but large enough to make sense on their own. Each chunk is turned into an embedding, a list of numbers that captures what the passage means, so that passages about similar things end up close together. Those embeddings go into a search index. That can be a dedicated vector database, or an extension to a database you already run, such as PostgreSQL.

Chunking sounds like plumbing, but it decides a lot. Split a table across two chunks, or cut a rule away from the heading that says who it applies to, and the right answer becomes impossible to find. Good chunking follows the structure of the document: its sections, headings, lists and tables.

Answering a question

When a question arrives, it’s turned into an embedding too, and the index returns the closest passages. Search by meaning alone can blur exact terms, such as a product code, a clause number or a surname. So many production systems use hybrid search: meaning-based (vector) search and classic keyword search together, often followed by a re-ranking step that puts the best passages first.

The chosen passages go into the prompt with clear instructions: answer only from these sources, say which ones you used, and say plainly when the answer isn’t there. This is called grounding, and it gives RAG three properties that matter in real products:

• Citations. You know exactly which passages the model was given, so you can show them next to the answer and a person can check it in one click. Models can still cite a passage that doesn’t say what they claim, so test for that too.
• Freshness. Change a document, re-index it, and the next answer reflects the change. No retraining.
• Access control. Each chunk can carry the permissions of the document it came from. Filter the search by who is asking, and the model never sees content that person isn’t allowed to read. Enforce this in the search, not in the prompt. A model can’t leak what it was never given.

The most common weakness: when a RAG system gives a bad answer, the usual cause is that search didn’t find the right passage, not that the model misread it. That’s good news, because search is ordinary engineering you can measure and improve.

Two other limits are worth knowing. Questions that need the whole collection, such as “how many contracts mention this clause?”, don’t suit a search that returns a handful of passages. And retrieved text is untrusted input: a document that contains instructions can try to steer the model, so treat it as data, not as commands.

How fine-tuning works

Fine-tuning takes a model that has already been trained and trains it a little more, on your examples. Each example is an input paired with the output you wish the model had given. Afterwards, the model leans towards producing outputs like yours. That makes fine-tuning good at changing behaviour:

• Format. Always producing a particular structure, such as a report layout or a set of extracted fields, without long instructions every time.
• Tone and style. Writing in your house voice, at the right reading level, and staying consistent over long outputs.
• Domain habits. Using your field’s terms the way your experts do, or sorting items into your own categories.

You often don’t need to retrain every weight in the model. Parameter-efficient methods such as LoRA (low-rank adaptation) freeze the original model and train a small set of extra weights alongside it. That makes training far cheaper, and it means you can keep several small adapters, one per task, on top of the same base model.

What it needs, and what it can’t do

The real cost of fine-tuning is the data. You need examples that show exactly the behaviour you want, and depending on the task that can mean hundreds of them or many more. If they are inconsistent, the model learns the inconsistency. Someone who knows the domain has to write or check them, and that work usually takes longer than the training itself.

What fine-tuning doesn’t do well is teach facts. A model may absorb some of what’s in your examples, but it can’t tell you where a fact came from, it can blend similar facts together, and when the facts change you have to prepare new data and train again. Anything in the training data can also surface for anyone who uses the model. Once it’s in the weights, there’s no way to say “this person may see that, and this one may not”.

One more practical cost: a fine-tune is tied to the base model it was built on. When a better base model comes out, or the old one is retired, you have to train and test it again. Keep your training data and evaluation set, and that’s far less work than the first time.

Side by side

Here is how the two compare on the questions that usually decide it.

The last row is the one to remember. Most of the trouble starts when a team uses one approach to solve the other’s problem.

Why prompts and retrieval come first

Before either, there’s a cheaper tool that’s easy to underestimate: the prompt. Clear instructions, a description of the output you want and a few worked examples fix many of the problems that make people reach for fine-tuning. Many model APIs can also enforce a structured output, where the response must match a schema you define. That alone removes a common reason to train a model.

So the order we recommend is:

1. Prompt design. Get the instructions, examples and output format right with a capable general model. A change takes minutes.
2. Retrieval. If the model needs your knowledge, add RAG. A change takes hours: re-chunk, re-index, adjust the search.
3. Fine-tuning. Only for problems that remain after the first two, and only once you can measure them. A change takes days or longer: new data, a new training run.

Each step is cheaper to change than the next, and the first two usually carry over when a stronger base model arrives: swap the model, rerun your evaluation set, and keep your prompts and index with small adjustments. It’s the same logic as any first version of a product: test your riskiest assumption the cheapest way first. More on that in How to Turn a Business Idea into an MVP.

When fine-tuning earns its place

Fine-tuning comes last in that order, but it isn’t a dead end. It’s a specific tool, and it tends to pay off when:

• You run a narrow, repetitive task at high volume, such as sorting incoming messages or pulling fields out of a standard document, and a smaller tuned model could do it faster and cheaper than a large general one. Your evaluation set will tell you if it does.
• You need a style or structure that prompting can’t keep consistent, especially over long outputs.
• Your prompts have grown long with examples you send on every call. Training those examples in can shorten every request. Check first whether your provider can cache a long, unchanging start to the prompt; that cuts the cost of resending it without any training.
• You need to run a model inside your own infrastructure, and a smaller open model needs help to reach the quality you need.

It’s usually the wrong answer when the request is:

• “So it knows our products.” That’s knowledge. Use retrieval.
• “So it stops making things up.” Fine-tuning can’t give the model facts it doesn’t have. Grounding in sources and testing are what reduce factual errors. A tuned model can even make confident errors in a more convincing voice.
• “So our data stays private.” Privacy depends on where the model runs and who processes your data, not on whether it has been tuned. Fine-tuning on a hosted service means sending your training data to that service.

Using both together

It isn’t either-or. You can use retrieval for facts and fine-tuning for behaviour. Picture a support assistant: retrieval supplies the current help articles for each question, and a fine-tune teaches the model your reply structure, your tone, and to say “I don’t have that information” when the passages don’t cover it.

There’s a lighter version too. Instead of tuning the model that writes the answers, you can tune the parts that do the finding, the embedding model or the re-ranker, on your own documents and questions, so that search understands your terminology. When retrieval is the weak link, that can help more than touching the answering model.

A sensible sequence is to launch with prompts and retrieval, collect real questions and reviewed answers in production (with your users’ consent and your privacy rules applied), and only then decide whether fine-tuning is worth it. By then you have real examples to train and test on.

Measure before you decide

Every choice above should be tested rather than argued. The tool for that is an evaluation set: real questions, each paired with the answer an expert would accept and the document where that answer lives. Start small, with questions your users actually ask, including awkward ones and ones your documents can’t answer. Then measure each part separately:

• Retrieval. Did the right passage come back near the top? If not, no model will save the answer.
• Answers. Is the answer correct, complete and supported by the sources it cites?
• Refusals. When the answer isn’t in your content, does the system say so instead of guessing?
• Permissions. Does anyone ever see a passage they shouldn’t? Test it with deliberately restricted documents.
• Cost and speed. What does each answer cost, and how long does it take under realistic load?

Run the whole set every time something changes: a prompt, the chunking, the base model, a fine-tune. Automated scoring, including using a model to grade answers, makes this quick, but check it against human judgement regularly. Without an evaluation set, every decision comes down to “it feels better”, and that’s how promising demos stall before they reach real users.

Where to start

A practical first pass looks like this:

1. Write down the questions people really ask, and where the answers live today.
2. Check the state of those documents. Are they current? Who owns them? Who is allowed to see which ones?
3. Build a small retrieval prototype with a capable general model and a careful prompt.
4. Turn your questions into an evaluation set, and measure retrieval and answers separately.
5. Fix retrieval first. Consider fine-tuning only for a gap you can see in your results.

Often, that path ends with a well-built RAG system and no fine-tuning at all. That’s a good outcome: it’s the simpler thing to run, change and explain.

If you’d like a second opinion on where your idea sits, our AI development page explains how we approach this kind of work, or you can tell us what you’re trying to build.