Fine-tuning or RAG
Most companies want retrieval
and are being sold fine-tuning
Somebody has told you that you need a model trained on your data. Usually what you want is a model that can read your data. The two are different builds.
Usually retrieval, which means the model reads your own documents when it answers instead of being changed by them. Fine-tuning means altering the model itself using your material. Most businesses are sold the second and want the first.
The words get used loosely, so here they are plainly. A model is the thing that understands language and writes an answer. Retrieval, often called RAG, leaves the model as it is and hands it the right pages out of your own files at the moment somebody asks a question. Fine-tuning takes your material and adjusts the model's own workings, so that it has absorbed a style or a narrow task.
Retrieval reads, fine-tuning absorbs
Retrieval hands the model your pages when the question is asked. Fine-tuning changes the model so that it writes or sorts the way your material does. They solve different problems.
Facts belong in retrieval
If the answer is in a document, retrieval will find it and can tell you which document it came from. A fine-tuned model cannot cite the page it learned something on.
A changed document breaks training
Your prices, your sites and your policies change. Retrieval picks up the new version the moment the file is replaced. A fine-tuned model knows what it was trained on until it is trained again.
Fine-tuning needs examples, not files
Owning twenty years of documents is not enough. Fine-tuning needs many examples of the exact job done correctly, in pairs, and somebody has to make them. That is the real cost.
You can do both
A fine-tuned model that is also given your documents at the moment of the question is right for some narrow internal jobs. It is rarely the place to start.
What retrieval actually is
Somebody asks a question. Before the model answers, the system searches your own files and records, pulls out the few passages that look relevant, and puts them in front of the model along with the question. The model reads them and answers from them. Your documents are never built into the model. They stay where they already sit, and they are read when they are needed.
This is why a retrieval system can tell you which record an answer came from, and why correcting it is a matter of correcting the document. It is also why it keeps up with a business that changes, which is all of them.
What fine-tuning actually is
Fine-tuning takes a model that already understands language and adjusts its internal workings using examples of the job you want done. The result is a model that has absorbed a way of working. It writes in your house style, or it sorts an enquiry the way your senior person would, without being told how each time.
What it does not do is give a model knowledge of your business. It picks up patterns, manner and judgement from the examples. It does not reliably remember facts from them, and it cannot tell you where it got one.
Reading against absorbing, in one table
The honest comparison, in the order the questions usually come up.
| Question | Retrieval | Fine-tuning |
|---|---|---|
| What changes | Nothing in the model. Your documents get searched when a question is asked | The model itself, permanently, using the material you gave it |
| What it is good at | Answering from facts in your records, and naming the record | Style, tone, judgement and one narrow repeated task |
| What it needs from you | Your files, somewhere readable, and permission to read them | Many worked examples of the job done correctly, in pairs |
| A document changes tomorrow | Replace the file and the answers change with it | The model is unchanged until it is trained again |
| Can it say where the answer came from | Yes, it names the record | No |
| The machine it needs | A small machine with no graphics card is usually enough | A bigger machine, with a graphics card |
| Where it usually goes wrong | Badly organised files, so the wrong passage is retrieved | Too few examples, or examples that disagree with each other |
Why retrieval is usually the answer
What a business almost always wants is for somebody to be able to ask a question and get the right answer out of material the company already owns. Where is that drawing. What did we quote this client last time. What does our leave policy say. What did the site report say on the day it rained. All of those are facts sitting in a file, and retrieval is the thing that finds them.
It also fails safely. If the right document is not there you find out, because the system says which records it looked at. A fine-tuned model asked a factual question it was never taught will answer anyway, fluently, and you will have no way of knowing.
When fine-tuning is the right call
There are real cases, and they are narrower than they are sold as. Fine-tuning earns its place when the job is one repeated task, the correct answer has a shape rather than a source, and you have or can make a large set of examples of it done properly.
This is corporate work more often than small-business work, and it belongs on your own hardware when the material is sensitive. No client has bought fine-tuning from me yet, so ask how I would do it rather than for an example. That is a separate service.
- Sorting or routing the same kind of document every day, your way
- Writing in a house style that is distinctive and that you can show many examples of
- Pulling the same fields out of the same kind of paperwork, at volume
- A narrow internal task where a smaller trained model runs more cheaply than a large general one
- Work that must run on your own machine, where a smaller model has to punch above its size
What each one needs from you
Cleaning up the records is the part owners underestimate. A system can only read what exists. If quotations live in a folder of photographs and attendance is on a register, that tidying is real work and it goes into the quote. How the pricing works.
Neither of them is a matter of handing over a folder and waiting. This is what each one actually needs from your side.
- Retrieval needs your files to be findable: in folders with names, rather than photographs of paper in a chat thread
- Retrieval needs a decision about what the system may read, which is a list you write
- Retrieval needs somebody to say which document wins when two of them disagree
- Fine-tuning needs many examples of the task done correctly, and somebody senior to approve them
- Fine-tuning needs those examples to agree with each other, which is harder than it sounds
- Both of them need somewhere to run, which is the hardware question
What I would build, and in what order
I start with retrieval on the records you already keep, running on a machine in your own office, for one team. It is the smaller build, it is the one that answers the questions people actually ask, and a normal week of use shows you whether the records are good enough to build more on.
If a narrow repeated job then turns out to need a model that has absorbed your way of doing it, that is a second project with its own quote. By then you will have the examples to train it with, because the first system collected them. See what sits in the room.
Built, not described
Questions people ask
Fine-tuning or RAG: which do we need?
Retrieval, in most cases. If the answers you want are facts sitting in your own documents and records, retrieval finds them and tells you which record it used. Fine-tuning is for one narrow repeated task where the job has a shape, and it needs many worked examples from you.
Can we have an Agent trained on our own documents?
You can have one that reads your own documents, which is what people usually mean. The documents stay where they are and the system searches them when a question is asked. Training the model on them is a different build, and for finding facts it is the weaker of the two.
How much data do you need to fine-tune a model?
Not documents. Examples, in pairs of a request and the correct output, and a great many of them even for a narrow task. Owning twenty years of files is not the same as having examples. Making them is usually the largest part of the work.
Can you fine-tune without a graphics card?
Not practically, for anything useful. Training needs a machine with a graphics card. Retrieval does not, which is one more reason it suits a small office. Hardware is covered here.
Will a fine-tuned model know our prices and policies?
Not reliably, and it will not know when they change. Facts belong in retrieval, where replacing the document changes the answer. A fine-tuned model answers from what it absorbed during training, and it cannot name the page it came from.
Does our data leave the building for either of these?
It does not have to. Retrieval and fine-tuning can both run on a machine in your own office, with an open model on it. That is how I would build either of them. No client has bought fine-tuning from me yet. Nothing is sent to an outside AI service unless you ask for it.
Can we do both?
Yes, and for some narrow internal jobs that is the right answer: a model trained on your way of doing one task, and given your current documents at the moment of the question. It is rarely where I would start, because retrieval alone answers most of what gets asked.
Next step