RAG vs fine-tuning for business knowledge: a decision framework

The short answer

RAG changes what the model knows at the moment it answers. Fine-tuning changes how the model behaves. If your problem is "the model doesn't know our policies, products, contracts or customers", you need retrieval. If your problem is "the model knows enough, but answers in the wrong format, tone or structure, or a cheaper model can't follow the task", fine-tuning is a candidate.

Most business knowledge projects are the first kind. Your documents change, different users may see different documents, and people need to check where an answer came from. Retrieval handles all three; fine-tuning handles none of them.

Decision table: RAG vs fine-tuning

We score a use case against these questions. If most answers land in the RAG column, start there.

The last row is where fine-tuning genuinely shines. OpenAI's own optimisation guide frames fine-tuning as a way to supply more examples than fit in a prompt, use shorter prompts at scale, and train a smaller, cheaper, faster model to excel at a particular task. Those are behaviour and cost goals — not "teach the model our handbook".

The misconception: fine-tuning to teach facts

Fine-tuning on your documents feels like it should produce a model that "knows" them. In practice it is a poor way to load facts:

• No citations. The model cannot point to the passage it learned something from, so reviewers have nothing to check against.
• Stale by default. Every document change means another training run, evaluation and deployment.
• No access control. Anything in the training data can potentially surface for any user who asks the right question. In a multi-tenant product, that alone rules it out for customer data.
• Confident blending. A model that has partially absorbed two versions of a policy will answer fluently from both. Retrieval at least lets you show which version was used.

There is also a practical signal in the market: OpenAI's model optimisation guide states that it is winding down its fine-tuning platform and that it is no longer accessible to new users. Fine-tuning remains available elsewhere, but it is not where the major providers are steering knowledge use cases.

When to combine them

The two approaches are not exclusive. The pattern we recommend when both are justified is retrieve the facts, tune the behaviour:

• A support assistant retrieves the relevant help-centre articles (RAG) and uses a tuned or heavily prompted model to answer in your house style and escalation format.
• A document-processing pipeline retrieves the relevant contract clauses, then a small tuned classifier labels them against your risk taxonomy at a fraction of a large model's cost.
• A high-volume workflow starts on a large model with retrieval; once you have thousands of reviewed outputs, those become training data for a smaller model doing the same narrow task.

Notice the order. You almost always build retrieval first, because it produces the reviewed examples you would later need for fine-tuning anyway.

Where RAG actually fails (it is rarely the model)

Choosing RAG is the easy part. When a RAG system gives a wrong answer, the cause is usually that the right passage was never retrieved. These are the failure modes we see most:

1. Ingestion destroys structure

PDFs parsed into a flat stream of text lose their headings, tables lose their column headers, and scanned pages come out empty. On the EdTech platform our founder works on, plain text extraction was not good enough for long course documents, so ingestion uses a vision model to read pages, anchored to the document's table of contents, with deterministic settings and content-addressed deduplication so re-ingesting the same file does not create duplicates.

2. Chunks lose their context

A chunk that says "the fee is waived after 30 days" is useless if it no longer says which product or plan it belongs to. Hierarchical chunking (keeping the section path with each chunk) helps. Anthropic's contextual retrieval technique goes further, prepending a short, model-generated description of where each chunk sits in its document before embedding it.

3. Pure vector search misses exact terms

Embeddings are good at meaning and bad at identifiers: part numbers, error codes, course codes, names. Keyword search (BM25 or PostgreSQL full-text) is the opposite. We run both and merge the results with reciprocal rank fusion (RRF), which scores each document by summing 1 / (k + rank) across the result lists. It needs no training and, in Cormack, Clarke and Büttcher's original SIGIR 2009 paper, outperformed both individual rankers and the Condorcet method. In Anthropic's published experiment, adding contextual BM25 to contextual embeddings cut the top-20 retrieval failure rate by 49%, and adding reranking took the reduction to 67%.

4. Filters applied too late

If you retrieve the top 20 chunks and then remove the ones the user may not see, you can end up with three results, all irrelevant. Tenant and permission filters belong inside the retrieval query. pgvector makes this natural: vectors live in the same PostgreSQL tables as your permission data, so a WHERE clause enforces isolation in the same query that ranks similarity — which is how per-school isolation works in the system described above.

-- Hybrid retrieval, tenant-filtered inside the query (simplified).
WITH semantic AS (
  SELECT id, ROW_NUMBER() OVER (ORDER BY embedding <=> $1) AS rank
  FROM chunks
  WHERE tenant_id = $2
  ORDER BY embedding <=> $1
  LIMIT 50
),
keyword AS (
  SELECT id, ROW_NUMBER() OVER (ORDER BY ts_rank_cd(tsv, query) DESC) AS rank
  FROM chunks, plainto_tsquery('english', $3) AS query
  WHERE tenant_id = $2 AND tsv @@ query
  ORDER BY ts_rank_cd(tsv, query) DESC
  LIMIT 50
)
SELECT id, SUM(1.0 / (60 + rank)) AS rrf_score   -- k = 60, as in the RRF paper
FROM (SELECT * FROM semantic UNION ALL SELECT * FROM keyword) AS ranked
GROUP BY id
ORDER BY rrf_score DESC
LIMIT 20;

How to measure which approach is working

Whatever you choose, decide how you will know it is right before you build. We measure two things separately, because they fail for different reasons:

Start with 50 to 200 real questions from real users, with the documents that should answer them. If retrieval recall is low, no amount of prompt tuning or fine-tuning will fix the answers — improve parsing, chunking or search first. If recall is high but faithfulness is low, then prompt and model changes are worth exploring.

A five-step way to decide

1. Write down the failure. Missing facts, wrong permissions, or wrong behaviour? Missing facts point to retrieval; wrong behaviour may point to tuning.
2. Measure the size of the knowledge. Under a few hundred pages and stable? Try long-context prompting with caching first.
3. Build retrieval with an evaluation set. Hybrid search, metadata filters, citations, and the metrics above.
4. Fix the weakest stage. Ingestion, chunking, search or prompt — whichever loses the most answers.
5. Consider fine-tuning only for behaviour or cost, once you have enough reviewed examples to train on and a measured reason to expect improvement.

If you already have a RAG prototype whose answers are not good enough to ship, the same framework applies — we start by building the evaluation set and finding where answers are lost. See our RAG development service, or, if the knowledge needs to live inside an existing product, our AI integration work.