RAG or fine-tuning: choose by the problem

RAG selects context at runtime. A model might receive passages from the applicable maintenance manual. When the manual changes, the application can update its search collection without retraining the model for that change. This still requires a functioning update process.

Fine-tuning uses examples to adapt model behavior. Google describes supervised fine-tuning as suitable for a defined task with available labelled data. Supported models and configurations are product-specific and need checking for a particular project. Google: supervised fine-tuning.

“RAG for knowledge, fine-tuning for behavior” is a useful shorthand, but it is incomplete. Training can also affect domain patterns, and RAG applications still need suitable instructions. The important question is which property can be changed in a controlled way and subsequently evaluated.

The following table compares RAG and fine-tuning on four points that the sections below explain in detail.

Task definition. Before adapting a model, inspect the task definition. Is the desired outcome unambiguous? Do subject specialists agree about a good answer? A model cannot reliably resolve contradictory expectations that remain unresolved in the reference examples themselves.

Prompt and format. Next inspect the prompt, output format and examples. If an application needs a specific schema, clear fields and structural validation belong in the baseline. Additional training is not the necessary first response to an occasionally missing field name.

Retrieval. Then inspect retrieval. Does the required information exist in the permitted collection and reach the model completely? A fine-tuned answer model cannot repair a missing document page or an incorrectly filtered source.

The following table maps observed failures to what to investigate first and a possible next step.

This is a diagnostic aid, not a universal performance ranking. Several causes can coexist. Address them separately where possible so a later comparison can show which change helped.

A fictional company regularly updates product instructions and responsibilities. Employees need answers tied to the applicable version and a usable source link. A maintained retrieval path is the natural first option to investigate.

Training changes exclusively into a model leaves important questions unresolved. How is one incorrect statement withdrawn? How is a historical version selected? How are different employees' reading rights respected? Retrieval and authorization systems can model these requirements explicitly.

RAG is not an automatic solution either. Source ownership, effective dates, deletion and test questions still need definition. For current transaction status, a direct query to the authoritative application may be more appropriate than a periodically updated document index.

Another synthetic example concerns service requests classified into clearly defined categories. Categories change infrequently, incoming wording varies widely and specialists can provide consistent reference decisions.

Begin with rules, a clear prompt and a suitable base model. Collect the remaining systematic failures. If enough consistent examples exist and an improvement would matter operationally, a fine-tuning experiment may be justified.

Include difficult boundary cases. Training only on obvious examples can leave a model overconfident on ambiguous inputs. An “insufficient information” outcome or human review route may matter more than a marginal improvement in an aggregate score.

Check category balance as well. A model that performs well on the dominant category may still fail on the rare requests that require urgent specialist attention. Report those cases separately rather than allowing overall volume to hide their errors.

A combined application might retrieve approved technical information and summarize it in a defined work format. RAG supplies current evidence; an adapted model might make a recurring domain-specific response pattern more consistent.

Build that combination in response to demonstrated need. First evaluate the unchanged model with the same sources and tasks. A training experiment becomes more targeted when remaining failures mainly concern processing the correct context.

The original RAG research already combined trained components with external information stores. Retrieval and training are not logically incompatible alternatives. Those historical research results do not establish the superiority of a present-day combined implementation. Lewis et al.: RAG.

Representative examples. A large collection of documents is not sufficient by itself. Fine-tuning examples must represent the intended task and correct outcome. Poor, contradictory or obsolete references can reinforce unwanted behavior.

Separate sets. Separate training, development and evaluation sets. Near-duplicate examples shared across training and testing can create misleadingly strong results. For time-dependent work, a later-period evaluation can help distinguish generalization from recognizing familiar wording.

Provenance and permitted use. Check data provenance and permitted use. Confidential material does not belong in training merely because the organization possesses it. For personal or protected information, assess the actual processing conditions and purpose. Calling the result “our own model” does not answer those questions.

Reference labels. Document how reference labels were produced. Examples agreed by qualified reviewers differ from labels generated automatically without review. If automatic labels are used, estimate their error through an appropriate sample before treating them as reliable training targets.

Same conditions. Keep questions, sources, permissions and scoring consistent. For example, compare a well-prompted base model with an adapted model given the same information. If one variant also receives cleaner documents, the comparison does not measure fine-tuning alone.

Separate criteria. Assess factual correctness, format, exceptions, unsupported claims and rework separately. A shorter answer may be cheaper but less useful. A perfectly structured output can still contain the wrong decision.

Untouched evaluation set. Reserve an untouched evaluation set for the final comparison. Repeatedly optimizing against the same cases gradually adapts the solution to that collection. Its scores then say less about future work questions.

Recorded configuration. Record the deployed configuration as well as the model name. Retrieval settings, prompt versions and generation parameters can change outcomes. Without those details, a later team cannot tell whether a difference reflects the model or another part of the application.

Adapting a model on confidential examples does not create user-specific disclosure control. The application still needs to determine which information and tools a person may use. In RAG, that boundary should operate before constructing answer context.

Withdrawing information also differs. A search record can be blocked or replaced if its derivatives are handled. Do not assume that information incorporated into model adaptation can be removed with an equally targeted operation.

Sensitive projects therefore need a documented lifecycle covering training data, model versions, retention, access and responses to withdrawal or errors. These responsibilities can materially change the scope of an apparently small training experiment.

Work on both sides. RAG creates work around source cleanup, ingestion, retrieval, context processing and maintenance. Fine-tuning introduces or substitutes work for example collection, annotation, training, evaluation and serving the adapted variant. Comparing the price of one model call misses much of this effort.

Change frequency. Include change frequency. Maintaining sources may be easier to plan when policies change regularly. For a stable task performed at high volume, a validated adaptation could be economically attractive. Calculate this with actual workload and measured outcomes.

Portability. A provider switch or base-model retirement may require renewed testing. Check whether training examples can be exported, which artifacts you control and how the application moves to another model version. Portability is something to verify rather than an automatic property of either approach.

Cost of failure. Also compare the cost of failure. A low-cost answer that requires expert correction may be more expensive than a slightly slower reliable workflow. Record human review and incident handling alongside infrastructure charges when evaluating the business case.

“We have many documents, so we need fine-tuning” confuses data volume with task type. A large, frequently changing collection may instead call for managed retrieval. The relevant distinction is missing access versus poor behavior when correct context is already available.

“RAG always requires another vector database” is also too broad. Information can come from existing search, direct queries or combined retrieval methods. Add a database when it demonstrably meets a requirement better, and choose architecture around the sources and the team's operating capabilities.

“An adapted model is automatically cheaper” omits fixed effort. Shorter prompts may reduce serving cost, while data preparation, training and evaluation add work. At low request volume, that effort can outweigh the saving. At high volume, confirm that difficult cases retain acceptable quality.

Turn each assumption into an observable question. Which real task needs a current source? Which systematic failure persists with the right information? Which cost actually changes? This turns a debate about methods into a manageable experiment with a clear decision. Record the conditions that would trigger reconsideration, such as higher volume, a new category of source or changed access requirements.

If a good prompt, clean sources and simple retrieval already meet the task, additional adaptation needs justification. A more complex solution should demonstrate fewer errors, less rework, better latency or lower total cost.

Choose the next step from the remaining failure pattern. Missing information points to source work, poor candidates to retrieval optimization and systematic processing errors to a controlled model comparison. The RAG acceptance plan helps ground that decision in inspectable cases rather than broad claims about either technique.

Keep the decision tied to the evaluated tasks and operating conditions.

These modernagents guides cover related decisions about AI knowledge assistants:

• AI knowledge assistants: sources, access and answers
• RAG explained: answers grounded in company knowledge
• Microsoft 365 Copilot or an additional RAG assistant?
• RAG costs: budget for setup, models and knowledge maintenance

For a workflow review, start with the task, systems involved and typical exceptions. Discuss it with modernagents.