RAG or fine-tuning? Start with the failure, not the technology label

Identify which layer is failing

A procurement assistant explains contract payment terms and returns purchase-request fields. Sometimes it misses the relevant clause; sometimes it receives the clause but omits a condition; sometimes the content is right but the application cannot consume its format. Users experience all three as bad answers, but they require different investigations.

Our proposed failure record distinguishes evidence access, evidence use and output behavior. Review real examples before selecting a technology. A single average score can conceal a training effort spent compensating for documents that never reached the model.

Understand the different levers

RAG supplies selected information at answer time; fine-tuning adapts model parameters through training data. AWS's comparison treats them as combinable and suggests retrieval as a starting point for questions over private documents. That is not a universal guarantee that every RAG system outperforms every alternative.

Retrieve the current contract from its authoritative source. First test application schema validation for request formatting. If repeatable behavioral failures remain with sufficient evidence and clear instructions, evaluate training separately rather than changing several layers at once.

Run an oracle-context diagnostic

Give the same model a complete, correct clause for a previously failed question. Improvement points toward document preparation or retrieval as priorities; persistent failure calls for examining instructions, task definition and model capability. This diagnostic suggests causes rather than proving a single explanation.

Record who selected the evidence and whether the experiment supplies information unavailable in real operation. Oracle context measures the answer stage's potential, not end-to-end performance. Final acceptance still requires the real retrieval, permission and document-version path.

For more on this topic, read Connecting AI answers to organizational documents in the Liyan knowledge base.

Prepare task evidence, not just raw files

Google Cloud's tuning documentation describes task-specific training examples. In this scenario, raw contracts do not replace explicit input/output examples and a consistent labeling policy. Examples should demonstrate the intended behavior, not merely repeat company vocabulary.

Separate evaluation data before preparing training examples and keep near-duplicate contract versions together. Include absent evidence, conflicting conditions and unauthorized requests. Resolve reviewer disagreement about the expected answer before asking training to reproduce it.

Evaluate a combined approach explicitly

RAFT studies training for domain-specific retrieval-augmented answering, including irrelevant retrieved documents. It illustrates that training and retrieval need not be competitors. Results on its experimental datasets do not guarantee the same gain on Persian procurement contracts.

Compare four controlled configurations: base model with reference evidence, base model with actual retrieval, adapted model with reference evidence, and adapted model with actual retrieval. This helps separate behavior gains from data-path changes. Record latency and operating cost with quality.

Keep freshness and authority outside assumptions

A contract can change today and become restricted tomorrow. Deleting a source row does not establish that a trained model has forgotten its contents. Define permitted training data, retention and deletion handling before creating the dataset. Those are organizational decisions, not benchmark outcomes.

A RAG source also does not automatically establish correct citation. Check valid versions and whether each claim is supported. Repeat document replacement and access-revocation tests in either architecture; a compliant-sounding model and a fresh index are separate parts of the trust chain.

Measure the whole lifecycle

Track collection, labeling, specialist review, evaluation, training, indexing and version maintenance. Token cost and response latency matter, but they are not the entire system cost. A frequently changing task also incurs repeated validation work.

Rehearse change: add a new contract, invalidate a training example and repair a consequential failure. Measure the human effort and time needed to restore acceptance. A cheaper static demo is not sufficient evidence of a cheaper maintainable system.

Keep the decision reviewable

Document the target failure, baseline, evaluation set, intervention and acceptance rule. Report amount extraction and approval-condition preservation separately from writing style. Choose criteria before comparing alternatives rather than selecting a convenient score afterward.

A useful outcome may be a bounded use case or human review rather than deployment of the most complex architecture. There is no permanent winner. Choose the approach that reduces the identified error within the organization's data and authority boundaries and leaves evidence that technical and business owners can examine.