AI uses for records
Context versus training data pilot: retrieve it, train on it, or both
By SourceX Editorial · Reviewed by Noah Loul ·
10 sections
Short answer
A context versus training data pilot gives one candidate dataset to the same base model two ways, as retrieved context and as fine-tuning data, plus both together and neither, under one budget, one grader and one sealed test. Split the test into lookup and behavior tasks first. If retrieval captures the gain, license for retrieval; if only training moves behavior tasks, license for training.
Retrieval
Context versus training data
Key takeaways
- A with-and-without test adds two different effects together: the dataset as knowledge the model reads and the dataset as behavior the model learns.
- A two-by-two design (base, context, training, both) plus two placebo arms separates the routes and shows whether they interact.
- Label each test item as lookup or behavior before any arm runs, by recording which single record, if any, holds the answer.
- Hold the model snapshot, budget rule, retrieval stack, training recipe, graders and repeat counts fixed, and analyze paired per-item differences.
- Each pattern of contrasts points to a license scope: retrieval, training, both or none; the pilot shows which rights a buyer needs, not that they exist.
What does a context versus training data pilot decide?
It decides how a buyer should use a candidate dataset: as context the model reads at inference time, as training data that changes the model's weights, as both, or not at all. That choice sets the license scope, the cost profile and whether the supplier must keep delivering updates, so it deserves a controlled test rather than a rule of thumb.
The two routes are different purchases. Retrieval keeps the records outside the model, where they can be refreshed, permissioned and cited, and it needs rights to store, index and show excerpts. Fine-tuning turns the records into weights, which needs training rights and a plan for the model if the license ends. A pilot that only asks whether the dataset helps leaves both questions open.
Useful evidence has three parts: an effect for each route measured against the same baseline on the same sealed test, that effect split by task type, and an interval from repeated runs showing whether the difference is larger than noise. The design below produces all three, along with the rules to write before any result is seen and a result table that maps each pattern to a license decision.
Knowledge or behavior: why a single with-and-without test hides the answer
A dataset can help a model in two different ways, and a plain with-and-without comparison adds them together. It can supply knowledge the model lacks, such as a clause in a supplier contract or the tolerance on a part number, or it can teach behavior, such as how an adjuster structures a denial letter or which queue a ticket belongs in.
OpenAI's guide to optimizing LLM accuracy draws the same line: context optimization covers what the model needs to know, including proprietary or out-of-date information, while LLM optimization covers how the model needs to act. The 2020 paper that named retrieval-augmented generation made the architectural version of the split, pairing a pretrained generator as parametric memory with a dense vector index of Wikipedia as non-parametric memory.
Published results give a working hypothesis, not an answer for your dataset. Ovadia and colleagues found in 2023 that retrieval consistently beat unsupervised fine-tuning at adding both familiar and new facts to 7B-class open models, and that adding fine-tuning on top of retrieval did not beat retrieval alone in their setup. Gekhman and colleagues reported in 2024 that fine-tuning examples carrying facts new to the model were learned more slowly and, once learned, raised its tendency to hallucinate. Neither study used your records, your task or this year's models, which is why the pilot exists.
Four arms and two placebos: one dataset delivered two ways
Give the same candidate dataset to the same base model through each route, alone and together, and compare every arm with a base arm that never sees it. This is a two-by-two factorial design: one factor is context (the dataset retrieved at inference or not), the other is training (the model fine-tuned on the dataset or not).
The factorial matters because the routes can interact. RAFT, a 2024 recipe, fine-tunes a model on questions paired with retrieved documents, including distractors it should learn to ignore, so that training teaches the model to use context better. If that happens with your records, the both arm beats the sum of the separate gains, and a buyer who tested only one route would undervalue them.
Two placebo arms keep the comparison honest. The context placebo retrieves from an off-task corpus of the same size, showing how much of the context gain comes from longer prompts alone. The training placebo fine-tunes on off-task examples in the same format, showing how much of the training gain is format learning. The arms and the contrast each one anchors are below:
| Arm | What the model gets | Contrast it anchors | Read it as |
|---|---|---|---|
| B: base | Current prompt and tools, no candidate data | Reference for every contrast | What you already have |
| C: context | Base model plus retrieval over the candidate dataset | C minus B | Value of the records as knowledge |
| T: training | Model fine-tuned on the candidate dataset, no retrieval of it | T minus B | Value of the records as behavior |
| CT: both | Fine-tuned model plus retrieval over the same dataset | CT minus C, and CT minus T | Whether the second route adds anything |
| Cp: context placebo | Retrieval over an off-task corpus of equal size | C minus Cp | Gain from these records, not from longer prompts |
| Tp: training placebo | Fine-tuning on off-task examples in the same format | T minus Tp | Gain from these records, not from format learning |
Lookup tasks and behavior tasks: split the test before the first run
Label every test item as a lookup task or a behavior task before any arm runs, because the two routes are expected to win different halves. A lookup task is answered by a fact that sits in one identifiable record of the candidate dataset; a behavior task needs a procedure, a format or a judgment that no single record states.
Make the label mechanical. For each item, a reviewer who has seen no arm's output records the identifier of the record that contains the answer, or writes none. Items with an identifier are lookup tasks, items without one are behavior tasks, and items that need both a fact and a procedure form a mixed stratum reported on its own rather than forced into either.
Then remove leakage. Search the candidate dataset for each item's answer text and its near duplicates; an item whose answer appears verbatim in a record rewards retrieval for copying, not for knowing. Drop those items or move them to a separate copy-check stratum, and keep the label, the record identifier and the deduplication result in the sealed test file so the strata cannot be redrawn after results arrive.
Write the expected pattern down too. On the published evidence, context should lead on lookup tasks and training on behavior tasks. A pilot that confirms the expectation still sizes each effect on your own records, and one that contradicts it is more valuable still, provided the rules below were fixed first.
The fixed conditions: what stays the same across every arm
Everything except the route the dataset takes must be held constant and written down before the first run. Change the base model snapshot, the retriever or the grader between arms and the comparison measures that change instead.
Budget is the hardest setting to equalize, because the routes spend in different places. Retrieval adds tokens and infrastructure to every call; fine-tuning front-loads training compute and adds a model to maintain. Fix a budget rule in advance, such as total cost at the planned call volume over twelve months, and record each arm's spend against it. Anthropic's contextual retrieval guidance adds a check for the context arm: a knowledge base under about 200,000 tokens, roughly 500 pages, can go into the prompt whole, so try that as the strongest form of context before building retrieval.
Repetition belongs in the protocol. Anthropic's guide to agent evaluations treats each attempt at a task as a trial and runs several, because model outputs vary between runs, and fine-tuning adds variance of its own, so train the T and CT arms more than once. Analyze per item: Evan Miller's 2024 paper on error bars for evals recommends comparing models through question-level paired differences and clustering standard errors when items come in related groups. The protocol fields are below:
| Field | Fixed setting | Why it is fixed |
|---|---|---|
| Base model | One named snapshot for every arm, recorded with its date | A model update mid-pilot changes every arm at once |
| Budget rule | Total cost at planned volume over 12 months, set before the first run | Retrieval and training spend money in different places |
| Retrieval stack | Same retriever, chunking and passage count for C, CT and Cp | Otherwise retrieval quality, not the records, decides |
| Training recipe | Same method and settings for T, CT and Tp, tuned on a development split only | Tuning against the test inflates the trained arms |
| Sealed test | Stratified, deduplicated against the dataset, read once at the end | Leakage rewards copying |
| Graders | Same code checks and rubric for all arms, blind to arm labels | Grader drift looks like an effect |
| Repeats | At least three inference trials per item and two training runs per trained arm | Single runs mistake noise for a gain |
| Analysis | Paired per-item differences, standard errors clustered by source record group | Related items are not independent evidence |
Rules for adopting, stopping or calling it inconclusive
Write four outcomes and their triggers into the plan before any arm is scored, with the buyer's own numbers in place of the placeholders. A rule written after the results only describes them.
At a glance
Rules for adopting, stopping or calling it inconclusive
Set the minimum effect from the business case for each route separately. Retrieval has to pay for its per-call cost and the supplier's update deliveries; training has to pay for compute, a model to maintain and the risk that the next base model makes it redundant. One threshold for both routes hides that difference.
- Success for a route: its contrast with the base arm, and with its placebo, exceeds the minimum effect the buyer would pay for on that stratum, with the interval clear of zero across repeats.
- Stop an arm: on lookup tasks, the rate of confidently stated wrong facts rises above the base arm by more than a preset margin, or on any stratum a severe error appears that the base arm never made.
- Inconclusive: the interval spans both zero and the minimum effect. The pilot neither supports nor rules out the license, and the next step is a larger sealed test, not a reinterpretation.
- Void: the test leaked, a grader changed, the base model snapshot changed or an arm broke the budget rule. Fix the cause and rerun; do not salvage the scores.
The result table: which pattern points to which license
Read the contrasts together, stratum by stratum, and let the pattern choose the license scope. SourceX has published no pilot results; the table gives the decision each pattern supports, so a buyer can fill in its own numbers after unsealing.
One contrast deserves attention first. The interaction, (CT minus T) minus (C minus B), shows whether retrieval helps more once the model has been trained on the same records. A positive interaction is the RAFT pattern and argues for licensing both uses together; a value near zero means the routes add independently and can be priced separately. The patterns and decisions are below:
| Pattern on the sealed test | What it means | License decision | Evidence before scaling |
|---|---|---|---|
| C beats B and Cp on lookup tasks; T adds nothing | The value is in the facts | License for retrieval, with update deliveries | A later time slice to test freshness |
| T beats B and Tp on behavior tasks; C adds nothing | The value is in the patterns | License for training, with terms for the trained model | A held-out supplier or period to test transfer |
| CT beats both C and T; interaction positive | Training teaches the model to use the records | License both uses together | Whether the extra gain of CT covers training cost |
| C roughly equals CT; T small | Retrieval captures the value | License retrieval only | A rerun against the strongest long-prompt baseline |
| C matches Cp, or T matches Tp | Longer prompts or format learning, not these records | Do not license on this evidence | Rewrite the prompt or use generic format data |
| No arm clears the minimum effect | The dataset does not fix this task | No license; respecify the task | Collect failures the base model actually makes |
Would the winner survive a stronger baseline, a fresh holdout and more runs?
Treat the first winner as fragile until three checks say otherwise. Each one can erase an apparent dataset gain, and each can be planned before the pilot starts.
A stronger baseline. The base arm should be the best setup available without the candidate data: a tuned prompt, the buyer's existing retrieval over its own documents and, where the dataset is small enough, the whole corpus placed in a long prompt. A context arm that beats a weak prompt but not a long one adds engineering, not value. Placement matters as well: the Lost in the Middle study found that models often do best when relevant information sits at the start or end of the input and worse when it is buried in the middle, so tune passage order on the development split for every context arm alike.
A fresh holdout. Rerun the winning contrast on items from a later period or from a different source company than the training records. Retrieval gains that rely on a fact repeated across the corpus, and training gains that rely on one supplier's house style, tend to shrink on items the dataset never saw.
More runs. Report the spread across training runs and inference trials, not the best run. If the ranking of arms flips between repeats, the result is inconclusive under the rules above, however large the best run looked.
What the pilot leaves unanswered
The design compares routes for one task, one base model and one dataset at one point in time. It does not show how a fine-tuned model ages as the business changes, how retrieval quality holds as the corpus grows, or whether the gain survives the next base model, which may close part of the gap on its own.
It also says nothing about rights. Whether a supplier's records may be indexed, cached, excerpted or used for training depends on the supplier's contracts and notices and on the license terms, and counsel should review those for the specific deal. The pilot tells a buyer which rights it needs; it cannot tell anyone that those rights exist.
Turning the pilot into a data request
A finished protocol is most of a good data request. State the intended use (retrieval, training or both), the task, the strata, the acceptance test and the minimum effect the buyer would pay for, so suppliers and their counsel can see exactly which rights are in question.
SourceX sources datasets for buyers on request from U.S. businesses rather than holding stock, and buyers describe scope, volume, language, format, timing and licensing needs through the request form. SourceX's role covers sourcing and qualification, rights review, contracting, data preparation, delivery and payment, and no data moves until the supplying company has approved the scope and terms and signed. Nothing here implies that a particular dataset is available or will improve a given model; finding that out is the pilot's job.
Frequently asked questions
Is retrieval or fine-tuning better for company data?
Neither wins in general. Ovadia and colleagues (2023) found retrieval more reliable for adding facts, while fine-tuning is the usual route for changing format and behavior. A four-arm pilot on your own records, split into lookup and behavior tasks, shows which route your dataset actually rewards.
Can the same records be licensed for retrieval and for training?
Often, but they are different uses that need different rights: retrieval stores, indexes and may display excerpts, while training turns the records into model weights. A license written for one use rarely covers the other, so name both uses in the request if the pilot points to both. Counsel should review the terms for the specific deal.
How large should the test set be for a context versus training data pilot?
Large enough that each stratum's interval can separate the minimum effect you care about from zero. OpenAI's accuracy guide suggests at least 20 questions with ground-truth answers as a starting baseline, but comparing four arms across two strata usually needs many more. Paired per-item analysis gets the most out of each item.
What if the whole dataset fits in the context window?
Test that version first. Anthropic's contextual retrieval guidance notes that a knowledge base under about 200,000 tokens can be placed in the prompt whole, which removes retrieval errors from the context arm. If the long-prompt version matches retrieval, it is the fairer context arm to set against training.
Does fine-tuning on new facts make a model hallucinate?
It can. Gekhman and colleagues (2024) found that fine-tuning examples with facts new to the model were learned more slowly and, once learned, increased its tendency to hallucinate. That is why the protocol carries a stop rule on confidently stated wrong facts for every trained arm.
Sources
- OpenAI's guide to optimizing LLM accuracy separates context optimization (what the model needs to know, including proprietary or out-of-date information) from LLM optimization (how the model needs to act) and suggests a baseline of 20 or more questions with ground-truth answers. Source
- Lewis and colleagues (2020) introduced retrieval-augmented generation, pairing a pretrained seq2seq generator as parametric memory with a dense vector index of Wikipedia as non-parametric memory. Source
- Ovadia and colleagues (2023) found that retrieval consistently outperformed unsupervised fine-tuning for injecting existing and new knowledge into 7B-class models, and that combining fine-tuning with retrieval did not beat retrieval alone in their setup. Source
- Gekhman and colleagues (2024) found that fine-tuning examples introducing new knowledge are learned more slowly and, once learned, linearly increase the model's tendency to hallucinate. Source
- RAFT (2024) fine-tunes a model on questions paired with retrieved documents, training it to ignore distractor documents and cite the relevant one. Source
- Anthropic's contextual retrieval guidance says a knowledge base smaller than about 200,000 tokens (about 500 pages) can be included in the prompt whole, without retrieval. Source
- Anthropic's guide to agent evaluations defines each attempt at a task as a trial and runs multiple trials because model outputs vary between runs. Source
- Evan Miller (2024) recommends comparing models through question-level paired differences and using clustered standard errors when evaluation questions come in related groups. Source
- The Lost in the Middle study found language models often perform best when relevant information is at the beginning or end of the input and degrade when it sits in the middle of long contexts. Source
Tell us what data you need
Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.