AI uses for records
Novel dataset, no improvement? Five gates that decide re-scope or stop
By SourceX Editorial · Reviewed by Noah Loul ·
9 sections
Short answer
A novel dataset that shows no improvement in a pilot has usually not failed; its new content has not met a failure the test measures. Check five gates in order: is the result conclusive, does the evaluation cover the cases where the data is new, does the model fail there, did enough arrive in a usable form, and does the task fit. Re-scope when an early gate fails; stop when all pass.
License terms
Novel dataset no improvement
Key takeaways
- Novelty describes data relative to what a model has seen; improvement describes a model relative to a task. A pilot links them only where new content meets a measured failure.
- Read the interval before the point estimate: a flat result whose interval still includes the agreed minimum gain is inconclusive, not negative.
- Aggregate thresholds hide slice arithmetic: if all the gain must come from a slice holding 12 percent of tasks, a 3-point overall target needs about 25 points on that slice.
- Fact-heavy records may help as retrieval context when fine-tuning on them shows little, so test the route before stopping preparation.
- A re-scope that changes the use, slice or records is a new scope decision for the data owner, and no technical fix settles rights, terms or a task the data does not fit.
What does a novel dataset with no improvement in a pilot actually show?
It shows that the dataset's new content did not move the score the pilot measured, which is a narrower finding than showing the data is useless. Novelty is a property of records relative to what a model has already seen. Improvement is a property of a model relative to a task. The two meet only where new content lands on a failure the evaluation counts, delivered in a form the model can use.
Buyers ask for novelty for sound reasons. OpenAI's data partnerships page, for example, asks for large datasets that are not already easy for the public to reach online and that express human intention, such as long-form writing or conversations rather than disconnected snippets. A purchase, though, is justified by a task gain, so a pilot that shows no improvement leaves the delivery owner with a narrow decision: re-scope the work, or stop preparing the rest of the dataset.
A defensible answer needs six pieces of evidence: the paired pilot result with its interval, the minimum gain both sides agreed before the run, a tag on every evaluation task saying whether it touches the case types the new records contain, the baseline's pass rate on those tasks, the route by which the records reached the model, and the license scope that any change of use would have to fit.
Why new content can leave the target score unchanged
Five mechanisms explain most flat results, and each points to a different fix. The new content may sit in case types the evaluation rarely exercises. The model may already handle those cases. The new records may be too small a share of the training mix to register. The content may be the wrong kind for the method used. Or the pilot may be too small to see a real effect.
The method mismatch is the least obvious. In a 2024 study, Gekhman and colleagues found that fine-tuning examples carrying facts a model did not already know were fitted substantially more slowly than examples consistent with its knowledge, and that once learned they increased the model's tendency to hallucinate. They concluded that models acquire factual knowledge mostly in pretraining, while fine-tuning mainly teaches them to use it. A separate comparison by Ovadia and colleagues found retrieval-augmented generation consistently ahead of unsupervised fine-tuning for both existing and entirely new knowledge.
Read together, those results suggest a working hypothesis for business records. When the novelty is mostly facts, such as carrier rules, part numbers or local policies, training on it can look flat while retrieving it at answer time does not. When the novelty is behavior, such as how a team sequences a refund exception, training is the stronger candidate. A pilot that tried only one route has not tested the dataset.
The pilot itself can also be at fault. Anthropic's guidance on agent evaluations treats a 0 percent pass rate across many trials with frontier models as more likely a broken task than an incapable agent, and the same suspicion applies to a test whose scores never move. Check the grader and the task set before blaming the records.
Is the null result conclusive enough to act on?
Only if its interval excludes the minimum gain you agreed to pay for. A difference of half a point with an interval from minus 2.5 to plus 3.5 points rules out neither zero nor a 3-point target, so it supports no preparation decision in either direction.
Anthropic's statistical guidance for model evaluations recommends reporting a standard error with every score, clustering standard errors when questions are related, such as several tasks drawn from one customer account, and comparing two models on the same questions through per-question differences, which strips out variance the two share. A dataset pilot is that paired comparison: the same tasks, scored with and without the new data.
Slices change the arithmetic. If the new records bear only on a slice that holds 12 percent of the evaluation tasks, and the other tasks stay flat, a 3-point overall target requires roughly a 25-point gain on the slice. Many flat pilots are this case: a real slice effect diluted below the threshold by tasks the data was never going to touch. Restate the threshold in slice terms before any rerun, or accept that the overall target is out of reach.
The novel-but-flat decision tree: five gates in order
Run the gates in order and stop at the first that fails, because its verdict sets the next step. Any verdict that re-scopes the work then passes the scope check in the next section before a rerun starts.
Each row names the evidence to pull, so someone who did not run the pilot can check the verdict.
| Gate | Question | Evidence to pull | Fails when | Verdict |
|---|---|---|---|---|
| 1. Resolution | Is the result conclusive against the agreed minimum gain? | Paired difference with a 95 percent interval, clustered by account or source | The interval includes both zero and the minimum gain | Inconclusive: enlarge or narrow the test before any preparation decision |
| 2. Coverage | Does the evaluation test the cases where the data is new? | Every task tagged by the case types in the new records; slice size and share | The slice is too small to resolve a slice-level threshold | Re-scope the evaluation: build a slice test and restate the threshold |
| 3. Headroom | Does the strongest current setup fail on that slice? | Baseline pass rate on the slice, with the baseline configuration recorded | The baseline already passes most slice tasks | Stop for this task; offer the records as an evaluation set only if the buyer wants one |
| 4. Dose and form | Did enough new content reach the model in a usable form? | New-record count after filtering, share of the training mix, facts versus behavior | The share is thin, or fact-heavy content was only trained on | Re-scope the use or slice: concentrate the records or test them as retrieval context |
| 5. Fit | With gates 1 to 4 passing, does the slice still not move? | Slice result with its interval at the restated threshold | The interval sits below the slice threshold | Stop preparation for this task; log the reason and route the data elsewhere |
Before re-scoping: does the new use fit the rights and terms?
Check that before any rerun, because most re-scope routes change something the data owner approved. Moving records from training into a retrieval index or an evaluation set changes the use. Preparing only a slice changes the delivered scope. Topping up a thin slice with outside data brings in material with its own rights.
None of these is an engineering question. A license may permit training but not storage in a retrieval system that serves answers, or permit evaluation only. SourceX's process shows the order that applies: the company's approval of scope and terms, and its signature, come first, and only then does any data move, so a change of use goes back to that approval rather than proceeding as a technical tweak. Outside data adds risk of its own: when the Data Provenance Initiative, publishing through the MIT Media Lab, examined over 1,800 text datasets, it found that popular hosting sites often gave license information wrongly or not at all.
Commercial terms can block a sensible re-scope too. If the price was set for a full archive and the value now sits in one slice, the honest move is to restate scope and price together. A genuine task mismatch survives every fix: when gate 5 fails, the data may suit a different task or buyer, but more preparation will not make it serve this one.
Illustrative pilot: carrier exception notes that moved one slice
Every name and number in this example is illustrative, not a measured result. A logistics software company offers 18 months of carrier exception notes to a buyer building an agent that drafts customer updates on delayed shipments. A novelty check finds almost no long-phrase overlap with a public web sample, using the 13-gram matching that EleutherAI's evaluation harness documents for contamination checks; OpenAI's GPT-3 analysis, which the harness follows, used N values between 8 and 13 depending on the dataset.
The pilot fine-tunes on 6,000 records and scores 400 tasks. The base setup passes 242 tasks and the tuned model 244: a gain of 0.5 points, with a paired 95 percent interval of about minus 2.5 to plus 3.5. Against a 3-point minimum agreed beforehand, gate 1 fails and the result is inconclusive.
Tagging the tasks shows why. Only 48 of the 400 tasks involve the exception types the notes cover, such as customs holds and weather reroutes. On those 48 the base setup passed 18 and the tuned model 24, a 12.5-point gain with an interval from about 0 to 25 points. On the other 352 tasks the tuned model lost 1.1 points, with an interval of about minus 4.1 to plus 1.8. The slice moved; the aggregate hid it.
The remaining gates sharpen the verdict. The base setup failed 30 of the 48 slice tasks, so headroom exists. Exception records made up 9 percent of the training mix, about 540 records, so the dose is thin. Much of the content is carrier-specific rules, the fact-like material that retrieval tends to handle better than training.
The verdict is re-scope, with a stop rule. Because the slice holds 12 percent of tasks, a 3-point overall gain would need about 25 points on it. The buyer restates value in slice terms, for example a 10-point minimum on exception tasks justified by what those cases cost. Preparation narrows to exception records, a retrieval arm joins the training arm, and a test of about 250 slice tasks narrows the interval to roughly plus or minus 5.5 points at the same rate of disagreement between the two models. If the upper bound falls below 10 points, preparation stops.
A rejection reason log for flat novel datasets
Log the verdict with its evidence, so the next buyer or the next model version starts from the reason rather than from a rumor that the data did not work. Name the mechanism, not the outcome: no improvement is an outcome, while an evaluation blind to the slice is a reason someone can act on.
Each entry also records the dates of the run, the model and evaluation versions, the denominator behind every rate, the rule that selected the tasks and records, and who approved the verdict on each side.
| Code | Reason | Evidence that proves it | Next action |
|---|---|---|---|
| NOV-1 | Inconclusive pilot | Interval spans zero and the agreed minimum gain | Rerun with a larger or slice-focused test; no preparation decision yet |
| NOV-2 | Evaluation blind to the new content | Few or no tasks tagged with the new case types | Build a slice test and restate the threshold in slice terms |
| NOV-3 | Already handled by the model | Baseline passes most slice tasks | Stop for this task; consider an evaluation-only offer |
| NOV-4 | Dose too thin | New records are a small share of the training mix after filtering | Concentrate the slice and rerun before preparing the rest |
| NOV-5 | Wrong route for the content | Fact-heavy records trained on but never retrieved | Add a retrieval arm, inside the approved use |
| NOV-6 | Task mismatch | Slice flat with resolution, coverage, headroom and dose confirmed | Stop preparation; route the data to another task or buyer |
| NOV-7 | Re-scope blocked by scope or terms | New use, slice or outside data not covered by the signed scope | Return to the data owner for scope and terms, or stop |
What this framework does not measure
The gates and codes are an editorial framework built from published research on knowledge injection, evaluation statistics and contamination checks, each cited on this page and checked on October 26, 2026. They have not been validated on a cohort of real pilots, and the worked example is illustrative. SourceX has not published pilot results for any dataset; this page describes a method, not measured outcomes.
Three limits matter most. The knowledge-injection findings come from question-answering studies, so their transfer to business records is a hypothesis to test, not a rule. Novelty measured by phrase overlap misses paraphrased content. And slice tags depend on someone judging which tasks a record type could affect, so two people should tag a sample independently and report how often they agree.
Writing a data request that tests novelty against the task
Put the slice, the use and the acceptance test in the request itself. A request that asks only for novel data invites exactly the flat pilot described here. One that names the case types the model fails on, the evaluation tasks that measure them and the minimum gain worth paying for lets a supplier judge fit before anyone prepares records.
SourceX does not sell from a shelf: it sources each dataset after a request comes in, and the request form is where a buyer spells out what it needs on six fronts, namely scope, volume, language, format, timing and licensing. Adding the intended use (training, retrieval or evaluation), the target slice and a written acceptance test makes the first conversation about fit rather than volume. No supplier can promise that a dataset will improve a given model; a request can make the test of that claim explicit from the start.
Frequently asked questions
Does a pilot with no improvement mean a novel dataset is worthless?
No. It means the new content did not move the score that was measured. Check whether the interval was narrow enough to decide, whether the evaluation tested the case types the data adds, and whether the content reached the model by a route it can use. Only when all of those hold and the slice still does not move is the data a poor fit for that task.
How can you tell whether a dataset is novel to a model?
A common first check is long-phrase overlap against a reference corpus, such as the 13-gram matching that EleutherAI's evaluation harness documents for contamination checks. Low overlap suggests the text is not already public, but it misses paraphrases and says nothing about usefulness. Treat novelty as a precondition for a gain, not evidence of one.
Why might fine-tuning on new information fail to improve a model?
A 2024 study found that language models fit fine-tuning examples containing unfamiliar facts substantially more slowly, and that learning them can increase hallucination. Another study found retrieval at answer time consistently ahead of unsupervised fine-tuning for injecting new knowledge. Fact-heavy records may therefore show more value as retrieval context than as training data.
How many tasks does a slice evaluation need?
Enough that the interval can separate zero from the slice threshold both sides agreed. In the illustrative example on this page, a 48-task slice produced an interval about 25 points wide, while roughly 250 tasks would narrow it to about plus or minus 5.5 points at the same rate of disagreement between the two models. Calculate it from your own pilot's paired differences.
Who should approve a decision to re-scope a dataset?
The buyer's acceptance owner and the data owner together. Re-scoping often changes the use, the slice or the records delivered, which are scope and terms decisions the data owner signed, not engineering choices. Counsel should review any change that touches the license.
Sources
- OpenAI's data partnerships page seeks large datasets not already easily accessible online that express human intention, such as long-form writing or conversations rather than disconnected snippets. Source
- Gekhman and colleagues (2024) found fine-tuning examples with facts unknown to the model were fitted substantially more slowly and, once learned, increased its tendency to hallucinate; models acquire factual knowledge mostly in pretraining. Source
- Ovadia and colleagues found retrieval-augmented generation consistently outperformed unsupervised fine-tuning for both existing and entirely new knowledge. Source
- Anthropic's agent evaluation guidance treats a 0 percent pass rate across many trials with frontier models as more likely a broken task than an incapable agent. Source
- Anthropic's statistical guidance recommends reporting standard errors, clustering them for related questions, and comparing models on the same questions through paired per-question differences. Source
- The Data Provenance Initiative audited more than 1,800 text datasets and found license information on popular hosting sites frequently missing or wrong. Source
- EleutherAI's evaluation harness documents 13-gram overlap for contamination checks, following GPT-3's analysis, which used N values between 8 and 13 depending on the dataset. Source
Tell us what data you need
Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.