Skip to content

AI uses for records

Specialist dataset pilot limits: which claims a small sample can carry

By SourceX Editorial · Updated

9 sections

Short answer

A small specialist dataset pilot supports claims only about the task population it sampled: the specialty, task types, sources, period and difficulty mix written down before any record is drawn. Hold the model, budget and grader fixed, preregister success, stop and inconclusive rules, and report each stratum with an interval. Claims about neighboring tasks, other suppliers or general capability need new evidence, so a coverage statement names what was not tested.

Agent run

Specialist dataset pilot

This page covers specialist dataset pilot limits: which claims a small sample can carry.Illustrative

Key takeaways

  • A pilot speaks for its sampled task population, defined in six fields before sampling, and for nothing wider.
  • Claims sit on five rungs, from sampled cases to general capability; a single small pilot reaches the second rung at most.
  • Small samples give wide intervals: 24 of 40 tasks solved has a Wilson 95% interval of roughly 45 to 74 percent.
  • Specialist fine-tuning can cost capability elsewhere, so every pilot needs a regression suite outside its specialty.
  • An illustrative 13-point gain on 60 cases is still inconclusive when the paired interval reaches below zero.

What can a small specialist dataset pilot claim?

A small specialist dataset pilot can claim an effect on the task population it actually sampled, measured under fixed conditions and reported with an interval. It cannot claim that the gain carries to tasks, suppliers or periods it never sampled. Specialist dataset pilot limits come from the sampling frame, not from how good the records look.

Take an illustrative pilot: 600 licensed adjuster notes from two regional property insurers, used to fine-tune a model for residential hail-damage claims, then scored on 60 held-out hail claims from the same two insurers. A pass shows that these notes helped this model on hail claims at those insurers during that period. It says nothing yet about flood claims, commercial property, other carriers or claims filed after a policy-form change.

The slide from 'helped on our sample' to 'improves claims handling' is where purchase decisions go wrong, because the narrow result is real and the wide claim sounds like a fair summary of it. Large benchmarks state their boundaries plainly. OpenAI's GDPval, published in 2025, covers 1,320 tasks across 44 occupations drawn from the nine industries contributing most to U.S. GDP, and OpenAI still notes that it tests one-shot work and does not capture tasks where a model builds context or improves over several drafts. A 60-task pilot owes its readers at least the same candor.

Write the sampled task population down before drawing a record

Define the population in six fields, freeze them before sampling, and treat them as the outer edge of every later claim: specialty, task types, record sources, time window, difficulty mix and exclusions. Whatever the fields leave out is the coverage gap, and the pilot report has to name it.

The population card also separates two things teams often blur. The target population is the work the buyer wants a model to do; the sampling frame is the set of records the pilot could actually draw from. A buyer who wants better claims handling across all perils but can only reach hail files from two insurers has a frame far narrower than the target, and the gap between them is exactly what the pilot cannot speak to.

Freeze the card with a date and an owner. If a field changes mid-pilot, for instance because a supplier sends extra records from a third insurer, log the change and either give those records their own stratum with separate results or leave them out entirely.

The population card follows; the example column is Illustrative.

Write the sampled task population down before drawing a record
Population fieldWhat to recordExample (Illustrative)What it bounds
SpecialtyThe work the records represent, in the buyer's own termsResidential hail-damage claimsAny claim about other perils or lines
Task typesEach model task scored, with its pass ruleCoverage decision, reserve estimate, next-action noteAny claim about untested tasks
Record sourcesOrganizations, systems and regions supplying casesTwo regional insurers on one claims systemAny claim about other suppliers or systems
Time windowResolution dates of the sampled casesClaims closed in 2023 and 2024Any claim after a rule or policy-form change
Difficulty mixShare of routine, ambiguous and disputed cases, and who assigned them60% routine, 30% ambiguous, 10% disputedAny claim about the hard tail
ExclusionsCase types removed, and whyFiles in litigation and reopened claimsAny claim about excluded case types

Four ways a narrow pilot ends up overclaiming

Overclaiming usually happens in the write-up rather than the experiment. The result is computed correctly, then reported without its population, averaged across strata that behaved differently, scored on cases that share sources with the training sample, or read as a general capability. Each failure has a cheap fix:

  • Qualifier drift. The result table says 'residential hail claims, two insurers, closed 2023 to 2024'; the slide says 'claims'. Fix: the population string travels with every number, including summaries and emails.
  • Pooled averages. A 13-point average gain can come almost entirely from routine cases while the disputed tail gets worse. Fix: report each stratum with its own count. The Model Cards framework (Mitchell and colleagues, 2019) recommends evaluation broken down across the groups that matter for the intended use, alongside a statement of that use.
  • Shared-source test sets. When training and test cases come from the same insurers and quarters, the score measures fit to those sources. Fix: hold out a later period or a whole source, or state plainly that none was held out.
  • Reading in-domain parity as capability. D'Amour and colleagues (2020) showed that many models reaching similar held-out accuracy in the training domain can behave very differently once deployed, a property they call underspecification. Fix: never infer behavior outside the sampled frame from scores inside it.

The claim ladder: five rungs, and which one a small pilot reaches

Every claim in a pilot report sits on one of five rungs, and each rung needs evidence that the rung below cannot supply. A single small pilot reaches rung two, the sampled population, and only when its test cases were drawn at random or by stratum from the frozen frame.

Treat each rung as an evidence requirement, not a wording choice. Rung three needs test cases from the neighboring strata themselves, such as claims closed after the training window. Rung four needs results with whole sources held out, repeated across several sources. Rung five needs a broad evaluation plus a regression suite outside the specialty, which a pilot built for one purchase decision rarely has.

Clinical trial reporting offers a tested analogy. The CONSORT guidance treats external validity as a judgment that depends on who a trial enrolled, its setting, the treatments tested and the outcomes measured. A dataset pilot has the same four limits: the sampled population, the deployment setting, the model and training recipe, and the metric.

Use the ladder to word every sentence of the pilot report:

The claim ladder: five rungs, and which one a small pilot reaches
RungClaim it allowsEvidence requiredReachable by a small pilot?Wording that fits
1. Sampled casesThe candidate beat the baseline on these test casesA fixed test set and a preregistered metricYesOn the 60 held-out hail claims tested
2. Sampled populationThe effect holds for the population the cases came fromA random or stratified draw from the frozen frame, with an interval per stratumYes, if the draw was random or stratifiedFor residential hail claims at the two sampled insurers, closed 2023 to 2024
3. Adjacent strataIt holds for neighboring task types or later periods inside the specialtyTest cases from each neighbor, such as a later quarterOnly with an added holdoutNot tested on claims closed after 2024
4. Other sourcesIt holds at organizations or systems not sampledResults with whole sources held out, across several sourcesNoNot tested at other insurers
5. General capabilityThe model is better at the job overallA broad evaluation plus a regression suite outside the specialtyNoNo general claim is made
Five nested claim rungs from sampled cases to general capability, with the two inner rungs marked as supportable by a small pilot and rungs three to five marked as needing new evidence.
The claim ladder for a small specialist pilot: rungs 3 to 5 each need evidence the pilot did not collect.

Preregistering the pilot: what stays fixed and what ends it

Freeze the model version, training budget, evaluation sets, grader and decision rules before training, and write the success, stop and inconclusive rules as numbers. Otherwise a narrow result gets reinterpreted until it supports the purchase someone already wanted.

Anthropic's guide to agent evaluations supplies the vocabulary for the fixed parts: a task is a single test with defined inputs and success criteria, a trial is one attempt at it, and a grader is the logic that scores some aspect of the result. The same guide offers a test for admitting a case at all: if two domain experts grading it separately could come back with different pass or fail calls, it is not yet a good task. Specialist work is full of cases that fail that test, so run it on a calibration subset before the pilot, not after.

Illustrative rules for the hail pilot: success means the candidate beats the baseline by at least 8 points on the sampled population, with the lower bound of the paired 95% interval above zero and no stratum of 15 or more cases worse by more than 5 points. Stop means any such stratum falls by more than 10 points, or the regression suite outside the specialty falls by more than 3 points. Strata under 15 cases are reported as counts and read case by case. Everything else is inconclusive, and the preregistered response to an inconclusive result is a larger or better-targeted sample, never a new metric.

The protocol card records each fixed condition and the reason it is fixed:

Preregistering the pilot: what stays fixed and what ends it
Protocol fieldFixed value or ruleWhy it is fixed
Model and versionOne pinned checkpoint for every armA model update moves scores on its own
Training budgetSame steps, tokens and schedule, with a volume-matched filler arm if the candidate adds dataSeparates the data's effect from extra training
Evaluation setsHeld-out cases from the frozen frame, plus a regression suite outside the specialtyMeasures both the claim and its cost elsewhere
GraderA written rubric, with two experts scoring a calibration subset and agreement reportedAmbiguous specialist cases inflate noise
TrialsThree or more runs per arm wherever outputs vary between runsA single run can mistake variance for effect
Decision rulesSuccess, stop and inconclusive thresholds written as numbersStops the bar from moving after results arrive

How wide is the interval on 40 or 60 tasks?

Wide enough to swallow most headline differences. Solving 24 of 40 tasks gives a Wilson 95% interval of roughly 45 to 74 percent, so a small pilot can separate two arms only when the true difference is large, and the report should say so.

Zero observed failures is weaker evidence than it sounds. With no severe errors in 40 cases, the Wilson upper bound is still near 9 percent; with 60 cases it is near 6 percent. 'No severe errors observed' belongs in a report only next to its denominator.

Two corrections keep small-sample intervals honest. When both arms answer the same test cases, compare paired differences case by case rather than two separate averages, as Evan Miller recommends in Anthropic's 2024 paper on adding error bars to evals. When cases share a source, such as several claims handled by one adjuster or drawn from one policy file, use clustered standard errors, which the same paper borrows from the social sciences. Specialist pilots nearly always cluster, so the effective sample sits closer to the number of sources than to the number of cases.

Wilson 95% intervals for common pilot sizes, computed with the method that the statsmodels proportion_confint function implements:

  • 12 of 20 solved: about 39 to 78 percent.
  • 24 of 40 solved: about 45 to 74 percent.
  • 45 of 60 solved: about 63 to 84 percent.
  • 0 failures in 40 cases: upper bound about 9 percent.
  • 0 failures in 60 cases: upper bound about 6 percent.

Four checks that make a narrow gain disappear

A positive specialist result is most often explained by a baseline that was not given the same help, a test set that shares sources with training, or a single lucky run, and it can also hide a cost outside the specialty. Run all four checks before any wording climbs the ladder:

  • A stronger baseline. Give the baseline what the candidate had: a tuned prompt, a few worked examples from the specialty and retrieval over the same records. If retrieval closes most of the gap, the records are useful as context, which is a different and usually cheaper purchase than training data.
  • A fresh holdout. Score on cases from a later period or a source used nowhere in training. A gain that halves on the fresh holdout was partly a within-source gain.
  • Repeat runs. Retrain each arm with at least three seeds. If the arms swap order across seeds, the result is inconclusive by rule, whatever the best run showed.
  • A regression suite outside the specialty. NVIDIA's Nemotron training FAQ reports that fine-tuning on agentic trajectories often degrades performance outside the target domain, and for mixture-of-experts models fine-tuned on a narrow domain it advises against training on that domain alone. A specialist gain that costs general behavior has to be priced with that cost attached.

Reporting the result: a per-stratum table and a coverage statement

Report one row per stratum with its count, both arms, the tasks fixed and broken, the paired change with its interval, and the highest rung the row supports. Then add a coverage statement that lists, in plain words, what was not tested.

The illustrative hail pilot shows why the rows matter more than the headline. The candidate solved 44 of 60 held-out claims against the baseline's 36, a 13-point gain that clears the 8-point bar. But the paired 95% interval runs from about -0.1 to +26.8 points, so its lower bound does not clear zero and the preregistered verdict is inconclusive. Routine claims improved with an interval just above zero, the 6 disputed claims went from 2 solved to 1, and the later-quarter holdout and the regression suite stayed within noise.

Because the primary rule was not met, the routine-claims result is reported as secondary, at rung two at most. The intervals are normal-approximation intervals on the paired differences; clustering by adjuster would widen them further.

A coverage statement for the same pilot reads: tested on residential hail claims from two regional insurers, closed 2023 to 2024, with one pinned model; not tested on other perils, commercial property, other insurers, claims closed after 2024, files in litigation or other models. The illustrative result table follows.

Reporting the result: a per-stratum table and a coverage statement
Stratum (cases)Solved: baseline to candidateFixed / brokenPaired change, 95% intervalHighest rung supported
Routine (36)25 to 318 / 2+16.7 points (+0.3 to +33.0)Rung 2, secondary: lower bound barely above zero
Ambiguous (18)9 to 125 / 2+16.7 points (-11.1 to +44.4)Rung 1 only: the interval includes zero
Disputed (6)2 to 10 / 1Counts only, read case by caseNone: too few cases
All sampled claims (60)36 to 4413 / 5+13.3 points (-0.1 to +26.8)Rung 1: inconclusive by the preregistered rule
Later-quarter holdout (30)18 to 204 / 2+6.7 points (-9.2 to +22.5)Rung 3 not reached
Regression suite outside the specialty (100)81 to 782 / 5-3.0 points (-8.2 to +2.2)Within the stop rule; watch it

What to ask a supplier for after a narrow pilot

Ask for the strata the pilot could not cover, in the counts the interval arithmetic says you need, with source and date keys so the next holdout is real. For the hail example that means more disputed and ambiguous claims, at least one additional insurer kept entirely for testing, and claims closed after the training window.

Put the population card and the acceptance test in the request itself, together with the intended use, including whether the records will serve as training data, evaluation data or both. Evaluation use matters because a held-out set is only useful while it stays unseen, so the terms should allow keeping a slice out of training.

SourceX does not keep datasets in stock: it looks for them among U.S. businesses when a buyer asks, and the request-data form collects the scope, volume, language, format, timing and licensing needs that search starts from. As a general guide, buyers want a consistent run of at least two years in the records and counts in the tens of thousands from a single system, though smaller sets can qualify when the work is rare, which is common in specialist pilots; each case is assessed deal by deal. The company supplying the records signs off on scope and terms before anything is delivered. A request that carries the population card and the acceptance test lets candidate data be described against the strata you need, without any promise about what is available.

Frequently asked questions

How many test cases does a specialist dataset pilot need?

Enough in each stratum to detect the difference that would change the decision, which for small pilots is usually more than teams expect. In an illustrative 60-case pilot, a 13-point gain still had a paired 95% interval reaching slightly below zero. Size every stratum you intend to report, and report strata under about 15 cases as counts rather than percentages.

Can a pilot on one company's records show the data will work at other companies?

Not on its own. Claims about other organizations need results with whole sources held out, repeated across several sources, because records from one company carry its rules, vocabulary and case mix. Until then, the report should say the data was not tested at other organizations.

What is a coverage statement in an AI data pilot?

It is a short paragraph listing what the pilot tested and what it did not: specialty, task types, sources, period, difficulty mix, exclusions and the model used. It travels with the headline number so the result cannot be quoted without its boundary.

Should a specialist dataset be mixed with general data during fine-tuning?

Often, yes. NVIDIA's Nemotron training FAQ advises against fine-tuning a mixture-of-experts model on a narrow domain alone and recommends keeping the mixture close to the original post-training blend. Whatever the mix, score a regression suite outside the specialty so any loss shows up in the report.

Is an inconclusive pilot a failed pilot?

No. An inconclusive result says the sample was too small, or the effect smaller than assumed, to meet the decision rule. The preregistered response is a larger or better-targeted sample, not a new metric chosen after seeing the results.

Sources

  • OpenAI's GDPval spans 1,320 tasks across 44 occupations selected from the nine industries contributing most to U.S. GDP; OpenAI notes it is one-shot and does not capture work where a model builds context or improves over multiple drafts. Source
  • Model cards document performance characteristics to clarify intended use and recommend evaluation disaggregated across relevant groups and their intersections. Source
  • Underspecification: many predictors with similar held-out performance in the training domain can behave very differently once deployed (D'Amour and colleagues, 2020). Source
  • CONSORT guidance treats external validity as a judgment that depends on the trial's participants, setting, treatments tested and outcomes assessed. Source
  • Anthropic defines a task as a single test with defined inputs and success criteria, a trial as one attempt and a grader as scoring logic, and says a good task is one where two domain experts would independently reach the same pass or fail verdict. Source
  • Evan Miller's 2024 paper on error bars for evals recommends paired, question-level differences when comparing two models on the same questions, and clustered standard errors when questions share a source. Source
  • The statsmodels proportion_confint function implements the Wilson score interval, among other methods, for a binomial proportion. Source
  • NVIDIA's Nemotron training FAQ says SFT on agentic trajectories often degrades performance outside the target domain, and advises not to fine-tune a mixture-of-experts model on a narrow domain alone. Source

Tell us what data you need

Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.

See if your company qualifies