Skip to content

AI uses for records

SourceX buyer dataset pilot: the evidence pack, locked before data

By SourceX Editorial · Reviewed by Noah Loul ·

8 sections

Short answer

A SourceX buyer dataset pilot is worth running only if its evidence pack is fixed before any records arrive: the decision it informs, the approved data scope, a model and budget held constant across arms, the baselines, and preset success, stop and inconclusive rules. After the run, the pack adds a result table of paired differences across repeated trials and the limits one pilot cannot escape.

Agent run

SourceX buyer dataset pilot

This page covers sourceX buyer dataset pilot: the evidence pack, locked before data.Illustrative

Key takeaways

  • Freeze the protocol before data exists: a time-stamped plan that cannot change after results is what turns a pilot into evidence rather than a demo.
  • Hold model, budget and evaluation fixed, and add an arm that spends the same budget on data you already hold, so any gain can be traced to the requested records.
  • Expect positive results to shrink under stronger baselines, leakage checks and repeated trials; Anthropic reports clustered standard errors on popular evals more than three times naive ones.
  • Report paired differences with intervals, regressions on work that already succeeds and a later-period holdout, then let preset rules choose license, extend once or walk away.
  • Datasets come through SourceX on request rather than from stock, so the pack's scope and acceptance test can travel with the request; SourceX has published no pilot results.

What should a SourceX buyer dataset pilot decide?

One purchase decision: whether records sourced to your written requirement improve a named task enough, against your best alternative, to justify a full license. Everything in the evidence pack exists to make that decision defensible to someone who was not in the room, such as a budget owner, a procurement lead or an auditor six months later.

The decision has three honest outcomes rather than one: license the full dataset, extend the pilot once under rules written in advance, or walk away. A pilot built to produce only a yes is a sales demo. The evidence that supports a useful answer is a comparison on held-out tasks from your own work, under conditions fixed before the data arrives, repeated enough to separate a real gain from noise and checked against a baseline strong enough to embarrass a weak dataset.

The SourceX context decides when the pack starts. SourceX does not keep datasets in stock. Each one is sourced from a U.S. business once a buyer has set out the scope, volume, language, format, timing and licensing terms it needs. With no catalog sample to try first, the protocol has to exist before the records do. SourceX has not published pilot results, and nothing here implies that any dataset will be accepted or will improve a model.

What goes in the evidence pack, and when is each part frozen?

Five parts, frozen at three moments: before the request, before data arrives and after the run. Freezing is the point. The Center for Open Science describes preregistration as posting a time-stamped, read-only version of a study plan before data collection or analysis begins, and the same discipline stops a pilot from being redesigned around its own results.

At a glance

What goes in the evidence pack, and when is each part frozen

Provenance belongs in the pack even though it is not a model result. Datasheets for Datasets, the documentation framework from Gebru and colleagues, groups its questions into seven categories, from motivation and composition through collection, preprocessing, distribution, maintenance, and legal and ethical considerations. A pilot that cannot say what the records are and where they came from cannot say what its result applies to. The five parts, in the order they freeze:

  • Decision record, frozen before the request: the purchase decision, the three permitted outcomes, the decision owner and who countersigns.
  • Requirement and acceptance test, frozen before the request: task, record types, systems, years, volume, format, permitted pilot use and the test the delivery must pass.
  • Protocol, frozen before data arrives: arms, fixed conditions, the held-out task set and its file hash, graders, trials, analysis and decision rules.
  • Provenance file, completed on delivery: the approved scope, the de-identification record, a datasheet and any departure from the requirement.
  • Result table and limits, completed after the run: the numbers the protocol named, a sample of transcripts read, every deviation from the plan and what the pilot cannot show.

The preregistered pilot protocol, field by field

Lock these eight fields before any record arrives, sign them and store the signed copy where nobody on the pilot can edit it. The illustrative entries show the level of detail that makes a field checkable; replace them with your own task.

Two fields do most of the work: the arms isolate what the requested records contribute, and the decision rules stop anyone moving the bar after seeing a number. The protocol:

The preregistered pilot protocol, field by field
FieldWhat you lockIllustrative entrySigned by
Decision and ownerThe purchase decision, the three outcomes and who decidesLicense, extend once or walk away; the evaluation lead decides and procurement countersignsDecision owner
Data requirementTask, record types, systems, years, volume, format and permitted pilot use, as written in the requestTwo years of closed support cases with resolution outcomes, in English, for evaluation and fine-tuning during the pilot onlyRequester
ArmsThe comparisons, each differing from the last in one respectA: current model; B: same budget on data already held; C: same budget on the requested dataEvaluation lead
Fixed conditionsModel version, harness or prompts, training steps or token budget and decoding settings, identical across armsSame base checkpoint, hyperparameters and step count for B and C; same agent harness for all threeEvaluation lead
Held-out task setTasks from your own work, frozen and hashed before data arrives and never shown to anyone preparing the data50 tasks drawn from recent real failures, stratified by case type, with the file hash recordedEvaluation lead
GradersCode checks first, model graders calibrated against experts, blinded human review for judgment callsOutcome check against the recorded resolution; blinded expert pairwise review on 20 tasksEvaluation lead and domain expert
Trials and analysisTrials per task, the metric and how differences and intervals are computed3 trials per task; pass rate; paired difference C minus B with a 95 percent interval clustered by caseEvaluation lead
Decision rulesSuccess threshold, stop rules and the inconclusive rule, all set before any resultLicense if the C minus B interval clears 5 points; stop on any regression over 3 points; one extension if it straddles 5Decision owner
Two panels: on the left, six protocol fields locked before data arrives; on the right, a result table with rows for arms A, B and C that fills after the run and feeds one of three outcomes.
The evidence pack: lock the protocol, then fill the result table. No results are shown.

Why hold the model, budget and evaluation fixed across arms?

Because any difference between arms other than the data becomes a rival explanation for the result. Fixing the model, the budget and the evaluation is what lets a gain in arm C be read as a property of the records rather than of the setup.

Tuning is the classic trap. Melis, Dyer and Blunsom re-evaluated popular language model architectures with large-scale automatic hyperparameter tuning and found that standard LSTMs, properly regularized, outperformed more recent models, contrary to published claims. A pilot that tunes arm C harder than arm B repeats that error. Budget works the same way: if arm C trains for more steps or tokens than arm B, the result measures more training, not better data.

Evaluation conditions have to match as well. Anthropic's January 2026 guidance on agent evaluations says each trial should start from a clean environment, because shared state between runs can cause correlated failures or inflate performance; in one internal case a model gained an unfair advantage by examining the git history of earlier trials. NVIDIA's Nemotron FAQ gives the matching advice for customized models: measure on your own task set, not on public leaderboards alone.

Could a positive pilot result disappear under a stronger test?

Yes, and the pack should be built to find out before the purchase rather than after it. Three checks remove most false wins: a stronger baseline, a proper holdout and repeated trials.

Holdouts fail more often than teams expect. Kapoor and Narayanan's survey of machine-learning-based science found leakage errors reported across 17 fields and proposed model info sheets to catch them before publication. In a dataset pilot the usual leaks are held-out tasks that resemble records in the delivery and outcome fields that give the answer away; a later-period holdout, built from cases that occurred after the delivered records end, tests for both.

Repetition and the right statistics matter as much. Anthropic's guidance on statistics for evals recommends reporting standard errors, clustering them when questions are related and analyzing paired differences. It reports that clustered standard errors on popular evals can be more than three times as large as naive ones, and that frontier models' question-level scores correlate between 0.3 and 0.7, which makes paired comparisons a nearly free gain in precision. Task difficulty matters too: for one evaluation set, NVIDIA selected 3,000 held-out problems with pass rates between 0.25 and 0.80, leaving room for scores to move either way. Run these checks before calling a win:

  • Add arm B, the same budget spent on data you already hold, and tune it as carefully as arm C.
  • Hash the held-out tasks before delivery and check the delivered records for near-duplicates of them.
  • Score a later-period holdout made of cases from after the delivered records end.
  • Run at least three trials per task and report the paired difference with a clustered interval.
  • Drop or replace tasks that every arm always passes or always fails, since they cannot show a change.
  • Read a sample of transcripts from each arm before trusting any score.

The result table the pack reports

Report one row per arm or check, each with its denominator, and let the preset rules read the table. The template below names what each row isolates, what to report and which reading blocks a purchase; it holds no results.

For judgment-heavy outputs such as drafted replies or reports, use blinded pairwise review. OpenAI's GDPval benchmark has occupational experts compare unlabeled deliverables without knowing which came from a model, and OpenAI states the benchmark's limit plainly: its tasks are one-shot, so it does not capture work that builds context over several drafts. A pilot's result table should declare its limits the same way. The rows:

The result table the pack reports
RowWhat it isolatesWhat to reportReading that blocks a purchase
A. Current model, no new dataThe starting level on the frozen tasksPass rate, tasks, trials and transcripts readArm A already passes most tasks, so the set cannot show a gain
B. Same budget, data already heldThe effect of more training of any kindPass rate and paired difference against AB matches C: the gain came from training, not from the requested records
C. Same budget, requested dataThe effect of the requested recordsPaired difference C minus B with a 95 percent interval, clustered by caseThe interval includes zero or sits below the preset threshold
Regression checkHarm on tasks that already workedCount and size of regressions against AAny regression beyond the stop rule
Repeat runStability across seeds or trialsC minus B on an independent second runThe sign flips or the interval widens past the threshold
Later-period holdoutWhether the gain survives newer casesC minus B on tasks from after the delivered records endThe gain vanishes on the later period

What can one pilot not tell you?

A pilot answers one question for one task, one supplier's records, one model and one period. Four limits belong in every pack, written before the result so they cannot be softened afterward.

  • Other suppliers: a gain on one company's records says little about another's, whose systems, coding habits and customers differ.
  • Full volume: a small sample can understate or overstate what the full dataset would do, because gains rarely scale in a straight line.
  • Future models: a gain on today's model may shrink after the next upgrade, so record the model version and plan a retest.
  • Production: automated evaluation is one view among several; Anthropic lists production monitoring, A/B testing, user feedback, manual transcript review and systematic human evaluation as complements.

Turning the evidence pack into a SourceX data request

Put the requirement and the acceptance test from the pack into the request itself. SourceX's request form collects scope, volume, language, format, timing and licensing needs; add the intended use, the held-out design and the test the delivery must pass, so sourcing starts from the same definition the pilot will judge.

Several SourceX facts belong in the provenance file. No record moves until the agreement is signed, with its scope and terms approved by the supplying company, and from that point SourceX may use, sublicense and distribute the de-identified dataset only as that agreement allows, so the pilot's permitted use should match the signed scope. SourceX carries out the de-identification itself or hands it to its processing partner, so the file should record what was removed and why. Large deliveries stay in the seller's own storage or ship on encrypted drives, which affects where a pilot can run.

Candidate data should fit the task rather than a volume target. SourceX's general guide is that buyers expect several years of consistent history, usually at least two, and one system holding tens of thousands of records or more, while smaller sets can qualify when the work is rare; each case is assessed deal by deal. None of this describes current stock, an acceptance rate or an expected improvement, which only your own pilot can measure.

Frequently asked questions

Can a pilot-sized scope be described in a SourceX data request?

Yes. The request form collects scope and volume, so a pilot-sized requirement and its acceptance test can be written into it. Datasets come through SourceX on request rather than from stock, and the supplying company approves scope and terms and signs before anything is delivered, so each request is assessed on its own.

How many tasks does a dataset pilot need?

Enough to resolve the decision rule, which depends on the size of gain you care about. Anthropic's agent-evaluation guidance says 20 to 50 tasks drawn from real failures is a great start when effects are large, while smaller effects need larger sets; paired analysis and repeated trials stretch a small set further.

What makes a dataset pilot preregistered?

Its plan is written, signed and frozen before any data arrives: the arms, fixed conditions, held-out tasks, analysis and decision rules. The Center for Open Science describes preregistration as a time-stamped, read-only plan posted before data collection or analysis, which keeps the planned test separate from anything explored afterward.

What happens if the pilot result is inconclusive?

The protocol decides in advance. A common rule allows one extension with a larger task set when the interval straddles the threshold and a walk-away if the second run is still inconclusive. An inconclusive result is not a soft yes.

Will data sourced through SourceX improve my model?

Nobody can promise that in advance, which is why the pilot exists. SourceX has not published pilot results, and the evidence pack is designed so that a weak dataset fails cleanly against your strongest baseline instead of passing on a favorable setup.

Sources

  • Anthropic's January 2026 guidance on agent evaluations recommends isolated trials from a clean environment, notes shared state can inflate results, suggests 20 to 50 tasks from real failures to start, advises reading transcripts and lists complementary methods such as production monitoring and A/B testing. Source
  • Anthropic's statistical guidance for evals recommends standard errors, clustered errors and paired differences, reporting clustered errors over three times naive ones on popular evals and question-score correlations of 0.3 to 0.7 between frontier models. Source
  • NVIDIA's Nemotron FAQ advises evaluating a customized model on your own task set rather than public leaderboards alone, and describes 3,000 held-out problems selected with pass rates between 0.25 and 0.80. Source
  • Datasheets for Datasets proposes that every dataset carry a datasheet, with questions in seven categories: motivation, composition, collection process, preprocessing, distribution, maintenance, and legal and ethical considerations. Source
  • The Center for Open Science describes preregistration as posting a time-stamped, read-only version of a study plan before data collection or analysis begins. Source
  • Melis, Dyer and Blunsom re-evaluated language model architectures with large-scale automatic hyperparameter tuning and found standard LSTMs, properly regularized, outperform more recent models. Source
  • Kapoor and Narayanan's survey found data leakage errors reported across 17 scientific fields that adopted machine learning and proposed model info sheets to detect leakage. Source
  • OpenAI's GDPval uses blinded comparisons by occupational experts of human and model deliverables, and OpenAI notes the benchmark is limited to one-shot tasks. Source

Tell us what data you need

Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.

See if your company qualifies