Skip to content

AI uses for records

Dataset pilot vs current model baseline: the rival and its 10 settings

By SourceX Editorial · Updated

9 sections

Short answer

A dataset pilot proves value only against the strongest current model baseline you could deploy without the data: today's best available model at its best setting, with tuned prompts, tools and retrieval, plus your production system. Record that rival's exact configuration, because prompt formatting alone moved one model's accuracy by up to 76 points (Sclar and colleagues, 2023). If the dataset's gain disappears against that rival, so does the purchase case.

Agent run

Dataset pilot

This page covers dataset pilot vs current model baseline: the rival and its 10 settings.Illustrative

Key takeaways

  • The baseline that counts is the strongest configuration you could ship today without the dataset, not whichever model you happened to fine-tune.
  • Configuration moves scores as much as data: Anthropic reports a model's CORE-Bench score rising from 42 to 95 percent after grading fixes and a less constrained scaffold (January 2026).
  • Tune the baseline as hard as the dataset arm; in 2017, properly regularized standard LSTMs outperformed newer architectures once all were tuned with the same care.
  • A ten-field baseline card, from snapshot ID to cost per task, lets anyone rerun the comparison, as long as the model has not been retired.
  • If the strongest baseline already passes your acceptance test, stop the pilot: the dataset has nothing left to prove.

What must a dataset pilot's current model baseline prove?

It must show what you could achieve today without buying the data, so that any gain the dataset adds is measured against a real alternative rather than a convenient one. The dataset passes only if it beats that rival on your held-out tasks by a margin set in advance; that margin is the gain that counts.

The evidence is a paired difference: the same held-out items scored by the dataset arm and by the strongest relevant baseline, across repeated trials, with an interval read against a minimum useful difference. A gain over last quarter's model, over a base checkpoint with default settings or over a vendor's published score is an apparent gain. It shows that something changed, not that the dataset is worth buying.

This protocol covers choosing the rival, recording its configuration, tuning it fairly, the rules for success, stopping and inconclusive results, and the result table. No measured results appear here, and SourceX has published none for this design.

Which rival counts as today's strongest relevant baseline?

The strongest configuration you could actually deploy for this task without the dataset. Four filters decide relevance: you can access it through the decision date, it fits your cost, latency and data-handling limits, its terms allow your use, and it has been tuned with the same effort as the dataset arm. Usually that means testing several candidates and keeping the best as the bar.

Do not assume the largest model is the strongest relevant one. NVIDIA's Nemotron training FAQ says the team has seen a customized Nemotron 3 Nano beat the proprietary models used as comparators in several specialized-domain evaluations, that heavily optimized domains such as math and coding leave customization the least headroom, and that the only way to know for a domain is to evaluate on your own task set rather than public leaderboards alone.

Relevance filters remove candidates as often as they add them. A frontier model that cannot receive customer records under your data-handling rules is not a rival for a task that needs those records in the prompt, however strong it is, and a model whose cost per task exceeds what the workflow can bear is a ceiling to report, not a bar to beat. Record why each excluded candidate failed a filter, so a reviewer can see the baseline was not chosen for weakness.

Do not leave people out either. OpenAI's GDPval paper (2025) makes head-to-head grading against deliverables from industry professionals its primary metric, a reminder that for some business tasks the rival a dataset must beat is the current human process. The five candidates, and how each is usually weakened:

Which rival counts as today's strongest relevant baseline?
Candidate rivalWhen it is the right barHow it gets weakenedFix before the pilot
Your production systemAlways; it is what the dataset must improve onCompared at an older snapshot or with features switched offFreeze the live configuration and copy it exactly
Best available frontier model via APIWhen the task data can go to a hosted modelDefault effort, a generic prompt, no toolsRun its highest practical effort with a tuned prompt and tools
Frontier model with retrieval over your documentsWhen missing knowledge is the failureRetrieval left out, so the dataset gets credit for contextIndex the documents you already hold and log the corpus version
Open-weight model customized on data you already holdWhen data must stay in-house or cost per task is tightA base checkpoint run with library defaultsStart from the post-trained checkpoint and tune with equal effort
Current human processWhen the output is a professional deliverableTimed on easy cases or graded by the person who did the workSample the same items and grade blind

How does a baseline get weakened without anyone noticing?

Through settings nobody wrote down: defaults, prompt format, harness bugs and unequal tuning. Each can move a score by more than a typical dataset gain, so a weak baseline makes a dataset look valuable for reasons unrelated to the data. Each fix raises the bar a step: defaults, a tuned prompt, maximum effort, then retrieval and tools.

Defaults differ by model and by library. Anthropic's effort documentation says most Claude models default to high effort, that Claude Opus 5.5 and Claude Haiku 5.5 default to medium, and that effort can be raised to max for the highest capability. Hugging Face's Transformers documentation says greedy search is the default decoding strategy and, unless the generation configuration says otherwise, produces at most 20 new tokens, which silently truncates long answers from an open-weight baseline.

Prompt format and the harness matter as much. Sclar and colleagues (2023) found that formatting changes in few-shot prompts moved LLaMA-2-13B's accuracy by up to 76 points and recommended reporting a range of performance across plausible formats. Anthropic's January 2026 evals guide describes Claude Opus 4.5 scoring 42 percent on CORE-Bench until researchers fixed rigid grading and ambiguous tasks and used a less constrained scaffold, after which it scored 95 percent; the same guide says that evaluating an agent means evaluating the harness and the model together.

Unequal tuning is the oldest form of the problem. Melis, Dyer and Blunsom (2017) retuned popular language model architectures with large-scale automatic hyperparameter search and found that standard LSTMs, properly regularized, outperformed more recent models, which they traced to poorly controlled hyperparameter variation. Even vendor comparisons mix settings: OpenAI's March 5, 2026 GPT-5.4 announcement reports GDPval results with GPT-5.4 at xhigh reasoning effort and GPT-5.2 at heavy, a slightly lower level.

An illustrative staircase of four baseline configurations, defaults, tuned prompt, max effort, and retrieval with tools, rising toward a fixed dataset-arm line, so the apparent gain over the weakest rung shrinks to the gain that counts over the strongest.
Why the rival matters: the same dataset arm against four baseline configurations. Illustrative; no measured values.

The baseline card: ten settings to record before the first run

Write one card per candidate rival, freeze it before the dataset arm trains, and attach it to the result. A card is complete when someone outside the team could rerun the baseline from it and get the same configuration.

Two fields carry deadlines. API models retire: Anthropic's deprecation documentation says requests to retired models fail and that it gives at least 60 days' notice before retiring a publicly released model, so record the retirement date and schedule any rerun before it. Harness and task versions change too: EleutherAI's evaluation harness versions its tasks so users can tell iterations apart, and its task guide cites a version bump that fixed an answer-extraction bug which had led to underestimated performance.

The ten fields, each with an illustrative entry:

The baseline card: ten settings to record before the first run
FieldWhat to recordWhy it moves the scoreIllustrative entry
Model and snapshotExact model ID or checkpoint hash, never an aliasAn alias can move to a newer version mid-pilotDated API model ID, or the checkpoint hash
Access and retirement datesDate first queried and any announced retirement dateA retired model cannot be rerunQueried 2026-10-26; no retirement announced
Reasoning effort or thinkingEffort level, thinking mode or thinking budgetDefaults differ across modelsEffort set explicitly to max
DecodingTemperature, maximum output tokens, stop sequencesLibrary defaults can truncate answersTemperature 0; output cap above the longest reference answer
Prompt and formatSystem prompt, template and the set of formats triedFormatting alone moves accuracyBest of five formats, with the range reported
Examples in contextNumber, selection rule and order of examplesShot count and order change behaviorFive examples drawn from the tuning split
Tools and harnessTool list, scaffold, harness version and commitThe harness is part of what is evaluatedHarness commit hash; three tools enabled
RetrievalCorpus, snapshot date, chunking and number of passagesMissing context gets credited to the datasetPolicy manual snapshot of 2026-10-01
Evaluation setupTask version, grader version and trials per itemGrader fixes change scoresTask version 1.0, grader version 2, five trials
Cost and latencyCost and seconds per task at this settingA baseline you cannot afford is not relevantMedian seconds and cost per task logged

How hard should the baseline be tuned?

As hard as the dataset arm, counted in the same units: the same number of prompt or hyperparameter trials, the same engineer hours and the same tuning split, kept apart from the holdout. Write the budget into the plan and log what each arm used.

Parity cuts both ways. If the dataset arm gets a learning-rate sweep and three rounds of prompt revisions while the baseline runs once at defaults, part of the gain belongs to tuning. If the baseline gets a careful search over prompt formats, report its range rather than its best single run, as Sclar and colleagues recommend, and score the dataset arm across the same formats.

Match the starting checkpoint as well. NVIDIA's Nemotron FAQ recommends starting a fine-tune from the post-trained model in most cases, because a domain fine-tune on its own should not be expected to reproduce what post-training adds. A dataset arm fine-tuned from a base checkpoint and compared with a post-trained baseline, or the reverse, mixes two changes into one result.

Separate the people where you can. When the team that hopes the dataset works also tunes the baseline, small choices drift toward the dataset arm without anyone intending it. Give baseline tuning to someone without a stake in the purchase, or freeze the baseline card before the dataset arm's first result is visible.

Which success, stop and inconclusive rules go in the plan?

Three rules against the strongest baseline, plus one for model releases, written and dated before any arm runs. Set the minimum useful difference with whoever owns the budget, in the units of the task metric: the smallest gain over the strongest rival that would repay the data and the integration work.

At a glance

Which success, stop and inconclusive rules go in the plan

Write the threshold as an absolute difference over the strongest rival, not as a relative improvement over a weak one. A large relative gain over a sandbagged baseline can shrink to almost nothing over a tuned one, and only the second number tells the budget owner what the data buys.

Score per item and over repeated trials. Anthropic's evals guide notes that model outputs vary between runs and that pass^k, the probability that all k trials succeed, falls as k rises; a dataset that lifts single-try success but not consistency improved the average case, not reliability. Report both when the deployment needs consistent behavior. The rules:

  • Success: on your held-out tasks, the lower interval bound of the dataset arm minus the strongest baseline clears the minimum useful difference, and no regression check fails.
  • Stop: the strongest baseline already meets the acceptance threshold, or a cheaper rival matches the dataset arm within the minimum useful difference. Either way the dataset has nothing left to prove.
  • Inconclusive: the interval crosses the minimum useful difference. Extend once, as preplanned, with more items or trials, apply the same rule, then stop.
  • Model release mid-pilot: finish against the frozen card, then run the new release as a separate, preplanned check rather than swapping the baseline halfway.

Would the gain survive a stronger rival, a new holdout and more trials?

A positive result earns belief only after three attempts to make it disappear, plus a format check, all built into the plan rather than run after someone dislikes the number. Each targets a different way the gain could be an artifact:

  • Stronger rival: rerun the best candidate at its highest practical effort, with retrieval and a format search. The GDPval paper reports that more reasoning effort, more task context and more scaffolding each improved model performance, so a gain that survives only against a lightly configured rival is not a gain over today's best.
  • New holdout: rescore on items written after the plan was frozen, or drawn from a later period, so neither arm was tuned against them.
  • More trials: repeat every arm with new seeds and more trials per item. A difference that shrinks as trials grow was noise.
  • Format range: report the dataset arm's margin across the same prompt formats used for the baseline. A margin that appears in one format only is fragile.

The preregistered result table against the strongest rival

Fill one row per rival after every arm finishes, read the primary row first, and publish the table with the frozen baseline cards. No values appear here because this article reports no pilot.

Methods note: the rival table, the baseline card and this result table were drafted for this article on 2026-10-26, drawing on Anthropic's guide to agent evals (January 2026) and its effort and model deprecation documentation, OpenAI's GDPval paper (2025) and GPT-5.4 announcement (March 5, 2026), NVIDIA's Nemotron training FAQ, Sclar and colleagues (2023), Melis, Dyer and Blunsom (2017), and the Hugging Face Transformers and EleutherAI harness documentation. Not measured: how often datasets beat the strongest current baseline, typical margins, how much tuning a baseline needs, or how quickly margins shrink across model releases.

The preregistered result table against the strongest rival
ComparisonBaseline cardRecord after the runPreset reading
Dataset arm minus production systemFrozen live configurationPaired mean difference, interval, items, trialsMust be positive; otherwise stop
Dataset arm minus strongest rivalBest of the tuned candidatesSame fields as the first rowPrimary result: success, stop or inconclusive
Dataset arm minus rival with retrievalRival plus your document indexSame fields as the first rowA gap that closes means the gain was missing context
Dataset arm minus customized open-weight modelPost-trained checkpoint tuned with equal effortSame fields plus cost per taskIf matched at lower cost, prefer the cheaper rival
Margin across prompt formatsOne format set for both armsRange of differences across formatsA margin in one format only is fragile
Dataset arm minus human processBlind-graded sample of current workWin, tie and loss countsContext for the business case, not the primary rule

Writing the strongest baseline into a data request

Send the baseline card with the request, so a supplier sources the cases your strongest rival still fails rather than cases it already handles. Because SourceX sources each dataset from U.S. businesses on request instead of from stock, the request is where the bar gets set: alongside the form's fields for scope, volume, language, format, timing and licensing needs, include the intended use, the rival's configuration and failure cases, and the acceptance test.

Describing where today's best model fails, for example multi-step exceptions with recorded outcomes, helps a supplier judge fit. That description is a sourcing brief, not a promise that such data exists, outperforms your baseline or improves your model. Terms come first on the supply side too: the record holder has to sign, on a scope and terms it approved, before releasing any data, so the pilot sample's terms are fixed before the comparison runs.

Frequently asked questions

What is a current model baseline in a dataset pilot?

It is the strongest configuration you could deploy for the task today without the candidate dataset: your production system, the best available model at its best setting with tools and retrieval, a customized open-weight model, or the current human process. A dataset is worth buying only if it beats that bar by a margin set in advance.

Should the baseline be a frontier API model or an open-weight model?

Test both if both fit your constraints, and keep the stronger as the bar. NVIDIA's Nemotron FAQ reports a customized small model beating proprietary comparators in several specialized domains, while heavily optimized domains left customization little headroom, so only your own task set can settle it.

How much tuning should the baseline get?

The same as the dataset arm, counted in the same units: prompt and hyperparameter trials, engineer hours and a shared tuning split kept apart from the holdout. Report the baseline's range across prompt formats rather than one favorable run.

What if the strongest baseline already passes the acceptance test?

Stop the pilot. The dataset has nothing left to prove for this task, and the budget is better spent on the cases the baseline still fails, which you can describe in a narrower data request.

What if the baseline model is retired before the pilot ends?

Record the retirement date on the baseline card from the start and schedule reruns before it. Anthropic, for example, says requests to retired models fail and gives at least 60 days' notice before retiring a publicly released model.

Sources

  • Anthropic's guide to agent evals (January 2026) says evaluating an agent evaluates the harness and model together, reports Claude Opus 4.5 moving from 42 to 95 percent on CORE-Bench after grading fixes and a less constrained scaffold, and explains multiple trials and pass^k. Source
  • Anthropic's effort documentation says most Claude models default to high effort, Claude Opus 5.5 and Claude Haiku 5.5 default to medium, and effort can be raised to max. Source
  • Anthropic's model deprecation documentation says requests to retired models fail and that publicly released models get at least 60 days' notice before retirement. Source
  • Sclar and colleagues (2023) found formatting changes in few-shot prompts moved LLaMA-2-13B accuracy by up to 76 points and recommend reporting a range across plausible formats. Source
  • Melis, Dyer and Blunsom (2017) found that standard LSTMs, properly regularized and tuned with large-scale hyperparameter search, outperformed more recent language model architectures. Source
  • OpenAI's GDPval paper (2025) uses head-to-head expert comparison as its primary metric and reports that more reasoning effort, task context and scaffolding improve model performance. Source
  • OpenAI's GPT-5.4 announcement (March 5, 2026) reports GDPval results with GPT-5.4 at xhigh reasoning effort and GPT-5.2 at heavy. Source
  • NVIDIA's Nemotron training FAQ reports a customized Nemotron 3 Nano beating proprietary comparators in several specialized domains, advises evaluating on your own task set, and recommends starting from the post-trained model in most cases. Source
  • Hugging Face's Transformers documentation says greedy search is the default decoding strategy and, unless set in GenerationConfig, generates a maximum of 20 new tokens. Source
  • EleutherAI's evaluation harness task guide describes versioning tasks, including a version bump that fixed an answer-extraction bug which had led to underestimated performance. Source

Tell us what data you need

Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.

See if your company qualifies