AI uses for records
Dataset pilot vs current model baseline: the rival and its 10 settings
By SourceX Editorial · Updated
9 sections
Short answer
A dataset pilot proves value only against the strongest current model baseline you could deploy without the data: today's best available model at its best setting, with tuned prompts, tools and retrieval, plus your production system. Record that rival's exact configuration, because prompt formatting alone moved one model's accuracy by up to 76 points (Sclar and colleagues, 2023). If the dataset's gain disappears against that rival, so does the purchase case.
Agent run
Dataset pilot
Key takeaways
- The baseline that counts is the strongest configuration you could ship today without the dataset, not whichever model you happened to fine-tune.
- Configuration moves scores as much as data: Anthropic reports a model's CORE-Bench score rising from 42 to 95 percent after grading fixes and a less constrained scaffold (January 2026).
- Tune the baseline as hard as the dataset arm; in 2017, properly regularized standard LSTMs outperformed newer architectures once all were tuned with the same care.
- A ten-field baseline card, from snapshot ID to cost per task, lets anyone rerun the comparison, as long as the model has not been retired.
- If the strongest baseline already passes your acceptance test, stop the pilot: the dataset has nothing left to prove.
What must a dataset pilot's current model baseline prove?
It must show what you could achieve today without buying the data, so that any gain the dataset adds is measured against a real alternative rather than a convenient one. The dataset passes only if it beats that rival on your held-out tasks by a margin set in advance; that margin is the gain that counts.
The evidence is a paired difference: the same held-out items scored by the dataset arm and by the strongest relevant baseline, across repeated trials, with an interval read against a minimum useful difference. A gain over last quarter's model, over a base checkpoint with default settings or over a vendor's published score is an apparent gain. It shows that something changed, not that the dataset is worth buying.
This protocol covers choosing the rival, recording its configuration, tuning it fairly, the rules for success, stopping and inconclusive results, and the result table. No measured results appear here, and SourceX has published none for this design.
Which rival counts as today's strongest relevant baseline?
The strongest configuration you could actually deploy for this task without the dataset. Four filters decide relevance: you can access it through the decision date, it fits your cost, latency and data-handling limits, its terms allow your use, and it has been tuned with the same effort as the dataset arm. Usually that means testing several candidates and keeping the best as the bar.
Do not assume the largest model is the strongest relevant one. NVIDIA's Nemotron training FAQ says the team has seen a customized Nemotron 3 Nano beat the proprietary models used as comparators in several specialized-domain evaluations, that heavily optimized domains such as math and coding leave customization the least headroom, and that the only way to know for a domain is to evaluate on your own task set rather than public leaderboards alone.
Relevance filters remove candidates as often as they add them. A frontier model that cannot receive customer records under your data-handling rules is not a rival for a task that needs those records in the prompt, however strong it is, and a model whose cost per task exceeds what the workflow can bear is a ceiling to report, not a bar to beat. Record why each excluded candidate failed a filter, so a reviewer can see the baseline was not chosen for weakness.
Do not leave people out either. OpenAI's GDPval paper (2025) makes head-to-head grading against deliverables from industry professionals its primary metric, a reminder that for some business tasks the rival a dataset must beat is the current human process. The five candidates, and how each is usually weakened:
| Candidate rival | When it is the right bar | How it gets weakened | Fix before the pilot |
|---|---|---|---|
| Your production system | Always; it is what the dataset must improve on | Compared at an older snapshot or with features switched off | Freeze the live configuration and copy it exactly |
| Best available frontier model via API | When the task data can go to a hosted model | Default effort, a generic prompt, no tools | Run its highest practical effort with a tuned prompt and tools |
| Frontier model with retrieval over your documents | When missing knowledge is the failure | Retrieval left out, so the dataset gets credit for context | Index the documents you already hold and log the corpus version |
| Open-weight model customized on data you already hold | When data must stay in-house or cost per task is tight | A base checkpoint run with library defaults | Start from the post-trained checkpoint and tune with equal effort |
| Current human process | When the output is a professional deliverable | Timed on easy cases or graded by the person who did the work | Sample the same items and grade blind |
How does a baseline get weakened without anyone noticing?
Through settings nobody wrote down: defaults, prompt format, harness bugs and unequal tuning. Each can move a score by more than a typical dataset gain, so a weak baseline makes a dataset look valuable for reasons unrelated to the data. Each fix raises the bar a step: defaults, a tuned prompt, maximum effort, then retrieval and tools.
Defaults differ by model and by library. Anthropic's effort documentation says most Claude models default to high effort, that Claude Opus 5.5 and Claude Haiku 5.5 default to medium, and that effort can be raised to max for the highest capability. Hugging Face's Transformers documentation says greedy search is the default decoding strategy and, unless the generation configuration says otherwise, produces at most 20 new tokens, which silently truncates long answers from an open-weight baseline.
Prompt format and the harness matter as much. Sclar and colleagues (2023) found that formatting changes in few-shot prompts moved LLaMA-2-13B's accuracy by up to 76 points and recommended reporting a range of performance across plausible formats. Anthropic's January 2026 evals guide describes Claude Opus 4.5 scoring 42 percent on CORE-Bench until researchers fixed rigid grading and ambiguous tasks and used a less constrained scaffold, after which it scored 95 percent; the same guide says that evaluating an agent means evaluating the harness and the model together.
Unequal tuning is the oldest form of the problem. Melis, Dyer and Blunsom (2017) retuned popular language model architectures with large-scale automatic hyperparameter search and found that standard LSTMs, properly regularized, outperformed more recent models, which they traced to poorly controlled hyperparameter variation. Even vendor comparisons mix settings: OpenAI's March 5, 2026 GPT-5.4 announcement reports GDPval results with GPT-5.4 at xhigh reasoning effort and GPT-5.2 at heavy, a slightly lower level.
The baseline card: ten settings to record before the first run
Write one card per candidate rival, freeze it before the dataset arm trains, and attach it to the result. A card is complete when someone outside the team could rerun the baseline from it and get the same configuration.
Two fields carry deadlines. API models retire: Anthropic's deprecation documentation says requests to retired models fail and that it gives at least 60 days' notice before retiring a publicly released model, so record the retirement date and schedule any rerun before it. Harness and task versions change too: EleutherAI's evaluation harness versions its tasks so users can tell iterations apart, and its task guide cites a version bump that fixed an answer-extraction bug which had led to underestimated performance.
The ten fields, each with an illustrative entry:
| Field | What to record | Why it moves the score | Illustrative entry |
|---|---|---|---|
| Model and snapshot | Exact model ID or checkpoint hash, never an alias | An alias can move to a newer version mid-pilot | Dated API model ID, or the checkpoint hash |
| Access and retirement dates | Date first queried and any announced retirement date | A retired model cannot be rerun | Queried 2026-10-26; no retirement announced |
| Reasoning effort or thinking | Effort level, thinking mode or thinking budget | Defaults differ across models | Effort set explicitly to max |
| Decoding | Temperature, maximum output tokens, stop sequences | Library defaults can truncate answers | Temperature 0; output cap above the longest reference answer |
| Prompt and format | System prompt, template and the set of formats tried | Formatting alone moves accuracy | Best of five formats, with the range reported |
| Examples in context | Number, selection rule and order of examples | Shot count and order change behavior | Five examples drawn from the tuning split |
| Tools and harness | Tool list, scaffold, harness version and commit | The harness is part of what is evaluated | Harness commit hash; three tools enabled |
| Retrieval | Corpus, snapshot date, chunking and number of passages | Missing context gets credited to the dataset | Policy manual snapshot of 2026-10-01 |
| Evaluation setup | Task version, grader version and trials per item | Grader fixes change scores | Task version 1.0, grader version 2, five trials |
| Cost and latency | Cost and seconds per task at this setting | A baseline you cannot afford is not relevant | Median seconds and cost per task logged |
How hard should the baseline be tuned?
As hard as the dataset arm, counted in the same units: the same number of prompt or hyperparameter trials, the same engineer hours and the same tuning split, kept apart from the holdout. Write the budget into the plan and log what each arm used.
Parity cuts both ways. If the dataset arm gets a learning-rate sweep and three rounds of prompt revisions while the baseline runs once at defaults, part of the gain belongs to tuning. If the baseline gets a careful search over prompt formats, report its range rather than its best single run, as Sclar and colleagues recommend, and score the dataset arm across the same formats.
Match the starting checkpoint as well. NVIDIA's Nemotron FAQ recommends starting a fine-tune from the post-trained model in most cases, because a domain fine-tune on its own should not be expected to reproduce what post-training adds. A dataset arm fine-tuned from a base checkpoint and compared with a post-trained baseline, or the reverse, mixes two changes into one result.
Separate the people where you can. When the team that hopes the dataset works also tunes the baseline, small choices drift toward the dataset arm without anyone intending it. Give baseline tuning to someone without a stake in the purchase, or freeze the baseline card before the dataset arm's first result is visible.
Which success, stop and inconclusive rules go in the plan?
Three rules against the strongest baseline, plus one for model releases, written and dated before any arm runs. Set the minimum useful difference with whoever owns the budget, in the units of the task metric: the smallest gain over the strongest rival that would repay the data and the integration work.
At a glance
Which success, stop and inconclusive rules go in the plan
Write the threshold as an absolute difference over the strongest rival, not as a relative improvement over a weak one. A large relative gain over a sandbagged baseline can shrink to almost nothing over a tuned one, and only the second number tells the budget owner what the data buys.
Score per item and over repeated trials. Anthropic's evals guide notes that model outputs vary between runs and that pass^k, the probability that all k trials succeed, falls as k rises; a dataset that lifts single-try success but not consistency improved the average case, not reliability. Report both when the deployment needs consistent behavior. The rules:
- Success: on your held-out tasks, the lower interval bound of the dataset arm minus the strongest baseline clears the minimum useful difference, and no regression check fails.
- Stop: the strongest baseline already meets the acceptance threshold, or a cheaper rival matches the dataset arm within the minimum useful difference. Either way the dataset has nothing left to prove.
- Inconclusive: the interval crosses the minimum useful difference. Extend once, as preplanned, with more items or trials, apply the same rule, then stop.
- Model release mid-pilot: finish against the frozen card, then run the new release as a separate, preplanned check rather than swapping the baseline halfway.
Would the gain survive a stronger rival, a new holdout and more trials?
A positive result earns belief only after three attempts to make it disappear, plus a format check, all built into the plan rather than run after someone dislikes the number. Each targets a different way the gain could be an artifact:
- Stronger rival: rerun the best candidate at its highest practical effort, with retrieval and a format search. The GDPval paper reports that more reasoning effort, more task context and more scaffolding each improved model performance, so a gain that survives only against a lightly configured rival is not a gain over today's best.
- New holdout: rescore on items written after the plan was frozen, or drawn from a later period, so neither arm was tuned against them.
- More trials: repeat every arm with new seeds and more trials per item. A difference that shrinks as trials grow was noise.
- Format range: report the dataset arm's margin across the same prompt formats used for the baseline. A margin that appears in one format only is fragile.
The preregistered result table against the strongest rival
Fill one row per rival after every arm finishes, read the primary row first, and publish the table with the frozen baseline cards. No values appear here because this article reports no pilot.
Methods note: the rival table, the baseline card and this result table were drafted for this article on 2026-10-26, drawing on Anthropic's guide to agent evals (January 2026) and its effort and model deprecation documentation, OpenAI's GDPval paper (2025) and GPT-5.4 announcement (March 5, 2026), NVIDIA's Nemotron training FAQ, Sclar and colleagues (2023), Melis, Dyer and Blunsom (2017), and the Hugging Face Transformers and EleutherAI harness documentation. Not measured: how often datasets beat the strongest current baseline, typical margins, how much tuning a baseline needs, or how quickly margins shrink across model releases.
| Comparison | Baseline card | Record after the run | Preset reading |
|---|---|---|---|
| Dataset arm minus production system | Frozen live configuration | Paired mean difference, interval, items, trials | Must be positive; otherwise stop |
| Dataset arm minus strongest rival | Best of the tuned candidates | Same fields as the first row | Primary result: success, stop or inconclusive |
| Dataset arm minus rival with retrieval | Rival plus your document index | Same fields as the first row | A gap that closes means the gain was missing context |
| Dataset arm minus customized open-weight model | Post-trained checkpoint tuned with equal effort | Same fields plus cost per task | If matched at lower cost, prefer the cheaper rival |
| Margin across prompt formats | One format set for both arms | Range of differences across formats | A margin in one format only is fragile |
| Dataset arm minus human process | Blind-graded sample of current work | Win, tie and loss counts | Context for the business case, not the primary rule |
Writing the strongest baseline into a data request
Send the baseline card with the request, so a supplier sources the cases your strongest rival still fails rather than cases it already handles. Because SourceX sources each dataset from U.S. businesses on request instead of from stock, the request is where the bar gets set: alongside the form's fields for scope, volume, language, format, timing and licensing needs, include the intended use, the rival's configuration and failure cases, and the acceptance test.
Describing where today's best model fails, for example multi-step exceptions with recorded outcomes, helps a supplier judge fit. That description is a sourcing brief, not a promise that such data exists, outperforms your baseline or improves your model. Terms come first on the supply side too: the record holder has to sign, on a scope and terms it approved, before releasing any data, so the pilot sample's terms are fixed before the comparison runs.
Frequently asked questions
What is a current model baseline in a dataset pilot?
It is the strongest configuration you could deploy for the task today without the candidate dataset: your production system, the best available model at its best setting with tools and retrieval, a customized open-weight model, or the current human process. A dataset is worth buying only if it beats that bar by a margin set in advance.
Should the baseline be a frontier API model or an open-weight model?
Test both if both fit your constraints, and keep the stronger as the bar. NVIDIA's Nemotron FAQ reports a customized small model beating proprietary comparators in several specialized domains, while heavily optimized domains left customization little headroom, so only your own task set can settle it.
How much tuning should the baseline get?
The same as the dataset arm, counted in the same units: prompt and hyperparameter trials, engineer hours and a shared tuning split kept apart from the holdout. Report the baseline's range across prompt formats rather than one favorable run.
What if the strongest baseline already passes the acceptance test?
Stop the pilot. The dataset has nothing left to prove for this task, and the budget is better spent on the cases the baseline still fails, which you can describe in a narrower data request.
What if the baseline model is retired before the pilot ends?
Record the retirement date on the baseline card from the start and schedule reruns before it. Anthropic, for example, says requests to retired models fail and gives at least 60 days' notice before retiring a publicly released model.
Sources
- Anthropic's guide to agent evals (January 2026) says evaluating an agent evaluates the harness and model together, reports Claude Opus 4.5 moving from 42 to 95 percent on CORE-Bench after grading fixes and a less constrained scaffold, and explains multiple trials and pass^k. Source
- Anthropic's effort documentation says most Claude models default to high effort, Claude Opus 5.5 and Claude Haiku 5.5 default to medium, and effort can be raised to max. Source
- Anthropic's model deprecation documentation says requests to retired models fail and that publicly released models get at least 60 days' notice before retirement. Source
- Sclar and colleagues (2023) found formatting changes in few-shot prompts moved LLaMA-2-13B accuracy by up to 76 points and recommend reporting a range across plausible formats. Source
- Melis, Dyer and Blunsom (2017) found that standard LSTMs, properly regularized and tuned with large-scale hyperparameter search, outperformed more recent language model architectures. Source
- OpenAI's GDPval paper (2025) uses head-to-head expert comparison as its primary metric and reports that more reasoning effort, task context and scaffolding improve model performance. Source
- OpenAI's GPT-5.4 announcement (March 5, 2026) reports GDPval results with GPT-5.4 at xhigh reasoning effort and GPT-5.2 at heavy. Source
- NVIDIA's Nemotron training FAQ reports a customized Nemotron 3 Nano beating proprietary comparators in several specialized domains, advises evaluating on your own task set, and recommends starting from the post-trained model in most cases. Source
- Hugging Face's Transformers documentation says greedy search is the default decoding strategy and, unless set in GenerationConfig, generates a maximum of 20 new tokens. Source
- EleutherAI's evaluation harness task guide describes versioning tasks, including a version bump that fixed an answer-extraction bug which had led to underestimated performance. Source
Tell us what data you need
Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.