AI data market
What should a post-training data pilot scorecard measure beyond model accuracy?
By SourceX Editorial · Updated
9 sections
Short answer
A post-training data pilot scorecard should show whether the candidate examples fix a named model failure on held-out business tasks, cohort by cohort, without regressing existing capabilities. Add lanes for data quality and lineage, residual identifiers and verbatim extraction, and cost per fixed failure. Fix pass rules before training, match the baseline budget and freeze the evaluation set, so results map to scale, narrow or stop.
Value drivers
Post training data pilot
Key takeaways
- Aggregate accuracy can rise while the failure you bought data for stays unfixed, so score held-out business tasks by failure cohort.
- A matched training budget, a rerunnable selection rule and a frozen, decontaminated test set keep the comparison about data rather than compute or leakage.
- Regression suites belong in every pilot because narrow fine-tuning data can erode capabilities the model already had.
- The privacy lane checks both training examples for residual identifiers and the trained model for verbatim completions of planted canaries.
- Pass rules written before training turn the result into a scale, narrow, fix or stop decision instead of a debate.
What should a post-training data pilot scorecard measure beyond model accuracy?
A post-training data pilot scorecard should measure whether the candidate examples fix a specific model failure on held-out business tasks, without breaking anything the model already did well, at a cost and privacy exposure the team can accept. Aggregate accuracy on a public benchmark answers none of those questions. It can rise while the failure you bought data for stays put, or fall slightly while the business task improves.
The scorecard in this guide has four lanes: task outcome by failure cohort, regression on existing capabilities, data quality and lineage, and privacy exposure, plus a cost line that turns the result into a per-fix figure. Each lane has a pass rule written before the pilot runs. The point is to make the scale, narrow or stop decision mechanical once results arrive, so nobody argues the numbers into a story afterward.
This framing follows a direction that evaluation practice has already taken. Stanford CRFM describes its HELM framework as including metrics for measuring aspects beyond accuracy, such as efficiency, bias and toxicity. A data pilot needs the same breadth, pointed at one decision: should this data source, selected this way, go into the next training run?
Start from one named failure and the cohorts behind it
Write the failure down as a sentence a support lead or claims manager would recognize, then split it into cohorts. A useful failure statement names the task, the wrong behavior and the business consequence, for example: the assistant approves refunds outside the return window when the order had a partial shipment. A vague target such as better customer support reasoning cannot be scored.
Cohorts are the slices where the failure concentrates. Build them from the model's own error log on real tasks: sort wrong outputs by the condition that triggered them, such as order type, policy version, document length or the presence of an attachment. Keep two to six cohorts that together cover most errors, plus one control cohort of tasks the model already handles well.
The Model Cards paper by Mitchell and colleagues argues for evaluation disaggregated by relevant groups rather than a single average, and the same logic applies here. An average over all tasks hides the case where the pilot data fixes the largest cohort and quietly damages a smaller one. The scorecard reports every cohort separately and treats the aggregate as a summary line, never the decision line.
The quality, cost, privacy and task outcome scorecard
Use the table below as the pilot's pre-registered scorecard. Fill the pass rule column before any training run, and attach the named evidence artifact to each row when results come in. A row without its artifact counts as not measured.
| Lane and measure | How to compute it | Pass rule to fix in advance | Evidence artifact |
|---|---|---|---|
| Task outcome: target cohort success rate | Score held-out tasks from each failure cohort against the verified business outcome, not a plausible-looking answer | Target cohorts improve by a stated margin beyond run-to-run noise | Per-task results file with cohort labels and grader notes |
| Task outcome: control cohort | Score tasks the baseline already passes | No drop beyond the noise band | Same results file, control rows |
| Regression: general capability suite | Rerun the team's standing internal suite and any public benchmarks it tracks | Each tracked suite stays within its noise band | Suite report for baseline and candidate on the same commit |
| Quality: label and outcome validity | Audit a random sample of pilot examples against source records | Agreed error tolerance met on the audited sample | Audit sheet with sample IDs and reviewer decisions |
| Quality: duplication and contamination | Check overlap between pilot data and every evaluation set | Zero overlapping evaluation items, or flagged items removed and rescored | Overlap report and list of excluded items |
| Lineage | Trace every example to a source system, export date, selection rule and license scope | Complete trace for all examples | Manifest file plus the selection rule text |
| Privacy: residual identifiers | Scan a sample of training examples for names, contact details and account numbers | Rate at or below the threshold in the data agreement | Scan log with sample size and reviewer confirmations |
| Privacy: extraction probe | Prompt the trained model with planted or known record prefixes and check for verbatim completions | No verbatim completion of a planted canary or real record | Probe script, prompts and outputs |
| Cost: per fixed failure | Total pilot effort in reviewer hours and compute hours, divided by target-cohort failures fixed | Below the ceiling the team set for scaling | Effort log and the cohort result counts |
Keeping the comparison fair: budget, selection rule and an untouched test set
A fair comparison holds everything constant except the data, which means a matched budget, a written selection rule and an evaluation set nobody touched while choosing examples. Without those three controls, any improvement could come from more training steps, from cherry-picked examples or from test items that leaked into training.
Match the budget in training tokens or examples, not in wall-clock time. Run the baseline arm with the same number of examples drawn from the team's current data mix, and keep hyperparameters, seeds and the base checkpoint identical. If a candidate source only helps when it gets more steps than the baseline, the pilot has measured compute, not data.
Write the selection rule as a filter another engineer could rerun: which source system, which date range, which record types and which exclusions. A rule such as closed tickets from one queue, with a resolution code and at least one agent reply, is reproducible. Hand-picked examples are not, and they will not scale to a full license.
Freeze the held-out evaluation set before anyone looks at candidate data. The EleutherAI evaluation harness documentation notes that downstream evaluations are less useful when test data also exists in the training set, and describes flagging contaminated test documents by n-gram overlap so a clean version of the benchmark can be scored. Run the same kind of overlap check between pilot examples and your own business evaluation set, and report both the raw and decontaminated scores.
Finally, measure noise. Run each arm with at least two seeds, or rescore the baseline twice, and use the spread as the noise band in every pass rule. A cohort gain smaller than the spread between two baseline runs is not a gain.
Which failure cohort improved, and did an existing capability regress?
Report the result as a cohort table first: baseline and candidate success on each failure cohort, the control cohort and each general suite, side by side. The question the scorecard answers is not whether the average went up but which cohort moved, by how much relative to noise, and what moved the other way.
Regression checks are not a formality. Luo and colleagues studied continual instruction tuning and report that catastrophic forgetting appears across models in the 1B to 7B range they tested, covering domain knowledge, reasoning and reading comprehension. A narrow pilot dataset can teach the refund rule and blunt the model's handling of unrelated policies, and only a standing regression suite will show it.
Treat three patterns differently. A target cohort gain with flat controls and flat suites is the clean case. A target gain with a control or suite drop beyond noise means the data needs mixing with general examples, or the selection rule is pulling in a skewed slice. A gain concentrated in one target cohort while the others stay flat usually means the source covers one condition well and the rest thinly, which points to narrowing the scope rather than buying more of everything.
NIST's AI Risk Management Framework asks that accuracy measurements be paired with clearly defined and realistic test sets representative of conditions of expected use, and that test sets, metrics and tools be documented. A pilot scorecard that records the cohort definitions, the frozen test set version and the grader instructions meets that bar and lets a later reviewer reproduce the decision.
Testing whether the pilot data leaks into model outputs
The privacy lane asks whether personal or confidential details survived preparation and whether the trained model can reproduce them. It is measured on both sides: the training examples before the run and the model's outputs after it.
On the data side, sample pilot examples and scan them for identifiers the de-identification step should have removed, then have a person confirm each hit. Record the sample size and the confirmed rate, because a scanner's raw hit count mixes true residue with false alarms. The threshold belongs in the data agreement, and the pilot should fail if the confirmed rate exceeds it, whatever the task results say.
On the model side, run an extraction probe. Carlini and colleagues showed at USENIX Security 2021 that an attacker could recover verbatim training sequences from a language model by querying it, including names, phone numbers and email addresses, some of which appeared in only one training document. A pilot does not need a full attack. Plant a few unique canary strings in the pilot data, prompt the trained model with their prefixes, and check whether it completes them; do the same with prefixes of a small sample of real records.
A canary completion is a stop signal for scaling, not a footnote. It means the training setup memorizes rare strings from this source, and the fix sits in preparation, deduplication or training configuration before any larger license is signed.
Illustrative: refund-policy errors in a support assistant
All figures in this example are illustrative counts and percentages, not results from a real pilot. A team running a customer support assistant finds it approves refunds outside policy in three situations: partial shipments, orders paid with store credit and returns past the window on subscription items. They define those as cohorts A, B and C and add a control cohort of standard in-window returns the model already handles.
They license a candidate set of 4,000 resolved tickets from one helpdesk queue, selected by a written rule: closed tickets with a refund decision code, an agent reply and a linked order record. The baseline arm gets 4,000 examples from their existing mix. Both arms train from the same checkpoint with two seeds each, and the frozen evaluation set holds 300 tasks per cohort.
In this illustration, cohort A success rises from 52% to 81% and cohort B from 47% to 70%, both far outside the 3-point spread between seeds. Cohort C barely moves, 40% to 42%, inside noise. The control cohort holds at 94%, but the team's general policy suite drops 5 points on warranty questions, outside noise. The overlap check finds 11 evaluation tasks that share long passages with pilot tickets; rescoring without them changes cohort A by 1 point. The privacy scan confirms identifiers in 2 of 500 sampled examples, under the agreed threshold, and no canary is completed.
The exception is cohort C. Reading the tickets shows the queue rarely handled subscription items, so the source cannot fix that failure at any volume. The team's decision is to scale the license for partial-shipment and store-credit tickets only, mix in general warranty examples to recover the regression, and look for a different source for subscription returns. The scorecard made that a narrow decision rather than a yes or no on the whole source.
Reading the result: scale the data, narrow it or stop
Map the cohort table to a decision with rules written before the pilot. The table below is a starting point; adjust thresholds to your noise band and risk tolerance, but do not change them after results arrive.
- Stop and request specialist review when a privacy probe fails, when the source's license scope does not cover the training use, or when the evaluation tasks involve regulated decisions your graders are not qualified to judge.
- Narrow rather than stop when one cohort improves cleanly, because a smaller, well-matched source is often worth more than a broad one that dilutes the fix.
| Pattern in the scorecard | Decision | What to do next |
|---|---|---|
| Target cohorts improve beyond noise, controls and suites flat, privacy and lineage pass | Scale | License more of the same source under the same selection rule and rerun the scorecard at the larger size |
| Gain in some target cohorts only | Narrow | Restrict the selection rule to the conditions that improved and find another source for the rest |
| Target gain with a regression beyond noise | Fix then retest | Mix in general examples or rebalance, then rerun the same arms before scaling |
| Gain disappears after decontamination | Stop | Treat the earlier result as leakage and rebuild the evaluation set before testing again |
| Confirmed identifiers above threshold or a canary completion | Stop scaling | Fix preparation or training configuration and repeat the privacy lane before any larger delivery |
| Lineage incomplete for part of the data | Hold | Remove untraceable examples, rescore, and decide on the traceable subset only |
Sourcing operational records for the next pilot round
The lineage and privacy rows of the scorecard depend on how the pilot data was sourced. Datasheets for Datasets, by Gebru and colleagues, poses questions a pilot manifest should answer, including whether information is missing from individual instances and why. Operational records licensed from a real business can answer them, because the export date, source system and selection rule are known.
Asked for records that match a set of failure cohorts, SourceX looks for U.S. businesses whose operational data fits, since it holds no stock, and the buyer's request can spell out the cohorts, volume, format and licensing needs. Rights review, preparation and delivery follow only after the supplying company has agreed the terms and signed, so the lineage row starts from a documented trail.
Bring the scorecard to that conversation. The cohort definitions and selection rule tell a supplier which records matter, and the pass rules set expectations for what a follow-on license must show.
What has to be frozen and registered before the first training run?
Freeze the configuration and the analysis plan in one dated record that someone outside the pilot team receives before any training starts. A skeptical reviewer's first question is whether anything moved after the results were seen: the evaluation set, the grader, the baseline, the metric or the decision threshold. The record answers it, and the scorecard's pass rules become one line of it rather than the whole protocol.
Registration can be simple. Send the record to the data supplier, a second internal team or the reviewer who will sign the decision memo, or commit it to a repository whose history cannot be quietly rewritten. Every later change goes into a deviations log with the date, the reason and whether any results had been seen. A change made after unblinding is reported as exploratory and never folded into the primary result.
| Frozen item | What the record states | Why a skeptic asks | Allowed after the freeze |
|---|---|---|---|
| Decision and minimum useful effect | The purchase decision the pilot informs and the smallest target-cohort gain worth paying for | Without a stated minimum, any positive gap can be called a win | Nothing; a new threshold means a new pilot |
| Model versions | Base checkpoint identifier, training code commit and inference settings for both arms | A silent model or code change can explain the whole difference | Bug fixes, logged as deviations and applied to both arms |
| Data snapshots | Candidate and baseline example files with content hashes, the selection rule and the sampling seed | Hand-swapped examples turn a test into a demonstration | Removal of contaminated or unlicensed items, logged with counts |
| Budget and runs | Examples or tokens per arm, steps, hyperparameters, seeds and the number of runs per arm | Extra runs until one looks good are a hidden search | None; any added run is reported with all earlier ones |
| Evaluation set and exclusions | Set version, task counts per cohort and the written rules for excluding a task | Tasks dropped after scoring can manufacture a gain | Exclusions under the written rules only, each listed with its reason |
| Grader | Rubric version, grader model and prompt, the human calibration sample and the agreement level required | A grader that drifts or favors one style changes the verdict | Recalibration only before unblinding |
| Analysis plan | Primary metric, paired comparison, interval method, clustering unit and decision rules | Choosing the statistic after seeing results inflates false wins | Extra analyses, labeled exploratory |
How do you randomize, blind and calibrate graders so the result holds up?
Randomize every choice a person could otherwise steer, hide which arm produced each output, and show that the grader agrees with qualified people before its scores count. These controls answer the reviewer who suspects the result came from selection, expectation or a lenient grader rather than from the data.
Calibrate the grader on a sample that two qualified people have already labeled. Anthropic's guide says LLM-as-judge graders should be closely calibrated with human experts. Measure grader-to-human agreement against human-to-human agreement on the same sample and write the allowed gap into the frozen record. GDPval sets a useful bar: its authors report automated grader agreement within 5 percent of human inter-rater agreement on their gold subset, yet kept head-to-head comparison by human experts as the primary metric because automated grading of real work is hard.
- Randomization: draw the candidate training sample from the selection rule's output with a recorded seed, and shuffle evaluation outputs before grading so neither arm is always scored first.
- Blinding: strip arm labels, model names and formatting tells from outputs before graders see them, and keep the key with someone who does not grade.
- Repeated trials: run each task more than once, each from a clean environment. Anthropic's engineering guide to agent evals (January 9, 2026) notes that model outputs vary between runs and that shared state between runs can cause correlated failures.
- Reliability as well as capability: the same guide defines pass@k as the likelihood of at least one correct solution in k attempts and pass^k as the probability that all k trials succeed. Report pass^k for the target cohort, because a buyer pays for a fix that holds every time.
- Excluded cases: list every task removed after the freeze with the rule that removed it, and report the primary result with and without those tasks.
What belongs in one result table, including the rows that went against you?
One table should carry every pre-registered metric for both arms, the paired difference with its interval, the threshold set in advance and a verdict, including the rows where the candidate lost. A reviewer shown only the improved cohort will assume the rest were dropped; regressions, cost and exclusions sitting in the same table are the best evidence that nothing was.
Compute the intervals the way evaluation statisticians recommend. Anthropic's research note on error bars for evals (November 19, 2024) advises reporting the standard error of the mean with each score, clustering standard errors on the unit of randomization, resampling answers several times to reduce variance within questions, and analyzing paired differences when two models answer the same questions. In a data pilot the clustering unit is usually the source case, because tasks built from one customer thread or one contract are not independent.
| Row | What to report for each arm | Comparison | Verdict rule |
|---|---|---|---|
| Target cohort success | Mean pass rate and pass^k per cohort, with standard errors | Paired difference on the same tasks, with a clustered interval | Lower bound above the minimum useful effect |
| Control cohort and regression suites | Pass rate per suite on the same code commit | Paired difference with interval | Interval rules out a drop larger than the noise band |
| Grader agreement | Grader-to-human and human-to-human agreement on the calibration sample | Gap between the two | Gap within the frozen limit, or the run is not scored |
| Cost per fixed failure | Reviewer hours, compute hours and licensing cost per arm | Cost divided by target failures fixed | Below the ceiling in the decision record |
| Exclusions and deviations | Count of excluded tasks and of logged deviations | Primary result with and without the exclusions | Verdict unchanged by the exclusions |
| Runs | Runs planned and runs performed | All runs pooled, none dropped | Performed equals planned, or the extra runs are disclosed |
Purchase, reject or gather more evidence: what does the decision memo conclude?
The memo concludes purchase when the interval clears the minimum useful effect and every guard holds, reject when the interval rules that effect out, and gather more evidence when the interval is too wide to decide. The third outcome is the one many pilots need and few write-ups admit; calling a wide interval a success is how a pilot fails review later. The scorecard's decision table above maps result patterns to actions, and the memo adds the interval test that decides whether a pattern is real.
Gathering more evidence means a plan, not a rerun. Anthropic's note recommends power analysis to calculate how many questions an evaluation needs to test a hypothesis; use the observed variance and the minimum useful effect to size the next round, then freeze that round the same way and report both rounds together. Repeating the pilot until one run clears the bar, and reporting only that run, is exactly what a skeptical reviewer is looking for.
- The decision and the verdict, with the frozen record's date and the deviations log attached.
- Every row of the result table, favorable or not, and the cohorts where the candidate lost.
- What the pilot could not show: conditions missing from the evaluation set, risks outside the scorecard and any rights or privacy question still open.
- The evidence that would reverse the verdict, so the next reviewer knows what to check.
| Target cohort interval | Guards | Memo verdict | Next step |
|---|---|---|---|
| Lower bound above the minimum useful effect | All pass | Purchase | License at scale under the same selection rule and rerun the scorecard on the larger delivery |
| Lower bound above the minimum useful effect | A regression, privacy or lineage guard fails | Hold | Fix the failed guard and repeat the frozen protocol before any purchase |
| Interval contains the minimum useful effect | Any | Gather more evidence | Size a second round by power analysis and freeze it before training |
| Upper bound below the minimum useful effect | Any | Reject | Record the negative result and stop testing this source under this selection rule |
| Upper bound below zero | Any | Reject | Check the pipeline for faults, then record the harm the candidate data caused |
Sources for this section
- Anthropic's engineering guide to agent evals (January 9, 2026) says model outputs vary between runs, that trials should start from a clean environment because shared state can cause correlated failures, defines pass@k and pass^k, and says LLM-as-judge graders should be closely calibrated with human experts. Source
- Anthropic's research note on statistical model evaluation (November 19, 2024) recommends reporting the standard error of the mean, clustering standard errors on the unit of randomization, resampling answers to reduce variance within questions, analyzing paired differences and using power analysis to set how many questions an eval needs. Source
- The GDPval paper reports automated grader agreement within 5 percent of human inter-rater agreement on its gold subset and uses head-to-head human expert comparison as its primary metric because automated grading is hard. Source
Frequently asked questions
How large should a post-training data pilot be?
Large enough that the target cohort result clears run-to-run noise and small enough to finish in one training cycle. Size it from the evaluation side: decide how many held-out tasks per cohort you need for a stable score, then choose a training sample that a matched baseline can also receive. Scale only after the scorecard passes.
Can a public benchmark replace the held-out business tasks?
No. Public benchmarks are useful in the regression lane to catch general damage, but they rarely contain the failure you are buying data to fix. The decision lane should be tasks drawn from your own error log and scored against verified business outcomes.
What if the improvement comes from more training steps rather than the data?
That is what the matched budget prevents. Give the baseline arm the same number of examples or tokens from your current mix, with the same checkpoint, seeds and settings. If the candidate only wins with extra steps, the pilot measured compute rather than data value.
Who should grade the task outcomes?
People who know the business rule being tested, working from written grading instructions and the verified outcome for each task. For regulated areas such as lending or insurance decisions, use qualified reviewers. Record grader notes so a disputed task can be rechecked.
Should privacy failures block a pilot that improves the model?
Yes for scaling. A confirmed identifier rate above the agreed threshold or a verbatim canary completion means the preparation or training setup leaks, and a larger license would multiply that exposure. Fix it, rerun the privacy lane, then decide on scale.
Sources
- Stanford CRFM's HELM framework includes metrics for aspects beyond accuracy, such as efficiency, bias and toxicity. Source
- Evaluations are less useful when test data also appears in training; contaminated test documents can be flagged by n-gram overlap and a clean score reported. Source
- NIST AI RMF 1.0 pairs accuracy with clearly defined, realistic test sets representative of expected use and asks that test sets, metrics and tools be documented. Source
- Model Cards propose evaluation disaggregated across relevant groups instead of a single average. Source
- Catastrophic forgetting was observed during continual instruction tuning across 1B to 7B models in domain knowledge, reasoning and reading comprehension. Source
- Querying a language model recovered verbatim training sequences, including names, phone numbers and email addresses, some present in only one document. Source
- Datasheets for Datasets asks whether information is missing from individual instances and why. Source
Tell us what data you need
Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.