AI uses for records
Dataset pilot hard case reporting: count every case before you average
By SourceX Editorial · Updated
9 sections
Short answer
Dataset pilot hard case reporting starts with the denominator: report every case you assigned, not only the ones that scored cleanly. Fix hard-case strata from attributes known before any run, allow exclusions only from a closed list decided blind to arm and applied to both arms, and count model-caused unscorable cases as failures. Then give the hard stratum its own row; a gain that disappears there is not a general gain.
Agent run
Dataset pilot hard case
Key takeaways
- Difficult cases rarely get deleted on purpose; they leave through timeouts, grader flags and missing references, and each exit inflates the headline.
- Define the hard stratum from attributes recorded before the first run, never from which cases the baseline happened to fail.
- Only three exclusion reasons survive review: leakage found before runs, a wrong reference confirmed blind to arm, and infrastructure failure that persists after reruns; each removes the case from both arms.
- Model-caused timeouts, malformed outputs and refusals are failures, not exclusions.
- In the illustrative pilot, dropping unscorable cases turns a 7-point gain into 12 and makes a slightly worse hard stratum look 10 points better.
What does a dataset pilot owe its hardest cases?
A full count and a separate row. A dataset pilot that compares a model trained with a candidate corpus against the same model without it should report results on every case it assigned, show how many cases left the analysis and why, and break the result out for a hard-case stratum defined before any run. Without those three things, a reader cannot tell a real gain from an easier test.
The decision this protects is narrow and expensive: whether to license or buy more of a corpus because a pilot said it helps. Hard cases are where that decision is usually won or lost. Multi-record disputes, policy exceptions and incomplete files are the work a buyer struggles to automate, and they are much of the reason operational records interest model developers at all.
Useful evidence has four parts: a case-flow ledger from sampling frame to analyzed cases, the exclusion reasons with who decided each one and when, the all-assigned result printed beside the as-scored result, and the hard stratum on its own row with its count. The protocol below fixes each part before the first training run.
How do difficult cases slip out of a pilot without anyone choosing to drop them?
Through the scoring pipeline. Hard cases are the ones most likely to time out in the harness, produce output the grader cannot parse, arrive without a usable reference answer or get flagged as ambiguous by a reviewer. When each of those events quietly removes a case from the denominator, the pilot ends up grading an easier test than the one it planned.
Some of those cases really are broken, which is what makes the problem hard. Anthropic's January 2026 guide to agent evaluations says a 0% pass rate across 100 trials most often signals a broken task rather than an incapable agent, and describes a model whose CORE-Bench score rose from 42% to 95% once grading bugs and ambiguous task specifications were fixed and the scaffold was loosened. In a post explaining why it no longer reports SWE-bench Verified (checked October 26, 2026), OpenAI says an audit of problems that models often failed found at least 59.4% had flawed tests that rejected functionally correct submissions.
The lesson is not that failed hard cases can be dropped. It is that a case can be declared broken only by a written test, applied before results are unblinded and to both arms at once. Otherwise every inconvenient failure starts to look broken, and only one arm's failures get forgiven.
Filtering by difficulty also happens on purpose, for good reasons. NVIDIA's Nemotron training documentation (repository checked October 26, 2026) describes dropping reinforcement-learning prompts that the starting checkpoint already solves every time, and choosing 3,000 held-out science problems with pass rates between 0.25 and 0.80 for evaluation. That is sensible for a training signal, but a set selected on pass rate cannot support a claim about a buyer's whole workload, and a pilot report has to say which kind of set it used.
How should the hard-case stratum be defined before the first run?
From attributes of the case that are known before any model sees it, never from the outcome. If the hard stratum is defined as the cases the baseline failed, the candidate will look better on them through regression to the mean alone, because some baseline failures were bad luck that would not repeat on a second run.
At a glance
How should the hard-case stratum be defined before the first run
Aggregate scores hide this kind of subgroup routinely. Oakden-Rayner and colleagues (arXiv, 2019) showed that medical imaging models with strong overall results could have relative performance differences of more than 20% on clinically important subsets that the training labels did not separate. Mitchell and colleagues' model card proposal (2019) asks for evaluation results broken down by groups and domain-relevant conditions for the same reason.
In business records, observable hardness usually comes from structure the system of record already stores. Pick two to four markers like the ones below, write the rule, and count the stratum before training:
- Linked records: a Jira issue with three or more linked issues, or a NetSuite sales order that touches more than one subsidiary or currency.
- Exception paths: a Zendesk ticket reopened after closure, escalated a tier, or closed with a policy override note.
- Missing or conflicting inputs: a claim file with an absent document, or two sources that disagree on a date or amount.
- Rare categories: case types below an agreed share of the sampling frame, for example under 2% of closed cases in the window.
- Long resolutions: cases whose reference resolution needed more than an agreed number of actions or tool calls.
Which exclusions are legitimate, and who decides them?
Three exclusion reasons survive scrutiny, and each removes the case from both arms; everything else stays in the denominator. Clinical trials settled the reporting standard long ago: CONSORT 2025 asks every trial to report, for each group, how many participants were randomly assigned, received the intended intervention and were analyzed for the primary outcome, with losses and exclusions reported as a separate item.
For a dataset pilot, the analogue is a closed list of reason codes written before training, an adjudicator who cannot see which arm produced a failing output, and a rule that unscorable output caused by the model counts as a failure. When OpenAI built SWE-bench Verified, professional developers screened the original tasks for underspecified issue descriptions and unit tests that could reject valid solutions before models were scored; that ordering, screening first and scoring second, is the one to copy.
The exclusion rule table below is the first half of the preregistered protocol. Copy it into the plan and edit the thresholds, not the logic:
| Reason code | Excludable? | Decided when, by whom | How it counts |
|---|---|---|---|
| Duplicate of, or leaked into, training data | Yes, from both arms | Before any run, by the data owner using hashed record keys | Removed and logged with a count per stratum |
| Reference answer wrong or missing | Yes, from both arms | Before unblinding, by an adjudicator who cannot see the arm | Removed and logged; repaired cases can rejoin a later round |
| Infrastructure failure unrelated to the output | Only after three reruns fail | During the run, by the harness owner from system logs | Removed from both arms if it persists; logged |
| Timeout, looping or malformed output from the model | No | Not applicable | Scored as a failure for that arm |
| Refusal or abstention where the reference expects an answer | No | Not applicable | Scored as a failure and reported as its own count |
| Grader disagreement or an ambiguous flag | No | Sent to adjudication, blind to arm | Stays in, scored by the adjudicated label |
| Out of scope, noticed after results | No | Not applicable after unblinding | Stays in; any re-analysis is labeled a sensitivity check |
The case-flow ledger: from sampling frame to analyzed cases
Every pilot report should open with a ledger that accounts for each case from the sampling frame to the final analysis, so a reviewer can rebuild the denominator without asking. It is the trial flow diagram adapted to a training experiment, with one column per arm once the arms diverge.
At a glance
The case-flow ledger: from sampling frame to analyzed cases
In the illustrative pilot used for the rest of this article, cases are sampled from a support system, the hard-stratum rule is applied before training, and the same pinned model is trained with and without a candidate corpus under one compute budget. The ledger records five checkpoints:
- Sampled frame: 412 closed cases drawn at random from the window, 308 routine and 104 meeting the hard-stratum rule, counted before training.
- Removed before runs: 8 routine cases whose records also appeared in the candidate corpus (reason code: leakage), leaving 404 assigned to both arms.
- Removed after adjudication: 4 hard cases whose reference answers were wrong, found by an adjudicator blind to arm and removed from both arms, leaving 400 cases: 300 routine and 100 hard.
- Unscorable, counted as failures: in the hard stratum, 20 baseline cases and 45 candidate cases timed out or produced output the grader could not parse; none did in the routine stratum.
- Analyzed: all 400 cases in each arm for the primary analysis, with the as-scored figures shown only as a secondary line.
Preregistered rules: adopt, reject, inconclusive or void
Write the verdict rules before training, with the model, compute budget, decoding settings, harness, grader and case list frozen, so the hard stratum cannot be renegotiated once numbers arrive. The thresholds here are editable examples; the structure is the point.
Each rule uses the all-assigned analysis and a paired interval from repeated trials. Anthropic's evaluation guide recommends multiple trials because model outputs vary between runs, so both arms run the same number of trials on the same case list:
- Adopt: the all-assigned gain is at least 5 points with its paired interval above zero, the hard stratum is no more than 3 points below baseline, and model-caused unscorable cases rise by no more than 5 per 100 in any stratum.
- Reject: the all-assigned gain is under 2 points or its interval includes zero, or the hard stratum falls more than 3 points below baseline.
- Inconclusive: anything between adopt and reject, including a gain that passes while a guard fails, or a hard stratum too small to detect the drop the plan says matters.
- Void: any exclusion was decided after unblinding, more than 5% of assigned cases left after runs began, or the arms were scored on different case lists.
Illustrative result: a 12-point gain that is really 7, and worse on hard cases
Counting every assigned case shrinks the headline gain from 12.2 to 7.0 points and reverses the direction of the hard stratum. The numbers are illustrative, constructed to show the arithmetic; they are not measured results from any pilot.
As scored, the baseline is graded on 380 cases and the candidate on 355, because each arm lost a different set of hard cases to timeouts. That is no longer a paired comparison: the candidate is graded on an easier mix. Counted as failures, the same timeouts show the candidate solving 22 hard cases against the baseline's 24.
Under the rules above, the verdict is inconclusive, not adopt. The overall gain passes and the hard stratum stays inside the 3-point margin, but model-caused unscorable cases in the hard stratum rose by 25 per 100, far past the guard of 5. The next step is diagnostic: read the transcripts of the 45 candidate timeouts, find whether longer outputs or loops caused them, and rerun both arms with the cause fixed.
| Line | Baseline | Candidate | Difference | Rule check |
|---|---|---|---|---|
| Overall, all 400 assigned cases | 234 of 400 (58.5%) | 262 of 400 (65.5%) | +7.0 points | Passes the 5-point gain test |
| Overall, as scored | 234 of 380 (61.6%) | 262 of 355 (73.8%) | +12.2 points | Secondary only; arms graded on different cases |
| Routine stratum, 300 cases | 210 (70.0%) | 240 (80.0%) | +10.0 points | Reported for context |
| Hard stratum, all 100 assigned | 24 (24.0%) | 22 (22.0%) | -2.0 points | Inside the 3-point margin |
| Hard stratum, as scored | 24 of 80 (30.0%) | 22 of 55 (40.0%) | +10.0 points | Misleading; not used for the verdict |
| Model-caused unscorable, hard stratum | 20 | 45 | +25 per 100 | Fails the 5-per-100 guard |
| Removed from both arms | 12 (8 leakage, 4 wrong references) | 12 (same cases) | None | 4 removed after runs began (1%), under the 5% limit |
Could the hard-case result be noise, label error or a weak baseline?
Any of the three can create or hide a hard-case effect, so check them before generalizing from the routine stratum's 10-point gain.
Noise first. A 100-case stratum cannot see small changes: for a pass rate near 25%, the standard error of a single proportion is about 4.3 points, so a 2-point difference sits well inside chance. Size the hard stratum for the smallest drop the buyer would act on, and pair cases across arms so per-case differences, not two separate averages, carry the test.
Label error second. Northcutt and colleagues (2021) estimated label errors averaging at least 3.3% across ten widely used test sets, and found that on corrected ImageNet labels a smaller model could overtake a larger one if the share of mislabeled examples rose by about 6%. Reference answers drawn from business records are no cleaner than benchmarks, and their errors tend to sit in the hardest files, which is why adjudication is blind to arm and applies to both arms.
Baseline strength and repetition last. A gain over a weak, untuned baseline can vanish against the strongest model the buyer could deploy without the data, and a single run can flatter either arm. Repeat the matched trials, report pass^k, the probability that all k trials succeed, beside the single-trial rate for workflows that must succeed every time, and treat a hard-stratum gain that appears in only one run as unconfirmed.
Putting hard-case reporting into a data request
State the hard-stratum rule, the exclusion list and the verdict rules in the request itself, so a supplier knows which records matter before sampling starts. That turns hard-case reporting from an argument after the fact into an acceptance test both sides can read.
For operational records, the useful characteristics are the ones that make cases hard and keep them scorable: linked records across systems, exception and override paths, the final outcome, and enough context for an adjudicator to confirm a reference answer. A request can ask for those characteristics without assuming any supplier already holds them.
SourceX does not hold datasets in stock: it sources them from U.S. businesses on request, its request form asks for scope, volume, language, format, timing and licensing needs, and adding the intended use and an acceptance test with a hard-case rule makes the requirement testable. SourceX has not published pilot results. This article describes a method, and a pilot run under it can still end inconclusive or negative.
Frequently asked questions
Is it ever acceptable to drop a test case from a dataset pilot?
Yes, for a short list of reasons written before training: the case leaked into the training data, its reference answer is wrong as confirmed by an adjudicator who cannot see the arm, or infrastructure failed repeatedly for reasons unrelated to the output. Each removal applies to both arms and appears in the case-flow ledger. Timeouts, malformed outputs and refusals caused by the model are failures, not exclusions.
Should a pilot oversample hard cases?
Often, because a small hard stratum cannot detect drops that matter. If you oversample, keep the sampling weights and report both the workload-weighted overall result and the per-stratum results, so the headline still reflects the real mix of work. Oversampling never replaces the separate hard-stratum row.
What is the difference between an as-scored and an all-assigned result?
An as-scored result divides passes by the cases each arm managed to score, so arms that lose different cases are graded on different tests. An all-assigned result divides by every case assigned to both arms and counts model-caused unscorable output as a failure. Use the all-assigned result for the verdict and show the as-scored one only as a secondary line.
How many hard cases does a dataset pilot need?
Enough to detect the smallest drop the buyer would act on. With 100 cases and a pass rate near 25%, one arm's standard error is about 4.3 points, so only large differences are visible; pairing cases across arms helps, and so does enlarging the stratum. Fix the number before training and record it in the plan.
Does SourceX publish dataset pilot results?
No. SourceX has not published pilot results, and this article describes a reporting method, not measured outcomes. Buyers can submit a task-specific requirement with an intended use and an acceptance test through the request-data form.
Sources
- Anthropic's January 2026 evaluation guide says a 0% pass rate across 100 trials most often signals a broken task, reports a CORE-Bench score rising from 42% to 95% after grading and task fixes, recommends multiple trials, and defines pass^k as the probability that all k trials succeed. Source
- OpenAI says an audit of SWE-bench Verified problems that models often failed found at least 59.4% had flawed test cases that reject functionally correct submissions. Source
- OpenAI built SWE-bench Verified by having professional developers screen SWE-bench tasks for underspecified issue descriptions and overly specific unit tests that could reject valid solutions. Source
- NVIDIA's Nemotron training FAQ describes dropping RL prompts the starting checkpoint already solves at a 100% pass rate and selecting 3,000 held-out science problems with pass rates between 0.25 and 0.80 for RL evaluation. Source
- Oakden-Rayner and colleagues found hidden stratification in medical imaging can produce relative performance differences of over 20% on clinically important subsets. Source
- Mitchell and colleagues' model card proposal (2019) calls for quantitative evaluation results broken down by groups, domain-relevant conditions and intersections of them. Source
- Northcutt and colleagues (2021) estimated at least 3.3% label errors on average across ten common test sets, and found ResNet-18 overtakes ResNet-50 on corrected ImageNet labels if the mislabeled share rises by about 6%. Source
- CONSORT 2025 item 22a asks trials to report, for each group, the numbers randomly assigned, receiving the intended intervention and analyzed for the primary outcome; item 22b covers losses and exclusions. Source
Tell us what data you need
Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.