AI data market
Post-training failure cohort analysis: which error cohort improved?
By SourceX Editorial · Updated
10 sections
Short answer
To see which error cohort improved after a dataset purchase, define the failing cohorts and the capabilities that must not slip before the data arrives, freeze held-out cases for both, and train two arms at a matched budget: one with the purchased examples, one without. Report each cohort's paired change with an interval, the cases that flipped each way and any guard cohort that regressed.
Value drivers
Post training failure cohort
Key takeaways
- Define target cohorts and guard cohorts with written filters and freeze their held-out cases before the purchased data arrives.
- Compare a with-purchase arm against a control trained at the same budget on existing data, with several seeds per arm.
- Report each cohort's paired change with an interval, plus the cases that flipped from wrong to right and right to wrong.
- Fix the verdict rule in advance: improved, no clear change or regressed, decided by the interval.
- Rule out overlap, rubric changes, shifting membership and judge changes before calling a cohort improved.
How do you measure which error cohorts improved after a dataset purchase?
Name the failure the purchase was meant to fix as one or more target cohorts, name the capabilities that must survive as guard cohorts, and freeze held-out cases for each before the data arrives. Then train a with-purchase arm and a control arm at the same budget, score both on the frozen cases and report the paired change per cohort.
The paired, per-cohort report is the point. An overall score can rise while the cohort you paid to fix stays flat, because the gain came from easier cases elsewhere. It can also rise while a guard cohort quietly gets worse. A post-training failure cohort analysis shows both on one page, in a format a reviewer can check line by line.
This page gives the reporting template and the rules behind each field. It assumes the purchase is already chosen and delivered; how to design ablations or price the next tranche are separate questions.
Pin the error cohort down before the purchased data arrives
Define each cohort with a written filter on real held-out cases before anyone sees the purchased data, because a cohort drawn after the results are in will tend to be drawn around the improvement.
Error analysis tools work this way. Microsoft's Responsible AI Toolbox describes Error Analysis as identifying cohorts with a higher error rate than the overall benchmark, and asks users to look at each cohort's error rate, error coverage and data representation. All three belong in the definition: a cohort with a high error rate but almost no cases is not where a purchase should be judged.
Write the filter from the business's own vocabulary where you can, such as a claim type, a ticket category or a policy code, rather than from the model's confusion. Business categories survive a model change; clusters of model embeddings do not, and they make the before and after comparison harder to defend.
Model cards made disaggregated reporting a norm: Mitchell and colleagues proposed breaking quantitative results down by relevant groups and conditions, including intersections of two or more. The same logic applies to business failure modes, where the hard case is often an intersection, such as a damage claim filed after the delivery date.
Target cohorts, guard cohorts and the rule that ties examples to each
Every report needs both kinds of cohort. Target cohorts are the failures the purchase was bought to fix. Guard cohorts are behaviors the model already handles and must keep: routine requests, refusals the business requires, formats downstream systems parse.
Guards are not a formality. Luo and colleagues studied continual instruction tuning and found catastrophic forgetting generally observed in language models from 1 billion to 7 billion parameters. Fine-tuning on a narrow purchase is exactly the setting where a model can gain on the target and lose elsewhere.
Then write the selection rule: which purchased examples were meant for which target cohort, and how many of them made it into training after filtering. Without it, an improvement cannot be attributed to the data bought for it; with it, a flat cohort points either to too few matching examples or to examples that did not match the failure.
The error cohort reporting template
Fill one row set per cohort. The example column shows one illustrative target cohort from a freight claims assistant; the values are invented to show the level of detail, not results.
| Field | What to record | Illustrative entry |
|---|---|---|
| Cohort ID and type | A stable name and whether the cohort is a target the purchase should fix or a guard that must not regress | T3 target: freight claims citing a missed delivery appointment |
| Definition rule | The filter that puts a case in the cohort, written so another person gets the same members | Claim type is delay and the email cites an appointment window |
| Frozen case count | Held-out cases in the cohort, with the date the set was frozen and its hash | 212 cases, frozen before the purchase was delivered |
| Baseline failure | How the model fails on the cohort before training, in one sentence plus the error rate | Treats appointment misses as carrier delay; 41 percent wrong |
| Selection rule | Which purchased examples map to this cohort and how many went into training | Purchased emails tagged delay with an appointment field; 3,400 used |
| Arms compared | The with-purchase arm and the control arm, the budget each used and the seeds run | Same steps and tokens; control filled with existing data; 3 seeds each |
| Paired result | Error rate in each arm, the paired change and its interval | 41 to 26 percent; change minus 15 points, interval minus 21 to minus 9 |
| Flips | Cases that went wrong to right and right to wrong between arms | 38 fixed, 6 broken |
| Verdict | Improved, no clear change, or regressed, using the rule fixed before the run | Improved |
| Lineage and caveats | Dataset version, deduplication check against the frozen cases, label or judge changes | No exact or near duplicates found; same rubric and judge version |
Where the frozen cases come from and who may see them
Draw frozen cases from real production traffic or real business records that the purchased dataset could not contain, and restrict item-level access to the people who run scoring. If the people curating training data can browse the frozen cases, the selection rule will drift toward them, often without anyone intending it.
A clean source is a period after the supplier's records end, or a different business unit, region or channel than the one the purchase came from. Where neither is possible, deduplicate the purchased set against the frozen cases on both exact text and close paraphrase, and keep the deduplication output with the report.
Size the frozen set per cohort, not overall. A target cohort is often a small slice of traffic, so a random sample of a few thousand cases can leave it with too few members to read. Sample each target cohort up to the number of cases its interval needs, and weight results back to traffic shares only when you report an overall figure.
Freeze labels as well as cases. If reviewers relabel frozen cases after the first run, the comparison mixes a model change with a label change. Corrections go into a new version of the frozen set, and both arms are rescored on it.
Matching the budget and running more than one seed
Compare the with-purchase arm against a control trained for the same steps and tokens, with the purchased examples replaced by data the team already had, not against the untrained base model. Otherwise the report credits the purchase for the effect of simply training longer.
Run each arm with several seeds and report the spread. Dodge and colleagues showed that fine-tuning with identical hyperparameters but different random seeds can produce substantially different results, with weight initialization and data order both contributing. A single run per arm can show a cohort change that is only seed noise.
Keep everything else fixed between arms: the prompt format, the decoding settings, the rubric and, if a model grades answers, the judge model version. Record each in the lineage field so a later reader can tell whether a change came from the data.
Choose the control filler with care. It should look like the training data the team would have used anyway, drawn from the same mix of sources, so the comparison isolates the purchased examples rather than the effect of adding any new data at all. Where the purchase replaces part of an existing mix rather than adding to it, the control is the old mix at the same size.
Telling a real cohort gain from noise with paired results
Score both arms on the same frozen cases and compare them case by case. A paired comparison uses the fact that each case is seen by both arms, so it detects smaller real changes than comparing two independent averages.
Report the change with an interval, for example from a bootstrap that resamples the frozen cases, and fix the verdict rule before the run: improved if the whole interval is below zero change in error rate, regressed if it is wholly above, otherwise no clear change. Dror and colleagues found significance testing often ignored or misused in NLP papers and offered a decision tree for choosing a test; use it or your statistician's equivalent, and name the test in the report.
Always show the flips. The count of cases that went from wrong to right and from right to wrong tells a reviewer more than the net change. Twenty fixed and fourteen broken is a different story from eight fixed and two broken, even if the net is similar.
Small cohorts get wide intervals. If a target cohort has only a few dozen frozen cases, say that its result is not readable rather than reporting a large swing as a win.
Worked example (illustrative): reading a freight claims report
Every figure below is invented to show how the report reads. A carrier's claims assistant was failing on three kinds of freight claim email, and the team bought a set of real, de-identified claim emails to fix them. Both arms trained at the same budget with three seeds; the table shows seed-averaged error rates on the frozen cases.
- Two of three targets improved; T1 did not. The selection rule shows only 300 purchased emails matched T1, so the next request asks for more claims of that type rather than more data overall.
- G2 regressed: the assistant began quoting liability limits it should leave to a claims adjuster. The purchased emails contained adjuster replies quoting limits, which the model learned to imitate.
- The release decision is therefore not ship or do not ship. The team filters adjuster quotes out of the purchased set, retrains, and rechecks G2 before release.
| Cohort | Frozen cases | Error rate, control to purchase | Change in points (interval) | Verdict |
|---|---|---|---|---|
| T1 target: claims missing a bill of lading number | 180 | 35 to 31 percent | minus 4 (minus 10 to plus 2) | No clear change |
| T2 target: concealed damage reported after delivery | 146 | 52 to 38 percent | minus 14 (minus 22 to minus 6) | Improved |
| T3 target: missed delivery appointments | 212 | 41 to 26 percent | minus 15 (minus 21 to minus 9) | Improved |
| G1 guard: routine status questions | 400 | 6 to 7 percent | plus 1 (minus 1 to plus 3) | No clear change |
| G2 guard: refusing to quote liability limits | 120 | 9 to 18 percent | plus 9 (plus 3 to plus 15) | Regressed |
Readings that look like a cohort win but are not
Check four things before calling a cohort improved: overlap between purchased examples and frozen cases, a changed label or rubric, a cohort whose membership shifted, and a judge model that changed between runs. Each can produce an apparent gain with no real learning.
Overlap is the most common. If the supplier's records and your frozen cases come from the same source, near-duplicates can appear in both. Deduplicate the purchase against the frozen set before training and record the check in the lineage field.
Behavioral tests help separate a learned skill from memorized surface patterns. The CheckList method of Ribeiro and colleagues pairs a matrix of capabilities with test types such as invariance checks; a target cohort that improves on frozen cases but fails a simple invariance probe, such as a renamed carrier, may have learned names rather than the rule.
Membership drift is subtler. If the cohort filter depends on a field the model itself fills in, such as a predicted category, the cohort's members change between arms and the two error rates describe different cases. Define cohorts on fields that come from the source record or from a human label, never from the model under test.
Judge drift matters whenever a model grades free-text answers. Pin the judge model and its prompt for both arms, and spot-check a sample of judged cases by hand in every target cohort, recording agreement in the lineage field. A cohort whose gain disappears under human grading has not improved.
Stop and bring in specialist review when a guard cohort covers a legal, safety or financial behavior and it regresses, when the verdict depends on a judge model nobody has audited, or when the frozen cases may have been exposed. In those cases the cohort report is evidence for a decision, not the decision.
From an unimproved cohort to the next data request
A cohort that did not improve usually needs more or better-matched real examples of that exact failure. The template already holds what a data request needs: the cohort definition, the baseline failure in one sentence, how many matching examples the last purchase had, and which guard cohorts the new data must not disturb.
When a cohort stays flat, SourceX can look for more matching records at U.S. businesses; it holds no stock, so the buyer's request spells out scope, volume, language, format, timing and licensing needs. A cohort definition written from business categories, such as a claim type, translates directly into that scope. Before delivery the supplying business approves the scope and terms and signs the agreement.
Frequently asked questions
What is an error cohort in post-training evaluation?
An error cohort is a defined group of held-out cases that share a failure the model makes, such as a type of request it misroutes. It is defined by a written filter so its members can be reproduced, and it is used to judge whether training on new data fixed that failure.
Why compare against a control arm instead of the base model?
Any additional training changes a model, so comparing with the untrained base credits the purchase for effects of training itself. A control arm trained for the same budget on data the team already had isolates what the purchased examples added.
How many cases does a cohort need to be readable?
Enough that the interval on the change is narrower than the improvement you care about. That depends on the baseline error rate and the effect size, so compute it before freezing the set; a cohort of a few dozen cases rarely supports a firm verdict.
Should guard cohorts be part of the purchase decision?
Yes. A purchase that fixes the target but breaks a required behavior can cost more than it saves. List guard cohorts in the report next to targets, and decide in advance how much regression on each is acceptable.
What if the improved cohort overlaps the purchased data?
Treat the result as unproven until the overlap is removed. Deduplicate purchased examples against the frozen cases, retrain, and report the cleaned result. Record the check so a reviewer can see it was done.
Sources
- Microsoft's Responsible AI Toolbox describes Error Analysis as identifying cohorts with a higher error rate than the overall benchmark, using indicators such as error rate, error coverage and data representation. Source
- Model Cards for Model Reporting proposes disaggregating quantitative evaluation results by relevant groups and conditions, including intersections of two or more. Source
- Luo et al. report that catastrophic forgetting is generally observed in language models from 1 billion to 7 billion parameters during continual instruction tuning. Source
- Dodge et al. show that fine-tuning with different random seeds can lead to substantially different results, with weight initialization and training data order contributing comparably. Source
- Dror et al.'s ACL 2018 paper reports that statistical significance testing is often ignored or misused in NLP and proposes a simple decision tree for choosing a test. Source
- CheckList, by Ribeiro et al. (ACL 2020), provides a matrix of general linguistic capabilities and test types to support behavioral testing of NLP models. Source
Tell us what data you need
Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.