AI uses for records
Success-only training data bias: a 7-step check by learning objective
By SourceX Editorial · Updated
9 sections
Short answer
Success-only training data bias arises when a team keeps only cases marked successfully resolved, so the model never sees failures, recoveries or cases still open. Whether that matters depends on the learning objective: imitating good replies can tolerate it, while outcome prediction, grading, escalation and evaluation cannot. Before filtering, sample the population by start date, sort cases into four outcome strata, measure what the filter removes, and decide per objective.
Agent run
Success only training data
Key takeaways
- A success filter always changes the population; it becomes a harmful bias when the objective needs the strata it removes.
- Sort cases into four strata before filtering: first-pass success, recovered success, failure and unresolved.
- Only reply imitation can do without failures in training, and even then the evaluation set needs the full outcome mix.
- Measure retention by stratum on a sample drawn by start date before the filter, not on the filtered output.
- Weights cannot restore a stratum the filter removed entirely: drop the filter, add a labeled sample, or narrow the use.
When does keeping only successful resolutions bias a dataset?
Always in the narrow sense, because a success filter changes which cases the dataset represents. It becomes a harmful bias when the learning objective needs the cases the filter removes, or when the evaluation set inherits the same filter and stops measuring them. The decision is not whether to filter in general but whether this filter fits this objective.
Three pieces of evidence settle it: what the filter removes, measured on a sample drawn before filtering; what the objective requires, written down before anyone looks at the numbers; and whether a model trained on the filtered set still performs on a holdout that keeps the full outcome mix. Record counts are not on the list. A filtered archive of a million cases can still contain no failures at all.
Research has met this problem before. In the STaR method for teaching models to reason (arXiv 2203.14465), a model fine-tunes only on its own rationales that reached correct answers, and the authors report that the loop eventually stops solving new problems because it receives no direct training signal for the problems it fails; they added a step that supplies the correct answer so failed problems can contribute. A business dataset filtered to successful resolutions has the same blind spot without the repair.
Four outcome strata a success filter collapses
Sort every case into one of four strata before deciding anything: first-pass success, recovered success, failure and unresolved. A success filter keeps the first, may keep part of the second and drops the other two; the exact cut depends on how success is defined.
First-pass success is a case resolved without a reopen, an escalation or a second attempt. Recovered success failed first and was fixed later, through a reopen, a handoff or rework. Failure ended without the outcome the customer needed: a refund instead of a fix, a cancellation, an abandoned request. Unresolved is a case still open, pending or inside its reopen window on the extract date; it is not a failure, and a later extract could move it into any of the other three.
Help desk statuses blur the strata. In Zendesk, a requester's reply reopens a solved ticket, while a closed ticket stays closed and a reply creates a new follow-up ticket; by default a ticket closes automatically four days after it is solved, and no later than 28 days. A filter on closed status therefore drops recently solved cases, and a recovery that arrived as a follow-up can look like two unrelated cases, one of them a clean first-pass success.
So the definition decides which strata survive. Status solved, status closed, a positive satisfaction rating, first-contact resolution and no reopen within a window each cut differently. Write the definition as a rule that can be run, not as a word.
Which learning objectives need failures, recoveries or unresolved cases?
In the matrix below, every objective except reply imitation needs at least one stratum a success filter removes, and the evaluation set needs all four.
Published work supports the rows. SQuAD 2.0 (arXiv 1806.03822, 2018) was built because reading-comprehension systems that could find answers in a passage tended to guess when the passage held none, and earlier datasets either held only answerable questions or used unanswerable ones that were easy to spot; that is the escalation row, where a model must learn when not to act. Preference methods such as DPO (arXiv 2305.18290, 2023) start from human labels of the relative quality of model outputs, which presupposes a worse example to compare; that is the grader row. Anthropic's guidance on agent evaluations recommends balanced problem sets that test both when a behavior should occur and when it should not, warning that one-sided tests produce one-sided optimization; that is the evaluation row.
Two cells deserve a note. Reply imitation can train without failures because the model copies what good work looked like, but the same model will meet failing cases in production, so its test set needs them. Outcome prediction treats unresolved cases as censored rather than missing: each one shows that a case lasted at least as long as its age on the extract date, information a success filter discards along with the case.
Read the matrix by row: find the objective, then check whether the filtered set still contains every stratum marked required:
| Objective | First-pass success | Recovered success | Failure | Unresolved |
|---|---|---|---|---|
| Reply imitation (fine-tuning on resolution replies) | Required | Helpful, with the failed attempt marked | Not needed for training; needed in the test set | Not needed |
| Triage and routing classifier | Required | Required | Required | Helpful, labeled as open |
| Outcome or reopen prediction | Required | Required | Required: it is the target class | Required, as censored cases |
| Grader, reward model or preference pairs | Required | Required | Required: the worse side of each pair | Not needed |
| Escalation or abstention policy | Required | Helpful | Required | Helpful, as hard cases to hand off |
| Recovery behavior (notice and fix own errors) | Helpful | Required, with the failed step kept | Required | Not needed |
| Evaluation set for any objective above | Required | Required | Required | Required, scored separately |
The bias check, step by step
Run the check once per objective, because the same filter can pass for one use and fail for another, and run it again whenever the success rule or the source system changes: a new status, automation or rating scale moves cases between strata without anyone touching the filter.
The check is reproducible because each step produces a rule or a count that someone else can rerun:
Checklist · 7 items
- Write the objective and the success rule as code before looking at any counts, for example: closed, no follow-up ticket within 14 days and no bad satisfaction rating.
- Define the population and draw a sample by case start date from the system of record, before any filter, with follow-up links attached so a reopen or follow-up joins its original case.
- Classify each sampled case into the four strata from status history and linked records, not from the final status alone, and mark cases still inside the reopen or auto-close window as unresolved.
- Run the success rule on the sample and record retention by stratum, plus retention by difficulty proxies such as channel, product, customer tier and number of handoffs.
- Compare the retained strata with the objective's row in the matrix; a required stratum at or near zero is a blocking gap.
- Decide: keep the filter, keep it and add a stratified sample of the missing strata, drop it, or narrow the intended use, and record who decided and why.
- Publish the limits with the dataset: the rule, the retention table, the removed strata and the claims the filtered set cannot support.
0 of 7 ticked. Tick each item as you go.
The sampling worksheet
Record the check on one sheet so the next team can rerun it. The entries are illustrative, for a help desk archive and an escalation objective:
| Field | What to record | Illustrative entry |
|---|---|---|
| Objective | The learning objective in one sentence | Train an escalation policy for a support agent |
| Success rule | The filter as runnable logic | Closed, no follow-up within 14 days, no bad rating |
| Population and unit | System, case type, start-date range and what one sampled item is | Help desk cases started January 2024 to June 2025; a case is a ticket plus its linked follow-ups |
| Sample design | Size, selection and random seed | 1,200 cases, simple random by start date, seed recorded |
| Extract date and maturity | When data was pulled and how old a case must be to count as settled | Extracted July 31, 2025; cases open, pending or solved but not yet closed on that date marked unresolved |
| Retention by stratum | Share of each stratum the rule keeps | First-pass 90%, recovered 30%, failure 0%, unresolved 0% |
| Difficulty proxies | Retention by channel, customer tier and handoff count | Two or more handoffs: 22% of the sample, 9% of the kept set |
| Decision and owner | Keep, add, drop or narrow, and who decided | Drop the filter for this objective; add failures and open cases as labeled strata |
Illustrative: what a closed-and-rated filter keeps from 1,200 cases
Illustrative: a team samples 1,200 help desk cases by start date and classifies them as 732 first-pass successes (61 percent), 168 recovered successes (14 percent), 204 failures (17 percent) and 96 unresolved (8 percent). Its success rule keeps closed cases with no follow-up within 14 days and no bad rating.
The rule keeps 709 cases, 59 percent of the sample. It keeps 659 first-pass successes, losing the other 73 to bad satisfaction ratings, and 50 recovered successes, mostly reopened tickets solved again before closing; recoveries that arrived as follow-up tickets fail the rule. It keeps no failures and no unresolved cases. The kept set is 93 percent first-pass success against 61 percent in the sample, and cases with two or more handoffs fall from 22 percent of the sample to 9 percent of the kept set.
The decision follows the matrix. For reply imitation the filter is defensible if the evaluation set keeps the full mix. For an escalation policy it is not: the filtered set holds no failures and no open cases, and multi-handoff cases are less than half as common as in the sample. The team drops the filter for that objective and adds failures and unresolved cases back as labeled strata.
The counts are invented to show the arithmetic; they do not measure any company's records.
Coverage limitations: what can the filtered set not support?
State, stratum by stratum, the claims the filtered set cannot support and the evidence that would restore each. Datasheets for Datasets (arXiv 1803.09010) asks whether a dataset holds every possible instance or a sample of a larger set, whether such a sample is representative, and whether information is missing from individual instances. A success filter makes all three answers specific:
| Removed or thinned | Claim the filtered set cannot support | Evidence that would restore it | Disclosure to write |
|---|---|---|---|
| Failures | Any estimate of failure rate or failure risk | A labeled, stratified sample of failed cases | Contains no failed cases by design |
| Unresolved cases | How long cases take, or how many never resolve | Open cases with extract date and age, marked as censored | Excludes cases open on the extract date |
| Failed attempts inside recovered cases | How the process notices and fixes its own errors | Status history and follow-up links that keep the failed step | Keeps final resolutions only; earlier attempts removed |
| Recent cases inside the auto-close window | Behavior in the most recent period | A later extract, or solved status accepted for recent cases | Cases solved shortly before the extract are underrepresented |
| Hard cases with several handoffs | Performance on difficult, multi-team cases | Sampling stratified by handoff count | Multi-handoff cases are thinned relative to the population |
Could a success filter be harmless? Testing the strongest alternative
For some objectives, yes, and the test is simple. The strongest argument for the filter is that failures teach bad behavior and volume makes up for what the filter removes. The first half has merit for reply imitation: a model fine-tuned on replies that failed may copy them. The second half does not hold, because no number of successful cases contains a failure, and repeated template replies can make a large archive smaller than its row count suggests.
Settle it with a holdout rather than an argument. Keep a population holdout drawn by start date with all four strata, train once on the filtered set and once on the filtered set plus a labeled sample of the missing strata, and compare both on the whole holdout and on each stratum. If the two models score alike on failures and open cases as well as on successes, the filter is harmless for that objective. If the filtered model looks better only on the successful slice, the filter has moved the bias into the score.
Run the comparison per objective rather than once for the dataset. A filter can be harmless for a reply model and damaging for an escalation policy trained from the same archive, so the worksheet should carry one decision row per objective, each with its own holdout scores.
Writing the outcome mix into a data request
If the archive in hand cannot supply the missing strata, ask for them by name when sourcing data. A request for resolved tickets returns resolved tickets; a request that names the strata, the success rule and the extract date returns data the team can check.
SourceX sources datasets for AI buyers on request from U.S. businesses rather than holding them in stock, and buyers set out what they need, from scope and volume to language, format, timing and license terms, in the request-data form. Stating the strata, the success rule and an acceptance test there gives a supplier something specific to check against its records before anything is agreed.
A request built for the bias check should state:
- The success definition the team will apply, as a rule, and the records needed to apply it: status history, follow-up links, ratings and timestamps.
- Target shares or minimum counts for each of the four strata, drawn by case start date.
- The extract date and the age below which cases are treated as unresolved.
- The difficulty proxies the team will check, such as channel, product and handoff count.
- An acceptance test: the retention table recomputed on the delivered sample, with the stratum counts it must meet.
Frequently asked questions
Is it ever fine to train only on successful resolutions?
Yes, for some objectives. Fine-tuning a model to imitate good resolution replies can work on successful cases alone, provided the evaluation set keeps failures and open cases so the score reflects real conditions. Outcome prediction, graders, escalation policies and recovery behavior need cases the filter removes.
What is the difference between a failed case and an unresolved case?
A failed case ended without the outcome the customer needed, such as a cancellation or a refund instead of a fix. An unresolved case was still open, pending or inside its reopen window when the data was extracted, so its outcome is unknown rather than negative. Treating unresolved cases as failures, or dropping them, distorts both outcome rates and timing.
Do recovered cases count as successes?
They end as successes, but they carry something first-pass cases do not: the failed attempt and the fix. Keep them as their own stratum with the failed step preserved, because graders and recovery behavior learn from that sequence, and a filter that keeps only the final reply throws it away.
Can reweighting fix a success-only dataset?
No. Weights can rebalance strata that are present but thin; they cannot create cases that are absent. If the filter leaves zero failures or zero open cases, the fixes are to drop the filter, add a labeled sample of the missing strata, or narrow what the model is used for.
How large should the pre-filter sample be?
Large enough that the smallest stratum you care about yields enough cases to estimate its retention with useful precision. Size the sample from that stratum's expected share rather than from the total, and if a stratum proves rarer than expected, oversample it in a second draw and keep the weights.
Sources
- In STaR, fine-tuning only on rationales that reached correct answers eventually stalls because the model gets no direct training signal for problems it fails; the authors added rationalization. Source
- SQuAD 2.0 was created because reading-comprehension systems guess when the context lacks the answer and earlier datasets had only answerable or easily spotted unanswerable questions. Source
- The DPO paper notes that preference-based fine-tuning starts from human labels of the relative quality of model generations. Source
- Anthropic recommends balanced evaluation problem sets that test both when a behavior should occur and when it should not, since one-sided tests produce one-sided optimization. Source
- In Zendesk, a requester's reply reopens a solved ticket, a closed ticket stays closed and a reply creates a follow-up, and tickets close automatically four days after being solved by default, within 28 days at most. Source
- Datasheets for Datasets asks whether a dataset contains all possible instances or a sample, whether a sample is representative, and whether information is missing from individual instances. Source
Tell us what data you need
Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.