AI uses for records
Dataset unresolved case censoring: open cases are not failures yet
By SourceX Editorial · Updated
10 sections
Short answer
Give every case three facts instead of one label: whether its outcome has been observed, how long it was observed, and the date the extract was taken. A case still open, pending or inside its reopen window is censored, not failed. Label it with its observed duration and censoring reason, then either train only on cases old enough to have matured or use a method built for censored outcomes.
Record timeline
Dataset unresolved case
Key takeaways
- An unresolved case is censored: its outcome is unknown at the extract date, not negative.
- Labeling open cases as failures invents a decline in recent data; dropping them hides the slow, hard cases.
- Record an event flag, observed duration, extract date, censoring reason and maturity window for every case.
- Status fields such as solved, pending or an empty resolution need explicit rules before they become labels.
- An open case is a real failure only when the target is deadline-based and the deadline passed before the extract.
- When most recent cases are censored or censoring is informative, use time-to-event methods with specialist review.
How do you distinguish unresolved cases from failed outcomes in training labels?
You separate what happened from what has been seen so far. A support case opened last week that is still open has not failed; its outcome is unknown at the moment the data was exported. Labeling it as a failure teaches a model that recent or slow cases fail, and dropping it quietly removes the slow, difficult cases from the training set.
The fix is a label scheme that records outcome maturity explicitly: an event indicator, an observed duration, an extract date, a censoring reason and a maturity window for each outcome type. With those fields a team can choose an honest training target, and a buyer can see how much of a dataset is still unresolved before paying attention to its failure rate.
This guide covers the definitions, how common business systems blur them, the label guide itself, a worked example, and the case where treating an open case as a failure is correct.
What censoring means when the subject is a business case
Censoring is the statistical name for an outcome that has not been observed yet, or could not be observed. The lifelines documentation describes individuals who have not experienced the event as right-censored: all that is known is their duration so far, which is shorter than their eventual duration. It also warns that a common analyst mistake is to ignore right-censored individuals, because doing so can badly underestimate durations.
scikit-survival describes the same structure from the data side. Each record needs a status field saying whether the event was observed and a time field holding either the time of the event or the last time the subject was checked. That pair translates directly to business records: an observed flag for whether the case reached a settled outcome, and a duration from opening to that outcome or to the extract date.
In business archives the event is rarely a single thing. A sales opportunity can be won, lost or withdrawn; a claim can be paid, denied or abandoned; a ticket can be solved, reopened or merged. Each of these is a different outcome, and an outcome that stops another from happening, such as a merge, needs its own value rather than being folded into failure.
How ticketing and tracking systems blur open, solved and failed
Status fields were designed to route work, not to label outcomes, and several standard behaviors produce misleading labels if read at face value.
| System behavior | What the field says | What is actually known | Label rule |
|---|---|---|---|
| Zendesk pending | Waiting for more information from the requester | Outcome not observed | Censored, reason: awaiting customer |
| Zendesk on-hold | Waiting on someone other than the requester; the requester sees it as open | Outcome not observed | Censored, reason: awaiting third party |
| Zendesk solved | Agent marked it resolved; a requester reply can reopen it | Provisional success until the reopen window ends | Immature success until it closes or the window passes |
| Zendesk closed | Locked by automation; a later reply creates a follow-up ticket | Settled for this ticket, but a linked follow-up may show the problem returned | Observed; check follow-ups before calling it success |
| Jira empty resolution | Jira treats an empty resolution field as unresolved | Outcome not observed, or a workflow that never set it | Censored unless the status history shows completion |
| Jira custom value named Unresolved | Any value in the resolution field is read as resolved | Misconfiguration; outcome unknown | Do not trust the field; derive from status history |
The outcome maturity and censoring label guide
This is the asset. Each case receives one of the label states below, plus the fields that justify it. The states are ordered from least to most known, and only the last two are usable as success or failure targets without a censoring-aware method.
- Every case also carries the extract_date, the system and status that produced the label, and the version of the maturity rules used.
- Durations are computed from the same clock for all cases, and time zones are fixed before computing them.
- Reopened cases keep their full history; the label is set from the last settled state, not the first.
| Label state | Definition | Fields required | Use in training and evaluation |
|---|---|---|---|
| Open, not observed | No settling event by the extract date | opened_at, extract_date, observed=false, duration_so_far | Censored; never a failure label |
| Waiting on external party | Paused for a customer, vendor or approver | pause_reason, pause_started_at | Censored; also report time paused |
| Provisionally settled | Marked solved, won or approved, still inside its reversal window | settled_at, window_length, window_ends_at | Censored until the window ends |
| Competing outcome | Ended by merge, transfer, withdrawal or account closure | competing_type, ended_at | Own class; not success, not failure |
| Lost to follow-up | Observation stopped for a reason tied to the case, such as the customer leaving | loss_reason, last_seen_at | Censored, flagged as possibly informative |
| Settled success | Settling event and window complete with no reversal | outcome=success, settled_at, window_ends_at | Usable positive label |
| Settled failure | Failure event observed, or the success deadline passed | outcome=failure, failure_event, failed_at | Usable negative label |
How long should a case age before its label counts?
Set a maturity window per outcome type from the data, not from a guess. Take older cohorts whose outcomes have had ample time to settle, measure how long reversals and late failures take to appear, and choose a window that captures nearly all of them. Then document the window, the cohort it was derived from and the share of late events it would miss.
| Outcome type | Event that settles it | Reversal to wait for | How to set the window |
|---|---|---|---|
| Support ticket | Closed after solved | Reopen or follow-up ticket | Time from solved to reopen in mature cohorts, plus the system's auto-close delay |
| Sales opportunity | Marked won or lost | Won deal canceled, lost deal revived | Time from close to cancellation or revival in older years |
| Field service job | Job completed | Callback visit for the same issue | Callback interval distribution for completed jobs |
| Insurance or warranty claim | Decision issued | Appeal or reopened claim | Appeal filing period in the policy terms, checked against observed appeals |
| Engineering issue | Resolution set | Issue reopened or regression linked | Time from resolution to reopen in mature releases |
Worked example: three months of support cases read on one day
Illustrative counts only, not from any real company. An extract taken at the end of a quarter covers three monthly cohorts of 1,000 support cases each. The reopen window is set at 30 days.
- Neither naive rule is right. The first invents failures; the second keeps the quick cases and hides the hard ones from both training and evaluation.
- A horizon label fixes this for a binary target: resolved within 14 days, scored only for cases opened at least 14 days plus the reopen window before the extract.
- A duration target keeps every case: observed cases contribute their time to resolution, censored cases contribute their time so far with observed set to false, and a survival method uses both.
- Either way, the dataset reports the censored share per cohort so a buyer sees that Month 3 is mostly unresolved.
| Cohort (1,000 cases each) | Settled success / failure | Censored | Failure rate if open counts as failed | Failure rate if open cases are dropped |
|---|---|---|---|---|
| Month 1 (oldest) | 940 / 40 | 20 | 6 percent | About 4.1 percent |
| Month 2 | 860 / 35 | 105 | 14 percent | About 3.9 percent |
| Month 3 (newest) | 600 / 30 | 370 | 40 percent | About 4.8 percent |
| All three months | 2,400 / 105 | 495 | 20 percent | About 4.2 percent |
Which cases go missing when unresolved records are dropped?
Dropping unresolved cases is not neutral, because the cases most likely to be open at any moment are the long ones. Complex billing disputes, multi-visit repairs, escalated claims and enterprise deals stay open longest, so they are overrepresented among censored cases and underrepresented in any set built only from settled ones. An agent trained on that set meets more of them in production than it ever saw in training.
Censoring can also carry information. If a case stops being observed because the customer left, the account closed or the case was migrated to another system, the stopping is related to the outcome rather than to the calendar. Treating those cases like ordinary open cases can make results look better than they are, which is why the guide gives them their own lost-to-follow-up state and asks for the reason.
Compare the censored share by segment, channel, product and customer size against the population the model will serve. A segment with a high censored share and few settled cases is a coverage gap, not a solved problem.
When an open case really is a failure
Some targets define failure by a deadline, and for those an open case past the deadline is correctly labeled failed. If the task is resolve within the service window, a case still open after that window has failed that task whatever happens later. The condition is that the deadline has fully elapsed before the extract date; a case opened two days before the extract cannot fail a five-day target.
The same logic applies to evaluation. When an agent is graded against historical outcomes, only cases whose maturity window ended before the extract can serve as ground truth, and the evaluation report should say how many cases were excluded for immaturity. Datasheets for Datasets asks dataset creators over what timeframe the data was collected and whether it matches when the underlying records were created; for outcome labels, the extract date and maturity rule are the answer that question needs.
Where a maturity cutoff stops being enough
- When most cases in the period of interest are censored, cutoffs leave too little recent data; a survival or time-to-event model is the better tool, and a statistician should choose it.
- When censoring is plausibly informative, such as cases ending because customers churned, standard censoring methods can mislead; model the loss as its own outcome and get specialist review.
- When outcomes compete, such as merged versus solved, a single success or failure label discards information; use a multi-class or competing-outcome target.
- When the status field itself is unreliable, as with a resolution field that workflows never set, rebuild outcomes from status history before applying any of the rules above.
- When policies or systems changed mid-period, maturity windows from older cohorts may not apply to newer ones; derive them per period.
Asking for extract dates and status history in licensed records
None of these labels can be rebuilt without three things: the date the records were extracted, the full status history of each case rather than its final status, and the links between a case and its reopenings or follow-ups. Specify them before a dataset is prepared, because a flattened export with only current status cannot be fixed later.
Through the SourceX request form a buyer can name the scope, volume, format and timing it needs, extract dates and status histories included; the records are then sourced from U.S. businesses for that request rather than taken from stock. The licensing company approves scope and terms before anything is delivered, and as a general guide buyers look for consistent history of typically two years or more, which also gives older cohorts enough time to mature.
Frequently asked questions
What is a censored case in a training dataset?
It is a case whose outcome had not been observed when the data was extracted, such as a ticket still open or a claim still within its appeal period. You know how long it has run so far, not how it ends, so it should be labeled as censored with its observed duration rather than as a success or failure.
Should unresolved cases be removed from a training set?
Usually not without a rule. Removing them drops the slowest and often hardest cases, which biases the set toward easy ones. Keep them with a censored label, or apply a maturity cutoff by opening date so whole cohorts are either mature or excluded.
What is outcome maturity?
Outcome maturity is whether enough time has passed for a case's outcome to settle, including any reversals such as reopenings, cancellations or appeals. A maturity window per outcome type, derived from older cohorts, defines when a label can be trusted.
Is a solved ticket a success label?
Not immediately. In systems such as Zendesk a requester reply can reopen a solved ticket until it closes, and a closed ticket can still lead to a follow-up. Treat solved as provisional until the reopen window ends and check for linked follow-ups.
When do you need survival analysis instead of simple labels?
When a large share of recent cases is still open, when you care about time to resolution, or when cases end for competing reasons. Survival methods use both observed and censored cases, which a binary label cannot do without discarding data.
Sources
- The lifelines documentation defines right-censored individuals as those whose event has not been observed and warns that ignoring them is a common mistake. Source
- The scikit-survival user guide stores survival data as a status field (event observed or censored) and a time field (event time or last contact). Source
- Zendesk defines pending as waiting on the requester and on-hold as waiting on someone else, shown to the requester as open. Source
- Zendesk explains that a reply can reopen a solved ticket, while a closed ticket is locked by automation and a later reply creates a follow-up ticket. Source
- Atlassian explains that Jira treats an empty resolution field as unresolved and any value, including a custom Unresolved value, as resolved. Source
- Datasheets for Datasets asks over what timeframe data was collected and whether it matches the creation timeframe of the underlying instances. Source
Tell us what data you need
Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.