Skip to content

AI uses for records

Dataset unresolved case censoring: open cases are not failures yet

By SourceX Editorial · Updated

10 sections

Short answer

Give every case three facts instead of one label: whether its outcome has been observed, how long it was observed, and the date the extract was taken. A case still open, pending or inside its reopen window is censored, not failed. Label it with its observed duration and censoring reason, then either train only on cases old enough to have matured or use a method built for censored outcomes.

Record timeline

Dataset unresolved case

This page covers dataset unresolved case censoring: open cases are not failures yet.Illustrative

Key takeaways

  • An unresolved case is censored: its outcome is unknown at the extract date, not negative.
  • Labeling open cases as failures invents a decline in recent data; dropping them hides the slow, hard cases.
  • Record an event flag, observed duration, extract date, censoring reason and maturity window for every case.
  • Status fields such as solved, pending or an empty resolution need explicit rules before they become labels.
  • An open case is a real failure only when the target is deadline-based and the deadline passed before the extract.
  • When most recent cases are censored or censoring is informative, use time-to-event methods with specialist review.

How do you distinguish unresolved cases from failed outcomes in training labels?

You separate what happened from what has been seen so far. A support case opened last week that is still open has not failed; its outcome is unknown at the moment the data was exported. Labeling it as a failure teaches a model that recent or slow cases fail, and dropping it quietly removes the slow, difficult cases from the training set.

The fix is a label scheme that records outcome maturity explicitly: an event indicator, an observed duration, an extract date, a censoring reason and a maturity window for each outcome type. With those fields a team can choose an honest training target, and a buyer can see how much of a dataset is still unresolved before paying attention to its failure rate.

This guide covers the definitions, how common business systems blur them, the label guide itself, a worked example, and the case where treating an open case as a failure is correct.

What censoring means when the subject is a business case

Censoring is the statistical name for an outcome that has not been observed yet, or could not be observed. The lifelines documentation describes individuals who have not experienced the event as right-censored: all that is known is their duration so far, which is shorter than their eventual duration. It also warns that a common analyst mistake is to ignore right-censored individuals, because doing so can badly underestimate durations.

scikit-survival describes the same structure from the data side. Each record needs a status field saying whether the event was observed and a time field holding either the time of the event or the last time the subject was checked. That pair translates directly to business records: an observed flag for whether the case reached a settled outcome, and a duration from opening to that outcome or to the extract date.

In business archives the event is rarely a single thing. A sales opportunity can be won, lost or withdrawn; a claim can be paid, denied or abandoned; a ticket can be solved, reopened or merged. Each of these is a different outcome, and an outcome that stops another from happening, such as a merge, needs its own value rather than being folded into failure.

How ticketing and tracking systems blur open, solved and failed

Status fields were designed to route work, not to label outcomes, and several standard behaviors produce misleading labels if read at face value.

How ticketing and tracking systems blur open, solved and failed
System behaviorWhat the field saysWhat is actually knownLabel rule
Zendesk pendingWaiting for more information from the requesterOutcome not observedCensored, reason: awaiting customer
Zendesk on-holdWaiting on someone other than the requester; the requester sees it as openOutcome not observedCensored, reason: awaiting third party
Zendesk solvedAgent marked it resolved; a requester reply can reopen itProvisional success until the reopen window endsImmature success until it closes or the window passes
Zendesk closedLocked by automation; a later reply creates a follow-up ticketSettled for this ticket, but a linked follow-up may show the problem returnedObserved; check follow-ups before calling it success
Jira empty resolutionJira treats an empty resolution field as unresolvedOutcome not observed, or a workflow that never set itCensored unless the status history shows completion
Jira custom value named UnresolvedAny value in the resolution field is read as resolvedMisconfiguration; outcome unknownDo not trust the field; derive from status history

The outcome maturity and censoring label guide

This is the asset. Each case receives one of the label states below, plus the fields that justify it. The states are ordered from least to most known, and only the last two are usable as success or failure targets without a censoring-aware method.

  • Every case also carries the extract_date, the system and status that produced the label, and the version of the maturity rules used.
  • Durations are computed from the same clock for all cases, and time zones are fixed before computing them.
  • Reopened cases keep their full history; the label is set from the last settled state, not the first.
The outcome maturity and censoring label guide
Label stateDefinitionFields requiredUse in training and evaluation
Open, not observedNo settling event by the extract dateopened_at, extract_date, observed=false, duration_so_farCensored; never a failure label
Waiting on external partyPaused for a customer, vendor or approverpause_reason, pause_started_atCensored; also report time paused
Provisionally settledMarked solved, won or approved, still inside its reversal windowsettled_at, window_length, window_ends_atCensored until the window ends
Competing outcomeEnded by merge, transfer, withdrawal or account closurecompeting_type, ended_atOwn class; not success, not failure
Lost to follow-upObservation stopped for a reason tied to the case, such as the customer leavingloss_reason, last_seen_atCensored, flagged as possibly informative
Settled successSettling event and window complete with no reversaloutcome=success, settled_at, window_ends_atUsable positive label
Settled failureFailure event observed, or the success deadline passedoutcome=failure, failure_event, failed_atUsable negative label

How long should a case age before its label counts?

Set a maturity window per outcome type from the data, not from a guess. Take older cohorts whose outcomes have had ample time to settle, measure how long reversals and late failures take to appear, and choose a window that captures nearly all of them. Then document the window, the cohort it was derived from and the share of late events it would miss.

How long should a case age before its label counts?
Outcome typeEvent that settles itReversal to wait forHow to set the window
Support ticketClosed after solvedReopen or follow-up ticketTime from solved to reopen in mature cohorts, plus the system's auto-close delay
Sales opportunityMarked won or lostWon deal canceled, lost deal revivedTime from close to cancellation or revival in older years
Field service jobJob completedCallback visit for the same issueCallback interval distribution for completed jobs
Insurance or warranty claimDecision issuedAppeal or reopened claimAppeal filing period in the policy terms, checked against observed appeals
Engineering issueResolution setIssue reopened or regression linkedTime from resolution to reopen in mature releases

Worked example: three months of support cases read on one day

Illustrative counts only, not from any real company. An extract taken at the end of a quarter covers three monthly cohorts of 1,000 support cases each. The reopen window is set at 30 days.

  • Neither naive rule is right. The first invents failures; the second keeps the quick cases and hides the hard ones from both training and evaluation.
  • A horizon label fixes this for a binary target: resolved within 14 days, scored only for cases opened at least 14 days plus the reopen window before the extract.
  • A duration target keeps every case: observed cases contribute their time to resolution, censored cases contribute their time so far with observed set to false, and a survival method uses both.
  • Either way, the dataset reports the censored share per cohort so a buyer sees that Month 3 is mostly unresolved.
Worked example: three months of support cases read on one day
Cohort (1,000 cases each)Settled success / failureCensoredFailure rate if open counts as failedFailure rate if open cases are dropped
Month 1 (oldest)940 / 40206 percentAbout 4.1 percent
Month 2860 / 3510514 percentAbout 3.9 percent
Month 3 (newest)600 / 3037040 percentAbout 4.8 percent
All three months2,400 / 10549520 percentAbout 4.2 percent

Which cases go missing when unresolved records are dropped?

Dropping unresolved cases is not neutral, because the cases most likely to be open at any moment are the long ones. Complex billing disputes, multi-visit repairs, escalated claims and enterprise deals stay open longest, so they are overrepresented among censored cases and underrepresented in any set built only from settled ones. An agent trained on that set meets more of them in production than it ever saw in training.

Censoring can also carry information. If a case stops being observed because the customer left, the account closed or the case was migrated to another system, the stopping is related to the outcome rather than to the calendar. Treating those cases like ordinary open cases can make results look better than they are, which is why the guide gives them their own lost-to-follow-up state and asks for the reason.

Compare the censored share by segment, channel, product and customer size against the population the model will serve. A segment with a high censored share and few settled cases is a coverage gap, not a solved problem.

When an open case really is a failure

Some targets define failure by a deadline, and for those an open case past the deadline is correctly labeled failed. If the task is resolve within the service window, a case still open after that window has failed that task whatever happens later. The condition is that the deadline has fully elapsed before the extract date; a case opened two days before the extract cannot fail a five-day target.

The same logic applies to evaluation. When an agent is graded against historical outcomes, only cases whose maturity window ended before the extract can serve as ground truth, and the evaluation report should say how many cases were excluded for immaturity. Datasheets for Datasets asks dataset creators over what timeframe the data was collected and whether it matches when the underlying records were created; for outcome labels, the extract date and maturity rule are the answer that question needs.

Where a maturity cutoff stops being enough

  • When most cases in the period of interest are censored, cutoffs leave too little recent data; a survival or time-to-event model is the better tool, and a statistician should choose it.
  • When censoring is plausibly informative, such as cases ending because customers churned, standard censoring methods can mislead; model the loss as its own outcome and get specialist review.
  • When outcomes compete, such as merged versus solved, a single success or failure label discards information; use a multi-class or competing-outcome target.
  • When the status field itself is unreliable, as with a resolution field that workflows never set, rebuild outcomes from status history before applying any of the rules above.
  • When policies or systems changed mid-period, maturity windows from older cohorts may not apply to newer ones; derive them per period.

Asking for extract dates and status history in licensed records

None of these labels can be rebuilt without three things: the date the records were extracted, the full status history of each case rather than its final status, and the links between a case and its reopenings or follow-ups. Specify them before a dataset is prepared, because a flattened export with only current status cannot be fixed later.

Through the SourceX request form a buyer can name the scope, volume, format and timing it needs, extract dates and status histories included; the records are then sourced from U.S. businesses for that request rather than taken from stock. The licensing company approves scope and terms before anything is delivered, and as a general guide buyers look for consistent history of typically two years or more, which also gives older cohorts enough time to mature.

Frequently asked questions

What is a censored case in a training dataset?

It is a case whose outcome had not been observed when the data was extracted, such as a ticket still open or a claim still within its appeal period. You know how long it has run so far, not how it ends, so it should be labeled as censored with its observed duration rather than as a success or failure.

Should unresolved cases be removed from a training set?

Usually not without a rule. Removing them drops the slowest and often hardest cases, which biases the set toward easy ones. Keep them with a censored label, or apply a maturity cutoff by opening date so whole cohorts are either mature or excluded.

What is outcome maturity?

Outcome maturity is whether enough time has passed for a case's outcome to settle, including any reversals such as reopenings, cancellations or appeals. A maturity window per outcome type, derived from older cohorts, defines when a label can be trusted.

Is a solved ticket a success label?

Not immediately. In systems such as Zendesk a requester reply can reopen a solved ticket until it closes, and a closed ticket can still lead to a follow-up. Treat solved as provisional until the reopen window ends and check for linked follow-ups.

When do you need survival analysis instead of simple labels?

When a large share of recent cases is still open, when you care about time to resolution, or when cases end for competing reasons. Survival methods use both observed and censored cases, which a binary label cannot do without discarding data.

Sources

  • The lifelines documentation defines right-censored individuals as those whose event has not been observed and warns that ignoring them is a common mistake. Source
  • The scikit-survival user guide stores survival data as a status field (event observed or censored) and a time field (event time or last contact). Source
  • Zendesk defines pending as waiting on the requester and on-hold as waiting on someone else, shown to the requester as open. Source
  • Zendesk explains that a reply can reopen a solved ticket, while a closed ticket is locked by automation and a later reply creates a follow-up ticket. Source
  • Atlassian explains that Jira treats an empty resolution field as unresolved and any value, including a custom Unresolved value, as resolved. Source
  • Datasheets for Datasets asks over what timeframe data was collected and whether it matches the creation timeframe of the underlying instances. Source

Tell us what data you need

Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.

See if your company qualifies