Privacy and preparation
Support dataset exception coverage: what is your sample missing?
By SourceX Editorial · Updated
10 sections
Short answer
To find support exceptions missing from a training sample, count exception types across the full case population using signals the help desk already stores: reopens, escalations, parent and child links, merges, SLA pauses, approved policy overrides and edits after resolution. Then compare each type's settled, fully linked cases in the sample against a floor set in advance. Thin or missing types are filled, labeled or declared.
Redaction
Support dataset exception
Key takeaways
- Exception gaps cannot be seen from inside a sample; count every exception type across the full population first.
- Most exception signals are structured fields, so the coverage matrix can be built from metadata before anyone reads ticket text.
- Only cases whose outcome window has passed without a reopen, repeat contact or changed resolution count as settled.
- Set the per-type floor in writing before counting, and report raw counts next to any share.
- Each thin or missing type gets a recorded action: fill it with a targeted draw, label it, or declare it in the dataset documentation.
How do you find the support exceptions a training sample left out?
You find them by counting exception types in the full case population first, then checking how many complete, settled cases of each type the sample actually holds. A gap is invisible from inside the sample: if a random draw of 2,000 tickets contains no merged multi-customer incidents, nothing in those 2,000 rows tells you the type exists.
The method has four steps. Define exception types by signals the help desk already stores, such as reopen counts, escalation fields, parent and child links and post-resolution edits. Count each type across the whole population for the period you plan to share. Count the same types in the sample, keeping only cases whose outcome has settled. Then mark every type as covered, thin, missing or unmeasurable, and decide whether to fill, label or declare each gap.
This is a customer operations task as much as a data task. The person who knows that refunds outside the return window need a team lead's approval is the person who can tell you which field records that approval, and whether it was recorded at all before a policy change.
Which help desk signals mark a case as an exception?
An exception is any case that left the standard path: it came back after resolution, needed someone with more authority, affected several customers at once, or was changed after it closed. Most help desks record these moves as structured fields or status history, so exception typing can run on metadata before anyone reads ticket text.
Two platform examples show where the signals sit. Zendesk's Ticket Metrics API returns a read-only reopens count for each ticket, and Zendesk notes that requester wait time resumes where it left off when a solved ticket is reopened. Dynamics 365 Customer Service can reopen a resolved case when a new email arrives about it, through a rule an administrator configures, and supports parent and child cases with a closure preference that either closes all child cases with the parent or blocks the parent until every child is closed.
| Exception type | Signal in the record | Where to look | What the sample loses without it |
|---|---|---|---|
| Reopened after resolution | Reopen count above zero, or a status move from solved back to open | Zendesk ticket metrics; Dynamics 365 status history and reopen rules | Cases where the first fix failed and the second attempt is the real answer |
| Escalated to another tier or team | Group or queue change after first response; escalation flag or linked engineering issue | Ticket audit trail; linked Jira or engineering issue | The handoff note, the receiving team's diagnosis and the delay it caused |
| Multi-customer incident | Parent case with child cases, or a problem ticket with linked incidents | Dynamics 365 parent and child cases; problem and incident links where the help desk has them | How one root cause was communicated and closed across many customers |
| Merged or duplicate | Merge marker, closed-as-duplicate reason or a pointer to a surviving case | Ticket audit trail and closure reason | The customer's first contact, often on a different channel |
| Paused for the customer or a third party | SLA paused status or a pending or on-hold state with a reason | Dynamics 365 SLA pause conditions; pending and on-hold states | Waiting periods that explain slow outcomes and should not be scored as agent delay |
| Policy exception approved | Override reason, approver field or approval task on the case | Custom fields, approval workflow records, refund or credit system | The rule that was bent, who had authority to bend it, and under which policy version |
| Edited after resolution | Modified date later than resolution date on a resolved or canceled case | Dynamics 365 case update settings; audit logs | Corrections that change the recorded outcome after the fact |
| Closed without confirmation | Automatic closure after silence or a default resolution code | Automation logs and closure reason | The difference between a fixed problem and a customer who stopped replying |
Why does a random ticket sample under-represent the cases that matter?
A random sample reproduces the population's proportions, so a type that makes up a fraction of a percent of cases arrives as a handful of rows, often with no settled outcome among them. A model trained or tested on that sample can look accurate overall while failing on exactly those cases.
Research on hidden stratification describes this failure. Oakden-Rayner and colleagues showed that models can perform well on average while performing poorly on subsets that were never labeled during training or testing, and reported relative performance gaps of over 20% on important subsets in their medical imaging work. Support data has the same structure: exception types are rarely labeled as such, so their failures disappear into an average.
Stratified sampling does not fix this on its own. scikit-learn's documentation notes that stratified splitters keep relative class frequencies approximately preserved in each fold, which only helps for classes you have labeled. Exception typing comes first; stratification can only use the labels it is given.
The task exception and outcome coverage matrix
The coverage matrix has one row per exception type and one column per question you need answered before the sample leaves the building. Build it from metadata exports, keep it beside the sample, and recompute it whenever the sample changes. The table below defines each column and the condition that should raise a flag.
Set the floor for each type before counting, in writing, so the matrix cannot be tuned to make a weak sample look complete. A floor is the minimum number of complete, settled cases a type needs before anyone draws conclusions from it; teams usually set it higher for types that carry money, safety or compliance consequences.
| Column | What it holds | How to compute it | Flag when |
|---|---|---|---|
| Exception type | One named type from the signal table, defined by a field rule | Rule written as a filter a second analyst can rerun | The rule depends on reading free text |
| Population cases | Count of cases of this type in the full period and scope | Run the rule over the complete export, not the sample | The count cannot be produced because the field did not exist for part of the period |
| Sample cases | Count of cases of this type in the sample | Run the same rule over the sample | Zero, or far below the population share |
| Settled cases in sample | Sample cases whose outcome window has passed with no reopen or follow-up | Apply the maturity rule in the next section | Below the floor set for this type |
| Handoff chain intact | Settled cases where every escalation, child case or merge can be followed to its end | Check linked records resolve to an existing case or issue | Links point to deleted, archived or unexported records |
| Policy version known | Settled cases where the policy in force on the case date can be identified | Join case dates to a dated policy register | Policy changes are undocumented or undated |
| Status | Covered, thin, missing or unmeasurable | Covered meets the floor on all checks; thin is below it; missing has zero; unmeasurable lacks the signal | Any type is thin or missing and the sample is described as representative |
| Action | Fill, label or declare | Chosen per type and recorded with a reason | No action is recorded for a thin or missing type |
When does a resolved support case count as settled?
A case counts as settled only when its outcome window has passed with no reopen, no follow-up contact from the same customer about the same issue, and no later edit that changed the recorded resolution. A solved status on its own proves only that someone pressed a button.
Pick the outcome window from your own reopen data: look at how long after resolution reopens and repeat contacts usually arrive, and choose a window that captures most of them. Cases resolved inside that window at the time of the export are immature. Keep them out of the settled count, or include them with an explicit immature label.
Check edits as well as reopens. Dynamics 365 blocks updates to resolved and canceled cases by default, but an administrator can allow updates through Power Automate flows or the API without reopening the case; Microsoft gives removing personal data under GDPR as one reason. A resolved case modified after its resolution date needs a look before its outcome is trusted.
Downstream evidence makes the strongest proof: a refund that posted in the billing system, a bug fix that shipped in a named release, a replacement that the warehouse received back. Satisfaction ratings help but are sparse and often reflect the conversation rather than the fix.
Which window, typing rule or scope choice flips a type's status?
Three assumptions move the matrix more than any other: the rule that assigns a case to an exception type, the length of the outcome window, and the boundary of the population. Change any of them and a type can move from covered to thin.
Test each one directly. Rerun the matrix with a shorter and a longer outcome window and note which types change status. Have a second person apply the typing rules to a small set of cases and compare. Recount the population with and without brands, channels or periods that were added late. Any type whose status flips under a reasonable change should be reported as assumption-dependent, not as covered.
Report counts, not just shares. A type with a handful of cases cannot support a precise percentage, so show the raw count next to any share and say plainly when a count is too small to estimate a rate. Where the signal did not exist for part of the period, report the type as unmeasurable for that part rather than treating the absence as zero.
Illustrative: a subscription software help desk checks its sample
The figures below are illustrative, not observed. A software company plans to share two years of support cases, about 40,000 in total, and has drawn a random sample of 2,000. Its operations lead sets a floor of 30 settled, fully linked cases per type and a 45-day outcome window based on when its reopens usually arrive.
Reopens and escalations clear the floor. Multi-customer incidents do not: the population holds about 160 parent cases, the sample holds 8, and only 3 have every child case exported. Refunds approved outside policy are unmeasurable for the first nine months, because the approver field was added partway through the period.
| Exception type | Population cases | Sample cases | Settled and linked in sample | Status and action |
|---|---|---|---|---|
| Reopened after resolution | About 2,400 | 118 | 96 | Covered; keep |
| Escalated to engineering | About 1,600 | 84 | 61 | Covered; keep, check issue links resolve |
| Multi-customer incident | About 160 | 8 | 3 | Thin; fill with a targeted draw of parent cases and all children |
| Refund outside policy | About 600 | 29 | 11 with approver recorded | Unmeasurable before field existed; label by period and declare |
| Edited after resolution | About 500 | 27 | Not applicable | Review edits; exclude cases whose outcome changed |
| Closed without confirmation | About 3,000 | 151 | 0 settled by definition | Label as unconfirmed; never count as resolved |
Fill the gap, label it, or declare it?
Fill a gap when the cases exist in the population and can be exported intact: draw extra cases of that type, keep a record of the sampling weight, and report the type separately so the oversampling does not distort overall rates. Label a gap when the cases are present but their outcome or approval is uncertain, so a user can include or exclude them deliberately. Declare a gap when the cases cannot be supplied, and write it into the dataset's documentation.
Declaring is not a failure. Datasheets for Datasets, the documentation framework proposed by Gebru and colleagues, asks dataset creators to describe composition, including whether a dataset is a sample of a larger set and how it was drawn. A plain statement that multi-customer incidents are under-represented and why is more useful to a buyer than a sample described as representative.
Some exception types need a privacy or legal reviewer before they are filled at all. Escalations to a legal team, chargeback disputes, account security cases and complaints that name employees tend to carry more personal and third-party data than routine tickets, and oversampling them concentrates that risk. Decide those types with the privacy reviewer before drawing extra cases.
Policy exceptions are the hardest type to supply and the most useful
Cases where staff bent a rule with approval are the type agent builders most often lack, because public benchmarks write their policies by hand. tau-bench, a benchmark for agents that follow domain policies in retail and airline settings, grades each task by comparing the final database state with an annotated goal state and measures consistency across repeated trials with a metric it calls pass^k.
Real policy-exception cases can supply that structure from actual operations, but only if three things are recorded: the policy text in force on the case date, the approval with the approver's role, and the action that followed. If any of the three is missing, the case can still show the conversation, but it cannot show what the correct end state was.
What the matrix changes when support records are licensed
Support teams that decide to license their history to AI developers can do it through SourceX, whose work runs from rights review and contracting to data preparation and delivery. A coverage matrix is useful evidence in that process because it shows what a dataset contains before anyone argues about what it is worth.
Checking eligibility does not require handing over tickets: the first step is a short company assessment, and no case leaves the help desk until the company has approved the scope and terms and signed. A finished matrix and its declared gaps make that scope conversation faster and more precise.
Frequently asked questions
What should the task exception and outcome coverage matrix include?
One row per exception type, defined by a rule on structured fields, with counts for the full population, the sample, settled cases, cases with an intact handoff chain and cases with a known policy version. Each row ends with a status of covered, thin, missing or unmeasurable and a recorded action of fill, label or declare.
Is a random sample of support tickets ever good enough?
It can be for describing routine contact reasons, where every type is common. It is rarely enough for training or testing an agent on exceptions, because rare types arrive in tiny numbers and often without settled outcomes. Check the matrix before deciding.
How long should we wait before a resolved case counts as settled?
Use your own reopen and repeat-contact history. Measure how long after resolution those usually arrive and choose a window that captures most of them, then rerun the matrix with a shorter and a longer window to see which types are sensitive to the choice.
Should auto-closed tickets count as resolved exceptions?
No. A case closed by automation after customer silence has no confirmed outcome. Keep it in the matrix as its own type, label it as unconfirmed, and never count it toward a settled floor.
Do we need to read ticket text to build the matrix?
Usually not. Reopen counts, group changes, parent and child links, merge markers, status history and modified dates are structured fields in most help desks, so the matrix can be built from metadata. Text review comes later, on the sampled cases, alongside privacy review.
Sources
- Zendesk's Ticket Metrics API returns a read-only reopens count for each ticket, and requester wait time resumes where it left off when a solved ticket is reopened. Source
- Dynamics 365 Customer Service can reopen a resolved case when an incoming email relates to it, through an automatic record creation rule an administrator configures. Source
- Dynamics 365 Customer Service supports parent and child cases, with a closure preference that either closes all child cases with the parent or blocks parent closure until all children are closed. Source
- Dynamics 365 lets administrators configure SLA pause conditions at the entity, SLA KPI and SLA item levels. Source
- Dynamics 365 does not allow updates to resolved or canceled cases by default, but an administrator can allow updates through Power Automate flows or the API without reopening, for example to remove data for GDPR compliance. Source
- Oakden-Rayner and colleagues showed that models can perform well overall but poorly on unlabeled subsets, with relative performance differences of over 20% on clinically important subsets. Source
- scikit-learn's stratified splitters keep relative class frequencies approximately preserved in each fold. Source
- tau-bench evaluates agents that follow domain policies by comparing the final database state with an annotated goal state, and introduces the pass^k reliability metric. Source
- Datasheets for Datasets proposes documenting a dataset's motivation, composition, collection process and recommended uses, including whether it is a sample of a larger set. Source
See if your company qualifies
A short company assessment. No data uploads are needed.