Skip to content

Privacy and preparation

Support dataset exception coverage: what is your sample missing?

By SourceX Editorial · Updated

10 sections

Short answer

To find support exceptions missing from a training sample, count exception types across the full case population using signals the help desk already stores: reopens, escalations, parent and child links, merges, SLA pauses, approved policy overrides and edits after resolution. Then compare each type's settled, fully linked cases in the sample against a floor set in advance. Thin or missing types are filled, labeled or declared.

Redaction

Support dataset exception

This page covers support dataset exception coverage: what is your sample missing.Illustrative

Key takeaways

  • Exception gaps cannot be seen from inside a sample; count every exception type across the full population first.
  • Most exception signals are structured fields, so the coverage matrix can be built from metadata before anyone reads ticket text.
  • Only cases whose outcome window has passed without a reopen, repeat contact or changed resolution count as settled.
  • Set the per-type floor in writing before counting, and report raw counts next to any share.
  • Each thin or missing type gets a recorded action: fill it with a targeted draw, label it, or declare it in the dataset documentation.

How do you find the support exceptions a training sample left out?

You find them by counting exception types in the full case population first, then checking how many complete, settled cases of each type the sample actually holds. A gap is invisible from inside the sample: if a random draw of 2,000 tickets contains no merged multi-customer incidents, nothing in those 2,000 rows tells you the type exists.

The method has four steps. Define exception types by signals the help desk already stores, such as reopen counts, escalation fields, parent and child links and post-resolution edits. Count each type across the whole population for the period you plan to share. Count the same types in the sample, keeping only cases whose outcome has settled. Then mark every type as covered, thin, missing or unmeasurable, and decide whether to fill, label or declare each gap.

This is a customer operations task as much as a data task. The person who knows that refunds outside the return window need a team lead's approval is the person who can tell you which field records that approval, and whether it was recorded at all before a policy change.

Which help desk signals mark a case as an exception?

An exception is any case that left the standard path: it came back after resolution, needed someone with more authority, affected several customers at once, or was changed after it closed. Most help desks record these moves as structured fields or status history, so exception typing can run on metadata before anyone reads ticket text.

Two platform examples show where the signals sit. Zendesk's Ticket Metrics API returns a read-only reopens count for each ticket, and Zendesk notes that requester wait time resumes where it left off when a solved ticket is reopened. Dynamics 365 Customer Service can reopen a resolved case when a new email arrives about it, through a rule an administrator configures, and supports parent and child cases with a closure preference that either closes all child cases with the parent or blocks the parent until every child is closed.

Which help desk signals mark a case as an exception?
Exception typeSignal in the recordWhere to lookWhat the sample loses without it
Reopened after resolutionReopen count above zero, or a status move from solved back to openZendesk ticket metrics; Dynamics 365 status history and reopen rulesCases where the first fix failed and the second attempt is the real answer
Escalated to another tier or teamGroup or queue change after first response; escalation flag or linked engineering issueTicket audit trail; linked Jira or engineering issueThe handoff note, the receiving team's diagnosis and the delay it caused
Multi-customer incidentParent case with child cases, or a problem ticket with linked incidentsDynamics 365 parent and child cases; problem and incident links where the help desk has themHow one root cause was communicated and closed across many customers
Merged or duplicateMerge marker, closed-as-duplicate reason or a pointer to a surviving caseTicket audit trail and closure reasonThe customer's first contact, often on a different channel
Paused for the customer or a third partySLA paused status or a pending or on-hold state with a reasonDynamics 365 SLA pause conditions; pending and on-hold statesWaiting periods that explain slow outcomes and should not be scored as agent delay
Policy exception approvedOverride reason, approver field or approval task on the caseCustom fields, approval workflow records, refund or credit systemThe rule that was bent, who had authority to bend it, and under which policy version
Edited after resolutionModified date later than resolution date on a resolved or canceled caseDynamics 365 case update settings; audit logsCorrections that change the recorded outcome after the fact
Closed without confirmationAutomatic closure after silence or a default resolution codeAutomation logs and closure reasonThe difference between a fixed problem and a customer who stopped replying

Why does a random ticket sample under-represent the cases that matter?

A random sample reproduces the population's proportions, so a type that makes up a fraction of a percent of cases arrives as a handful of rows, often with no settled outcome among them. A model trained or tested on that sample can look accurate overall while failing on exactly those cases.

Research on hidden stratification describes this failure. Oakden-Rayner and colleagues showed that models can perform well on average while performing poorly on subsets that were never labeled during training or testing, and reported relative performance gaps of over 20% on important subsets in their medical imaging work. Support data has the same structure: exception types are rarely labeled as such, so their failures disappear into an average.

Stratified sampling does not fix this on its own. scikit-learn's documentation notes that stratified splitters keep relative class frequencies approximately preserved in each fold, which only helps for classes you have labeled. Exception typing comes first; stratification can only use the labels it is given.

The task exception and outcome coverage matrix

The coverage matrix has one row per exception type and one column per question you need answered before the sample leaves the building. Build it from metadata exports, keep it beside the sample, and recompute it whenever the sample changes. The table below defines each column and the condition that should raise a flag.

Set the floor for each type before counting, in writing, so the matrix cannot be tuned to make a weak sample look complete. A floor is the minimum number of complete, settled cases a type needs before anyone draws conclusions from it; teams usually set it higher for types that carry money, safety or compliance consequences.

The task exception and outcome coverage matrix
ColumnWhat it holdsHow to compute itFlag when
Exception typeOne named type from the signal table, defined by a field ruleRule written as a filter a second analyst can rerunThe rule depends on reading free text
Population casesCount of cases of this type in the full period and scopeRun the rule over the complete export, not the sampleThe count cannot be produced because the field did not exist for part of the period
Sample casesCount of cases of this type in the sampleRun the same rule over the sampleZero, or far below the population share
Settled cases in sampleSample cases whose outcome window has passed with no reopen or follow-upApply the maturity rule in the next sectionBelow the floor set for this type
Handoff chain intactSettled cases where every escalation, child case or merge can be followed to its endCheck linked records resolve to an existing case or issueLinks point to deleted, archived or unexported records
Policy version knownSettled cases where the policy in force on the case date can be identifiedJoin case dates to a dated policy registerPolicy changes are undocumented or undated
StatusCovered, thin, missing or unmeasurableCovered meets the floor on all checks; thin is below it; missing has zero; unmeasurable lacks the signalAny type is thin or missing and the sample is described as representative
ActionFill, label or declareChosen per type and recorded with a reasonNo action is recorded for a thin or missing type

When does a resolved support case count as settled?

A case counts as settled only when its outcome window has passed with no reopen, no follow-up contact from the same customer about the same issue, and no later edit that changed the recorded resolution. A solved status on its own proves only that someone pressed a button.

Pick the outcome window from your own reopen data: look at how long after resolution reopens and repeat contacts usually arrive, and choose a window that captures most of them. Cases resolved inside that window at the time of the export are immature. Keep them out of the settled count, or include them with an explicit immature label.

Check edits as well as reopens. Dynamics 365 blocks updates to resolved and canceled cases by default, but an administrator can allow updates through Power Automate flows or the API without reopening the case; Microsoft gives removing personal data under GDPR as one reason. A resolved case modified after its resolution date needs a look before its outcome is trusted.

Downstream evidence makes the strongest proof: a refund that posted in the billing system, a bug fix that shipped in a named release, a replacement that the warehouse received back. Satisfaction ratings help but are sparse and often reflect the conversation rather than the fix.

Which window, typing rule or scope choice flips a type's status?

Three assumptions move the matrix more than any other: the rule that assigns a case to an exception type, the length of the outcome window, and the boundary of the population. Change any of them and a type can move from covered to thin.

Test each one directly. Rerun the matrix with a shorter and a longer outcome window and note which types change status. Have a second person apply the typing rules to a small set of cases and compare. Recount the population with and without brands, channels or periods that were added late. Any type whose status flips under a reasonable change should be reported as assumption-dependent, not as covered.

Report counts, not just shares. A type with a handful of cases cannot support a precise percentage, so show the raw count next to any share and say plainly when a count is too small to estimate a rate. Where the signal did not exist for part of the period, report the type as unmeasurable for that part rather than treating the absence as zero.

Illustrative: a subscription software help desk checks its sample

The figures below are illustrative, not observed. A software company plans to share two years of support cases, about 40,000 in total, and has drawn a random sample of 2,000. Its operations lead sets a floor of 30 settled, fully linked cases per type and a 45-day outcome window based on when its reopens usually arrive.

Reopens and escalations clear the floor. Multi-customer incidents do not: the population holds about 160 parent cases, the sample holds 8, and only 3 have every child case exported. Refunds approved outside policy are unmeasurable for the first nine months, because the approver field was added partway through the period.

Illustrative: a subscription software help desk checks its sample
Exception typePopulation casesSample casesSettled and linked in sampleStatus and action
Reopened after resolutionAbout 2,40011896Covered; keep
Escalated to engineeringAbout 1,6008461Covered; keep, check issue links resolve
Multi-customer incidentAbout 16083Thin; fill with a targeted draw of parent cases and all children
Refund outside policyAbout 6002911 with approver recordedUnmeasurable before field existed; label by period and declare
Edited after resolutionAbout 50027Not applicableReview edits; exclude cases whose outcome changed
Closed without confirmationAbout 3,0001510 settled by definitionLabel as unconfirmed; never count as resolved

Fill the gap, label it, or declare it?

Fill a gap when the cases exist in the population and can be exported intact: draw extra cases of that type, keep a record of the sampling weight, and report the type separately so the oversampling does not distort overall rates. Label a gap when the cases are present but their outcome or approval is uncertain, so a user can include or exclude them deliberately. Declare a gap when the cases cannot be supplied, and write it into the dataset's documentation.

Declaring is not a failure. Datasheets for Datasets, the documentation framework proposed by Gebru and colleagues, asks dataset creators to describe composition, including whether a dataset is a sample of a larger set and how it was drawn. A plain statement that multi-customer incidents are under-represented and why is more useful to a buyer than a sample described as representative.

Some exception types need a privacy or legal reviewer before they are filled at all. Escalations to a legal team, chargeback disputes, account security cases and complaints that name employees tend to carry more personal and third-party data than routine tickets, and oversampling them concentrates that risk. Decide those types with the privacy reviewer before drawing extra cases.

Policy exceptions are the hardest type to supply and the most useful

Cases where staff bent a rule with approval are the type agent builders most often lack, because public benchmarks write their policies by hand. tau-bench, a benchmark for agents that follow domain policies in retail and airline settings, grades each task by comparing the final database state with an annotated goal state and measures consistency across repeated trials with a metric it calls pass^k.

Real policy-exception cases can supply that structure from actual operations, but only if three things are recorded: the policy text in force on the case date, the approval with the approver's role, and the action that followed. If any of the three is missing, the case can still show the conversation, but it cannot show what the correct end state was.

What the matrix changes when support records are licensed

Support teams that decide to license their history to AI developers can do it through SourceX, whose work runs from rights review and contracting to data preparation and delivery. A coverage matrix is useful evidence in that process because it shows what a dataset contains before anyone argues about what it is worth.

Checking eligibility does not require handing over tickets: the first step is a short company assessment, and no case leaves the help desk until the company has approved the scope and terms and signed. A finished matrix and its declared gaps make that scope conversation faster and more precise.

Frequently asked questions

What should the task exception and outcome coverage matrix include?

One row per exception type, defined by a rule on structured fields, with counts for the full population, the sample, settled cases, cases with an intact handoff chain and cases with a known policy version. Each row ends with a status of covered, thin, missing or unmeasurable and a recorded action of fill, label or declare.

Is a random sample of support tickets ever good enough?

It can be for describing routine contact reasons, where every type is common. It is rarely enough for training or testing an agent on exceptions, because rare types arrive in tiny numbers and often without settled outcomes. Check the matrix before deciding.

How long should we wait before a resolved case counts as settled?

Use your own reopen and repeat-contact history. Measure how long after resolution those usually arrive and choose a window that captures most of them, then rerun the matrix with a shorter and a longer window to see which types are sensitive to the choice.

Should auto-closed tickets count as resolved exceptions?

No. A case closed by automation after customer silence has no confirmed outcome. Keep it in the matrix as its own type, label it as unconfirmed, and never count it toward a settled floor.

Do we need to read ticket text to build the matrix?

Usually not. Reopen counts, group changes, parent and child links, merge markers, status history and modified dates are structured fields in most help desks, so the matrix can be built from metadata. Text review comes later, on the sampled cases, alongside privacy review.

Sources

  • Zendesk's Ticket Metrics API returns a read-only reopens count for each ticket, and requester wait time resumes where it left off when a solved ticket is reopened. Source
  • Dynamics 365 Customer Service can reopen a resolved case when an incoming email relates to it, through an automatic record creation rule an administrator configures. Source
  • Dynamics 365 Customer Service supports parent and child cases, with a closure preference that either closes all child cases with the parent or blocks parent closure until all children are closed. Source
  • Dynamics 365 lets administrators configure SLA pause conditions at the entity, SLA KPI and SLA item levels. Source
  • Dynamics 365 does not allow updates to resolved or canceled cases by default, but an administrator can allow updates through Power Automate flows or the API without reopening, for example to remove data for GDPR compliance. Source
  • Oakden-Rayner and colleagues showed that models can perform well overall but poorly on unlabeled subsets, with relative performance differences of over 20% on clinically important subsets. Source
  • scikit-learn's stratified splitters keep relative class frequencies approximately preserved in each fold. Source
  • tau-bench evaluates agents that follow domain policies by comparing the final database state with an annotated goal state, and introduces the pass^k reliability metric. Source
  • Datasheets for Datasets proposes documenting a dataset's motivation, composition, collection process and recommended uses, including whether it is a sample of a larger set. Source

See if your company qualifies

A short company assessment. No data uploads are needed.

See if your company qualifies