Skip to content

Privacy and preparation

How to validate whether a resolved status is a reliable outcome label

By SourceX Editorial · Updated

10 sections

Short answer

Validate a resolved status by testing it against independent evidence that the problem was solved, on random samples drawn separately for each way cases reach resolved, such as agent, customer and automated closures. Define verified in writing, size each stratum's sample from the precision you need, estimate validity with intervals, and ship each stratum as a verified label, a weak label, or closed but unverified.

Grader rubric

Dataset outcome label

This page covers how to validate whether a resolved status is a reliable outcome label.Illustrative

Key takeaways

  • Resolved and closed are workflow states set by people and rules; they do not by themselves show that a problem was fixed.
  • Define verified with an evidence ladder, from independent downstream confirmation down to rule closures and contradictions.
  • Stratify by how a case reached resolved, because automated and bulk closures behave differently from agent closures.
  • Size samples per stratum from the interval width you need, and decide from the interval's lower bound.
  • Ship each stratum with its measured validity, and relabel what cannot be verified instead of dropping it silently.

How do you validate whether a resolved status is a reliable outcome label?

Validate a resolved status by testing it against evidence that the problem was actually solved, on a random sample drawn separately from each way a case can reach resolved. Write down what a verified outcome means, rank the evidence that can confirm it, sample within strata such as agent closures and automated closures, have reviewers check each sampled case against that evidence, and estimate validity per stratum with a confidence interval.

The result is not a single accuracy number for the whole dataset. It is a validity estimate for each stratum, and a decision for each: ship the status as a verified outcome label, ship it as a weaker label with its measured validity, relabel it as closed but unverified, or exclude it.

This page is the supplier's protocol for running that test before records are offered, so the outcome field in a dataset says what it can support. It complements buyer-side verification methods by fixing the sampling design, sample sizes and decision rules in advance.

Why is resolved a workflow state rather than an outcome?

A resolved or solved status records that a workflow step ended, not that the customer's problem went away. Helpdesk and project tools are built to move work along, and their statuses carry the rules of the tool.

Zendesk is a clear example. Its documentation says agents can set a ticket to solved but cannot close it; closing is done by triggers or automations, and the standard automation closes tickets four days after they are set to solved. A customer reply to a closed ticket creates a follow-up ticket instead of reopening the original. A solved status therefore says an agent ended the conversation, and a closed status mostly says time passed, while the evidence that the fix worked, or did not, may sit in a different ticket.

Jira separates the idea further. Atlassian's guidance says any issue with the Resolution field populated counts as resolved for Jira apps, whatever its status, and recommends checking for issues in a done status with no resolution and the reverse. In a poorly configured project, resolved and done can disagree on the same issue. Before sampling anything, find out which field your system treats as resolution and how it is set.

What counts as a verified outcome? The evidence ladder

A verified outcome is one confirmed by evidence independent of the person or rule that closed the case. Rank the evidence available in your systems into tiers, then decide in writing which tiers count as verified for this dataset. The ladder below is the definition half of the protocol.

  • Recontact window: choose it from how long problems in this business take to resurface, such as days for a password reset and weeks for a billing or hardware fault, and write the reason down.
  • Same-issue test: decide how a later contact is matched to the original, for example same customer and same product or asset, and whether a reviewer confirms the match.
  • Evidence independence: count a tier only if the evidence was produced by someone other than the person or rule that closed the case.
  • Contradiction precedence: a contradiction found later overrides any confirmation found earlier.
What counts as a verified outcome? The evidence ladder
TierEvidenceSupport exampleField service or finance example
E1 Independent confirmationA downstream record shows the outcome happenedRefund transaction posted; account setting changed in product logsPart installed and no callback in the warranty system; payment applied
E2 Customer confirmationThe customer states the issue is fixedReply confirming the fix; positive rating tied to the ticketSigned completion form; customer acceptance
E3 Absence of recontactNo related contact within an agreed windowNo follow-up ticket on the same issueNo return visit for the same asset
E4 Agent assertionOnly the closing person's notes describe a fixClosing note says the fix was appliedTechnician note says repaired
E5 Rule or bulk closureA rule, script or migration closed the caseAutomation closed after customer silenceBatch close at year end
E6 ContradictionLater evidence says the outcome was wrongFollow-up ticket, reopen or refund after resolutionCallback, chargeback or reversal

The outcome label validity sampling protocol

The protocol is ten steps, and steps 1 to 4 are written down before any record is reviewed. Fixing the definition, strata, sample sizes and decision rule in advance keeps the result from being shaped by what reviewers happen to see first.

Two practical choices make the review honest. Hide the status and resolution code from reviewers where the review tool allows it, so they judge the evidence rather than confirm the label. And give reviewers a fixed time budget per case with an explicit option to record that the evidence was insufficient, because a forced verdict on a thin record inflates the verified share.

  • 1. Label definition: state the outcome the label should mean, for example the customer's issue was fixed without a later related contact within a set window, and which evidence tiers count as verified.
  • 2. Frame: define the population (systems, period, case types), list exclusions with reasons and record the export it was drawn from.
  • 3. Strata: split by how a case reached resolved (agent, customer, automation, bulk job, migration), and by period or system version, outcome class and team where they differ.
  • 4. Sample size and decision rule: set the target interval width per stratum and the acceptance thresholds before drawing records (see the next two sections).
  • 5. Draw: select records at random within each stratum using a recorded seed, and keep the list of selected IDs.
  • 6. Evidence pull: for each sampled case, gather the event log, linked records, later contacts and downstream transactions into one review view.
  • 7. Review: assign each case the highest evidence tier found and a verdict of verified, unverified or contradicted, without relying on the status field itself.
  • 8. Agreement: have a second reviewer independently review an overlapping subset, measure agreement and adjudicate every disagreement.
  • 9. Estimate: compute the verified share per stratum with a confidence interval, plus the contradicted share.
  • 10. Decide and record: apply the decision rule from step 4 and write the results, exceptions and rule into the dataset documentation.

How large should the sample be in each stratum?

Size each stratum's sample from how precisely you need to know its validity, not from a percentage of the stratum. A small stratum and a large one need roughly the same sample for the same precision.

The figures below use the standard normal approximation for a 95 percent confidence interval on a proportion, the method statsmodels implements in its sample size function for confidence intervals. They are arithmetic, not observations. When you do not yet know the stratum's validity, plan for the worst case of one half; if earlier evidence suggests validity near 90 percent, a smaller sample gives the same width. Strata of only a few hundred cases need a finite population adjustment, or a full review.

How large should the sample be in each stratum?
Interval half-widthSample per stratum, validity unknownSample per stratum, validity near 90 percentUse when
Plus or minus 10 points9735Screening a stratum to decide whether it needs closer study
Plus or minus 7 points19671Routine documentation of a mid-sized stratum
Plus or minus 5 points385139Strata that a buyer will use as evaluation ground truth
Plus or minus 3 points1,068385Large, high-stakes strata where small differences change the decision

How do you turn the estimates into an acceptance decision?

Decide from the lower bound of each stratum's interval, not from its point estimate, so a small sample cannot pass on luck. For the interval itself, prefer the Wilson score or the Clopper-Pearson interval over the plain normal one when validity is close to 0 or 1 or samples are small. The statsmodels documentation notes that the Clopper-Pearson interval has coverage of at least the nominal level but is generally conservative, while most other methods have average coverage at the nominal level and can fall below it in some cases.

Check reviewer agreement before trusting any estimate. The scikit-learn documentation describes Cohen's kappa as a measure for comparing labelings by different human annotators and notes that scores above 0.8 are generally considered good agreement. Low agreement means the label definition from step 1 is unclear, and the fix is to rewrite it and re-review, not to average the reviewers.

The thresholds in the table are an example to agree with the buyer, not a standard. Set them for the buyer's use: evaluation ground truth needs a stricter bar than a weak training signal.

How do you turn the estimates into an acceptance decision?
Result for the stratumExample thresholdDecisionLabel to ship
High verified share, no meaningful contradictionLower bound at or above the agreed bar for ground truthAccept as verified outcomeOutcome label with stratum validity documented
Moderate verified shareLower bound below the ground-truth bar but above the weak-label barAccept as weak labelOutcome label marked weak, with interval
Low verified share, few contradictionsLower bound below the weak-label barRelabelClosed, unverified
Contradicted share materialContradictions above the agreed toleranceInvestigate the cause, then relabel or excludeContradicted or excluded
Reviewer agreement poorKappa below the agreed barRewrite the definition and re-reviewNone until resolved

Which closure pattern passes every format check yet sinks the label?

The defect most likely to change a buyer's acceptance decision while every file opens is a resolved class dominated by rule or bulk closures. The status, codes and timestamps all look normal, but the outcome was never judged by anyone. Stratifying by closure actor in step 3 is what exposes it; a sample drawn across the whole dataset can bury a large automated stratum inside an average that looks acceptable.

Two other defects behave the same way. Follow-up contacts that live in separate tickets make a failed fix look final unless the review links them, which is why step 6 pulls later contacts. And a resolution field that disagrees with status, as in a misconfigured Jira workflow, can mark unfinished work as resolved.

Reopen handling deserves its own check. Some systems overwrite the resolution code when a case is reopened and closed again, so the final value describes the last attempt while the history of earlier failed fixes survives only in the event log. If the delivered outcome field comes from current values, sample reopened cases separately and decide whether the label should describe the first resolution, the last one or the whole case.

Do not assume curated labels are clean either. A study of ten widely used machine learning test sets estimated label errors averaging at least 3.3 percent, and operational outcome fields are recorded under far less review than benchmarks.

Illustrative worked example: a software company's support history

Illustrative example, invented to show the method: a business software company prepares four years of support tickets so buyers can train and grade support agents on resolved cases. Tickets reach solved in three ways: an agent solves them, a customer marks them solved in the portal, or an automation solves them after a period of customer silence.

The team defines verified as an E1 or E2 confirmation, or E3 with no related contact within the agreed window. It samples 196 tickets from each of the three strata, has a second reviewer check a quarter of them, adjudicates disagreements and computes Wilson intervals. In this invented example, customer-solved tickets clear the ground-truth bar, agent-solved tickets fall in the weak-label band because many have only agent notes, and automation-solved tickets fall below the weak-label bar.

The company ships agent-solved and customer-solved tickets with their stratum validity documented, relabels automation-solved tickets as closed, unverified, and links follow-up tickets to their originals in the delivery. None of these figures describe a real company or dataset.

When is a status only workflow data, not a label?

Stop treating a status as an outcome label when no independent evidence exists for most cases: no linked transactions, no customer confirmations and no reliable way to detect recontact. The records may still be useful as process data, but the outcome field should be described as a workflow status.

Steps

When is a status only workflow data, not a label

Narrow the scope when validity is concentrated: offer the strata, periods or teams that pass, and document the rest as unverified. Request specialist review when outcomes affect people, such as claims, credit, hiring or complaints, where recorded outcomes can carry the policies of the people who made them, and when reviewers need domain expertise to judge whether a fix worked. Document the label definition, strata, sample sizes, intervals and decisions alongside the dataset, following the practice Datasheets for Datasets recommends of recording how data was collected and labeled.

  • Stop: no independent evidence for most cases, or reviewer agreement stays poor after rewriting the definition.
  • Narrow: passing and failing strata are clearly separable by closure actor, period or team.
  • Specialist review: decisions about people, regulated records or fixes that need domain judgment.
  • Proceed: each stratum has a decision and the results are written into the documentation.

Outcome validity in a SourceX licensing review

Among the stages SourceX runs for businesses licensing records directly to SourceX, from sourcing and qualification through rights review, contracting, data preparation, delivery and payment, a validity test like this one belongs to preparation, where its per-stratum results become part of the documentation a buyer reads before deciding how to use the outcome field.

Validity estimates are only stable with enough cases in each stratum, which is one reason SourceX's general guide points to two or more years of consistent history and tens of thousands of records or more in one system, while still judging each deal individually.

Whether your outcome-labeled records qualify is answered first by a short company assessment that requires no uploaded data.

Frequently asked questions

What should the outcome label validity sampling protocol include?

A written label definition with the evidence tiers that count as verified, a sampling frame, strata by closure actor and period, sample sizes and decision rules fixed in advance, random draws with a recorded seed, evidence-based review, a second-reviewer agreement check, per-stratum intervals and a documented decision for each stratum.

Is a closed ticket the same as a resolved outcome?

No. In many helpdesks a closed status is set by an automation after time passes, and a customer reply can create a new follow-up ticket instead of reopening the original. Closure says the workflow ended; only independent evidence says the problem was fixed.

Why stratify instead of sampling the whole dataset at random?

Because validity differs sharply by how a case was closed. A single random sample can hide a large stratum of automated closures inside an average that looks acceptable, while per-stratum estimates show exactly which part of the label can be trusted.

How many records should I review to validate an outcome label?

It depends on the precision you need per stratum, not on the dataset's size. With a 95 percent interval and unknown validity, about 97 records give plus or minus 10 points and about 385 give plus or minus 5 points; small strata need a finite population adjustment.

Can I keep cases whose outcome cannot be verified?

Yes, if they are labeled honestly, for example as closed, unverified, and their share is documented. Buyers can still use them as process or context data, but they should not be mixed silently with verified outcomes used for grading.

Sources

  • Zendesk documents that agents can set tickets to solved but not closed, that triggers or automations close tickets, and that replies to closed tickets create follow-up tickets. Source
  • Zendesk's standard automations include one that closes tickets four days after their status is set to solved. Source
  • Atlassian says any Jira issue with the Resolution field populated counts as resolved regardless of status, and recommends checking for mismatches between status and resolution. Source
  • The statsmodels documentation says the Clopper-Pearson interval has coverage of at least the nominal level but is generally conservative, while most other methods have average coverage at the nominal level. Source
  • The scikit-learn documentation describes Cohen's kappa as comparing labelings by different human annotators, with scores above 0.8 generally considered good agreement. Source
  • Northcutt, Athalye and Mueller estimate an average of at least 3.3 percent label errors across the test sets of 10 widely used datasets. Source
  • Datasheets for Datasets proposes documenting each dataset's motivation, composition, collection process and recommended uses. Source

See if your company qualifies

A short company assessment. No data uploads are needed.

See if your company qualifies