Skip to content

AI uses for records

Support resolution ground truth, graded the way medicine grades tests

By SourceX Editorial · Updated

9 sections

Short answer

Support resolution ground truth should come from a reference standard, not from a closed status. Borrow the structure diagnostic research uses: treat closure, survey scores and the agent's note as index tests, and verify each claim in the reply against the policy in force, the action record and the account state after a fixed interval, applying the same check to every sampled case with graders blind to the closure code.

Grader rubric

Support resolution ground

This page covers support resolution ground truth, graded the way medicine grades tests.Illustrative

Key takeaways

  • A closed status records which status someone chose; ground truth needs evidence independent of the agent who closed the case.
  • Treat closure, resolution codes, survey scores and wrap-up notes as index tests, and build a reference standard from policy, action records, resulting state and an interval.
  • Each claim in a reply needs its own record: a refund can be created, then fail, as Stripe's refund lifecycle shows.
  • Verify a random sample within each closure code, not just reopened or complained-about cases, or silent failures keep their resolved label.
  • Grade into five verdicts, report the unverifiable share separately and count a different path that solves the problem within policy as verified.

What counts as support resolution ground truth?

Support resolution ground truth is a verdict, reached from evidence independent of the agent who closed the case, that the customer's problem was fixed correctly, within the policy in force at the time, and stayed fixed. A closed status is not that verdict. It records which status someone selected: in Salesforce's Case data model, for example, the closed flag is controlled by the Status field and cannot be set directly.

The gap matters because closure is cheap to produce and easy to imitate. A model trained on closed tickets as positives learns whatever ends conversations, such as a confident promise, an apology or a help link, and a model graded against closure is rewarded for the same habits. Neither score says whether the refund landed, the account changed or the fault stopped coming back.

Diagnostic medicine met the same problem decades ago, with quick tests on one side and slow, expensive truth on the other, and built a vocabulary for it. This guide borrows that vocabulary, applies it to the chain from reply to system action to resulting state, and ends with an annotated packet and a five-verdict rubric a team can run on its own cases.

Closure is an index test: the diagnostic accuracy frame for support

In diagnostic accuracy research, the quick signal being evaluated is the index test, and the best available procedure for establishing the truth is the reference standard; a study is only as trustworthy as that standard and the way it was applied. QUADAS-2, the appraisal tool for such studies hosted by the University of Bristol, asks whether the reference standard is likely to classify the target condition correctly, whether its results were read without knowledge of the index test, whether every patient received a reference standard and the same one, and "Was there an appropriate interval between index test(s) and reference standard?" Each question is answered yes, no or unclear.

Support data maps onto that structure almost term for term. Closure status, the resolution code, the satisfaction score and the agent's wrap-up note are index tests: cheap, present on every case and produced by the very process under judgment. The reference standard is the evidence that the outcome happened: the policy version in force, the action record in the system that executes the fix, the account state afterward and any contrary event inside a set interval. The translation:

Closure is an index test: the diagnostic accuracy frame for support
Diagnostic accuracy termSupport equivalentWhat goes wrong when it is skipped
Target conditionThe customer's problem fixed correctly, within the policy in force, and still fixed after the intervalGraders reward different things, and low agreement gets blamed on the graders
Index testClosure status, resolution code, satisfaction score, the agent's wrap-up noteThe proxy ends up graded against itself
Reference standardPolicy version, action record, resulting account state, contrary events in the intervalLabels inherit every mistake the proxy hides
Interval between test and referenceA wait before the resulting state is checked, fixed per issue type in advanceFailures that surface later, such as a refund that bounces, are scored as wins
Reading the reference blind to the index testGraders see the case evidence but not the closure code or survey scoreGraders anchor on the closed flag and the five-star rating
Same reference standard for every caseEvery sampled case checked against the same kinds of system recordsScores move with the mix of issue types instead of with quality
Every case verifiedAll sampled cases get the check, not only reopened or complained-about onesSilent failures keep their resolved label

From reply to action to resulting state: what each claim needs

Every resolution reply bundles several claims, and each claim needs its own record. A reply saying the duplicate charge has been refunded asserts that a refund was created, that it reached the customer and, implicitly, that the duplicate will not recur. The first is settled by an action record, the second by the resulting state and the third only by time.

Payments show how far apart those links can drift. In Stripe's refund lifecycle a refund can be requires_action, pending, succeeded, failed or canceled. A card refund can sit pending until the balance covers it, and a refund is marked failed when, for instance, the customer's bank account has been closed, with a failure_reason such as expired_or_canceled_card or lost_or_stolen_card and a refund.failed event sent to the merchant. Stripe adds that returned funds can take up to 30 days from the post date to come back. A ticket closed when the refund was created is closed before two of its three claims can be checked.

The pattern holds beyond payments. A plan downgrade is only a request until the subscription record shows the new plan on the next invoice; a replacement shipment is only a label until the carrier shows delivery; a password reset is only a link until the customer signs in. For each issue type, write down which system holds the action record, which holds the resulting state and how long effects can stay open.

A 2026 preprint on agent evaluation states the timing problem formally. Casheekar and Tangirala argue that a score taken when a run stops supports a final verdict only if later events cannot change it; in their controlled replay, labels taken at the endpoint differed from final labels for every delayed operation, and they propose an open-effects record for operations that may still matter afterward (arXiv 2608.14940). In support terms a pending refund, a scheduled cancellation and a parcel in transit are open effects, and a case that still has one is not ready for a final label.

Why partial verification is the default, and how to undo it

Most support teams verify the cases that complain. Quality reviews pull escalations, reopens and low survey scores, so the cases that get a second look are the ones that already failed loudly, while a silent failure, such as a refund that bounced after a five-star rating, keeps its resolved label. That is why QUADAS-2 asks whether every patient received the reference standard: when only part of a sample is verified, the unverified part is rarely a random part.

Undo it by sampling before looking. Draw the audit sample at random within each closure code and issue type, apply the full reference standard to every drawn case, including those closed as resolved with top scores, and report one number per issue type: the share of closed-as-resolved cases the reference standard contradicts. That number says how much the closure labels are worth for that issue type.

Two neighboring traps need their own checks. The first is circular evidence: accepting the agent's note that a refund was processed as the action record, when the note is part of what is under test; require the record from the system that executed the action. The second is mixed standards: billing cases checked against the payment processor while the rest are checked only against the helpdesk, so scores shift with the issue mix rather than with quality; report verdicts separately for each reference standard used.

An annotated resolution packet for a duplicate charge (illustrative)

The packet is illustrative: an invented subscription software customer, not a real record. The customer was billed twice after changing plans; the agent replied that the duplicate charge had been refunded and set the ticket to solved, and the customer rated the interaction five stars. Each row names the record that supplies the field and whether the field may enter the label.

Notice where the evidence lives. Four of the six rows depend on records outside the helpdesk: invoice history, a versioned policy store and two kinds of payment processor record. A dataset that exports only tickets can describe this case but cannot grade it, and a grader shown the closure code and the five stars first would be tempted to stop reading before the row that decides it.

The deciding row is the resulting state. When the case was assembled after the interval the team had fixed for billing issues, the payment record showed the refund as failed with an expired-card reason, a refund.failed event on file and no second attempt, while the ticket still read solved. Under the rubric in the next section the case is contradicted, which is exactly the kind of case a closure-based label counts as a success:

An annotated resolution packet for a duplicate charge (illustrative)
Packet fieldIllustrative contentRecord that supplies itEnters the label?
Customer problemTwo charges for one billing period after a plan changeInvoice history and the customer's first messageYes: defines the target condition
Policy in forceDuplicate charges refunded in full to the original payment method, in the version effective on the contact datePolicy store versioned by effective dateYes: defines what a correct fix was
Reply claimThe agent writes that the duplicate charge has been refundedTicket transcriptYes, as the claim under test
Action recordRefund created against the duplicate charge the same dayPayment processor refund objectYes: shows the action happened
Resulting stateRefund failed, reason expired or canceled card, no retryPayment processor eventsYes: the deciding evidence
Closure code and surveySolved; five starsHelpdesk fields and survey responseNo: kept as index tests and hidden from graders

The grading rubric: five verdicts and what each requires

Grade every case into one of five verdicts and keep the verdict apart from the closure code. A case earns a positive label only when every claim in the reply is matched by an action record and a confirmed resulting state, and the interval has passed with no contrary event.

Two rules keep the rubric honest. Unverifiable is not a success, so its share is reported beside the results instead of being folded into them. And a different path that solves the customer's problem within policy counts as verified even when it departs from the expected action, because a fixed expected answer can punish a better fix: Anthropic's guide to agent evaluations recounts a model that solved a tau2-bench flight-booking problem through a loophole in the policy, failing the evaluation as written while producing a better outcome for the user. The verdicts:

The grading rubric: five verdicts and what each requires
VerdictEvidence requiredUse in training or evaluationIllustrative trigger
Verified, durableAction record matches every claim; resulting state confirmed; no contrary event before the interval endsPositive label; eligible for the evaluation setRefund succeeded, no dispute or repeat contact within the interval
Verified, different pathProblem solved within policy by an action other than the expected one, with the customer's agreementPositive label; the expected-action field is revised, not the verdictCredit on the next invoice instead of a card refund, accepted by the customer
ProvisionalAction and state confirmed, but an open effect remainsHeld out of labels until the effect resolves; counted separatelyRefund still pending when the data is exported
ContradictedA record shows a claim false: no action, a failed or reversed action, or a contrary event inside the intervalNegative label, with the failing link namedRefund failed with an expired-card reason while the ticket stayed solved
UnverifiableA record needed for any step is missing or cannot be joinedExcluded from both labels; its share reportedNo payment processor export for the ticket's account
Decision tree for grading a support resolution: a missing record makes the case unverifiable, records that do not confirm the fix make it contradicted, remaining open effects make it provisional, and otherwise it is verified, with closure code and satisfaction score kept out of the label.
The rubric from this article, with its two verified verdicts drawn as one leaf.

Testing the reference standard on held-out cases

Before the verdicts become labels, test whether two people applying the reference standard reach the same verdict on cases neither used to design the rubric. Draft the rubric on one period, then have two graders, blind to closure codes and survey scores, grade a random sample from a later period. Report agreement per verdict and per issue type, plus the contradicted share among cases closed as resolved.

Fix the interval in writing before grading. The explanation document for QUADAS-3, the tool's successor, says the interval should suit the question, short enough that the condition has not changed, and that reviewers should prespecify what counts as appropriate. Support adds a twist: the condition can change in either direction, so the interval must also be long enough for the payment rail, carrier or fault to reveal a failure. The separate SourceX guide on support outcome measurement lag covers how to set that window from reopen and follow-up timing.

Then check that the labels make a difference. Score one model, or one candidate dataset, on the held-out set twice, once against closure-based labels and once against reference-standard verdicts. If the rankings or pass rates diverge for an issue type, the reference standard is carrying information the proxy lacks; if they match, the cheaper proxy may be adequate there, and the audit has earned its cost by showing it.

Could closure plus a high satisfaction score be close enough?

Sometimes, and the audit described above is how to find out where. For issue types fixed inside the conversation, such as a how-to question, closure and a good rating may track the reference standard closely. For issue types whose fix runs through another system, such as refunds, cancellations, replacements and account changes, the payment example shows how the two can split without anyone noticing.

The strongest form of the objection is economic: verifying cases costs money, and a large pile of noisy closure labels might train a better model than a small verified set. That is a real tradeoff, but it argues for a verified sample, not against one. The sample is what reveals how noisy the closure labels are and for which issue types they can be trusted, and it is the only sound basis for an evaluation set. Training can tolerate label noise that evaluation cannot, because an evaluation set with silent failures marked as wins rewards exactly the confident, unverified replies a team is trying to remove.

Writing resolution evidence into a data request

Ask for support histories joined to the systems where fixes execute and settle: payment processor events, subscription and order records, shipment tracking and account audit logs, each timestamped and tied to the ticket by a stable key. Without those joins, a resolution can be described but not verified.

SourceX looks for U.S. companies whose records fit a buyer's written requirement instead of offering datasets from a shelf, so the requirement sets what counts as a match. Name the issue types in scope, the action and state systems for each, the interval for each, the minimum share of cases you need in each verdict class and an acceptance test, such as blind agreement between two graders on a held-out sample. Candidate records are then assessed company by company, and the supplying company approves the scope and signs before any data is delivered.

Frequently asked questions

Is a solved or closed status ever good enough as ground truth?

It can be for issue types fixed inside the conversation, but only an audit shows which ones. Verify a random sample of closed cases against the action record and resulting state, and use closure as a label only where the contradicted share is low enough for the intended use. Evaluation sets should use the reference standard either way.

What is partial verification in support data?

It means checking only some cases against the full evidence, usually reopens, escalations and low ratings. Because those cases already failed visibly, silent failures among the rest keep their resolved label. Draw the sample at random within each closure code before deciding which cases to verify.

How long should the wait be before the resulting state is checked?

Set it per issue type before grading, long enough for the systems involved to reveal a failure. A payment rail that can take weeks to return a failed refund needs a longer interval than a password reset. Cases with open effects at export stay provisional.

Should graders see the closure code or satisfaction score?

No. Graders should see the customer's problem, the policy in force, the reply and the system records, but not the closure code or survey score, which are the signals under test. Showing them invites graders to anchor on the outcome the agent claimed.

What happens to cases whose records are missing?

They are graded unverifiable and left out of both positive and negative labels. Report their share, because a high unverifiable rate in one issue type can make the remaining verdicts unrepresentative of that issue type.

Sources

  • The QUADAS-2 tool's signalling questions ask whether the reference standard is likely to classify the target condition correctly, whether it was interpreted without knowledge of the index test, and whether there was an appropriate interval between index test(s) and reference standard. Source
  • The QUADAS-2 background document explains that signalling questions are answered yes, no or unclear, with yes indicating low risk of bias, and the flow and timing domain covers whether patients received a reference standard and the same one. Source
  • The QUADAS-3 explanation and elaboration says the interval between index test and reference standard should suit the question, be short enough that the condition has not changed for some questions, and be prespecified by review authors. Source
  • Anthropic's guide to agent evaluations describes a model that solved a tau2-bench flight-booking problem through a policy loophole, failing the evaluation as written while producing a better solution for the user. Source
  • Casheekar and Tangirala (arXiv, August 2026) argue that an endpoint score supports a final verdict only with outcome finality; in a controlled replay, endpoint labels differed from final labels for every delayed operation, and they propose an open-effects record. Source
  • Stripe's refund documentation lists refund states including requires_action, pending, succeeded, failed and canceled, marks a refund failed when, for example, the customer's bank account is closed, records a failure_reason, sends a refund.failed event, and says returned funds can take up to 30 days from the post date. Source
  • In Salesforce's Case data model object, the closed flag shows whether the case is closed or open, is controlled by the Status field and cannot be set directly. Source

Tell us what data you need

Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.

See if your company qualifies