AI uses for records
Support resolution ground truth, graded the way medicine grades tests
By SourceX Editorial · Updated
9 sections
Short answer
Support resolution ground truth should come from a reference standard, not from a closed status. Borrow the structure diagnostic research uses: treat closure, survey scores and the agent's note as index tests, and verify each claim in the reply against the policy in force, the action record and the account state after a fixed interval, applying the same check to every sampled case with graders blind to the closure code.
Grader rubric
Support resolution ground
Key takeaways
- A closed status records which status someone chose; ground truth needs evidence independent of the agent who closed the case.
- Treat closure, resolution codes, survey scores and wrap-up notes as index tests, and build a reference standard from policy, action records, resulting state and an interval.
- Each claim in a reply needs its own record: a refund can be created, then fail, as Stripe's refund lifecycle shows.
- Verify a random sample within each closure code, not just reopened or complained-about cases, or silent failures keep their resolved label.
- Grade into five verdicts, report the unverifiable share separately and count a different path that solves the problem within policy as verified.
What counts as support resolution ground truth?
Support resolution ground truth is a verdict, reached from evidence independent of the agent who closed the case, that the customer's problem was fixed correctly, within the policy in force at the time, and stayed fixed. A closed status is not that verdict. It records which status someone selected: in Salesforce's Case data model, for example, the closed flag is controlled by the Status field and cannot be set directly.
The gap matters because closure is cheap to produce and easy to imitate. A model trained on closed tickets as positives learns whatever ends conversations, such as a confident promise, an apology or a help link, and a model graded against closure is rewarded for the same habits. Neither score says whether the refund landed, the account changed or the fault stopped coming back.
Diagnostic medicine met the same problem decades ago, with quick tests on one side and slow, expensive truth on the other, and built a vocabulary for it. This guide borrows that vocabulary, applies it to the chain from reply to system action to resulting state, and ends with an annotated packet and a five-verdict rubric a team can run on its own cases.
Closure is an index test: the diagnostic accuracy frame for support
In diagnostic accuracy research, the quick signal being evaluated is the index test, and the best available procedure for establishing the truth is the reference standard; a study is only as trustworthy as that standard and the way it was applied. QUADAS-2, the appraisal tool for such studies hosted by the University of Bristol, asks whether the reference standard is likely to classify the target condition correctly, whether its results were read without knowledge of the index test, whether every patient received a reference standard and the same one, and "Was there an appropriate interval between index test(s) and reference standard?" Each question is answered yes, no or unclear.
Support data maps onto that structure almost term for term. Closure status, the resolution code, the satisfaction score and the agent's wrap-up note are index tests: cheap, present on every case and produced by the very process under judgment. The reference standard is the evidence that the outcome happened: the policy version in force, the action record in the system that executes the fix, the account state afterward and any contrary event inside a set interval. The translation:
| Diagnostic accuracy term | Support equivalent | What goes wrong when it is skipped |
|---|---|---|
| Target condition | The customer's problem fixed correctly, within the policy in force, and still fixed after the interval | Graders reward different things, and low agreement gets blamed on the graders |
| Index test | Closure status, resolution code, satisfaction score, the agent's wrap-up note | The proxy ends up graded against itself |
| Reference standard | Policy version, action record, resulting account state, contrary events in the interval | Labels inherit every mistake the proxy hides |
| Interval between test and reference | A wait before the resulting state is checked, fixed per issue type in advance | Failures that surface later, such as a refund that bounces, are scored as wins |
| Reading the reference blind to the index test | Graders see the case evidence but not the closure code or survey score | Graders anchor on the closed flag and the five-star rating |
| Same reference standard for every case | Every sampled case checked against the same kinds of system records | Scores move with the mix of issue types instead of with quality |
| Every case verified | All sampled cases get the check, not only reopened or complained-about ones | Silent failures keep their resolved label |
From reply to action to resulting state: what each claim needs
Every resolution reply bundles several claims, and each claim needs its own record. A reply saying the duplicate charge has been refunded asserts that a refund was created, that it reached the customer and, implicitly, that the duplicate will not recur. The first is settled by an action record, the second by the resulting state and the third only by time.
Payments show how far apart those links can drift. In Stripe's refund lifecycle a refund can be requires_action, pending, succeeded, failed or canceled. A card refund can sit pending until the balance covers it, and a refund is marked failed when, for instance, the customer's bank account has been closed, with a failure_reason such as expired_or_canceled_card or lost_or_stolen_card and a refund.failed event sent to the merchant. Stripe adds that returned funds can take up to 30 days from the post date to come back. A ticket closed when the refund was created is closed before two of its three claims can be checked.
The pattern holds beyond payments. A plan downgrade is only a request until the subscription record shows the new plan on the next invoice; a replacement shipment is only a label until the carrier shows delivery; a password reset is only a link until the customer signs in. For each issue type, write down which system holds the action record, which holds the resulting state and how long effects can stay open.
A 2026 preprint on agent evaluation states the timing problem formally. Casheekar and Tangirala argue that a score taken when a run stops supports a final verdict only if later events cannot change it; in their controlled replay, labels taken at the endpoint differed from final labels for every delayed operation, and they propose an open-effects record for operations that may still matter afterward (arXiv 2608.14940). In support terms a pending refund, a scheduled cancellation and a parcel in transit are open effects, and a case that still has one is not ready for a final label.
Why partial verification is the default, and how to undo it
Most support teams verify the cases that complain. Quality reviews pull escalations, reopens and low survey scores, so the cases that get a second look are the ones that already failed loudly, while a silent failure, such as a refund that bounced after a five-star rating, keeps its resolved label. That is why QUADAS-2 asks whether every patient received the reference standard: when only part of a sample is verified, the unverified part is rarely a random part.
Undo it by sampling before looking. Draw the audit sample at random within each closure code and issue type, apply the full reference standard to every drawn case, including those closed as resolved with top scores, and report one number per issue type: the share of closed-as-resolved cases the reference standard contradicts. That number says how much the closure labels are worth for that issue type.
Two neighboring traps need their own checks. The first is circular evidence: accepting the agent's note that a refund was processed as the action record, when the note is part of what is under test; require the record from the system that executed the action. The second is mixed standards: billing cases checked against the payment processor while the rest are checked only against the helpdesk, so scores shift with the issue mix rather than with quality; report verdicts separately for each reference standard used.
An annotated resolution packet for a duplicate charge (illustrative)
The packet is illustrative: an invented subscription software customer, not a real record. The customer was billed twice after changing plans; the agent replied that the duplicate charge had been refunded and set the ticket to solved, and the customer rated the interaction five stars. Each row names the record that supplies the field and whether the field may enter the label.
Notice where the evidence lives. Four of the six rows depend on records outside the helpdesk: invoice history, a versioned policy store and two kinds of payment processor record. A dataset that exports only tickets can describe this case but cannot grade it, and a grader shown the closure code and the five stars first would be tempted to stop reading before the row that decides it.
The deciding row is the resulting state. When the case was assembled after the interval the team had fixed for billing issues, the payment record showed the refund as failed with an expired-card reason, a refund.failed event on file and no second attempt, while the ticket still read solved. Under the rubric in the next section the case is contradicted, which is exactly the kind of case a closure-based label counts as a success:
| Packet field | Illustrative content | Record that supplies it | Enters the label? |
|---|---|---|---|
| Customer problem | Two charges for one billing period after a plan change | Invoice history and the customer's first message | Yes: defines the target condition |
| Policy in force | Duplicate charges refunded in full to the original payment method, in the version effective on the contact date | Policy store versioned by effective date | Yes: defines what a correct fix was |
| Reply claim | The agent writes that the duplicate charge has been refunded | Ticket transcript | Yes, as the claim under test |
| Action record | Refund created against the duplicate charge the same day | Payment processor refund object | Yes: shows the action happened |
| Resulting state | Refund failed, reason expired or canceled card, no retry | Payment processor events | Yes: the deciding evidence |
| Closure code and survey | Solved; five stars | Helpdesk fields and survey response | No: kept as index tests and hidden from graders |
The grading rubric: five verdicts and what each requires
Grade every case into one of five verdicts and keep the verdict apart from the closure code. A case earns a positive label only when every claim in the reply is matched by an action record and a confirmed resulting state, and the interval has passed with no contrary event.
Two rules keep the rubric honest. Unverifiable is not a success, so its share is reported beside the results instead of being folded into them. And a different path that solves the customer's problem within policy counts as verified even when it departs from the expected action, because a fixed expected answer can punish a better fix: Anthropic's guide to agent evaluations recounts a model that solved a tau2-bench flight-booking problem through a loophole in the policy, failing the evaluation as written while producing a better outcome for the user. The verdicts:
| Verdict | Evidence required | Use in training or evaluation | Illustrative trigger |
|---|---|---|---|
| Verified, durable | Action record matches every claim; resulting state confirmed; no contrary event before the interval ends | Positive label; eligible for the evaluation set | Refund succeeded, no dispute or repeat contact within the interval |
| Verified, different path | Problem solved within policy by an action other than the expected one, with the customer's agreement | Positive label; the expected-action field is revised, not the verdict | Credit on the next invoice instead of a card refund, accepted by the customer |
| Provisional | Action and state confirmed, but an open effect remains | Held out of labels until the effect resolves; counted separately | Refund still pending when the data is exported |
| Contradicted | A record shows a claim false: no action, a failed or reversed action, or a contrary event inside the interval | Negative label, with the failing link named | Refund failed with an expired-card reason while the ticket stayed solved |
| Unverifiable | A record needed for any step is missing or cannot be joined | Excluded from both labels; its share reported | No payment processor export for the ticket's account |
Testing the reference standard on held-out cases
Before the verdicts become labels, test whether two people applying the reference standard reach the same verdict on cases neither used to design the rubric. Draft the rubric on one period, then have two graders, blind to closure codes and survey scores, grade a random sample from a later period. Report agreement per verdict and per issue type, plus the contradicted share among cases closed as resolved.
Fix the interval in writing before grading. The explanation document for QUADAS-3, the tool's successor, says the interval should suit the question, short enough that the condition has not changed, and that reviewers should prespecify what counts as appropriate. Support adds a twist: the condition can change in either direction, so the interval must also be long enough for the payment rail, carrier or fault to reveal a failure. The separate SourceX guide on support outcome measurement lag covers how to set that window from reopen and follow-up timing.
Then check that the labels make a difference. Score one model, or one candidate dataset, on the held-out set twice, once against closure-based labels and once against reference-standard verdicts. If the rankings or pass rates diverge for an issue type, the reference standard is carrying information the proxy lacks; if they match, the cheaper proxy may be adequate there, and the audit has earned its cost by showing it.
Could closure plus a high satisfaction score be close enough?
Sometimes, and the audit described above is how to find out where. For issue types fixed inside the conversation, such as a how-to question, closure and a good rating may track the reference standard closely. For issue types whose fix runs through another system, such as refunds, cancellations, replacements and account changes, the payment example shows how the two can split without anyone noticing.
The strongest form of the objection is economic: verifying cases costs money, and a large pile of noisy closure labels might train a better model than a small verified set. That is a real tradeoff, but it argues for a verified sample, not against one. The sample is what reveals how noisy the closure labels are and for which issue types they can be trusted, and it is the only sound basis for an evaluation set. Training can tolerate label noise that evaluation cannot, because an evaluation set with silent failures marked as wins rewards exactly the confident, unverified replies a team is trying to remove.
Writing resolution evidence into a data request
Ask for support histories joined to the systems where fixes execute and settle: payment processor events, subscription and order records, shipment tracking and account audit logs, each timestamped and tied to the ticket by a stable key. Without those joins, a resolution can be described but not verified.
SourceX looks for U.S. companies whose records fit a buyer's written requirement instead of offering datasets from a shelf, so the requirement sets what counts as a match. Name the issue types in scope, the action and state systems for each, the interval for each, the minimum share of cases you need in each verdict class and an acceptance test, such as blind agreement between two graders on a held-out sample. Candidate records are then assessed company by company, and the supplying company approves the scope and signs before any data is delivered.
Frequently asked questions
Is a solved or closed status ever good enough as ground truth?
It can be for issue types fixed inside the conversation, but only an audit shows which ones. Verify a random sample of closed cases against the action record and resulting state, and use closure as a label only where the contradicted share is low enough for the intended use. Evaluation sets should use the reference standard either way.
What is partial verification in support data?
It means checking only some cases against the full evidence, usually reopens, escalations and low ratings. Because those cases already failed visibly, silent failures among the rest keep their resolved label. Draw the sample at random within each closure code before deciding which cases to verify.
How long should the wait be before the resulting state is checked?
Set it per issue type before grading, long enough for the systems involved to reveal a failure. A payment rail that can take weeks to return a failed refund needs a longer interval than a password reset. Cases with open effects at export stay provisional.
Should graders see the closure code or satisfaction score?
No. Graders should see the customer's problem, the policy in force, the reply and the system records, but not the closure code or survey score, which are the signals under test. Showing them invites graders to anchor on the outcome the agent claimed.
What happens to cases whose records are missing?
They are graded unverifiable and left out of both positive and negative labels. Report their share, because a high unverifiable rate in one issue type can make the remaining verdicts unrepresentative of that issue type.
Sources
- The QUADAS-2 tool's signalling questions ask whether the reference standard is likely to classify the target condition correctly, whether it was interpreted without knowledge of the index test, and whether there was an appropriate interval between index test(s) and reference standard. Source
- The QUADAS-2 background document explains that signalling questions are answered yes, no or unclear, with yes indicating low risk of bias, and the flow and timing domain covers whether patients received a reference standard and the same one. Source
- The QUADAS-3 explanation and elaboration says the interval between index test and reference standard should suit the question, be short enough that the condition has not changed for some questions, and be prespecified by review authors. Source
- Anthropic's guide to agent evaluations describes a model that solved a tau2-bench flight-booking problem through a policy loophole, failing the evaluation as written while producing a better solution for the user. Source
- Casheekar and Tangirala (arXiv, August 2026) argue that an endpoint score supports a final verdict only with outcome finality; in a controlled replay, endpoint labels differed from final labels for every delayed operation, and they propose an open-effects record. Source
- Stripe's refund documentation lists refund states including requires_action, pending, succeeded, failed and canceled, marks a refund failed when, for example, the customer's bank account is closed, records a failure_reason, sends a refund.failed event, and says returned funds can take up to 30 days from the post date. Source
- In Salesforce's Case data model object, the closed flag shows whether the case is closed or open, is controlled by the Status field and cannot be set directly. Source
Tell us what data you need
Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.