Skip to content

AI uses for records

Support evidence sufficiency dataset: what to ask after a failure report

By SourceX Editorial · Updated

10 sections

Short answer

Request only the evidence that separates the remaining plausible causes, starting with what the customer can observe without changing anything: exact error, start time, expected result, steps and the affected account, device or version. Each question must be able to change the next action. Once no answer would, act or hand off, then confirm the outcome after a maturity window rather than at ticket closure.

Evaluation split

Support evidence sufficiency

This page covers support evidence sufficiency dataset: what to ask after a failure report.Illustrative

Key takeaways

  • Evidence is sufficient when no further answer could change the agent's next action; asking past that point is cost, not diligence.
  • Ask for read-only observations before asking a customer to reset, reinstall or change settings.
  • Prefer sanitized, narrow evidence such as an error code and timestamp over full logs or session captures.
  • Label each decision point as sufficient, insufficient, or sufficient but blocked by authority.
  • A closed ticket proves nothing on its own; check reopens, linked follow-ups and repeat cases after a documented maturity window.

What evidence should a support agent request when a customer reports failure?

A support agent should request only the evidence that separates the remaining plausible causes of the failure, starting with what the customer can observe without changing anything: the exact error, when it started, what they expected, the steps that produce it, and the account, device or version involved. Every question should be able to change the next action. If no answer could change it, the agent already has enough and should act or hand off.

That rule turns a vague instinct into something a team can measure. For each reported failure there is a point where the evidence on hand is sufficient to pick a fix, an escalation path or a known-issue response. Asking past that point wastes the customer's time; acting before it produces wrong fixes and reopened tickets.

This page gives a benchmark for that judgment: a set of failure-report patterns with the evidence that is sufficient for each, the cheapest next question when it is missing, and a task record that lets a team score an agent, human or AI, against real resolved cases.

Why one failure report hides several different problems

"It failed" is a symptom, not a diagnosis, so the first job is to decide which class of failure the report belongs to. A login that fails, an export that produces the wrong numbers and a feature that was never available on the customer's plan all arrive as the same sentence, yet each needs different evidence.

Software teams have studied this gap for years. In a survey of developers and users of the Apache, Eclipse and Mozilla projects, Bettenburg and colleagues found that developers rate steps to reproduce, stack traces and test cases as the most useful parts of a bug report, and that these are also the items reporters find hardest to provide. A support conversation inherits the same mismatch: the customer knows what they saw, while the agent needs what caused it.

Structured intake helps but does not close the gap. GitHub's documented example of a bug issue form asks for current behavior, expected behavior, steps to reproduce and environment details, and it lets maintainers mark fields as required. A form like that fixes the order of questions in advance. An agent has to choose the next question after each answer, which is exactly the decision this benchmark tests.

The evidence sufficiency and next question benchmark

The benchmark pairs each common report pattern with the evidence that is enough to act and the single cheapest question that moves an incomplete case toward it. Use the table as the answer key when labeling cases, then adapt the rows to your own product and policies.

The evidence sufficiency and next question benchmark
Report patternEvidence that is sufficient to actCheapest next question if it is missingSignal to stop asking and hand off
Cannot sign inAccount identifier confirmed, exact error text, sign-in method (password, SSO, magic link), whether other users in the same organization are affectedWhat exact message do you see, and does a colleague using the same sign-in method see it too?Error points to the identity provider or an account lock that only an administrator can lift
Feature missing or disabledAccount plan or entitlement record, user role, the screen where the feature was expectedWhich screen were you on, and which role does your user have?Entitlement and role match policy, so the request is a sales or plan question rather than a fault
Data looks wrong after an import, sync or exportRecord identifiers for one wrong example, the expected value, the source of truth, the time of the last successful runCan you share one record ID with the value you expected and where that value came from?The wrong value already exists in the customer's source system
Action fails with an errorExact error text or code, timestamp with time zone, steps that reproduce it, product version or browserWhat were the last two steps before the error, and roughly what time was it?Error reproduces internally, so engineering owns the next step
Intermittent or slow behaviorTwo or more timestamps of occurrences, network or location context, whether it happens on another device or networkWhen did it last happen, and does it also happen on a different network?Monitoring shows a platform incident in the same window
Integration stopped workingWhich connected system, when it last worked, any credential, token or permission change since thenDid anyone change the connected account, password or permissions recently?The third-party system reports an outage or a revoked authorization
Billing or charge looks wrongInvoice or transaction reference, the line in dispute, the plan change history for the periodWhich invoice and which line item do you believe is wrong?The dispute needs a refund or credit decision outside the agent's authority
Previous fix did not holdLink to the earlier case, what was done, what changed since, whether the symptom is identicalIs this the same message and the same step as last time, or something new?Second recurrence of the same symptom after the documented fix

Fields in one benchmark task record

Each benchmark task is one real decision point inside a resolved case: the conversation so far, the evidence available, and what the agent should do next. Recording it at the field level lets a reviewer check the label and lets a model be scored on more than its final message.

Fields in one benchmark task record
FieldWhat it holdsValidation checkIllustrative value
case_id and turn_indexStable case identifier and the position of the decision point in the threadUnique pair; turn_index is within the thread lengthCase 4471, turn 3 (illustrative)
report_patternOne row from the benchmark tableValue is from the controlled listData looks wrong after a sync
evidence_presentEach evidence item already in the thread, with the turn where it appearedEvery item cites a turn index at or before the decision pointExpected value given at turn 1; record ID missing
sufficiency_labelSufficient, insufficient or sufficient-but-blocked (enough evidence, no authority to act)Exactly one value; reviewer initials recordedInsufficient
gold_next_moveThe cheapest question, the action, or the handoff that the case history shows was rightMust be consistent with the later turns and the final outcomeAsk for one record ID and its source value
acceptable_alternativesOther moves a reviewer would accept, such as an equivalent questionAt least one reviewer agreed; empty is allowedAsk for a screenshot of the record that includes its ID
policy_versionThe support policy or runbook version in force on the case dateVersion date is on or before the case dateRunbook revision dated within the case month
outcome_label and outcome_checked_atResolved and held, reopened, follow-up raised, or unresolved, with the date the label was setLabel set only after the maturity window described belowResolved and held

Which requests cost the customer more than they reveal?

Ask for observations before asking the customer to change anything, and never ask for evidence that carries secrets when a safer version exists. A request that makes the customer reboot, reinstall or reset settings destroys the state the agent needed to see, and a request for raw logs can collect far more personal data than the question needs.

The tau2-bench telecom domain, a public benchmark for customer service agents, makes this distinction explicit. Its troubleshooting workflow splits the actions a user can take on a phone into read-only diagnostic actions, such as checking network status or SIM status, and fix actions that modify the device, such as toggling airplane mode or resetting APN settings. Labeling your own cases the same way lets the benchmark penalize a fix request that came before the diagnostic that would have justified it.

Browser network captures are a common example of evidence that needs care. Chrome DevTools documentation says a HAR export is sanitized by default, excluding Cookie, Set-Cookie and Authorization headers, and that exporting those headers requires turning on a separate setting. An agent should ask for the sanitized export, and the benchmark should treat a request for passwords, full session captures or payment card numbers as an error regardless of whether the case was later solved.

  • Prefer a description or a cropped screenshot of one record over a full data export.
  • Prefer an error code and timestamp over a complete log file.
  • Prefer a read-only check over a reset, reinstall or reboot until the diagnostic points there.
  • Never request a password, one-time code or full card number; route identity checks through the documented verification step.

Worked case: a one-line sync complaint, and the outage exception

This case is illustrative and does not describe any real customer. A customer writes: "The sync failed again, nothing matches." The pattern is data looks wrong after a sync. Evidence present: the word "again", which suggests an earlier case. Sufficiency label: insufficient. The cheapest next question is not "please send your logs" but a request for one record ID with the value the customer expected and where that value came from, plus a check of the earlier case.

The customer replies with a record ID and says the expected value is in their accounting system. The agent looks up the record, sees that the last successful run was two days earlier, and finds a permission change on the connected account the same day. Now the evidence is sufficient: the gold next move is to explain the permission change and ask the customer's administrator to restore access, a handoff to someone with authority over the connected system. Three turns, one question, no logs.

The exception matters as much as the rule. Suppose the same message arrives while monitoring already shows a platform incident affecting syncs in that window. Here the evidence is sufficient at turn one. The gold move is to confirm the incident, give the documented status update and link the case to the incident record. A benchmark that rewards asking for a record ID in that situation would teach agents to interrogate customers during an outage, so label the decision point as sufficient and score extra questions as cost.

After closure: reopens, follow-ups and repeat cases that show whether a fix held

A closed ticket is not proof of a fix; the proof is the absence of a reopen, a linked follow-up or a repeat case for the same symptom within a defined window after closure. The window has to reflect how your help desk actually handles late replies, or the outcome labels will be wrong.

Zendesk is a useful example because its rules are documented. Its ticket lifecycle article says a requester's reply reopens a solved ticket, that agents cannot close tickets manually, that an automation closes solved tickets after four days by default, and that tickets are closed automatically 28 days after being set to solved regardless of other rules. Once closed, a requester cannot reopen the ticket but can create a follow-up request that references it. A separate Zendesk article notes that tags, subject, priority and some custom fields can still be edited on closed tickets, so a label can change after closure.

For the benchmark this means three practical rules. Follow follow-up references and treat a linked follow-up with the same symptom as a failed resolution. Freeze outcome labels only after a maturity window that you set and document, and record outcome_checked_at so later edits are visible. Search for repeat cases from the same account and symptom outside the formal link, because customers often open a new ticket instead of replying.

Software engineering research treats reopening as its own signal. Zimmermann and colleagues at Microsoft Research combined a survey of 358 Microsoft employees with a quantitative study of Windows bug reports to characterize why bugs get reopened. Support teams can borrow the idea: reopen reasons are worth labeling, because "new information arrived" and "the fix did not work" teach an agent very different lessons.

Scoring an agent on sufficiency, question cost and final state

Score an agent on whether it reached the right state with the fewest necessary questions, not on whether its last message sounded helpful. A polite closing message after a wrong fix should fail.

At a glance

Scoring an agent on sufficiency, question cost and final state

The tau2-bench environment evaluator shows one way to check end state. It compares the agent-side database and the user-side device state after a simulated conversation with the gold end state and assigns the database reward only when both match. Teams building from real support histories rarely have a simulator, but the same principle applies: compare what changed in the account, configuration or ticket fields with what the resolved case shows should have changed.

  • Sufficiency accuracy: how often the agent correctly judges that it has enough evidence or not at each decision point, compared with sufficiency_label.
  • Next-move match: whether the chosen question, action or handoff matches gold_next_move or an acceptable alternative.
  • Question cost: the number of questions asked beyond the gold path, counted separately for questions that required the customer to change state.
  • Unsafe requests: any request for secrets or unsanitized captures, scored as a failure of the task regardless of outcome.
  • Matured outcome: whether cases the agent handled end in resolved and held after the maturity window, when the benchmark is replayed against live or simulated outcomes.

Where the benchmark is unfair to the agent, and what to do about it

The benchmark measures judgment against what a past team did, so it is only as fair as those past decisions. Some resolved cases took the long way round; if the gold path includes unnecessary questions, the benchmark will reward them. Have a customer operations lead review gold_next_move on a sample of tasks before using scores to compare agents.

Three situations call for narrowing scope or bringing in a specialist rather than forcing a label. Cases that turn on security incidents, account takeover or legal requests belong with the security or legal team, and the benchmark should only test whether the agent hands off. Cases where the support policy changed mid-thread need the policy_version field checked by someone who knows the old runbook. Cases with heavy personal data in attachments should be reviewed by a privacy reviewer before any record leaves the help desk.

Finally, keep the label set small. Sufficient, insufficient and sufficient-but-blocked are enough for most teams; adding fine-grained grades usually lowers agreement between reviewers without improving the agent.

Assembling resolved support histories into this benchmark

The raw material for this benchmark already sits in most help desks: threads with their turn order, linked follow-ups, reopen events, policy versions and the account changes that resolved each case. What makes it valuable outside the company is completeness, with every decision point tied to an outcome that had time to mature.

Help desk histories like these can be licensed for AI with SourceX handling rights review, contracting, preparation and delivery. A short company assessment, with no data upload, shows whether a given history qualifies. As a general guide, buyers look for consistent history over two or more years and tens of thousands of records in one system, although rarer work can qualify with less. Customer permission, privacy obligations and contract rights are reviewed separately; removing names does not by itself create a right to license the records.

Frequently asked questions

How many questions should a support agent ask before acting?

There is no fixed number. The right count is the number of questions needed to separate the plausible causes, and each one should be able to change the next action. When no answer would change what the agent does, it should act or hand off.

Should agents always ask for logs or screenshots?

No. Logs and full captures often contain far more personal data and secrets than the question needs. Ask for the narrowest observation that answers the question, such as an error code with a timestamp or a cropped screenshot of one record, and use sanitized exports where the tool offers them.

What is an evidence sufficiency label?

It is a reviewer's judgment, at one point in a support thread, of whether the evidence present is enough to choose the next action. A simple scheme uses three values: sufficient, insufficient, and sufficient but blocked, where the agent knows enough but lacks authority to act.

How long after closing should a support outcome be considered final?

Set a maturity window that reflects how your help desk handles late replies and follow-ups, document it, and record the date each label was set. In Zendesk, for example, solved tickets reopen on a requester reply until they close, and closed tickets can receive linked follow-ups.

Can old support tickets be used to benchmark an AI support agent?

Yes, if the threads keep their turn order, linked follow-ups and resolution history, and if the company has the contractual and privacy basis to use them that way. Reviewers should confirm gold answers on a sample, because past agents did not always take the shortest path.

Sources

  • The tau2-bench telecom troubleshooting workflow separates read-only diagnostic actions a user can take on a phone from fix actions that modify the device. Source
  • The tau2-bench environment evaluator compares agent-side and user-side database state with the gold end state and gives the database reward only when both match. Source
  • GitHub's example bug issue form asks for current behavior, expected behavior, steps to reproduce and environment, and form fields can be marked required. Source
  • Zendesk: a requester reply reopens a solved ticket; agents cannot close tickets manually; the default automation closes solved tickets after four days; tickets close automatically 28 days after being solved; closed tickets can receive follow-up requests that reference them. Source
  • Zendesk allows tags, subject, priority and a subset of custom fields to be edited on closed tickets. Source
  • Chrome DevTools exports a sanitized HAR by default that excludes Cookie, Set-Cookie and Authorization headers; including them requires a separate setting. Source
  • Bettenburg and colleagues surveyed Apache, Eclipse and Mozilla developers and users and found steps to reproduce, stack traces and test cases most useful to developers and hardest for reporters to provide. Source
  • Zimmermann and colleagues characterized why bugs get reopened using a survey of 358 Microsoft employees and a quantitative study of Windows bug reports. Source

See if your company qualifies

A short company assessment. No data uploads are needed.

See if your company qualifies