Skip to content

AI uses for records

Support sentiment resolution dataset: 505 failed chats said thanks

By SourceX Editorial · Updated

8 sections

Short answer

Customer sentiment cannot stand in for resolution in a support dataset. In 1,980 public tau-bench agent conversations graded against the final database, 63.4% of failed tasks ended with the customer saying thanks, against 54.7% of completed ones, and a thanks-based label matched the real outcome only 47.4% of the time. Store emotion as its own field; grade resolution on facts, policy, resulting state and durability.

Document stack

Support sentiment resolution

This page covers support sentiment resolution dataset: 505 failed chats said thanks.Illustrative

Key takeaways

  • A thankful closing message is not evidence of resolution: in all four public tau-bench trajectory files, failed conversations thanked the agent at least as often as completed ones.
  • Thanks used as a label matched the database-checked outcome in 939 of 1,980 trajectories (47.4%), below the 59.7% scored by calling every case resolved.
  • Sentiment tools score words, not systems: Contact Lens rates speaker turns and averages call periods into a score from -5 to +5 without reading any order or account record.
  • Grade resolution on factual accuracy, policy compliance, resulting state and durability, and keep emotion as a fifth field that never changes the grade.
  • Split held-out tests by task or issue, never by conversation, and oversample the thanked-but-failed and unhappy-but-correct cells.

Can customer sentiment stand in for resolution in a support dataset?

No. Sentiment records how the customer felt about the conversation; resolution records whether the account, order or system ended up in the state the policy required. The two come apart often enough that a dataset which labels success by tone teaches a model to sound helpful rather than to be right.

The failure is easy to miss because the signals usually travel together on a dashboard. A ticket marked solved, with a warm closing line and a good survey score, looks finished. None of those three fields checks whether the refund posted, the address changed or the replacement shipped. A support sentiment resolution dataset worth training on keeps the emotional record and the outcome record in separate columns, each backed by its own evidence.

This page measures the gap on public data, then builds a grading rubric and a worked record that keep the two apart. Neighboring pages cover behavior-coded emotion labels and maturity windows for resolution labels. The question here is narrower: what goes wrong when emotion is used as the grade, and which evidence should replace it.

1,980 public agent conversations: thanks measured against the final database

Failed conversations ended with thanks more often than successful ones. The tau-bench benchmark (Yao and colleagues, 2024) pairs a language-model agent with a customer simulated by another language model in retail and airline service tasks. It scores each run by comparing the database state at the end of the conversation with an annotated goal state, so a run passes only if the orders and reservations actually changed as required (and, where a task demands it, the agent stated specific information). Its public repository ships four files of historical trajectories, and every record carries that reward.

The counting rule, applied exactly as written to the repository at commit 59a200c: a trajectory is completed when its reward field equals 1.0, and failed otherwise. Its closing customer turn is the last simulated-customer message that still has text once the stop token ###STOP### and surrounding spaces are removed. The trajectory counts as thanked when that turn matches thank, thanks or appreciate at the start of a word, ignoring case. All 1,980 trajectories are counted, with no sampling.

Across the four files, 505 of 797 failed trajectories (63.4%) closed with thanks, against 647 of 1,183 completed ones (54.7%). The direction held in every file, including the two where the simulated customers thanked almost every time. A broader rule that also accepts the whole words great, perfect, wonderful, awesome, helpful or excellent gives the same picture: 65.5% of failures against 58.4% of completions.

Used as a label, thanks would have agreed with the database-checked outcome in 939 of 1,980 trajectories, or 47.4%. Calling every trajectory resolved would have scored 59.7%. The signal did not just add noise; it leaned slightly the wrong way, since only 56.2% of thanked trajectories were completed, below the 59.7% base rate. File by file:

1,980 public agent conversations: thanks measured against the final database
Trajectory fileTrajectoriesCompleted (reward 1)Thanked at close, completedThanked at close, failed
gpt-4o-retail460278248 of 278 (89.2%)167 of 182 (91.8%)
gpt-4o-airline2008471 of 84 (84.5%)104 of 116 (89.7%)
sonnet-35-new-retail920637287 of 637 (45.1%)154 of 283 (54.4%)
sonnet-35-new-airline40018441 of 184 (22.3%)80 of 216 (37.0%)
All four files1,9801,183647 of 1,183 (54.7%)505 of 797 (63.4%)
Dumbbell chart of the share of tau-bench trajectories whose closing customer turn says thanks, completed versus failed, for four trajectory files and all 1,980 combined; failed runs thank at least as often in every file.
Thanks at close by benchmark outcome, 1,980 public tau-bench trajectories (commit 59a200c). Customers are language-model simulators.

Why a warm close and a correct outcome come apart

The customer judges the conversation, and the conversation is all the customer can see. Nobody on the phone can inspect the order table, the refund ledger or the policy manual. They hear a confident agent, an apology and a promise, and they respond to that. When an agent misreads a status or skips a step, the customer usually finds out days later, if ever.

Sentiment tooling inherits the same blind spot. Amazon's documentation for Contact Lens conversational analytics says it classifies each speaker turn as positive, negative or neutral, then weighs how often each occurs and how long the streaks run to score every period of a call from -5 to +5; the overall score averages those periods. Its data lake keeps a separate end-of-contact sentiment score for the customer as well. Every input is speech or text. None of it reads the system of record.

Policy can push sentiment and correctness in opposite directions. ASAPP built its Action-Based Conversations Dataset, more than 10,000 human-to-human dialogues across 55 user intents, around the point that an agent must balance what the customer asks for against constraints set by company policy. A correct refusal, such as declining a refund the policy does not allow, tends to earn a cooler close than a wrong yes, so a grader that rewards warmth ranks the violation above the compliant answer.

Timing opens a third gap. Zendesk's help center describes a default email satisfaction survey that goes out 24 hours after a ticket is solved. The rating therefore captures the customer's view a day after the reply, often before a reopen, a reversal or a failed delivery can show whether the fix held.

Reading one failed upgrade the customer thanked the agent for

Task 20, trial 0 in the gpt-4o-retail file shows the whole problem in one record. The simulated customer's goal is to upgrade every item to its most expensive option and pay the difference by gift card. The conversation ends warmly; the database shows that nothing changed.

The record also shows why a grade based on the final state is stricter than a quick read of the transcript. Skimmed, the exchange looks like a polite agent handling an impossible request. Only the order records reveal that one of the customer's three orders was pending the whole time and could have been changed under the written policy.

Two details matter for anyone building labels. The agent described two processed orders as delivered, a factual error no tone score would catch. And the customer's closing line announces a future contact, which in a live helpdesk would arrive days later as a reopen or a follow-up ticket and change the outcome after the label was written.

The packet below lays the record out field by field, using only what the trajectory and the benchmark's written retail policy contain:

Reading one failed upgrade the customer thanked the agent for
Packet fieldWhat the record showsGrade on this axis
Customer goalUpgrade every item to the most expensive option, keep the same shoe size, pay the difference by gift card, or PayPal if that is impossibleSets the target state
Policy in forceItems on a pending order can be modified; only delivered orders can be exchanged; status must be checked before actingSets the permitted path
Agent statementsTold the customer that two orders were delivered; the order records in the same conversation show both as processedFactual accuracy: fail
Actions takenTwo exchange calls on the processed orders, both rejected by the system; no change requested on pending order #W9911714Policy compliance: fail
Resulting stateNo order changed; the annotated goal required modifying four items on the pending orderResulting state: fail (reward 0)
Customer closeOkay, thank you for letting me know. I'll reach out again later. Then: Thank you. You too!Emotion: positive, stored but not graded
DurabilityThe customer states that they will contact support again about the same requestDurable resolution: fail

Five grading axes, and the one that never enters the score

Grade resolution on four evidence-backed axes and carry emotion as a fifth field that is never scored. Each graded axis has its own source of truth, so a reviewer can dispute one grade without reopening the others, and a model trained on the labels learns which part of its behavior failed.

Store the emotion field with enough provenance to reuse it honestly: what produced it (a survey, a sentiment model and its version, or a keyword rule), whether it describes one turn or the whole contact, and when it was captured relative to the solve time. A survey answered before a reopen and a closing-turn score taken at the end of a chat are different measurements, and a buyer comparing archives needs to know which one sits in the column.

A case counts as resolved only when all four graded axes pass. An unknown on any axis keeps the case out of the success set instead of defaulting to a pass. Emotion stays useful elsewhere, for routing upset customers to senior staff, studying service recovery or training tone; it simply never decides whether the work was done. The rubric:

Five grading axes, and the one that never enters the score
AxisQuestion it answersEvidence that settles itPass rule
Factual accuracyDid every statement the agent made match the systems of record at that moment?Order, billing or account records as of each message timestampNo statement contradicted by the records
Policy complianceDid the agent take only permitted actions, in the required order?The policy version in force on the case date, plus the action logEvery action permitted and every required check done first
Resulting stateDid the account, order or system end where the request and the policy required?A snapshot of the affected records after the case, compared with an annotated goal stateState matches the goal
DurabilityDid the fix hold?Reopens, follow-up tickets, reversals or repeat contacts on the same issue inside a window fixed in advanceNo same-issue failure signal inside the window
Customer emotion (not graded)How did the customer sound at the close and in any survey?Closing turns, sentiment scores, CSAT with its send timeStored as context; never changes the grade

Proving the separation on your own tickets before training on them

Run the same two-by-two count on your own records before any label reaches training. Join each closed case to its verified outcome, then cross-tabulate the outcome against whatever emotion signal you hold: closing-turn sentiment, a CSAT score or a thank-you flag. If the signal predicts outcome no better than the base rate, as it failed to in tau-bench, it cannot serve as a proxy, and every case needs outcome evidence of its own.

Hold out by task or issue, not by conversation. In the historical files, each of the 115 retail and 50 airline tasks appears four or eight times, so a random split by trajectory would put near-identical cases on both sides. In a helpdesk the matching unit is the issue type or the customer account; a split by ticket lets repeat contacts leak across. Freeze the held-out set before tuning the rubric, have two reviewers grade it independently, and report agreement for each axis, because policy compliance is where graders most often part ways.

Then read the rubric's results in the two cells where emotion and outcome disagree. Thanked-but-failed and unhappy-but-correct cases are exactly where a sentiment-trained model learns the wrong lesson, so oversample them in evaluation even when they are rare in production.

Set the decision rule before looking at the results. One workable rule: if an emotion signal agrees with verified outcomes no more often than calling every case resolved, it is excluded from labeling and kept only as context. If it beats that baseline, it may help decide which cases reviewers check first, but it still never replaces the four graded axes.

Simulated customers are not real ones: where this evidence stops

The tau-bench counts show a mechanism, not a production rate. Every customer in those files is a language model following a written goal, and the repository does not say which simulator produced each file; its README names gpt-4o as the default user simulator. Thank-you rates differ sharply between files, from 89.2% of completed gpt-4o-retail runs to 22.3% of completed sonnet-35-new-airline runs, which says more about the simulators than about people.

The strongest objection is that real customers notice failure. Some do, and a customer told that an order cannot be changed will often push back. But real customers share the simulator's central limit: they cannot see the back end, and many failures surface only after the conversation ends. The counting rule is also blunt. A keyword match misses sarcasm and polite dissatisfaction, and it hears nothing of tone in a voice call. Both limits argue for measuring the gap on real, outcome-linked records, not for trusting sentiment.

What this evidence cannot tell you: how wide the gap is in a given company's queue, whether it differs between chat, email and voice, or whether a trained sentiment model does better than a keyword rule. Each of those needs real tickets joined to verified outcomes.

Specifying sentiment and resolution fields in a data request

Ask for both layers, and for the evidence behind the resolution layer. A useful request names the emotion fields to keep (closing turns, survey score with its send time, any sentiment model output with its version) and, separately, the outcome evidence: the policy version in force, the action log, a post-case snapshot of the affected records, and reopen or follow-up links over a stated window.

Put the acceptance test in the request itself. For example: in a stratified sample of 200 cases, at least 95% carry a resolution grade traceable to system records, and no case is marked resolved on survey or sentiment evidence alone. Those thresholds are illustrative; set your own.

SourceX does not hold support datasets in stock; it sources them from U.S. businesses against a written request that states scope, volume, format, timing and licensing needs. The business that owns the records approves the scope and terms and signs before any delivery. Whether a particular archive carries outcome evidence is established case by case, and no effect on model quality is promised.

Frequently asked questions

Is CSAT a reliable outcome label for support AI?

Not on its own. A CSAT score records the customer's view when the survey was answered, which in Zendesk's default email setup is a day after the ticket is solved and before most reopens or reversals. Keep it as context beside a resolution grade built from system records.

Can a sentiment model replace human grading of support resolutions?

No. Sentiment models score the words exchanged, not whether the order, refund or account change happened. In 1,980 public tau-bench trajectories, a simple thank-you rule matched the database-checked outcome less often than labeling every case resolved.

How should a case be labeled when the customer was unhappy but the agent was right?

Grade it resolved if factual accuracy, policy compliance, resulting state and durability all pass, and record the negative emotion separately. These cases make strong evaluation examples, because a tone-based grader would score them as failures.

What counted as a thank-you in the tau-bench analysis?

The last simulated-customer turn with text after the stop token was removed had to contain thank, thanks or appreciate at the start of a word, ignoring case. A broader rule adding six positive words was run as a check and gave the same direction.

Should sentiment be stripped out of a support training dataset?

No. Keep it as a separate, dated field with the tool or survey that produced it. It helps with routing, escalation and tone training; it causes harm only when it becomes the success label.

Sources

  • The tau-bench repository publishes four historical trajectory files with a reward per run; its reward code sets reward to 1 only when the final database hash matches the goal state and required outputs were said; its README says users are simulated by language models with gpt-4o as the default simulator; its retail policy limits modification to pending orders and exchange to delivered orders. Source
  • The tau-bench paper evaluates by comparing the database state at the end of a conversation with the annotated goal state. Source
  • ABCD contains over 10K human-to-human dialogues with 55 user intents, built so that agent actions balance customer desires against company policy constraints. Source
  • Contact Lens classifies each speaker turn as positive, negative or neutral and scores each period of a call from -5 to +5 using frequency and sentiment streaks; the overall score averages the periods. Source
  • The Amazon Connect data lake stores overall and end sentiment scores for agent and customer. Source
  • Zendesk's default email CSAT survey is sent 24 hours after a ticket is solved, and the delay can be changed. Source
  • In Zendesk, a reopened ticket results when the requester adds a new comment to a solved ticket, for example because they disagree the issue was resolved. Source

Tell us what data you need

Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.

See if your company qualifies