AI uses for records
Customer promise tracking benchmark: how to measure whether promises were kept
By SourceX Editorial · Reviewed by Noah Loul ·
10 sections
Short answer
A benchmark can verify whether customer promises were kept by giving a system a conversation plus the later operating records, then grading three things: whether it found each commitment, whether it cited the action record that settled it, and whether it chose the right outcome class. This page is a protocol for measuring that, with rules for labels, baselines, splits and uncertainty; it reports no results.
Evaluation split
Customer promise tracking
Key takeaways
- A promise is a commitment to a specific action for a specific customer with a deadline; policy quotes and customer undertakings are excluded.
- Evidence of a kept promise lives in action records such as shipments, credit memos and call logs, not in a ticket status.
- Seven outcome classes, including superseded by agreement and not verifiable, keep the grader from forcing kept or broken on incomplete records.
- Deterministic grading from records, distractor evidence and company-level splits make a later result defensible.
- This is a method for measuring; no results are reported here.
What would a customer promise tracking benchmark actually measure?
It would measure whether an AI system, given a customer conversation and the operating records that follow it, can find each commitment the business made, locate the records that fulfilled or failed it, and label the outcome correctly. The unit being tested is the verification, not the conversation.
That is a different job from the one most customer service agent benchmarks score. The tau-bench family, published by Sierra researchers, simulates a user talking to an agent that has tools and a policy, then grades the episode by comparing the final database state with the state an annotated correct action sequence produces. The grading ends when the conversation ends.
Promises live longer than conversations. A support agent says a replacement will ship by Thursday, a billing rep says a credit will appear on the next statement, a field service dispatcher says a technician will call before arriving. Whether those things happened is written days or weeks later, in other systems, by other people. A benchmark for this question has to follow the promise across that gap.
The practical use is plain. Operations teams want to know which commitments are slipping before customers complain, and AI teams building agents that make commitments need a way to check that the agent's promises are honored by the systems behind it. Both need a verifier they can trust, and trust starts with a test whose design was fixed before anyone saw a score.
This page publishes a protocol only. It reports no results, because none have been collected under this design, and every count in the examples is illustrative.
Which statements count as a promise, and which are only polite noise?
A promise, for this benchmark, is a statement by someone acting for the business that commits the business to a specific action for a specific customer, with an explicit or reasonably inferable deadline. Policy quotes, conditional offers the customer never accepted and the customer's own undertakings are excluded.
Annotators need the boundary written down before labeling starts, because the hard cases decide the score. The table below is the inclusion rule set; the example wording is illustrative, not taken from any real record.
| Illustrative wording | Promise? | Fields to extract | Reason |
|---|---|---|---|
| I have issued the replacement and it will ship by Thursday | Yes | Action: ship replacement; object: order line; due: Thursday; owner: fulfillment | Specific action, customer and deadline |
| Someone from billing will call you within two business days | Yes | Action: outbound call; owner: billing; due: two business days from message time | Deadline is relative, so the anchor timestamp must be stored |
| We will look into this | Borderline, labeled vague | Action: investigate; due: none | No checkable deliverable; kept as its own class so models are not rewarded for inventing one |
| Our policy allows returns within the return window | No | None | A quoted policy, not a commitment to this customer |
| If you send photos, we can offer a replacement | Only if the condition is met | Condition: photos received; action: replacement | Conditional promise becomes active only when the record shows the condition happened |
| I will send the signed form tomorrow (customer) | No | None | A customer undertaking; tracked separately if at all |
Where does the evidence that a promise was kept live?
The evidence sits in action records outside the conversation: a shipment record in the order system, a credit memo in the ERP, a logged outbound call in the phone platform, a resolved engineering issue in Jira. A ticket status of Solved is not evidence that the promised action happened.
Help desks already track some commitments, but they track the business's own service targets rather than what an individual agent said. Salesforce entitlement processes attach milestones to cases and fire success, warning and violation actions as targets are met or missed. Zendesk SLA policies define metrics such as first reply time and requester wait time, and requester wait time pauses while a ticket is Pending. Those fields help build the timeline, yet a met SLA can sit next to a broken spoken promise.
Some promises also carry legal weight. Under the FTC's Mail, Internet, or Telephone Order Merchandise Rule, as the agency's business guide explains it, a seller needs a reasonable basis for a stated shipping time, or for 30 days when none is stated, and must seek the customer's consent to a delay with a revised date and the option to cancel for a refund. A benchmark case that includes such a notice has to treat the revised date, once consented to, as the operative commitment.
So each case packet links four kinds of record: the conversation turn that contains the promise, any later message that changes it, the action records that could satisfy it, and the record that shows the customer-facing result.
The promise action and outcome evaluation design
The design below is the asset: a fixed specification a lab or vendor can pre-register before collecting a single case. Each row states what is fixed in advance and why it matters for the validity of a later result.
| Design element | Specification | Why it is fixed in advance |
|---|---|---|
| Research question | Given a conversation and linked operating records, does the system extract each promise and assign the correct outcome class with a correct evidence citation? | Separates extraction errors from judgment errors |
| Population and frame | Closed support, billing and service cases from consenting, rights-cleared companies, listed by company, channel and promise type before sampling | Defines what any later number describes |
| Unit of analysis | One promise, nested in one case, nested in one company | Supports clustered uncertainty instead of treating promises as independent |
| Inputs given to the system | Conversation transcript, timestamps, and a bounded set of candidate action records including distractors | Tests evidence selection, not only reading comprehension |
| Outputs required | Promise fields, outcome class, cited record IDs and the date the outcome was established | Lets a grader check the reasoning chain, not just the label |
| Reference labels | Two trained annotators per case, adjudication by a domain reviewer from the same function | Produces a gold label with a known disagreement rate |
| Grader | Exact match on outcome class and promise fields; set overlap on cited record IDs; human review of disagreements | Keeps scoring deterministic where the records allow it |
| Baselines | Status-field baseline, keyword-and-date rule baseline, time-boxed human reviewer, systems under test | Shows whether a model beats the shortcuts a team already has |
| Splits | Development and test split by company and by promise family; test set sealed until pre-registration is filed | Measures transfer to unseen businesses |
| Release conditions | Rights confirmed, de-identification verified, license permits evaluation use, change log for every label correction | A benchmark that cannot be shared or audited cannot be cited |
How should each promise outcome be labeled?
Use a closed set of seven outcome classes, each with a minimum evidence rule, and require the label to cite the record that settles it. Kept and broken are not enough, because most real disputes sit in the middle classes.
Revised promises are scored as new promises linked to the one they replace, so the benchmark counts two units in that case. Recording the chain is what lets an analyst later separate businesses that miss promises from businesses that renegotiate them openly.
An illustrative packet shows why. A customer reports a damaged item in a chat on a Monday. The agent writes that a replacement will ship by Thursday. On Wednesday the warehouse system shows the item out of stock, and the agent sends a message offering a later date or a refund. The customer replies choosing the later date. The order system then shows a shipment two days after the new date. The correct label is not broken and not kept on time: the original promise was superseded by agreement, and the revised promise was kept late. A system that only reads the first and last messages will get it wrong both ways.
| Outcome class | Minimum evidence | Common false signal |
|---|---|---|
| Kept on time | Action record dated on or before the due time and matching the promised object | Ticket closed on time with no action record |
| Kept late | Matching action record dated after the due time | Later apology message without an action |
| Partly kept | Action record covering part of the promised object, such as one of two items shipped | Shipment record for the order, not the line |
| Superseded by agreement | Later message with a revised commitment and the customer's assent | Unilateral delay notice with no reply where assent is required |
| Released by customer | Customer message canceling or declining the promised action | Silence after a follow-up |
| Broken | Due time passed, no matching action, no supersession or release in the linked records | Missing records counted as broken |
| Not verifiable | The records needed to decide are absent from the packet | Forcing a kept or broken label on incomplete evidence |
How are cases drawn and graded without leaking the answer?
Sample promises within strata of company, channel and promise type, then build each packet with distractor records so the system has to choose evidence rather than read the one record it is handed. Graders compare structured outputs first and send only disagreements to people.
Distractors matter more than they seem. If a packet contains exactly one shipment record, a model can score well by assuming the shipment satisfies the promise. Real ERP and order histories contain earlier shipments, other order lines and canceled attempts, and the packet should too.
The tau-bench code shows one way to make grading deterministic: its reward function replays the annotated action sequence on a fresh copy of the database, hashes the resulting data, and compares that hash with the agent's end state. The successor tau2-bench repository makes the reward the product of listed components, by default database end state and required communicated information, and marks natural-language assertions judged by a model as experimental. A promise benchmark can borrow the principle: decide outcome labels from records wherever the records allow it, and keep model judges out of the reference labels unless they are calibrated against human adjudication.
Annotators should label the promise list before they see any action records, then label outcomes. Doing both at once lets knowledge of what happened shape what they decide was promised.
Channel affects both stages. Phone promises reach the record only through call notes or transcripts, which may paraphrase what was said, while chat and email preserve the exact wording. Keep channel as a stratum and report results per channel, so a strong score on email does not hide weak extraction from summarized call notes.
How many promises, which baselines and what error bars?
Set the sample size from a power calculation for the smallest difference between systems that would change a decision, report every metric with a confidence interval resampled by company, and run each system several times so reliability is visible as well as accuracy.
The status-field baseline is the one most likely to embarrass a model. It labels a promise kept whenever the case closed before the due time. If a system cannot beat that shortcut on the late, partly kept and superseded classes, it adds little to what a help desk report already shows.
For repeated runs, the tau-bench paper introduced pass^k, the chance that all k independent trials of a task succeed. The tau2-bench metrics code estimates it from n trials with c successes as the number of ways to choose k from c divided by the number of ways to choose k from n. Report it alongside per-class precision and recall and a confusion matrix, because a single accuracy figure hides whether errors fall on broken promises, the class a business cares about most.
Uncertainty has to respect clustering. Promises from one company share agents, systems and habits, so resample whole companies, not individual promises, when computing intervals.
Which result would still hold on companies the system never saw?
Only a result measured on a sealed test split of whole companies and whole promise families, collected after the system's training cutoff, should be read as general. Scores on promises from companies already represented in development describe memory of local conventions as much as skill.
Each company writes promises in its own idiom and records fulfillment in its own systems. A model tuned on a retailer's shipping promises may fail on a utility's callback promises even if both use Zendesk. Report results per held-out family and per held-out company before any pooled number, and say how many companies sit behind each figure.
Time also leaks. Hold out the most recent period of each company's records, so that a system cannot exploit policy changes it saw described in earlier cases.
A useful robustness check is to swap the systems of record while holding the promise wording fixed. If the same promise is verified correctly when fulfillment is logged in NetSuite but not when it is logged in a home-grown order table, the system has learned a schema rather than the task. Report that gap as its own finding, because it predicts how much setup work a deployment at a new company will need.
Pre-registration checklist before any score is reported
At a glance
Pre-registration checklist before any score is reported
Checklist · 10 items
- Research question and the seven outcome classes, with evidence rules, frozen and dated.
- Sampling frame listed by company, channel and promise type, with exclusion reasons counted.
- Annotation guide, annotator training record and a planned agreement statistic with its threshold for relabeling.
- Packet construction rules, including how distractor records are chosen.
- Baselines named and versioned, including the status-field and rule baselines.
- Model versions, prompts, temperature and number of runs per case fixed before the test split is opened.
- Primary metric, secondary metrics and the company-level resampling method for intervals.
- Held-out companies, held-out promise families and the time cut recorded.
- Rights review, de-identification method and evaluation-use license terms recorded for every contributing company.
- Statement that the page reports a protocol, and a changelog for every later label correction.
0 of 10 ticked. Tick each item as you go.
Rights, privacy and sourcing the case records
Promise cases are built from customer conversations, so every packet needs a confirmed right to use the records for evaluation and de-identification that keeps the timeline intact. Removing names while shifting dates inconsistently can turn a kept promise into a late one.
Access to a help desk is not permission to use its contents. The company's customer contracts, privacy notices and data processing terms decide whether records can be licensed for evaluation, and counsel should review those terms for each contributing company. De-identification reduces privacy risk but does not create that right.
De-identification for this benchmark has a specific constraint: intervals must survive. If dates are shifted, shift every record in a case by the same offset so the gap between the promise, its due time and the action stays exact. Free-text fields need the same care, because agents often paste order numbers, addresses and phone numbers into messages, and those identifiers can link a case back to a person even after the structured fields are cleaned.
Teams that need such cases can commission them. SourceX does not hold a catalog; for each request it finds suitable U.S. businesses, and the request sets out scope, volume, language, format, timing and the license terms needed. SourceX sits between the record holder and the requesting team, covering the search, the rights check, the contract, the preparation and the hand-over.
Frequently asked questions
Is this the same as tau-bench?
No. Tau-bench grades whether an agent reaches the correct database state inside one simulated conversation. This protocol grades whether a system can verify, after the fact and across several systems, that a commitment made in a conversation was later fulfilled, changed or broken.
Why not use the ticket status as the label?
A ticket can be marked solved while the promised refund, shipment or callback never happened, and a ticket can stay open after the promise was kept. The protocol requires a cited action record for every kept or broken label and uses the status field only as a baseline to beat.
Can a model judge be used to create the reference labels?
Only with care. The protocol uses deterministic checks on structured records wherever possible and two human annotators with adjudication for the rest. A model judge can help triage, but its agreement with human adjudication should be measured and reported before any of its labels enter the reference set.
What results does this page report?
None. It publishes the design, outcome classes, grading rules and a pre-registration checklist. Any result should come from collected, rights-cleared cases, with the population, period, sample and uncertainty stated alongside it.
What should the promise action and outcome evaluation design include?
At minimum a promise definition with inclusion rules, a closed set of outcome classes with evidence rules, packet construction with distractors, baselines, company-level splits, a clustered uncertainty method and documented rights for every case. The design table on this page lists each element and why it is fixed in advance.
Sources
- The original tau-bench reward replays the annotated actions on a fresh database, hashes the data and compares it with the agent's end state. Source
- In tau2-bench the final reward is the product of listed components, by default database end state and communicated information; model-judged natural-language assertions are marked experimental. Source
- The tau2-bench metrics code estimates pass^k from n trials and c successes as comb(c, k) divided by comb(n, k), citing the tau-bench paper. Source
- Tau-bench emulates conversations between a simulated user and an agent with domain API tools and policy guidelines, compares the final database state with an annotated goal state, and proposes pass^k for reliability across trials. Source
- Salesforce milestone actions include success actions when a milestone is completed, warning actions before it is violated and violation actions when it is violated. Source
- Zendesk SLA policies define requester wait time as time in New, Open and On-hold, pausing while a ticket is Pending. Source
- The FTC business guide says sellers need a reasonable basis for a stated shipping time, or 30 days if none is stated, and must seek consent to a delay with a revised date and a cancellation and refund option. Source
Tell us what data you need
Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.