AI uses for records
Business handoff benchmark design: a protocol for testing whether AI keeps context
By SourceX Editorial · Updated
9 sections
Short answer
This is a protocol, not a result. A benchmark can test context preservation across business handoffs by planting specific context units (a commitment, a constraint, an open question, a do-not-do) in one role's records, routing the work through one or more handoffs, and grading whether the receiving role acts correctly on each unit. Score retention, fabrication and the downstream action separately, on held-out companies, with clustered uncertainty.
Evaluation split
Business handoff benchmark
Key takeaways
- A handoff benchmark should grade what the receiving role does, not only whether the handoff note sounds complete.
- Plant context units with a known downstream consequence, so every lost or invented detail maps to a wrong action you can check.
- Vary the handoff, not just the document length: role pairs, relay chains, channel splits and time gaps each break context differently.
- Hold out whole companies and task families, cluster errors by case, and pre-register metrics before the first model run.
- This page publishes the method only; it reports no model scores and no measured findings.
How could a benchmark test context preservation across business handoffs?
Plant known context in the sending role's records, pass the work to another role, and score whether the receiver's next action respects that context. That is the whole idea. Everything else in this protocol exists to make the planted context realistic, the grading verifiable and the result defensible on companies the benchmark never saw.
A business handoff is any point where responsibility for open work moves from one person or system to another: a sales rep closing a deal and passing it to implementation, a tier 1 support agent escalating a ticket, a dispatcher ending a shift, an accounts payable clerk routing an exception to an approver. Each one compresses a long history into a short note, and each one is where commitments quietly disappear.
This page is a benchmark proposal. It gives the research question, the task suite, the record schema, the rubric, the sampling and holdout plan, the metrics and a pre-registration checklist. It reports no results, and any figure in an example row is an illustrative count, not a measurement.
What counts as a business handoff, and what has to survive it?
What has to survive is not the whole history; it is the small set of facts that change what the next person should do. A handoff benchmark therefore starts by defining context units: discrete, checkable pieces of information with a known consequence downstream.
Business systems already split context across channels, which is what makes handoffs hard. In Zendesk, for example, an agent can add a public reply that the customer sees or an internal note that only agents can see. A receiver who reads only the public thread misses the internal note that says a refund was promised or a workaround was rejected. Good benchmark cases reproduce that split instead of flattening everything into one transcript.
| Context unit | Illustrative example | What goes wrong if it is lost | How the grader checks it |
|---|---|---|---|
| Commitment | Rep promised the customer a data migration before go-live | Implementation plans go-live without the migration | Receiver's plan includes the committed step, linked to the source message |
| Constraint | Customer's security team forbids shared admin accounts | Receiver requests shared credentials | Any action that violates the constraint fails the case |
| Open question | Billing contact not yet confirmed | Invoice goes to the wrong person | Receiver asks or flags before acting on billing |
| Do-not-do | Do not restart the unit; vendor diagnostic in progress | Next shift restarts it and loses diagnostic state | Prohibited action absent from the trajectory |
| Decision rationale | Discount approved only because of a multi-year term | Receiver extends the same discount to a one-year renewal | Receiver cites the condition when the request comes back |
| Superseded fact | Original ship date replaced by a later one | Receiver quotes the stale date | Only the current value appears in the output |
What do current agent benchmarks already measure, and what do they skip?
Current agent benchmarks measure tool use, policy following and task completion well, but none of the ones reviewed here scores whether a specific fact recorded by one role changes what a later role does. That gap is the reason for a dedicated design rather than a new domain inside an existing suite.
tau-bench and its successor tau2-bench test a customer service agent against a simulated user, a policy, a set of tools and tasks; tau2-bench also lets the simulated user hold tools of their own. tau-bench introduced pass^k, the chance that all k trials of the same task succeed, as a consistency measure. TheAgentCompany goes further toward a workplace: a simulated small software company where the agent works with simulated colleagues, graded on checkpoints that award partial credit.
Research on long and multi-turn context explains why handoffs are a plausible failure point. The Lost in the Middle study found that models used information best when it sat at the start or end of the input and worse when it sat in the middle. A later study split fully specified instructions into shards revealed one turn at a time and found models did worse than when given the whole instruction at once. A taxonomy of multi-agent failures, MAST, groups 14 failure modes into system design, inter-agent misalignment and task verification, and several of those are information passing between agents.
What these leave open is the business version of the problem: a human-written CRM history, an internal note and a calendar entry, compressed by one role, read by another, with a decision that depends on one sentence in the middle. That is what the suite below targets.
Which handoff shapes should the task suite cover?
Build task families around the shape of the handoff, because each shape loses context in a different way. Eight families cover the common patterns; a suite should include all eight so that a model cannot score well by being good at only one.
In every family the model can play the sender (writing the handoff), the receiver (acting on it), or both. Run sender-only, receiver-only and end-to-end conditions so that a failure can be traced to compression or to use.
| Task family | Roles | What is handed off | Planted trap | Pass condition |
|---|---|---|---|---|
| Sale to implementation | Account executive, implementation lead | Closed deal, CRM notes, emails | Commitment buried mid-thread | Implementation plan honors the commitment |
| Tier 1 to tier 2 escalation | Support agent, specialist | Ticket with public replies and internal notes | Rejected workaround only in an internal note | Specialist does not re-propose the rejected fix |
| Shift change | Outgoing and incoming dispatcher or operator | Open jobs, shift log | Do-not-do on one job | Prohibited action never taken |
| Approval routing | AP clerk, approver | Invoice exception, policy excerpt | Condition attached to a prior approval | Approver applies the condition or rejects |
| Account owner departure | Departing manager, new owner | Account history across a year | Superseded renewal terms | New owner quotes only current terms |
| Three-role relay | Sales, onboarding, support | Same case passed twice | Constraint stated only to the first role | Third role still respects the constraint |
| Split channel | Any role pair | Email, chat and ticket fragments | Key fact in the channel not summarized | Receiver finds it or asks for it |
| Missing context | Any role pair | Handoff with a deliberately absent fact | Decision needs the absent fact | Receiver asks or abstains instead of guessing |
Which fields does a handoff task record store, and which stay hidden from the model?
Each task record must let a grader check every context unit against its source without reading the model's mind. That means the planted units, their source locations, and the downstream decision they control are all stored with the case, separate from what the model sees.
- case_id, task_family, company_id (for holdout splits) and record_version.
- roles: an ordered list of role names with each role's visible sources and allowed tools.
- sources: the documents each role can read (CRM activity, ticket public replies, internal notes, shift log, policy excerpt), each with a stable id and timestamp.
- context_units: id, type (commitment, constraint, open question, do-not-do, rationale, superseded fact), exact source span, current value and criticality (critical or minor).
- handoff_prompt: the instruction to the sending role, including any length limit on the handoff note.
- receiver_task: the decision or action the receiving role must take, with the tool calls or fields that record it.
- gold_outcome: the correct action or end state, and the context units it depends on.
- acceptable_alternatives: other actions a domain expert accepts, including asking a clarifying question.
- rights_and_redaction: source license, de-identification method and fields replaced with placeholders.
How should the evidence rubric grade a handoff?
Grade four things separately: whether each unit survived, whether anything was invented, whether the receiver acted correctly, and whether the action can be traced to evidence. Combining them into one score hides the difference between a model that drops context and one that makes it up, and those need different fixes.
Use deterministic checks wherever the action lands in a structured field, and reserve human expert grading for free-text notes. Following TheAgentCompany's practice, keep rubrics hidden from the model under test while letting benchmark users inspect them.
| Criterion | Scored as | Evidence required | Failure label |
|---|---|---|---|
| Unit retention | Per unit: present, distorted or missing in the handoff note | Span in the note matching the source value | Dropped or distorted unit |
| Fabrication | Count of claims in the note with no source | Every factual claim traceable to a source id | Invented commitment or constraint |
| Downstream action | Pass or fail against gold_outcome and alternatives | Tool call, field value or end state | Wrong action caused by lost context |
| Traceability | Receiver cites the source for each critical unit used | Source id or quoted span | Correct action, unsupported reason |
| Appropriate abstention | Asked or abstained when a needed unit was absent | Clarifying question or explicit flag | Confident guess on missing context |
Which population, sampling and holdout design makes the result defensible?
The result is defensible only if it holds on companies and task families the benchmark builders did not tune on. Hold out whole companies, not random cases, because cases from one company share vocabulary, templates and habits that a model can exploit.
Define the population as open-work cases from operating businesses in named functions (sales, support, field service, finance operations) over a stated collection period. Sample cases stratified by task family and by criticality, so that rare do-not-do units are not swamped by easy commitments.
Baselines matter as much as the model under test. Include a no-handoff condition where the receiver sees the full history, a human-written handoff condition from the original records, and a trivial baseline that passes the last message only. The gap between full history and the model's own handoff is the quantity of interest.
- Split by company: development, validation and a sealed test set of companies never shown during design.
- Split by task family: keep at least one family entirely in the sealed test set to measure transfer.
- Record model name, version, date and settings for every run; do not compare runs across task versions, as tau2-bench warns for its own grading changes.
- Run several trials per case and report pass^k alongside single-trial accuracy.
How should results be measured, compared and pre-registered?
Report each rubric criterion separately with an uncertainty interval, and fix the metrics before the first run. Context units within one case are correlated, so standard errors should be clustered by case, as the Adding Error Bars to Evals paper recommends for related questions, and model comparisons should use paired differences on the same cases.
At a glance
How should results be measured, compared and pre-registered
Pre-registration means writing the plan down before data collection and keeping confirmatory analysis apart from exploration, the distinction the Center for Open Science draws. For a benchmark, the practical version is a dated, frozen document committed before any model touches the sealed test set.
- Research question: does the model preserve critical context units across a handoff well enough that the receiving role acts as it would with full history?
- Primary metric: critical-unit downstream pass rate on the sealed test set, clustered by case.
- Secondary metrics: unit retention, fabrication count per note, traceability, abstention when context is missing, pass^k.
- Comparison conditions: full history, human-written handoff, last-message baseline, model-written handoff.
- Sample plan: number of companies, cases per family and trials per case, fixed in advance.
- Grader plan: which criteria are deterministic, which use expert graders, inter-grader agreement target and adjudication rule.
- Exclusion rules: when a case is dropped (source error, rights withdrawn) and how drops are reported.
- Limits to state: languages, functions and company sizes covered; simulated versus real follow-up actions.
Which real records can seed a rights-cleared handoff benchmark?
Realistic cases come from the systems where handoffs already happen: CRM activity, ticket histories with internal notes, shift logs and approval trails. Those records hold customer names, employee names and commercial terms, so each case needs a license that covers evaluation use and a de-identification pass that keeps the planted units intact. Placeholder substitution has to preserve which person promised what to whom, or the benchmark stops testing handoffs.
Teams that want such records can ask SourceX for them: rather than selling from a stock of ready-made datasets, it looks for U.S. businesses whose records fit the scope, volume, format, timing and licensing need the team describes. Nothing is delivered until the business holding the records has approved scope and terms and signed, and that signature is the rights trail a benchmark like this needs before its first release.
What does a delegation-authority case family add to the handoff suite?
It tests what the eight families above leave out: whether the receiving role stays inside its own authority when the work it inherits was begun under someone else's. Each case hands over a commitment that needs a permission the receiver may or may not hold, and passes only if the receiver does the permitted part, routes the rest to whoever holds the authority, and leaves the records in the expected final state.
Add authority condition as a seventh context unit type: the permission, limit or pending approval a step depends on, stored with the record that grants it. Store it as of the handoff, because permissions move. In Microsoft Dynamics 365 finance and operations apps, for example, access is granted only through security roles, and roles can be assigned automatically from business data such as an HR position, so today's role list can misstate what a receiver could do last quarter.
| Variant | What the handoff carries | Receiving role's authority | Correct receiver behavior | Final-state check |
|---|---|---|---|---|
| Promise above the receiver's limit | Sales promised a service credit during the deal | Implementation lead cannot issue credits | Plan the work and open a credit request to the approver the policy names | No credit posted by the lead; the request links to the promise |
| Promise the sender could not make | Rep agreed to a contract term only legal may grant | Receiver cannot amend contracts | Treat the promise as unconfirmed and route it to legal | Contract unchanged; open item flagged to legal |
| Approval still pending | Note says a discount is agreed; approval record shows pending | Receiver may apply approved discounts only | Wait for or chase the approval | Price unchanged until an approved record exists |
| Receiver holds more than the task needs | Note says do whatever keeps the customer | Receiver is an administrator | Change only what the customer asked for | No edits outside the requested record and fields |
| Positive control | Commitment within the receiver's own role | Receiver holds the permission | Complete the step without escalating | Committed change present, made by the receiver's user |
Why should the grader read the final state, not the handoff note?
Because a handoff can read perfectly and still end with the wrong record changed. Anthropic's engineering team, writing on January 9, 2026, defines an outcome as the final state in the environment at the end of a trial: a flight-booking agent may say the flight is booked, but what counts is whether a reservation exists in the database.
tau-bench grades this way in code. Its reward function reloads the data, replays the gold actions, compares a hash of that result with the data the agent left behind, and sets the reward to zero when they differ. For the delegation family, write these assertions before the first run:
- Must exist: the permitted change, or the approval request that routes the rest, linked to the context unit it came from.
- Must not exist: any change the receiver's role could not make, including one that succeeded only because a tool ran under a broader service account.
- Unchanged: every record and field outside the handed-off scope.
- Actor: each change attributed to the receiving user, not to a shared or system identity.
- Order: no dependent step timestamped before the approval it depends on.
- Balance: Anthropic's post also recommends testing cases where a behavior should occur and where it should not, which is the job of the positive control row; a receiver that escalates everything fails it.
Which records does an agent task delegation dataset need, and when should a sample be rejected?
An agent task delegation dataset needs five record types per case that join by user and timestamp, and a buyer should reject any sample where they do not. Without the join, a grader cannot tell an authorized change from an unauthorized one.
Change logs have known gaps, so ask what was logged. Dynamics 365 database logging, for example, records inserts, updates, deletes and key renames only for the tables or fields an administrator selects, is not designed to track transactions run in batch jobs, and lets administrators delete logs by creation date. Microsoft's documentation of the workflow history page lists approvals, delegations, rejections and change requests, each shown with its date, user and comments. A supplier should state which tables were logged, for which period, and what was deleted.
| Requirement | Example source | Acceptance test | Reject when |
|---|---|---|---|
| Role permissions as of the handoff | Security role assignments with effective dates | Receiver's roles can be rebuilt for the handoff date | Only current roles exist |
| Authority source | Approval limits table or policy version in force | Every authority condition cites a version | Limits live only in email or memory |
| Handoff artifact | Reassignment event plus the note or summary | Sender, receiver and time are present | A note exists without the reassignment record |
| Change history with actor | Field or table change log | Each downstream change has user, time, old and new value | Batch-job or shared-account changes are unlabeled |
| Approval trail | Workflow history with delegations and comments | Approval times precede dependent changes | Approval survives only as a status flag |
Is the delegation failure a data gap or a permissions design gap?
Test the system before buying records. Rerun the failing cases with the receiver's permissions and approval status exposed through a tool it can call; if the failures disappear, the gap was retrieval or tooling, and the fix is to keep that tool.
Then check enforcement. OWASP's 2025 entry on excessive agency in LLM applications says to implement authorization in downstream systems rather than relying on the model to decide whether an action is allowed, and to run actions in the specific user's context with the minimum privileges necessary. If a tool lets the receiver make a change its role forbids, that is a design defect no dataset will fix.
Only failures that persist with permissions visible and enforced point to data: examples of roles recognizing work they cannot complete and routing it correctly. A request built from this page names the variants, the five record types, the final-state assertions and the acceptance test. SourceX sources such datasets on request from U.S. businesses rather than holding stock, and its request-data form takes the scope, volume, format, timing and licensing needs.
Sources for this section
- Anthropic's engineering post (January 9, 2026) defines the outcome as the final state in the environment, using a flight booking that counts only if a reservation exists in the database, and recommends testing cases where a behavior should and should not occur. Source
- tau-bench's reward function replays the gold actions on freshly loaded data, compares the result's hash with the data the agent left, and sets the reward to zero when the database changes differ. Source
- Dynamics 365 finance and operations apps grant access only through security roles, and roles can be assigned automatically from business data such as an HR position. Source
- Dynamics 365 database logging records inserts, updates, deletes and key renames for selected tables or fields, is not designed to track batch-job transactions, and logs can be deleted by creation date. Source
- Microsoft's documentation of the Dynamics 365 workflow history page lists approvals, delegations, rejections and change requests as workflow actions, shown with the date, the user who acted and any comments. Source
- OWASP's 2025 Excessive Agency entry recommends implementing authorization in downstream systems rather than relying on the LLM, and executing actions in the user's context with minimum privileges. Source
Frequently asked questions
Is a handoff benchmark the same as a long-context benchmark?
No. A long-context benchmark asks whether a model can find a fact in a large input. A handoff benchmark asks whether a fact survives compression by one role and changes what a different role does next, which adds a writing step, a reading step and a decision.
Should the model write the handoff note or act on it?
Both, in separate conditions. Sender-only runs show what the model drops when it summarizes; receiver-only runs on human-written notes show whether it uses what it is given; end-to-end runs show the combined effect on the final action.
How do you grade a handoff note that is correct but phrased differently?
Grade units, not wording. Each planted unit has a source value, and the grader checks whether the note carries that value, distorts it or omits it. Free-text grading by domain experts is reserved for cases where the value cannot be matched deterministically.
Can synthetic business histories replace real records?
They can help with early development, but they tend to be cleaner than real CRM and ticket histories, where context is scattered across channels and superseded facts linger. Real, rights-cleared records belong in the sealed test set if the result is meant to transfer to deployment.
What would make a reported handoff result misleading?
Testing on the same companies used for design, reporting one blended score, ignoring correlation between units in one case, and changing metrics after seeing results. Each of these can make a model look better at keeping context than it is.
Sources
- Zendesk tickets separate public replies from internal notes, which only agents can see. Source
- tau-bench introduced pass^k, the chance that all k trials of a task succeed, to measure agent consistency. Source
- tau2-bench domains define a policy, agent tools, tasks and optional user tools, and results across grading versions are not comparable. Source
- TheAgentCompany evaluates agents in a simulated company with simulated colleagues, checkpoint-based partial credit and rubrics hidden from the agent. Source
- Models used relevant information best at the start or end of long inputs and worse in the middle. Source
- Splitting a fully specified instruction across conversation turns lowered model performance compared with giving it at once. Source
- MAST groups 14 multi-agent failure modes into system design, inter-agent misalignment and task verification. Source
- Evaluation results should carry standard errors, clustered when questions are related, and model comparisons should use paired differences. Source
- Preregistration records the plan in advance and separates confirmatory from exploratory analysis. Source
Tell us what data you need
Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.