AI uses for records
Agent trajectory expert disagreement: when two valid paths should both stay
By SourceX Editorial · Reviewed by Noah Loul ·
10 sections
Short answer
Keep two different expert trajectories when both replay from the same starting state, both reach an outcome the task's checks accept, and neither breaks a written rule. Collapse them when one path fails verification or differs only by noise. Escalate to the policy owner when the outcomes differ and the rules do not decide. Record each verdict with both rationales so every valid solution survives into the dataset.
Agent run
Agent trajectory expert
Key takeaways
- A single reference trajectory is one way to solve a task, not the answer key; outcome checks are what decide whether a different path is correct.
- Classify every split first: order-only, different route, different permitted outcome, rule break, policy gap or slip. Each type has a different default verdict.
- Adjudicate on replayed evidence, not on which expert is more senior. If a path cannot be replayed, it cannot be adjudicated.
- Verdict codes should change how the pair is used: both paths kept for training, outcome sets widened for evaluation, or the case held for a policy ruling.
- Expert identities belong in the record only as pseudonymous IDs, and the right to use expert work product is a contract question, not a data question.
When should a trajectory dataset keep two different expert paths?
Keep both when each path replays from the same logged starting state, each ends in a state the task's outcome checks accept, and neither breaks a written rule. In that case the difference is information about the task: there is more than one correct way to do it, and an agent trained or graded on only one will be wrong about the other.
The tau2-bench maintainers make the same point about their own task files. The actions listed in a task are one reference trajectory, replayed on a fresh environment to derive the target database state; any sequence of tool calls that produces an equivalent end state passes. The listed path documents a solution, it does not define the only one.
Collapse to one path when the other fails outcome verification, breaks a rule, or differs only in harmless ordering that adds nothing. Escalate when the two paths end in different outcomes and the written policy does not say which is acceptable. That three-way rule is the core of the protocol below.
Why a single reference path makes a poor answer key
Matching against one recorded path punishes correct work that took another route. The tau2-bench documentation warns that its partial action score measures similarity to one reference path, not correctness, and that an agent can match none of the listed actions and still be fully correct. It reserves path-gated rewards for tasks where every valid solution has actually been enumerated.
Mature benchmarks grade outcomes for this reason. The tau-bench paper compares the database state at the end of a conversation with an annotated goal state. WebArena centers its tasks on the functional correctness of task completions. SWE-bench ships a gold patch with each instance, yet it grades a submitted patch by running the repository's tests and counts an instance resolved when the required tests pass, so a different patch can succeed.
Expert demonstrations inherit the same problem. When two experts solve a task differently, the question is not which one matches the reference, but whether each one meets the outcome checks and the rules.
Six kinds of expert divergence and the default verdict for each
Classify the split before anyone argues about it. Most disagreements fall into one of six types, and the type usually predicts the verdict. Compare the two paths step by step against the replayed state, then pick the first row that fits.
| Divergence type | What it looks like | What it usually means | Default verdict |
|---|---|---|---|
| Order only | Same reads and writes, in a different order, with no step depending on the other | No real choice was made | Keep one path as canonical and note the order freedom |
| Different route, same outcome | Different tools or lookups, equivalent end state | More than one valid method exists | Keep both as equivalent solutions |
| Different permitted outcome | End states differ, and the policy allows either | A legitimate judgment call | Keep both, each tied to the condition that made it acceptable |
| Rule break | One path exceeds a limit, skips a required check or writes something it should not | An error, however experienced the expert | Reject that path, keep it as a labelled negative example if useful |
| Policy gap | End states differ, and the written policy does not decide | The rules are incomplete | Hold the case for a policy owner ruling |
| Slip or missing information | One expert misread a field, lacked a document or hit a tool error | Collection defect, not disagreement | Fix the collection issue and redo the task |
The adjudication protocol, stage by stage
This is the asset: a protocol that decides each disagreement on evidence and keeps every valid solution. Run it on every task where two experts' trajectories differ in end state or in any write action, and on a sample of tasks where they differ only in reads.
- Keep the order of stages. Showing experts each other's work before stage 6 turns independent evidence into negotiation.
- Never let seniority settle stage 7. The verdict cites an outcome check or a policy clause, or it is not a verdict.
| Stage | Who acts | Input | Output and stop condition |
|---|---|---|---|
| 1. Independent capture | Each expert, without seeing the other's work | Same task, same starting state, same tools | Two complete trajectories with a short rationale at every write step |
| 2. Replay | Data engineer | Both trajectories and the logged starting state | Each path reproduces its own end state; a path that will not replay goes to stage 8 as not replayable |
| 3. Outcome check | Data engineer, using the task's checks | Both end states | Pass or fail per path against outcome checks and records that must not change |
| 4. Rule check | Task expert who did not produce either path | Both paths and the written policy version in force | Each write step marked permitted, not permitted or not covered by policy |
| 5. Classify | Adjudicator | Results of stages 2 to 4 | One divergence type from the six-row table |
| 6. Rationale exchange | Both experts, after classification | The other expert's path and rationale | Each expert says whether the other path is acceptable and why; changed minds are recorded, not overwritten |
| 7. Verdict | Adjudicator, or policy owner for policy gaps | Everything above | One verdict code with a written reason and the policy clause relied on |
| 8. Route and audit | Dataset owner | Verdict and both paths | Paths stored with verdict; a sample of verdicts re-adjudicated each release by a second adjudicator |
Verdict codes and what each one does to the dataset
A verdict only matters if it changes how the data is used. Each code below maps to a specific treatment for training and for evaluation, so nobody has to reinterpret free text later.
The approach follows the wider annotation literature. Davani and colleagues showed that majority voting discards real disagreement and proposed modeling each annotator's judgments separately; trajectory data can keep the same information by storing each expert's path with its verdict rather than one merged path.
| Verdict code | Meaning | Training use | Evaluation use |
|---|---|---|---|
| EQUIVALENT | Both paths valid, same outcome | Keep both as positive demonstrations | Grade on outcome; never require either path |
| ALTERNATIVE_OUTCOME | Both valid, different permitted outcomes | Keep both, each with the condition that justifies it | Accept either outcome; add the condition to the task if it should decide |
| PREFERRED | Both acceptable, policy or the business prefers one | Use both, mark the preferred one; usable as a preference pair | Accept both, report preferred-path rate separately |
| REJECTED | One path breaks a rule or fails outcome checks | Keep the valid path; keep the other only as a labelled negative | The rejected outcome becomes a failing case |
| POLICY_HOLD | Policy does not decide | Exclude until ruled | Exclude until ruled; log the gap for the policy owner |
| NOT_REPLAYABLE | A path cannot be reproduced | Exclude | Exclude; fix logging and recollect |
Can a second team rebuild both paths and reach the same outcomes?
If not, the disagreement cannot be adjudicated, because nobody can check which outcome each path really produced. Replay is the precondition for every stage after the first.
Benchmarks that grade by outcome show what replay needs. OSWorld gives each task an initial state setup configuration and an execution-based evaluation script. AppWorld defines a task's initial state as the app databases plus the current date and time. A pair of expert trajectories needs the same: a starting state another team can restore, and a check another team can run.
- Case ID and step IDs, stable across both paths so steps can be aligned.
- Actor: a pseudonymous expert ID and role, never a name.
- Time of each step, and the clock value the environment showed.
- Observation: what the expert saw before acting, including documents opened.
- Action: the tool call or interface action with its arguments.
- Tool response: the full response, including errors and retries.
- Artifact links: files, messages or records created or changed.
- Verified outcome: the end state and the result of each outcome check.
- Replay instructions: environment version, starting snapshot, policy version and any stubbed services.
Worked example (illustrative): two support leads, one damaged-item claim
This case is hypothetical. A customer reports a damaged item and says in the chat that a replacement or a refund would both be fine. Expert A checks the order, confirms the item is in stock and ships a replacement. Expert B checks the order, notes a delivery delay on the replacement stock and issues a refund.
Both paths replay from the logged state. Both pass the outcome checks: the customer is made whole and no other record changes. The rule check finds both permitted, because the policy allows either remedy when the customer accepts both. The adjudicator classifies the split as a different permitted outcome and records ALTERNATIVE_OUTCOME, with the condition that the customer accepted either remedy, and notes that stock delay explains B's choice.
A third expert, C, refunds the item and also adds a goodwill credit above the limit an agent may grant without approval. C's outcome check passes for the customer but the rule check fails on the credit. That path is REJECTED and kept only as a labelled negative example. If the policy had been silent on goodwill credits, the case would have gone to POLICY_HOLD instead.
When expert disagreement is a defect, not a signal
Disagreement is useful only when it reflects a real choice in the task. Plank argues that treating every task as having one ground truth ignores genuine disagreement and multiple plausible answers; the converse also holds, and some disagreements are just errors.
Treat a split as a defect when one expert lacked information the other had, when the instruction was ambiguous in a way the task author did not intend, when a tool failed during one session, or when the two experts were working against different policy versions. Fix the collection, not the verdict. A high share of slip verdicts in a batch is a sign that the task design needs work before more experts are paid to repeat it.
Stop and narrow the scope when policy holds pile up in one task family: the business has not decided its own rules, and no amount of expert labeling will decide them.
Expert identities, contracts and what the record should leave out
Keep who-did-what as pseudonymous IDs with role and tenure band, not names. Prabhakaran and colleagues recommend releasing annotator-level labels, ideally anonymized, and documenting how annotators were recruited and how work was assigned; the same documentation lets a buyer judge whether a disagreement reflects expertise or assignment.
Trajectories captured on real business systems also carry customer and employee data in observations and tool responses, so de-identification has to cover those fields, not only the expert ID. Whether the experts' work product may be licensed depends on their employment or service contracts, and whether customer records may be used depends on the business's notices and agreements. Counsel should review both before release.
Sourcing workflow histories with more than one expert's hand on them
Real businesses already produce natural pairs: second reviews, escalations, supervisor overrides and audits where two people handled the same kind of case differently. Those records are where valid alternative paths, and real policy gaps, show up without staging them.
Companies can license selected review histories like these through SourceX, which handles the licensing work on their side. A buyer asks for multi-reviewer workflow histories by giving scope, volume, language, format, timing and licensing needs; such records are found among U.S. businesses for the specific request, and nothing is delivered before the supplier has approved the scope and signed.
Frequently asked questions
Should agent training data contain more than one correct trajectory per task?
Yes, when the task genuinely has more than one valid method or permitted outcome. Training on one path teaches the agent that the others are wrong, and grading on one path marks correct agents as failures. Keep each valid path with a verdict that says why it is valid.
What should the adjudication protocol preserving multiple valid solutions include?
Independent capture by each expert, replay of both paths, outcome checks, a rule check against the written policy, classification of the divergence, an exchange of rationales, a coded verdict with the reason, and a periodic re-adjudication sample. The output is each path stored with its verdict, not one merged path.
Could another team replay both expert paths and confirm each outcome?
Only if each trajectory carries the starting snapshot, environment and policy versions, every observation and tool response, artifact links and the outcome checks. With those, another team can restore the state, replay each path and confirm both end states before adjudicating.
Who should adjudicate when two experts disagree?
A task expert who produced neither path, working from replayed evidence and the written policy. When the policy does not decide, the policy owner rules instead, and the ruling is recorded so later cases follow it.
Is majority vote a good way to settle trajectory disagreements?
Usually not. A vote picks the more common path, not the correct one, and it discards valid alternatives. Outcome checks and policy clauses decide correctness; votes only measure popularity.
Sources
- tau2-bench treats a task's listed actions as one reference trajectory; any tool-call sequence reaching an equivalent database end state passes, and path-gated rewards are reserved for tasks with all valid solutions enumerated. Source
- tau-bench compares the database state at the end of a conversation with the annotated goal state. Source
- WebArena's benchmark tasks focus on evaluating the functional correctness of task completions. Source
- Each SWE-bench instance includes a gold patch and FAIL_TO_PASS and PASS_TO_PASS test lists. Source
- SWE-bench grades a generated patch by running the repository's tests and counts an instance resolved when the patch made the required tests pass. Source
- OSWorld gives each task an initial state setup configuration and a custom execution-based evaluation script. Source
- AppWorld defines a task's initial state as the state of the app databases and the current date and time. Source
- Plank argues that assuming a single ground truth ignores genuine disagreement, subjectivity and multiple plausible answers. Source
- Davani and colleagues show majority voting discards real disagreement and model each annotator's judgments as separate subtasks. Source
- Prabhakaran and colleagues recommend releasing annotator-level labels, ideally anonymized, and documenting annotator recruitment and work assignment. Source
Tell us what data you need
Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.