Skip to content

AI uses for records

Agent trajectory expert disagreement: when two valid paths should both stay

By SourceX Editorial · Reviewed by Noah Loul ·

10 sections

Short answer

Keep two different expert trajectories when both replay from the same starting state, both reach an outcome the task's checks accept, and neither breaks a written rule. Collapse them when one path fails verification or differs only by noise. Escalate to the policy owner when the outcomes differ and the rules do not decide. Record each verdict with both rationales so every valid solution survives into the dataset.

Agent run

Agent trajectory expert

This page covers agent trajectory expert disagreement: when two valid paths should both stay.Illustrative

Key takeaways

  • A single reference trajectory is one way to solve a task, not the answer key; outcome checks are what decide whether a different path is correct.
  • Classify every split first: order-only, different route, different permitted outcome, rule break, policy gap or slip. Each type has a different default verdict.
  • Adjudicate on replayed evidence, not on which expert is more senior. If a path cannot be replayed, it cannot be adjudicated.
  • Verdict codes should change how the pair is used: both paths kept for training, outcome sets widened for evaluation, or the case held for a policy ruling.
  • Expert identities belong in the record only as pseudonymous IDs, and the right to use expert work product is a contract question, not a data question.

When should a trajectory dataset keep two different expert paths?

Keep both when each path replays from the same logged starting state, each ends in a state the task's outcome checks accept, and neither breaks a written rule. In that case the difference is information about the task: there is more than one correct way to do it, and an agent trained or graded on only one will be wrong about the other.

The tau2-bench maintainers make the same point about their own task files. The actions listed in a task are one reference trajectory, replayed on a fresh environment to derive the target database state; any sequence of tool calls that produces an equivalent end state passes. The listed path documents a solution, it does not define the only one.

Collapse to one path when the other fails outcome verification, breaks a rule, or differs only in harmless ordering that adds nothing. Escalate when the two paths end in different outcomes and the written policy does not say which is acceptable. That three-way rule is the core of the protocol below.

Why a single reference path makes a poor answer key

Matching against one recorded path punishes correct work that took another route. The tau2-bench documentation warns that its partial action score measures similarity to one reference path, not correctness, and that an agent can match none of the listed actions and still be fully correct. It reserves path-gated rewards for tasks where every valid solution has actually been enumerated.

Mature benchmarks grade outcomes for this reason. The tau-bench paper compares the database state at the end of a conversation with an annotated goal state. WebArena centers its tasks on the functional correctness of task completions. SWE-bench ships a gold patch with each instance, yet it grades a submitted patch by running the repository's tests and counts an instance resolved when the required tests pass, so a different patch can succeed.

Expert demonstrations inherit the same problem. When two experts solve a task differently, the question is not which one matches the reference, but whether each one meets the outcome checks and the rules.

Six kinds of expert divergence and the default verdict for each

Classify the split before anyone argues about it. Most disagreements fall into one of six types, and the type usually predicts the verdict. Compare the two paths step by step against the replayed state, then pick the first row that fits.

Six kinds of expert divergence and the default verdict for each
Divergence typeWhat it looks likeWhat it usually meansDefault verdict
Order onlySame reads and writes, in a different order, with no step depending on the otherNo real choice was madeKeep one path as canonical and note the order freedom
Different route, same outcomeDifferent tools or lookups, equivalent end stateMore than one valid method existsKeep both as equivalent solutions
Different permitted outcomeEnd states differ, and the policy allows eitherA legitimate judgment callKeep both, each tied to the condition that made it acceptable
Rule breakOne path exceeds a limit, skips a required check or writes something it should notAn error, however experienced the expertReject that path, keep it as a labelled negative example if useful
Policy gapEnd states differ, and the written policy does not decideThe rules are incompleteHold the case for a policy owner ruling
Slip or missing informationOne expert misread a field, lacked a document or hit a tool errorCollection defect, not disagreementFix the collection issue and redo the task

The adjudication protocol, stage by stage

This is the asset: a protocol that decides each disagreement on evidence and keeps every valid solution. Run it on every task where two experts' trajectories differ in end state or in any write action, and on a sample of tasks where they differ only in reads.

  • Keep the order of stages. Showing experts each other's work before stage 6 turns independent evidence into negotiation.
  • Never let seniority settle stage 7. The verdict cites an outcome check or a policy clause, or it is not a verdict.
The adjudication protocol, stage by stage
StageWho actsInputOutput and stop condition
1. Independent captureEach expert, without seeing the other's workSame task, same starting state, same toolsTwo complete trajectories with a short rationale at every write step
2. ReplayData engineerBoth trajectories and the logged starting stateEach path reproduces its own end state; a path that will not replay goes to stage 8 as not replayable
3. Outcome checkData engineer, using the task's checksBoth end statesPass or fail per path against outcome checks and records that must not change
4. Rule checkTask expert who did not produce either pathBoth paths and the written policy version in forceEach write step marked permitted, not permitted or not covered by policy
5. ClassifyAdjudicatorResults of stages 2 to 4One divergence type from the six-row table
6. Rationale exchangeBoth experts, after classificationThe other expert's path and rationaleEach expert says whether the other path is acceptable and why; changed minds are recorded, not overwritten
7. VerdictAdjudicator, or policy owner for policy gapsEverything aboveOne verdict code with a written reason and the policy clause relied on
8. Route and auditDataset ownerVerdict and both pathsPaths stored with verdict; a sample of verdicts re-adjudicated each release by a second adjudicator

Verdict codes and what each one does to the dataset

A verdict only matters if it changes how the data is used. Each code below maps to a specific treatment for training and for evaluation, so nobody has to reinterpret free text later.

The approach follows the wider annotation literature. Davani and colleagues showed that majority voting discards real disagreement and proposed modeling each annotator's judgments separately; trajectory data can keep the same information by storing each expert's path with its verdict rather than one merged path.

Verdict codes and what each one does to the dataset
Verdict codeMeaningTraining useEvaluation use
EQUIVALENTBoth paths valid, same outcomeKeep both as positive demonstrationsGrade on outcome; never require either path
ALTERNATIVE_OUTCOMEBoth valid, different permitted outcomesKeep both, each with the condition that justifies itAccept either outcome; add the condition to the task if it should decide
PREFERREDBoth acceptable, policy or the business prefers oneUse both, mark the preferred one; usable as a preference pairAccept both, report preferred-path rate separately
REJECTEDOne path breaks a rule or fails outcome checksKeep the valid path; keep the other only as a labelled negativeThe rejected outcome becomes a failing case
POLICY_HOLDPolicy does not decideExclude until ruledExclude until ruled; log the gap for the policy owner
NOT_REPLAYABLEA path cannot be reproducedExcludeExclude; fix logging and recollect

Can a second team rebuild both paths and reach the same outcomes?

If not, the disagreement cannot be adjudicated, because nobody can check which outcome each path really produced. Replay is the precondition for every stage after the first.

Benchmarks that grade by outcome show what replay needs. OSWorld gives each task an initial state setup configuration and an execution-based evaluation script. AppWorld defines a task's initial state as the app databases plus the current date and time. A pair of expert trajectories needs the same: a starting state another team can restore, and a check another team can run.

  • Case ID and step IDs, stable across both paths so steps can be aligned.
  • Actor: a pseudonymous expert ID and role, never a name.
  • Time of each step, and the clock value the environment showed.
  • Observation: what the expert saw before acting, including documents opened.
  • Action: the tool call or interface action with its arguments.
  • Tool response: the full response, including errors and retries.
  • Artifact links: files, messages or records created or changed.
  • Verified outcome: the end state and the result of each outcome check.
  • Replay instructions: environment version, starting snapshot, policy version and any stubbed services.

Worked example (illustrative): two support leads, one damaged-item claim

This case is hypothetical. A customer reports a damaged item and says in the chat that a replacement or a refund would both be fine. Expert A checks the order, confirms the item is in stock and ships a replacement. Expert B checks the order, notes a delivery delay on the replacement stock and issues a refund.

Both paths replay from the logged state. Both pass the outcome checks: the customer is made whole and no other record changes. The rule check finds both permitted, because the policy allows either remedy when the customer accepts both. The adjudicator classifies the split as a different permitted outcome and records ALTERNATIVE_OUTCOME, with the condition that the customer accepted either remedy, and notes that stock delay explains B's choice.

A third expert, C, refunds the item and also adds a goodwill credit above the limit an agent may grant without approval. C's outcome check passes for the customer but the rule check fails on the credit. That path is REJECTED and kept only as a labelled negative example. If the policy had been silent on goodwill credits, the case would have gone to POLICY_HOLD instead.

When expert disagreement is a defect, not a signal

Disagreement is useful only when it reflects a real choice in the task. Plank argues that treating every task as having one ground truth ignores genuine disagreement and multiple plausible answers; the converse also holds, and some disagreements are just errors.

Treat a split as a defect when one expert lacked information the other had, when the instruction was ambiguous in a way the task author did not intend, when a tool failed during one session, or when the two experts were working against different policy versions. Fix the collection, not the verdict. A high share of slip verdicts in a batch is a sign that the task design needs work before more experts are paid to repeat it.

Stop and narrow the scope when policy holds pile up in one task family: the business has not decided its own rules, and no amount of expert labeling will decide them.

Expert identities, contracts and what the record should leave out

Keep who-did-what as pseudonymous IDs with role and tenure band, not names. Prabhakaran and colleagues recommend releasing annotator-level labels, ideally anonymized, and documenting how annotators were recruited and how work was assigned; the same documentation lets a buyer judge whether a disagreement reflects expertise or assignment.

Trajectories captured on real business systems also carry customer and employee data in observations and tool responses, so de-identification has to cover those fields, not only the expert ID. Whether the experts' work product may be licensed depends on their employment or service contracts, and whether customer records may be used depends on the business's notices and agreements. Counsel should review both before release.

Sourcing workflow histories with more than one expert's hand on them

Real businesses already produce natural pairs: second reviews, escalations, supervisor overrides and audits where two people handled the same kind of case differently. Those records are where valid alternative paths, and real policy gaps, show up without staging them.

Companies can license selected review histories like these through SourceX, which handles the licensing work on their side. A buyer asks for multi-reviewer workflow histories by giving scope, volume, language, format, timing and licensing needs; such records are found among U.S. businesses for the specific request, and nothing is delivered before the supplier has approved the scope and signed.

Frequently asked questions

Should agent training data contain more than one correct trajectory per task?

Yes, when the task genuinely has more than one valid method or permitted outcome. Training on one path teaches the agent that the others are wrong, and grading on one path marks correct agents as failures. Keep each valid path with a verdict that says why it is valid.

What should the adjudication protocol preserving multiple valid solutions include?

Independent capture by each expert, replay of both paths, outcome checks, a rule check against the written policy, classification of the divergence, an exchange of rationales, a coded verdict with the reason, and a periodic re-adjudication sample. The output is each path stored with its verdict, not one merged path.

Could another team replay both expert paths and confirm each outcome?

Only if each trajectory carries the starting snapshot, environment and policy versions, every observation and tool response, artifact links and the outcome checks. With those, another team can restore the state, replay each path and confirm both end states before adjudicating.

Who should adjudicate when two experts disagree?

A task expert who produced neither path, working from replayed evidence and the written policy. When the policy does not decide, the policy owner rules instead, and the ruling is recorded so later cases follow it.

Is majority vote a good way to settle trajectory disagreements?

Usually not. A vote picks the more common path, not the correct one, and it discards valid alternatives. Outcome checks and policy clauses decide correctness; votes only measure popularity.

Sources

  • tau2-bench treats a task's listed actions as one reference trajectory; any tool-call sequence reaching an equivalent database end state passes, and path-gated rewards are reserved for tasks with all valid solutions enumerated. Source
  • tau-bench compares the database state at the end of a conversation with the annotated goal state. Source
  • WebArena's benchmark tasks focus on evaluating the functional correctness of task completions. Source
  • Each SWE-bench instance includes a gold patch and FAIL_TO_PASS and PASS_TO_PASS test lists. Source
  • SWE-bench grades a generated patch by running the repository's tests and counts an instance resolved when the patch made the required tests pass. Source
  • OSWorld gives each task an initial state setup configuration and a custom execution-based evaluation script. Source
  • AppWorld defines a task's initial state as the state of the app databases and the current date and time. Source
  • Plank argues that assuming a single ground truth ignores genuine disagreement, subjectivity and multiple plausible answers. Source
  • Davani and colleagues show majority voting discards real disagreement and model each annotator's judgments as separate subtasks. Source
  • Prabhakaran and colleagues recommend releasing annotator-level labels, ideally anonymized, and documenting annotator recruitment and work assignment. Source

Tell us what data you need

Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.

See if your company qualifies