Skip to content

AI data market

Decision history agent learning: what one gold label erases

By SourceX Editorial · Reviewed by Noah Loul ·

9 sections

Short answer

Decision histories can help agents handle ambiguity only if they record the readings a request allowed, the one the business chose, why, and what followed. Single-answer labels erase exactly that. In a recount of AmbiEnt's 542 ambiguous examples, nine crowdworkers split on 78.6 percent, and writing out each reading lifted Fleiss' kappa from 0.34 (the paper reports 0.12) to 0.67. No public study has yet tested the business-record version.

Agent run

Decision history agent

This page covers decision history agent learning: what one gold label erases.Illustrative

Key takeaways

  • A single-answer dataset stores one accepted output per input; a decision history also stores the options a request allowed, the choice, its reason, the decider and the outcome.
  • In AmbiEnt's crowdworker file, a one-label scheme left nine annotators split on 426 of 542 ambiguous examples, and majority vote would keep one label where the dataset's experts accepted two or more.
  • Writing out each reading lifted agreement to a Fleiss' kappa of 0.67, from 0.12 as the AmbiEnt paper reports or 0.34 in a standard computation on the released crowdworker file, so most of the disagreement was ambiguity rather than careless labeling.
  • Adjacent research points the same way: models trained on human label distributions generalized better, and post-training reduced GPT-4's calibration, but none of it tests business decision records.
  • The scarce, investable layer is reasons and outcomes captured at decision time; request text and final states are plentiful.
  • Ask any supplier for the share of sampled records with more than one documented option, a timestamped reason and a linked outcome.

Do decision histories help agents handle ambiguous requests?

They can, but only when the history records more than the final action. An agent facing an ambiguous request needs to know which readings were admissible, which one the business preferred, what tipped the choice and whether it held up. A record that keeps all four can teach when to choose, when to ask and how to explain; a record that keeps only the final action teaches a guess.

Two dataset types are being compared. A single-answer dataset pairs each input with one accepted output: a closed ticket with its resolution code, a request with the action an employee took. A decision history adds the options that were open, the constraint or preference that ranked them, the person who decided, the stated reason and the downstream outcome, including any reversal. The second type is rarer and costlier to assemble, and the investor question is whether it earns a premium for agent training and evaluation.

Public evidence answers part of that question. It shows what single answers do to ambiguous items, how models behave on them and whether keeping disagreement helps a learner. It does not yet show that business decision histories make agents better at ambiguity, because no published controlled study located for this article has tested that. This article builds the case from the evidence that exists, adds a recount of public annotation data and names the study that would settle it.

A recount of 542 ambiguous cases: what one label erases

Majority vote over single labels discards a valid reading in every ambiguous item, and the discarded reading then looks like annotator noise. The cleanest public measurement comes from AmbiEnt, a linguist-annotated benchmark of 1,645 examples published at EMNLP 2023. Its repository includes a crowdworker file of 542 ambiguous premise-hypothesis pairs, each labeled by nine crowdworkers under the usual one-label scheme and paired with expert-written interpretations of each reading.

Recounting that file for this article: all 542 items carry at least two interpretations with different correct labels (538 have two, four have three). On the original, ambiguous wording the nine crowdworkers agreed unanimously on 116 items, 21.4 percent, and split on 426, 78.6 percent. The most common label was one of the experts' valid labels in 517 items, and the crowd's labels together covered every valid label in 387. A majority-vote gold label would keep one reading and mark the others wrong in all 542.

The decisive number is what happened after disambiguation. When crowdworkers labeled each written-out reading, agreement measured by Fleiss' kappa rose from 0.34 on the 542 ambiguous originals to 0.67 on the 1,091 interpretations, and the crowd majority matched the expert label on 1,039 of them, 95.2 percent. The starting figure comes with a caveat: the AmbiEnt paper reports a kappa of 0.12 on the original wording and 0.67 after disambiguation; a standard Fleiss computation on the released crowdworker file gives 0.34. From either starting point agreement rose to 0.67 once each reading was explicit, so most of the split came from ambiguity in the input, not carelessness in the annotators.

Models stumble on the same items. The AmbiEnt authors report that GPT-4's generated disambiguations were judged correct 32 percent of the time in crowdworker evaluation, against 90 percent for the disambiguations in their dataset. The table restates the recount as a dataset buyer would read it:

A recount of 542 ambiguous cases: what one label erases
Recount measure (AmbiEnt crowd file)ValueWhat it means for a single-answer dataset
Ambiguous items with two or more valid labels542 of 542One gold label marks a valid reading wrong in every item
Nine-annotator split on the original wording426 of 542 (78.6%)Disagreement is the normal case, not the exception
Plurality label is one of the valid labels517 of 542 (95.4%)The majority is usually right, just incomplete
Crowd labels cover every valid label387 of 542 (71.4%)The minority vote often holds the second reading
Fleiss' kappa, ambiguous wording vs written-out readings0.34 vs 0.67 (paper: 0.12 vs 0.67)Recording the reading restores agreement
Crowd majority matches the expert label per reading1,039 of 1,091 (95.2%)Residual noise is small once readings are explicit
Two semicircle gauges of crowd agreement on AmbiEnt's ambiguous examples, Fleiss' kappa 0.34 on the original wording in a standard recount of the released file (the paper reports 0.12) and 0.67 once each reading is written out, above a bar splitting 542 items into 426 split and 116 unanimous.
Recount of the public AmbiEnt crowdworker file (Liu et al., 2023), computed for this article on 2026-10-27.

Why investors should expect ambiguity in real requests, not just in benchmarks

Ambiguity is common wherever people phrase requests in their own words, and systems trained toward single answers handle it poorly. The AmbigQA authors found that over half of the questions in NQ-open, an open-domain question-answering benchmark, were ambiguous, with sources such as unclear event and entity references (Min et al., 2020). Business requests add sources of their own: an order number that matches two shipments, a 'next week' that depends on the time zone, a discount that could apply per item or per order.

Disagreement also predicts where models fail. ChaosNLI collected 100 annotations per example for 4,645 inference examples and found that models did well on high-agreement items but barely beat random guessing on low-agreement ones (Nie et al., 2020). A dataset that hides agreement levels hides exactly the items an agent is most likely to get wrong.

Keeping the distribution appears to help, at least where it has been tested. Training image classifiers against full human label distributions from CIFAR10H improved accuracy on every generalization set the authors tried, with larger gains further from the training distribution, and left the models far less confident when they were wrong (Peterson et al., 2019). Training toward single preferred answers can push the other way: OpenAI's GPT-4 technical report (2023) says the pre-trained model was well calibrated and that calibration was reduced after post-training.

None of these studies used business records, and two are not about agents at all. Together they support a mechanism, not a result: labels that preserve the admissible options give a learner something to be uncertain about, and graders that accept every valid option stop punishing an agent for picking a legitimate one.

The agent-task input map for an ambiguous request

An ambiguous business request becomes a usable agent task when seven inputs are present and checkable. The map lays them out as states, actions, constraints and verifiable outcomes, the four parts any agent environment needs, and marks what a single-answer record usually keeps.

Illustrative: a customer writes 'please move my delivery to next week'. One reading postpones the whole order; another postpones only the backordered item while the rest ships. A single-answer record stores 'rescheduled order to June 12'. A decision history also stores both readings, the rule that split shipments need the customer's consent, the question the employee asked, the customer's answer and whether the delivery then arrived without a complaint.

Each row names one input, its role in the task, what each record type holds and how a buyer can check it in a sample:

The agent-task input map for an ambiguous request
InputRole in the taskSingle-answer recordDecision historyHow a buyer verifies it
Request as received, verbatimInitial stateOften paraphrased or droppedKept with timestamp and channelCompare with the source message
System state at decision timeInitial stateCurrent state onlySnapshot or replayable logReplay to the decision moment
Admissible readings or optionsAction spaceAbsentListed before the choiceOptions timestamped before the action
Rule or preference that ranked themConstraintAbsent or implicitLinked policy version or stated preferenceVersion in force on that date
Choice, decider and authorityActionFinal action onlyWho chose, under what authorityApproval or role record
Stated reasonConstraint evidenceAbsentWritten at decision timeTimestamp precedes the outcome
Outcome and any reversalVerifiable outcomeClosed statusLinked follow-up, complaint or reversalLater record in another system

Where documented tradeoffs add value in the agent data chain

Value concentrates in the two links that cannot be produced after the fact: the stated reason and the linked outcome. Requests and final states are abundant in every ticketing and order system. Reasons written at decision time, and outcomes joined back to the decision, exist only where a process forced someone to record them.

At a glance

Where documented tradeoffs add value in the agent data chain

An evaluation buyer ultimately pays for a grader that accepts every valid reading and scores the cost of an invalid one. The investor read is that suppliers whose records already carry reasons and outcomes hold the scarce links, while suppliers offering volume of requests and final actions compete on price. That ordering is an editorial judgment, built from the evidence above rather than from deal data.

The chain, link by link, from most abundant to scarcest:

  • Records: requests, actions and statuses. Plentiful, low in value per record, and the raw material every competitor can reach.
  • Episode assembly: joining request, state, action and outcome across systems. An engineering cost that better tooling keeps lowering.
  • Reading annotation: listing the admissible readings and marking the chosen one. Experts can add this to old records, as AmbiEnt's linguists did to machine-generated examples, so it is labor rather than scarcity.
  • Reason and rule linkage: why this option won, under which rule version. Scarce, because a reason reconstructed months later is a guess about a guess.
  • Outcome linkage: whether the choice held, was reversed or drew a complaint. Scarce, because it needs later records that most exports leave behind.

Could single-answer data and a clarifying question be enough?

Possibly, and that is the strongest case against paying for decision histories. An agent can be instructed to ask whenever readings diverge, and a benchmark can grade the asking rather than the choice. If asking were always cheap and always welcome, a business's documented preference would matter less.

Three facts limit that escape. Asking costs time and goodwill, so an agent still needs to know which ambiguities the business settles by default and which ones justify a question. The ChaosNLI finding that low agreement predicts failure applies to the decision to ask as much as to the answer. And the default itself is a business choice that differs from one company to the next, which is precisely what a decision history records.

The opposite objection concerns evidence rather than demand: a record collection may lack the environment or outcome an agent needs. A folder of approvals with reasons but no replayable state cannot test an agent, and a reason without an outcome cannot show that the choice was good. Decision histories earn a premium only when state and outcome arrive with them, which a buyer can check on a sample before paying.

What could make the decision-history thesis wrong

Five developments would weaken the thesis. Each can be tested, and a supplier that cannot say which one worries it most has not stress-tested its own case.

  • No transfer. A controlled study could show that agents trained on business decision histories handle ambiguous requests no better than agents given single answers plus a clarification policy. None was found while researching this article.
  • Experts beat records. Most AmbiEnt examples were built by generating unlabeled examples with InstructGPT and having linguists annotate label sets and disambiguations. If expert annotation of synthetic cases teaches ambiguity as well as real histories do, the premium moves to annotation labor.
  • Reasons are thin. If sampled rationales turn out to be one-word codes or notes written after the outcome was known, the scarce link is missing even where the field exists.
  • Cheap graders win. Buyers may keep single-outcome graders because they cost less to run and accept some unfairness to agents; acceptable-set labels would then serve a smaller market.
  • Rights narrow the supply. Reasons often sit in employee notes and customer messages, which need privacy and contract review before any license, and some will not clear it.

Diligence questions for decision-history suppliers, SourceX included

Ask for measurements on a sample you choose, with definitions fixed in advance, so the answers cannot be curated. The questions work for every supplier of agent data.

Treat SourceX as one supplier to examine this way, not as evidence for the thesis. It arranges licenses for business records that a company decides to offer, taking on the sourcing and qualification, rights review, contract, preparation, delivery and payment. The supplying company approves scope and terms and signs before any record reaches SourceX. SourceX has published no figures on how many records it handles carry readings, reasons or outcomes; that kind of company and commercial metric goes to investors directly.

SourceX is raising from accredited investors under Rule 506(c). Its investor page is where to request materials and to put a specific question, such as the reading-coverage share in the first item:

  • In a random sample of 100 decision records, what share lists more than one admissible option before the choice was made?
  • What share carries a reason written at decision time, and how is that timestamp proven?
  • What share links to a later outcome, such as a delivery, a complaint, a refund or a reversal?
  • Can the system state at decision time be replayed, or only described?
  • When two experts disagree about which option the business preferred, what happens to the label?
  • Does the grader accept a set of valid answers, and how is the cost of each wrong answer set?
  • Which contracts and notices cover the reason text, and what share of sampled records cleared review?

Methods note: how the AmbiEnt recount and input map were built

The recount used AmbiEnt/analysis/crowdworker_annotations.jsonl from the AmbiEnt repository (github.com/alisawuffles/ambient, commit 1fcb43e of February 5, 2024, CC BY 4.0), read on 2026-10-27. Valid labels are the distinct non-null labels of the written interpretations; a null label marks a distractor. Original-wording labels are the nine q0 labels per item. Fleiss' kappa uses the standard formula with nine raters, over the 542 originals and over the 1,091 non-distractor interpretations, the same item selection as the repository's crowdworker statistics script. The AmbiEnt paper (EMNLP 2023, arXiv 2304.14399) reports a Fleiss' kappa of 0.12 on the original wording and 0.67 after disambiguation, and the repository's saved notebook output (notebooks/crowdworker_experiment statistics.ipynb) gives 0.117 and 0.670; a standard Fleiss computation on the released crowdworker file gives 0.34 and 0.67. The cause of the gap on the original wording was not established; the rise to 0.67 holds from either starting figure.

Other evidence: the AmbiEnt paper (EMNLP 2023), AmbigQA (2020), ChaosNLI (2020), Peterson and colleagues on CIFAR10H (2019) and the GPT-4 technical report (2023), each confirmed through a search excerpt on its publisher's site, plus the AmbiEnt README read in the clone. The input map and the value chain are editorial frameworks written for this article.

Not measured: how often real business records carry readings, reasons or outcomes; whether decision histories improve agents on ambiguous requests; and what buyers pay for acceptable-set labels. AmbiEnt's items are short English sentence pairs, not business tasks, so the recount shows what a labeling scheme does, not what agents learn.

Frequently asked questions

What is a decision history in AI training data?

A decision history is a record of one business choice that keeps the request, the system state at the time, the options that were open, the rule or preference that ranked them, who decided, the stated reason and what happened next. It differs from a log of actions because it records why one option beat the others. For an agent, that difference turns a guess into a learnable choice.

Is annotator disagreement just noise in a dataset?

Often it is not. In AmbiEnt's crowdworker file, nine annotators split on 78.6 percent of ambiguous items, yet agreement rose to a Fleiss' kappa of 0.67 once each reading was written out, from 0.12 as the paper reports or 0.34 when the released file is recounted with the standard formula, so most of the split came from the input. Keeping only the majority label throws a valid reading away.

How is a decision history different from an agent trajectory?

A trajectory records the steps an agent or person took, such as tool calls and messages, from start to finish. A decision history focuses on the choice points inside that path: which options were admissible, which one was taken, why, and whether it held. A trajectory can contain decision histories, but many record only the path taken.

Can single-answer datasets still be used to evaluate agents?

Yes, for tasks whose correct outcome is unique, such as a refund amount fixed by a rule. For ambiguous requests, a single answer marks valid alternatives wrong and rewards agents for guessing the majority reading. Graders for those items should accept a set of valid outcomes and score the cost of invalid ones.

What evidence would show decision histories deserve a premium?

A controlled comparison: agents trained or evaluated with decision histories against agents given single answers plus a clarification policy, on held-out ambiguous business requests, with outcomes checked in later records. None was found while researching this article. Until one appears, price the scarce links, reasons and outcomes captured at decision time, rather than record counts.

Sources

  • AmbiEnt's crowdworker file holds 542 ambiguous examples, each labeled by nine crowdworkers on the original wording and on each written interpretation; this article's recount of it gives the split, coverage and Fleiss' kappa figures. Source
  • AmbiEnt examples were mostly created by generating unlabeled examples with InstructGPT, then annotated with label sets and disambiguations by linguists and validated by the authors; the crowdworker file records annotations under a scheme that does not account for ambiguity. Source
  • AmbiEnt is a linguist-annotated benchmark of 1,645 examples; GPT-4's generated disambiguations were judged correct 32 percent of the time in crowdworker evaluation, against 90 percent for the dataset's own disambiguations (EMNLP 2023). Source
  • Over half of the questions in NQ-open are ambiguous, with sources such as event and entity references (AmbigQA, 2020). Source
  • ChaosNLI collected 100 annotations per example for 4,645 NLI examples; models do well on high-agreement items but barely beat random guessing on low-agreement ones (2020). Source
  • Training classifiers against full human label distributions from CIFAR10H improved accuracy on all generalization datasets tested, with larger gains further out of distribution, and made models less confident when wrong (2019). Source
  • The GPT-4 technical report says the pre-trained model was highly calibrated and that calibration was reduced after the post-training process (2023). Source

Request SourceX investor materials

SourceX shares company and commercial metrics with investors directly. Any offering is made only to verified accredited investors.

See if your company qualifies