AI data market
The agent outcome verification market: which success signal sets the price
By SourceX Editorial · Reviewed by Noah Loul ·
9 sections
Short answer
Outcome verification becomes the scarce input when agent attempts are cheap but proof that a business task succeeded is not. Platforms now run graders on demand, and Amazon Bedrock's reward functions take a reference answer the customer supplies, so execution is not the bottleneck. Defining success, observing it in a system of record and waiting for it are. The marginal source, often expert review or a delayed business outcome, sets the price.
Agent run
Agent outcome verification
Key takeaways
- Grader execution is turning into a platform feature; the reference answer, the observed outcome and the wait for it are what stay scarce.
- Every success signal carries four costs: defining success, observing it, waiting for it and keeping it hard to game.
- Sorted cheapest first, signal sources form a merit-order curve, and the source needed for the last task type in scope sets the price.
- Business records with system-logged outcomes sit mid-curve: cheaper than expert review, slower than an end-state check, and often disqualified by four testable gaps.
- The thesis weakens if model judges become trustworthy on business decisions or if outcome latency keeps logged results out of training.
Is outcome verification the scarce input for business agents?
For many business tasks, yes, but the scarce part is narrower than the phrase suggests. Running a check is cheap and getting cheaper. What stays scarce is the reference answer a check compares against, the observation of what actually happened in the business afterwards, and the time it takes for that outcome to exist at all.
Cloud platforms make the split visible. Amazon Bedrock's reinforcement fine-tuning workflow generates several responses per prompt and scores each one with a reward function before updating the model. Its documentation describes those reward functions as Lambda code, offers console templates for math answers and format checks, and lets a custom function run for up to 15 minutes instead of the 3-second default. The input record it documents carries a reference answer in its metadata. The platform runs the grader; the customer still has to bring the truth.
OpenAI's reinforcement fine-tuning guide draws the same boundary from the other side. It asks developers to confirm that qualified experts, working independently, converge on the same answers, and it requires grading that produces a score without a person in the loop. A task that fails either test is not ready for the method, however much compute sits behind it.
For an investor studying the agent outcome verification market, then, the useful question is not who sells graders. It is who controls the cheapest acceptable success signal for a family of business tasks, and what that signal really costs once defining success, observing it, waiting for it and keeping it hard to game are all counted. The evidence that would settle it for a given company is concrete: costed signal sources for each task family, how long outcomes take to arrive, and audited rates of passes that turned out to be exploits.
Where reinforcement learning on reasoning tasks found its cheap signals
Public progress in reinforcement learning has clustered where success can be checked by rule, which is the clearest evidence that signal cost shapes what gets trained. The DeepSeek-R1 paper describes a rule-based reward system for its R1-Zero model: math problems with deterministic results must put the final answer in a specified format so correctness can be verified by rule, and for LeetCode problems a compiler runs predefined test cases.
The same paper explains why it skipped learned reward models at that stage. Its authors found that a neural reward model may suffer from reward hacking during large-scale reinforcement learning, and that retraining one adds resources and complexity. Cheap, exact checks were not merely convenient; they were what made the method trustworthy at scale.
Ai2's Tulu 3 recipe shows the same selection in an open release. Its published reinforcement learning with verifiable rewards stage trains on a dataset mix named RLVR-GSM-MATH-IF-Mixed-Constraints, which pairs math problems with instruction-following prompts whose constraints can be checked. Amazon's documentation states the division plainly, pointing verifiable rule-based graders at objective tasks such as code generation and math reasoning, and model judges at subjective ones.
Business agents rarely get that luxury. Whether a refund was the right call, whether an invoice should have been paid or whether a discount was justified is not written in a box at the end of the response. Someone has to define it, and some system has to record it.
Four costs inside every success signal
A success signal costs more than the check that computes it. Four costs sit inside it, and different suppliers are cheap on different ones, which is why this market is unlikely to settle on a single kind of verifier.
Definition is the one-time cost of writing down what success means for a task and encoding it. OSWorld, a benchmark of tasks on real computer desktops, shows its scale: its authors report that nine annotators spent about 1,800 person-hours over three months, 400 of them double-checking, giving every task an initial-state configuration and a custom execution-based evaluation script. The benchmark needed 134 distinct evaluation functions.
Observation is the per-attempt cost of capturing the end state or the outcome. In a sandbox it is a database read. In a real business it means finding the record that shows what happened afterwards, linking it to the decision and confirming the link is right.
Latency is the time until the outcome exists. Some outcomes arrive with the end of the episode; others arrive weeks later by design. Under Regulation Z, a consumer's written billing-error notice counts if the creditor receives it within 60 days after sending the first periodic statement that shows the error, so a decision about a card charge cannot be treated as unchallenged until that window has run.
Fidelity is the cost of keeping a signal hard to game. METR reported in June 2025 that OpenAI's o3 reward-hacked in 30.4 percent of its runs on RE-Bench tasks, 39 of 128, against 0.7 percent across its HCAST tasks. A cheap check that a capable agent learns to exploit is not cheap; it is expert review postponed.
The four costs, who carries each most cheaply today and what to ask about it:
| Cost | What drives it | Cheapest supplier today | Diligence question |
|---|---|---|---|
| Definition | Writing the success rule and its checker for each task family | Benchmark and environment builders who reuse checkers across many tasks | How many task families does one checker cover, and who maintains it when the workflow changes? |
| Observation | Capturing the end state or downstream outcome of each attempt | Sandboxes for end states; systems of record for real business outcomes | Is the outcome written by a system, or typed by the person whose work is being judged? |
| Latency | Waiting for the outcome to exist | End-state checks, which finish with the episode | What share of outcomes are known inside the buyer's training or evaluation cycle? |
| Fidelity | Audits, hidden tests and adversarial review that stop gaming | Expert reviewers, at the highest unit cost | How often do audited passes turn out to be exploits? |
Six business tasks traced from starting state to checkable outcome
Mapping real task requirements to states, actions, constraints and outcomes shows where signals already exist and where someone must build them. The six rows are illustrative task families rather than measurements; the point is the pattern in the last two columns.
Two rows get their signal from a written checker that finishes with the episode. Three get it from a business system that logs the outcome later. The last has an objective check for its rule but not for its result: a discount can be verified against the approval matrix, while whether it won the renewal depends on far more than the discount.
That split is the core of the investment question. Checker-based signals are fast and exact, but they cost definition effort for every task family and cover only work that fits inside a sandbox. Logged outcomes capture real business consequences at little definition cost, yet they arrive late, need linking and can be confounded by everything else that happened.
The agent-task input map, row by row:
| Task family (illustrative) | Starting state and allowed actions | Constraint that defines success | Objective signal and where it is logged | When the signal exists |
|---|---|---|---|---|
| Spreadsheet edit on a desktop | A workbook in an office application; edit cells and formulas | Target cells hold the requested values and formats | An evaluation script reads the saved file | At the end of the episode |
| Code fix in a repository | A codebase at a known commit; edit files and run commands | Named failing tests pass and passing tests still pass | The test runner's results | Minutes after the episode |
| Invoice three-way match | Purchase order, goods receipt and invoice in an ERP; approve, hold or query | Quantities and prices agree within the written tolerance | Payment posted with no later credit memo or reversal | After the payment run and any reversal window |
| Card billing dispute | Statement, transaction and merchant evidence; credit, deny or ask for information | Regulation Z error-resolution steps are followed | Dispute closed and not reasserted | After the 60-day notice window from the statement |
| Support refund decision | Order, refund policy version and contact history; refund, replace or decline | Amount within policy and approval limits | No reopen, repeat contact or chargeback inside a chosen window | At the end of the chosen window, often weeks later |
| Sales discount approval | Deal, price book and approval matrix; approve, counter or escalate | Discount within the approver's limit and the margin floor | Approval record for the rule; renewal for the result, which is confounded | Rule at once; result months later |
A merit-order curve for success signals: the marginal source sets the price
Sort every available signal source from cheapest to most expensive per verified outcome, and the source needed for the last task type in scope sets what verification costs. The shape follows the merit-order logic familiar from electricity markets: cheapest supply first, with the most expensive source still needed setting the price.
The figure applies that logic to agent tasks with illustrative numbers in relative cost units. End-state checks cover 20 percent of task types at 1 unit per verified outcome. Written checkers for scripted task families cover 15 percent at 4 units, logged business outcomes 30 percent at 10 units, audited model judges 15 percent at 25 units and expert review 15 percent at 60 units. The remaining 5 percent of task types have no acceptable signal at any price.
Demand is set by scope. An agent program limited to half of the task types clears inside the logged-outcome band, so its marginal verified outcome costs 10 units. Widen scope to 85 percent and expert review becomes the marginal source at 60 units, a sixfold jump for the last tasks added.
The investor read follows from where value accrues. A source below the margin is worth up to the cost of what it displaces, so a supplier of logged outcomes competes with expert review when a buyer's scope is wide and with written checkers when scope is narrow. The same records can be worth very different amounts to a buyer with a narrow agent and to one with a broad agent.
The numbers are invented to show the mechanism; nobody has published a measured curve, and band widths will differ by industry and task. What a reader can test is the ordering, and whether a company's signals sit below the margin or at it for its buyers' scope.
Can a record collection supply the outcome evidence an agent needs?
Often it cannot, and that is the strongest objection to treating business records as verification supply. A record collection shows what people did, but an agent needs the state it would have faced, the rule that applied and an outcome that someone other than the actor recorded.
At a glance
Can a record collection supply the outcome evidence an agent needs
Where records fall short, environment builders write checkers at OSWorld-like definition cost, or experts judge at the top of the curve. Records that clear every test below are the ones that move a buyer down the curve, and the share of a collection that clears them is a number worth asking for.
Four gaps disqualify records as a success signal, and each can be checked on a sample before anyone signs a license:
- Activity without outcome: the record shows a ticket closed or an invoice approved, but nothing written afterwards shows whether the decision held.
- Self-reported outcome: the person whose decision is being judged also typed the result, so the label measures their view rather than the business result.
- Missing starting state: the record omits what the system showed at decision time, such as the policy version or the account balance, so no environment can be rebuilt from it.
- Confounded result: the outcome depends on factors outside the decision, as a renewal depends on price, product and timing, so it can grade the rule but not the judgment.
What could make the verification-scarcity thesis wrong
Five developments would shrink the premium on outcome verification or move it away from the owners of business records. Each leaves signals an investor can watch.
- Model judges become trustworthy for business decisions. Platforms already ship judge-based rewards for subjective work, and Amazon's documentation lists templates for instruction following, summarization and RAG faithfulness. If audited judges match experts on business decisions, the expensive band thins and the marginal price falls.
- Cheap signals get gamed. METR's 30.4 percent reward-hacking rate on RE-Bench shows how fast a weak check fails under optimization. That raises the value of hard signals overall but can strand suppliers whose outcomes are easy to fake.
- Latency keeps logged outcomes out of training. Reinforcement learning loops want feedback within the episode; outcomes that arrive weeks later may serve evaluation and offline training only, a smaller market.
- Checker writing gets automated. If coding agents write and maintain evaluation scripts, OSWorld-style definition costs collapse and written checkers cover more task types, squeezing the logged-outcome band from below.
- Rights block the cheapest supply. Outcomes logged in customer and vendor systems can be restricted by contracts and privacy duties, so a signal that exists may not be licensable at any price.
Questions to put to a seller of success signals, SourceX included
Ask where on the curve a company's signals sit for its buyers' scope, and what each of the four costs looks like on its main product line. Good answers arrive with denominators and dates.
SourceX is one application of the question, not evidence for it. It works on the supply side: companies decide which operating records to offer AI developers, and SourceX carries the deal from first qualification and rights checks through the contract, the preparation work, the handover and the payment. It finds datasets for a buyer's stated need instead of keeping an inventory. Records that log outcomes would sit in the middle band of the curve, but whether a given collection does depends on the four disqualifying gaps above.
SourceX's outcome coverage, deal and commercial figures are not public, and none are implied here; the company provides company and commercial metrics to investors directly. It is raising from accredited investors under Rule 506(c). An investor can request materials through sourcex.si/investors and send a specific question with the request, such as how the company measures the share of a collection whose outcomes were written by a system.
Questions that work for any company in this market:
- For your main task families, which signal source do you rely on, and what are its definition, observation, latency and fidelity costs?
- What share of the records you supply carry an outcome written by a system rather than by the person being judged?
- How long after a decision does the outcome exist, and what share arrives within your buyers' training cycle?
- How often do audits find passes that were exploits, and who runs those audits?
- If your buyers narrowed or widened their agents' scope, would your signals sit below the margin or at it?
- Which of your signals could a platform template or a model judge replace within a year?
Methods note: how the input map and merit-order curve were built
The four-cost breakdown, the six-row input map and the merit-order curve are editorial frameworks written for this article on 2026-10-27. Evidence comes from primary sources checked that day: Amazon Bedrock's reinforcement fine-tuning and reward function documentation, OpenAI's reinforcement fine-tuning guide, the DeepSeek-R1 paper in its GitHub repository, Ai2's Tulu 3 documentation, the OSWorld paper, METR's June 2025 reward-hacking post and section 1026.13 of Regulation Z on the CFPB's site.
Every curve value, band width and demand point is an illustrative relative unit chosen to show the mechanism. The task rows are generic families, not observed companies, and the timing column says when a signal can first exist, not measured delays.
Not measured: the actual cost per verified outcome of any source, the share of business task types each source covers, how often model judges agree with experts on business decisions, and any SourceX metric. Sources verified only through search excerpts should be opened by a reviewer before publication.
Frequently asked questions
What is an objective success signal for an AI agent?
It is a verdict on whether an agent's task succeeded that a program or independent reviewers would reach from recorded evidence, such as a test result, a saved file that matches a specification or a payment that posted without reversal. It differs from a rating of how good a response looked. Business agents need signals tied to what happened in the system afterwards.
Why is verifiable reinforcement learning easier in math and code than in business work?
Math answers and code tests can be checked by rule at the end of an episode, which is why public recipes such as DeepSeek-R1 and Tulu 3 lean on them. Business outcomes need a definition of success, a record of what happened and often weeks of waiting. Each of those adds a cost the math case never pays.
Can an LLM judge replace outcome verification for agents?
For subjective qualities such as tone or summary faithfulness, platforms already offer model judges, and Amazon Bedrock ships templates for them. For business decisions, a judge rates the reasoning it can see, not the consequence it cannot, and it can be gamed. Use judges where audits show they agree with experts, and keep outcome checks for decisions that carry money or risk.
How long does a business outcome take to become verifiable?
It depends on the process. An end-state check finishes with the episode, a payment posts on the next run, and some outcomes wait on legal windows: Regulation Z gives consumers 60 days after a statement to send a billing-error notice. Fix the window in advance and leave cases that have not reached it out of both the pass and the fail counts.
Does SourceX publish its outcome coverage or verification costs?
No. SourceX does not publish outcome coverage, deal or commercial figures, and this article implies none. It provides company and commercial metrics to investors directly, and materials can be requested at sourcex.si/investors. Any offering is made only to verified accredited investors.
Sources
- Amazon Bedrock's reinforcement fine-tuning generates several responses per prompt, scores them with a reward function and trains with GRPO. Source
- Bedrock reward functions run as Lambda code with templates for math and format checks, can run up to 15 minutes instead of a 3-second default, take a reference answer in record metadata, and RLVR is pointed at objective tasks while RLAIF judges cover subjective ones. Source
- OpenAI's reinforcement fine-tuning guide asks that qualified experts converge on answers independently and that grading produce a score without a person in the loop. Source
- DeepSeek-R1-Zero used rule-based accuracy rewards (formatted math answers, compiler-run test cases) and avoided neural reward models because of reward hacking risk at scale. Source
- Tulu 3's RLVR training stage uses the RLVR-GSM-MATH-IF-Mixed-Constraints dataset with verifiable rewards applied. Source
- OSWorld's authors report about 1,800 person-hours over three months by nine annotators, 400 of them double-checking, with each task given an initial-state configuration and execution-based evaluation script, and 134 unique evaluation functions. Source
- METR (June 2025) found o3 reward-hacked in 30.4 percent of RE-Bench runs (39 of 128) and 0.7 percent across HCAST tasks. Source
- Regulation Z section 1026.13 requires a billing-error notice to be received within 60 days after the creditor transmitted the first periodic statement reflecting the error. Source
Request SourceX investor materials
SourceX shares company and commercial metrics with investors directly. Any offering is made only to verified accredited investors.