AI data market
Agent tool access vs judgment data: the five rungs above a valid call
By SourceX Editorial · Reviewed by Noah Loul ·
10 sections
Short answer
Agent tool access teaches an agent how to make a call, not whether a business would stand behind it. The interface layer is standardizing: MCP moved into the Linux Foundation's new Agentic AI Foundation in December 2025, and strict schemas constrain arguments. Judgment is not: ST-WebAgentBench reports agents losing up to 38% of raw successes once policies are enforced. The scarce input is judgment data, meaning decisions recorded with their rule, alternatives and consequence.
Agent run
Agent tool access
Key takeaways
- Tool access covers one rung of five, the interface; permission, policy, tradeoff and consequence must be learned from other evidence.
- The interface rung is standardizing under neutral governance and schema enforcement, which makes connectors a weak place to look for durable data value.
- ST-WebAgentBench pairs 375 enterprise tasks with 3,057 policy instances, and success drops by up to 38% once violations count against the agent.
- None of the benchmarks reviewed here grades business tradeoffs or delayed consequences directly, and those are the rungs where real decision records are hardest to replace.
- Records teach judgment only when the decision point, the options open at the time and the later consequence can be rebuilt.
Does tool access teach an agent business judgment?
No. Tool access teaches an agent the interface: which functions exist, what arguments they take and how to call them without an error. Business judgment is a different skill. It means deciding whether this actor may take the action, what the company's rules require in this case, which permitted option serves the goal at the lowest cost and risk, and what will happen afterward.
The protocols that grant access draw the same line. The Model Context Protocol specification, version 2025-11-25, defines a tool by a name, a description, a JSON Schema for its inputs and optional annotations about its behavior. It tells clients to treat those annotations as untrusted unless they come from a trusted server, and says there should always be a human able to deny a tool invocation. The schema describes what a valid call looks like. Whether the call should happen is left to someone else.
For an investor, that boundary is the thesis. If the interface is becoming cheap and shared while judgment stays expensive and private, value in agent infrastructure moves toward whoever holds evidence of judgment: records of real decisions, the rules they applied and what followed. This article tests that claim against public benchmarks, maps the five rungs between a valid call and a sound decision, and names the developments that would prove it wrong.
Why connecting an agent is turning into shared plumbing
The interface rung is being standardized, measured and enforced at the same time, which is what early commoditization looks like. On December 9, 2025, the Linux Foundation announced the Agentic AI Foundation with three founding contributions: MCP from Anthropic, goose from Block and AGENTS.md from OpenAI. A later Linux Foundation release reports that the foundation passed 300 members in less than a year.
Enforcement has moved into the model APIs. OpenAI's Structured Outputs, introduced alongside its gpt-4o-2024-08-06 model, lets a developer mark a function definition as strict so that the model's arguments follow the supplied schema; OpenAI contrasts this with its older JSON mode, which only promised valid JSON. Measurement is mature as well. The Berkeley Function Calling Leaderboard tests simple, parallel and multi-turn calls, calls offered only irrelevant function documentation, and multi-turn exchanges where a needed function or parameter is missing.
None of this makes integration free. Authentication, rate limits and untidy enterprise systems still take engineering. But engineering effort can be bought and repeated by any well-funded team; it is not a scarce input. A diligence memo that credits an agent company's moat to its connector count should ask what remains of that moat when the protocol is open and schema compliance ships with the model.
The judgment ladder: five rungs above a valid call
Score any agent task, or any agent data supplier, on five rungs. Tool access supplies the first fully, the second partly and the top three not at all. Each rung needs different evidence, and the higher the rung, the more of that evidence lives only inside operating companies.
Read the last column as a market signal. Interface skill is graded at scale; policy compliance has dedicated public benchmarks; tradeoffs and consequences get little or no direct measurement in the benchmarks reviewed here. Where measurement is thin, a buyer cannot point to a leaderboard score and has to buy or build the underlying evidence instead.
Methods note for the ladder: the five rungs are an editorial framework written for this article on 2026-10-27 from the MCP tools specification, the Berkeley Function Calling Leaderboard's test categories, ST-WebAgentBench's policy dimensions, CRMArena-Pro and ToolEmu. It has not been validated against purchases or training results, and the measurement column describes the cited benchmarks, not production deployments.
| Rung | Question the agent must answer | What tool access supplies | Records that teach it | Public measurement |
|---|---|---|---|---|
| Interface | Which function, with which valid arguments? | All of it: names, schemas and descriptions | API documentation and call logs | Berkeley Function Calling Leaderboard; strict schema modes |
| Permission | May this actor take this action now? | Partly: scopes and access errors at call time | Role grants, delegation rules, denial and escalation logs | ST-WebAgentBench scope, consent and hierarchy policies |
| Policy | What do this company's rules require in this case? | Only what a tool description happens to state | Policy versions linked to the cases they decided, with exceptions | ST-WebAgentBench Completion under Policy; CRMArena-Pro |
| Tradeoff | Which permitted option is best given cost, speed and risk? | Nothing | Decisions with the options open at the time, the decider and the reason | None among the benchmarks reviewed here |
| Consequence | What happens next, and who bears it? | Nothing | Outcomes joined back to the decision weeks or months later | ToolEmu risk scoring with emulated tools; little else |
What happens to agent success when policy counts
Agent success falls once policy compliance is part of the score, and by enough to matter commercially. ST-WebAgentBench, accepted at ICLR 2026, pairs 375 enterprise tasks in GitLab, a shopping admin panel and SuiteCRM with 3,057 policy instances across six dimensions, from staying within permitted scope to asking for consent before irreversible actions. Its core metric, Completion under Policy, counts a task only when it is finished with zero violations, and its maintainers report that agents lose up to 38% of their raw successes once policies are enforced.
The benchmark's design isolates the cause. Its CRM difficulty tiers keep each task's intent identical and vary only the policy load, which the maintainers say shows that policy complexity, not task complexity, drives failures. That is the ladder in miniature: the same calls, more rules, fewer acceptable outcomes.
Business-specific results point the same way. Salesforce AI Research's CRMArena-Pro reports leading agents at about 58% success on single-turn CRM tasks, falling to about 35% in multi-turn settings. Workflow execution was the exception, with top agents above 83% in single-turn tasks. The paper also reports near-zero inherent confidentiality awareness, which targeted prompting improved, often at a cost to task performance. Running a known workflow sits close to the interface rung; knowing what not to reveal is judgment.
Consequences are the least measured rung. ToolEmu uses a language model to emulate 36 high-stakes toolkits across 144 test cases; by its automated evaluator, even the safest agent tested failed in risky ways 23.9% of the time, and human reviewers judged 68.8% of the failures it identified to be valid real-world failures. Emulation is a proxy, but it is a proxy aimed at the top rung, where real outcome data is rarest.
Tradeoffs and consequences: the rungs benchmarks rarely grade
Tradeoffs and consequences sit outside most public evaluation because they have no single correct answer a benchmark author can write down. A policy rule can be checked: consent was requested or it was not. A tradeoff is a choice among permitted options whose quality depends on costs, customer value and risk appetite that differ by company and by month.
Take an illustrative case. A distributor's service agent can expedite a backordered part by paid freight, offer an in-stock substitute, or issue a partial credit and wait. Policy allows all three and every tool call is valid. The better choice depends on the account's history, the margin on the order, how often substitutes come back as returns and whether late deliveries have preceded cancellations before. None of that is in a tool schema and little of it is in a policy manual; it sits in order histories, returns records, account notes and the results of earlier choices.
Consequences arrive later still. Whether the substitute was kept, the credit prevented a cancellation or the freight cost was recovered shows up weeks afterward, usually in another system. An agent that never sees those links can learn the rules but not their cost. This is the most defensible part of the thesis and the hardest to supply, because it needs decisions joined to later outcomes and prepared so that the outcome never leaks into the inputs of the decision it judges.
Could a pile of records still miss what an agent needs?
Yes, and this is the strongest objection to the thesis. A record collection can be large, licensed and relevant yet teach no judgment, because it lacks the environment an agent acts in or the evidence of what a decision changed. A ticket archive shows what staff wrote; it rarely shows which options were open, which rule applied on that date or what happened to the account afterward.
At a glance
Could a pile of records still miss what an agent needs
The benchmarks above hint at the gap from the other side. ST-WebAgentBench and CRMArena-Pro run agents inside working applications so that actions change state and results can be checked against what happened. A static archive offers no such sandbox. Records can seed one, or become evaluation cases, only if each decision can be rebuilt.
Three checks separate judgment data from reference text, and each failure moves a dataset down the ladder:
- Decision point: can the moment of choice be rebuilt from what was known then, including the policy version in force on that date?
- Alternatives: does the record show the options that were available, or only the one taken? Without rejected options a model learns habits, not tradeoffs.
- Consequence link: can a later outcome be joined to the decision by a stable key, with a window long enough for the outcome to arrive?
Where margin can sit in the agent value chain
An agent product combines several inputs: a protocol to reach tools, a model to reason, rules to follow, decision evidence to learn from and an environment to be tested in. Different kinds of business supply each one, and they are not equally easy to replace.
If the ladder is right, margin collects where evidence is scarce and hard to rebuild. The layers below run from most shared to least, and the claim an investor can test is that durable pricing power rises down the list:
- Protocols and connectors: open specifications under neutral governance, where competition turns on reliability and coverage that rivals can copy.
- Schema compliance: increasingly a built-in feature of model APIs rather than a product of its own.
- General reasoning: supplied by model developers and improving with each release; it lifts every rung but carries no company's rules or history.
- Permission and policy evidence: role grants, policy versions and exception logs held by each operating company and reachable only through it.
- Tradeoff and consequence evidence: decisions joined to outcomes across systems, the rarest input and the one no reviewed benchmark stands in for.
- Evaluation environments: sandboxes where decisions change state, strongest when seeded with real cases and weaker when every case is invented.
What could make the judgment-data thesis wrong
Four developments would weaken the case for paying more for judgment data than for interface data. Each can be tracked in public, which makes the thesis testable rather than a matter of conviction.
- Written rules could prove enough. If stronger models close the gap between raw completion and Completion under Policy when handed policies in the prompt, policy evidence becomes text a buyer can write rather than data it must license. ST-WebAgentBench's tiered results across new model releases are the marker to watch.
- People could keep the judgment. The MCP specification recommends a human who can deny tool invocations; if deployments keep consequential approvals human, demand shifts toward assistive agents that need less autonomous judgment data.
- Emulation could substitute. ToolEmu's human reviewers judged over 80% of 200 emulated tool trajectories realistic; if emulated consequences train judgment as well as recorded ones, real outcome records lose their premium.
- Rights and privacy could block supply. Decision records name employees and customers and often carry confidential terms. CRMArena-Pro's finding of near-zero inherent confidentiality awareness also means a buyer has to trust the data pipeline, not the model, to keep secrets.
Diligence questions for any agent data supplier, SourceX included
Ask each supplier to place its records on the ladder and prove the placement with a sample, then put the same questions to SourceX.
SourceX is an application of this thesis, not evidence for it. It works between businesses that hold operating records and AI developers that want them, handling rights review, contracting, preparation and payment, and the business approves scope and terms and signs before any record reaches SourceX. Datasets are sourced when a buyer asks rather than kept in stock, so whether a given rung can be supplied is a question about specific records, not inventory.
SourceX has not published metrics on decision records, outcome links or buyer acceptance; it shares company and commercial figures with investors directly. It is raising from accredited investors under Rule 506(c), and an investor can request materials at sourcex.si/investors and bring any of these questions:
- Which rung do your records reach, and what share of a sample actually reaches it?
- For each decision, can you show the options open at the time, or only the action taken?
- Which policy version applied to each decision, and how is that established?
- What share of decisions link to a later outcome, over what window, and what is excluded?
- How do you keep outcomes out of a decision's inputs during preparation?
- Which license terms cover training, evaluation and seeding a test environment?
Methods note: what the judgment ladder rests on and leaves out
This article was written on 2026-10-27. Benchmark facts come from the ST-WebAgentBench repository and the Berkeley Function Calling Leaderboard documentation, read from GitHub on that date, from the CRMArena-Pro and ToolEmu papers on arXiv, from MCP specification version 2025-11-25, from OpenAI's Structured Outputs announcement and from two Linux Foundation releases. Repository and specification text was read directly; the papers, the OpenAI page and the Linux Foundation releases were confirmed through search excerpts on each publisher's own site, so a reviewer should open them before release.
Not measured: how often real business records reach each rung, whether judgment data improves agents more than policies placed in a prompt, and what buyers pay for any rung. Benchmark percentages describe the agents, tasks and dates their authors report, not rates in production. The judgment ladder and the value-chain ordering are editorial frameworks.
Frequently asked questions
Is tool use the same as judgment for an AI agent?
No. Tool use is the ability to pick a function and call it with valid arguments. Judgment is deciding whether the action is permitted, what the business's rules require, which allowed option is best and what will follow. Public benchmarks grade the first far more thoroughly than the rest.
Does MCP make agent training data less valuable?
MCP makes reaching tools cheaper and more uniform, which lowers the value of data that only teaches calls. It does not say which call a business should make, and its specification recommends a human who can deny invocations. Data that records decisions, rules and outcomes is untouched by the protocol and may gain relative value.
What does Completion under Policy measure?
It is ST-WebAgentBench's core metric: the share of tasks an agent completes with zero policy violations. Comparing it with the plain completion rate shows how much apparent success depends on breaking rules. The benchmark's maintainers report losses of up to 38% of raw successes once policies are enforced.
Which business records teach an agent judgment?
Records that keep a decision together with its context: the options available, the policy version in force, who decided and the outcome that followed. Approval histories with reasons, exception logs, and returns or credit decisions linked to later account activity are common sources. Records that show only the action taken teach habits rather than judgment.
Has SourceX published metrics on judgment data?
No. SourceX has not published figures on decision records, outcome links or acceptance rates, and shares company and commercial metrics with investors directly. Investor materials can be requested at sourcex.si/investors; any offering is made only to verified accredited investors.
Sources
- The MCP specification (2025-11-25) defines a tool by name, description, input JSON Schema and optional annotations, tells clients to treat annotations as untrusted unless from trusted servers, and says a human should be able to deny tool invocations. Source
- The Linux Foundation announced the Agentic AI Foundation on December 9, 2025, anchored by MCP from Anthropic, goose from Block and AGENTS.md from OpenAI. Source
- A Linux Foundation release reports that the Agentic AI Foundation passed 300 members in less than one year. Source
- OpenAI's Structured Outputs lets a strict function definition make model arguments follow the supplied schema, unlike JSON mode, which only ensures valid JSON. Source
- The Berkeley Function Calling Leaderboard tests simple, parallel and multi-turn calls, irrelevance detection and multi-turn cases with a missing function or parameter. Source
- ST-WebAgentBench pairs 375 enterprise tasks with 3,057 policy instances in six dimensions; agents lose up to 38% of raw successes under its Completion under Policy metric. Source
- CRMArena-Pro reports about 58% single-turn and 35% multi-turn success for leading agents, workflow execution above 83% single-turn, and near-zero inherent confidentiality awareness. Source
- ToolEmu emulates 36 high-stakes toolkits over 144 test cases; the safest agent failed in risky ways 23.9% of the time, and 68.8% of identified failures were judged valid. Source
Request SourceX investor materials
SourceX shares company and commercial metrics with investors directly. Any offering is made only to verified accredited investors.