Systems and records
Agent trajectory action granularity: clicks, API calls or business decisions
By SourceX Editorial · Updated
9 sections
Short answer
Choose trajectory granularity by the action your model will emit in production. Train computer-use agents on interface events, tool-calling agents on API calls, and planners or graders on business decisions. Record one level finer than the training target when you can, link every step to the decision it served, and accept the dataset only if a second team can replay it at the level you train on.
Agent run
Agent trajectory action
Key takeaways
- There are three useful levels: interface events, tool or API calls, and business decisions; each teaches a different skill.
- The training level is the action your model emits at inference; recording one level finer keeps the option to re-derive it.
- The decision level carries what the lower levels cannot: the choice, the alternatives and the evidence behind them.
- Alignment identifiers linking every click to its call and every call to its decision are what make a mixed-level dataset usable.
- Replay requirements differ by level, so the acceptance test must match the level you will train on.
How do you choose the right action granularity for agent trajectories?
Choose the granularity at which your model will act in production, record the level below it when the source allows, and attach the business decision each step served. A computer-use agent emits clicks and keystrokes, so it trains on interface events. A tool-calling agent emits function calls, so it trains on API calls. A planner, an orchestrator or a grader reasons about decisions, so it needs the decision level.
Getting this wrong is expensive in both directions. A dataset delivered only as business decisions cannot teach an agent where to click in a vendor portal. A dataset delivered only as raw clicks makes a tool-calling model learn screen geometry it will never see, and buries the few steps that mattered among hundreds that did not.
The rest of this page compares the three levels on one task, shows how to link them, and gives a replay test for each, so a buyer can write the granularity into a data request rather than discovering the mismatch after delivery.
What does each level record in systems people already use?
Each level already exists in public agent environments and business software, which is the easiest way to see what it contains and what it leaves out.
Interface events are the finest level. OSWorld defines an action space of typed actions such as MOVE_TO, CLICK, TYPING, PRESS, HOTKEY, WAIT, FAIL and DONE, with x and y coordinates bounded by the screen size. Anthropic's computer use tool exposes comparable actions, such as left_click with a coordinate, mouse_move, type and key. Web environments often raise the level slightly: WebArena's actions refer to element ids from the page's accessibility tree, as in click [id], and Mind2Web records each step as a target element plus an operation, either click, type or select option.
Tool or API calls sit in the middle. In tau-bench's retail environment, cancel_pending_order takes an order id and a reason and changes the order record, and its tool description tells the agent to get explicit user confirmation first. The Model Context Protocol standardizes this level: a client sends tools/call with arguments, and business logic failures come back in the result with isError set to true, so a refusal by the system is itself an observable step.
Business decisions are the coarsest level. Process mining logs show the shape: in the XES standard, an event's concept:name names the executed activity and lifecycle:transition records its phase, such as start or complete. A decision record adds what an event log usually lacks, which is the options considered, the evidence used and the reason for the choice.
How does one task look at all three granularities?
The comparison below records one illustrative accounts payable task at all three levels: a clerk applies a vendor credit memo to an open invoice in a web-based ERP, after checking that the credit matches a returned shipment. Step counts are illustrative, chosen to show proportions, and would differ by system.
Read the table row by row and the trade-off becomes concrete. The interface record is roughly forty steps long, yet only a handful of those steps carry the judgment that made the task correct: opening the return receipt and confirming its lines match the credit. A model trained to imitate all forty steps spends most of its learning on scrolling and field focus.
The call record compresses the same work into six steps, each with a typed response the next step depends on. It is the natural level for an agent that will reach the ERP through an API or an MCP server, and it is where errors such as a credit already applied or an invoice on hold show up as structured results.
The decision record has only two steps, but it is the only level that states why the credit was applied in full instead of partly, and which evidence justified it. That is the information a grader needs to tell a correct application from a lucky one.
| Dimension | Interface events | Tool or API calls | Business decisions |
|---|---|---|---|
| Steps in the episode (illustrative) | About 40 clicks, keystrokes and scrolls | 6 calls: search vendor, get invoice, get credit, get return, apply credit, add note | 2 decisions: credit is valid for this invoice; apply in full |
| Action vocabulary | A handful of verbs with coordinates, element ids or text | Named functions with typed arguments | Business verbs: approve, apply, hold, reject, escalate |
| Observation the step needs | Screenshot or accessibility tree at that moment | Previous tool responses and the request | The evidence set: invoice, credit memo, return receipt, policy |
| What it teaches | Operating this screen in this application | Choosing and sequencing calls and handling errors | Judgment: when the credit applies and when to stop |
| What a grader can check | Final screen state or the record behind it | Calls made, arguments, end state of records | Decision against an expert's decision and the outcome |
| What replay needs | Same application version, layout and screen size | Same API version and seeded record state | The same evidence available to the decision-maker |
| Typical failure if it is the only level | Noise drowns the decisions; tied to one interface | Misses the screens a human used to find the information | Cannot teach execution in any system |
| Privacy exposure | Highest: screens show unrelated records | Moderate: payloads hold the fields called | Lowest per step, but rationale text can name people |
| Typical source in a business | Screen recordings with event logs | Integration logs, audit trails, API gateways | Approval records, notes, ticket comments, sign-offs |
How do you align the levels step by step?
A mixed-level dataset is only usable if every finer step points to the coarser step it served. The alignment table below shows the same task with each business decision linked to the calls that carried it out and the interface events that produced those calls. Identifiers are illustrative.
Notice that one decision can own several calls, and one call can own a dozen interface events. The links run in one direction, from the finer step to its parent, so a step never needs to know its children. When a supplier cannot produce a link with confidence, the step should carry an explicit unlinked flag rather than a guessed parent, because a wrong link teaches a planner that a decision caused actions it did not cause. The evidence column matters as much as the links: it records what the clerk could see when deciding, which is what a decision-level replay will hand to a second expert.
Checklist · 7 items
- Case id and step id at every level, with the parent id of the coarser step.
- Actor and timestamp for each step, from the source system rather than the recorder where possible.
- The observation at each step: screen or tree for events, prior responses for calls, the evidence set for decisions.
- The action as emitted, and the tool or system response, including errors and refusals.
- Links to artifacts the step created or changed, such as the note or the updated invoice.
- The verified outcome of the case, taken from the system of record, not from the agent's or clerk's last message.
- Replay instructions for each level delivered: application version, API version, seed state and screen size.
0 of 7 ticked. Tick each item as you go.
| Decision id and decision | Call ids and calls | Interface event range | Evidence linked |
|---|---|---|---|
| D1 identify the documents | C1 search vendor, C2 get invoice, C3 get credit | E1 to E14: search box, typing, result clicks | Vendor record, open invoice, credit memo |
| D1 check the credit is valid | C4 get return receipt | E15 to E22: open returns tab, filter, open receipt | Return receipt matching the credit memo lines |
| D2 apply the credit in full | C5 apply credit to invoice | E23 to E33: apply dialog, amount field, confirm | Policy that matched returns allow full application |
| D2 document the reason | C6 add note to invoice | E34 to E40: notes panel, typing, save | Note text referencing the return receipt |
Which level should you train on for each model use?
Match the training level to the model's output, and use the other levels as context or labels. The matrix below covers the common cases; it assumes the dataset includes alignment identifiers.
Two questions settle most borderline cases. First, what will the model output at inference: coordinates, function calls or a choice among business options? Second, what will the grader check: the screen, the records or the judgment? When the answers point to different levels, train on the first and keep the second as labels for evaluation.
Hierarchical agents, where a planner hands subgoals to an executor, need two training levels at once. In that design the decision level trains the planner, the call or interface level trains the executor, and the alignment table is the training signal that connects them. Without the parent identifiers, the two models are trained on data that never shows how a decision became actions.
| Model use | Train on | Use as supporting data | Why |
|---|---|---|---|
| Computer-use agent for a specific application | Interface events | Decisions as subgoal labels | The model must emit clicks and keys on that interface |
| Tool-calling agent over business APIs or MCP servers | Tool calls with responses | Decisions as plan labels; events rarely | The model emits calls; screen detail is noise |
| Planner or orchestrator that delegates to sub-agents | Business decisions | Calls to show how each decision was executed | The model chooses what to do, not how to click |
| Grader, reward model or process reward model | Decisions and outcomes | Calls as evidence of what happened | It must judge whether the work met the requirement |
| Policy-following evaluation | Decisions with the cited policy | Calls to confirm the decision was carried out | Scoring depends on the rule applied, not the path |
| Imitation of expert operators across systems | Calls and decisions | Events for a sample to catch missing calls | Experts often use screens that have no API |
Can a delivered trajectory be replayed at the level you train on?
Only if the replay requirements of the training level are met, and they differ sharply by level. Run the replay test for the level you train on before accepting a delivery; a dataset that replays at the decision level can still fail completely at the interface level.
At a glance
Can a delivered trajectory be replayed at the level you train on
For interface events, replay needs the same application version, layout and screen geometry, since coordinate actions such as OSWorld's are bounded by the screen size and an element id only means something in the page state where it was read. For tool calls, replay needs the API version and the seeded record state, so that each call returns what it returned the first time, including errors. For decisions, replay means giving a second expert the same evidence and checking whether they reach the same choice; disagreement there is information about the task, not a defect in the data.
- Interface replay test: run a sample of episodes in the stated application version and screen size; every action should land on the element the original step targeted.
- Call replay test: reset the seeded state, issue the recorded calls in order, and compare each response and the end state with the recorded ones.
- Decision replay test: give a second expert the linked evidence without the original decision, and record agreement and the reasons for any difference.
- Cross-level test: for each decision, confirm its calls exist and its interface events produced those calls, with no orphan steps at any level.
When is the obvious level the wrong one?
Several real situations break the simple rule. The first is a decision with no action: a clerk who reviews a credit and decides not to apply it leaves no call and few events. A dataset recorded only at the call level silently drops these cases, and an agent trained on it learns that every review ends in an action. Record decisions explicitly, including decisions to hold or do nothing.
The second is a call with no interface: integrations, scheduled jobs and other automations generate calls no person made. Mixing them into a human-imitation dataset at the call level teaches the model to act like a script. Label the actor on every step and filter by it.
The third is interface work that matters in itself. When the target system has no API, or the API lacks a field the screen exposes, the interface level is the only faithful record, even for a model that will mostly call tools. In that case keep the interface episodes for those steps and say so in the dataset documentation, rather than inventing calls that do not exist.
A fourth exception is batching. Many business systems let a person apply one action to many records at once, such as approving a page of invoices. At the interface level that is one click; at the call level it may be one bulk call or many single calls; at the decision level it is many separate judgments made quickly. Record the batch as one action with a list of affected records, and record a decision per record only when the source shows the person looked at each one.
What does this comparison not show?
The comparison uses one accounts payable task, and the proportions between levels change by domain. Customer support work has far more text and fewer screens; field service and logistics have more systems per decision. Treat the step counts as illustrative and measure your own source before choosing.
Finer levels also carry more personal and unrelated data, because a screen shows whatever else was open. Choosing a coarser training level does not remove that exposure if the finer level is delivered alongside it; de-identification has to cover every level that leaves the source company.
The three levels are also not the only possible cuts. Some teams add a subtask level between calls and decisions, such as find the documents or verify the return, which suits planners that delegate in small units. The same rules apply: name the level, give every step a parent, and test replay at the level you train on.
Finally, granularity is a recording decision made at the source, and it is hard to change after the fact. Interface events that were never captured cannot be recovered from an API log, and reasons that were never written down cannot be recovered from either. Decide before collection starts, and check a sample before volume is delivered.
How should a trajectory request specify granularity?
Write the training level, the supporting levels and the alignment fields into the request, and ask for a sample that passes the replay test at the training level. Without that, suppliers tend to deliver whatever their recorder captures, which is usually the interface level for screen capture vendors and the call level for log exports.
When a company licenses approval histories, audit trails or ticket notes to AI developers, SourceX runs the rights review, contracting, preparation and delivery. A team writes the granularity it needs, plus scope, volume, language, format and timing, into the request-data form; SourceX then looks for matching trajectories at U.S. businesses rather than selling stock.
Frequently asked questions
What is action granularity in an agent trajectory?
It is the level of abstraction at which each step's action is recorded: a single click or keystroke, a single tool or API call, or a single business decision. The same task can be recorded at any of these levels with very different step counts. Granularity decides what a model trained on the data learns to emit.
Should a computer-use agent ever be trained on API calls?
Not as its main target, because it acts through the interface. API calls and decisions are still useful as subgoal labels that tell the model what each run of clicks was for. They also help a grader check the end state behind the screen.
Can you derive coarser levels from finer ones later?
Partly. Calls can sometimes be reconstructed from interface events plus network logs, but decisions and their reasons usually cannot be inferred from clicks. That is why the decision level should be captured from the source records, such as approvals and notes, rather than derived.
How do you handle steps that belong to no business decision?
Keep them and label them, for example as navigation or recovery after an error. They are part of how real work is done. Orphan steps that cannot be linked even to a navigation goal are a quality signal worth reporting.
Does finer granularity mean higher data quality?
No. Finer data is larger and harder to de-identify, and it is only better when the model must act at that level. A well-linked call-level dataset is usually more useful for a tool-calling agent than a raw screen recording of the same work.
Sources
- OSWorld's action space defines typed actions such as MOVE_TO, CLICK, TYPING, PRESS, HOTKEY, WAIT, FAIL and DONE, with x and y bounded by the screen size. Source
- Anthropic's computer use tool exposes interface actions such as left_click with a coordinate, mouse_move, type and key. Source
- WebArena actions refer to element ids from the page's accessibility tree, as in click [id]. Source
- Mind2Web records each action as a target element plus an operation (click, type or select option), with a value for type and select. Source
- tau-bench's cancel_pending_order tool takes an order id and reason, changes the order record, and its description requires explicit user confirmation first. Source
- MCP clients invoke tools with tools/call, and tool execution errors, including business logic errors, are reported in results with isError true. Source
- In XES event logs, an event's concept:name names the executed activity and lifecycle:transition records its lifecycle phase. Source
Tell us what data you need
Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.