Skip to content

Systems and records

Agent trajectory action granularity: clicks, API calls or business decisions

By SourceX Editorial · Updated

9 sections

Short answer

Choose trajectory granularity by the action your model will emit in production. Train computer-use agents on interface events, tool-calling agents on API calls, and planners or graders on business decisions. Record one level finer than the training target when you can, link every step to the decision it served, and accept the dataset only if a second team can replay it at the level you train on.

Agent run

Agent trajectory action

This page covers agent trajectory action granularity: clicks, API calls or business decisions.Illustrative

Key takeaways

  • There are three useful levels: interface events, tool or API calls, and business decisions; each teaches a different skill.
  • The training level is the action your model emits at inference; recording one level finer keeps the option to re-derive it.
  • The decision level carries what the lower levels cannot: the choice, the alternatives and the evidence behind them.
  • Alignment identifiers linking every click to its call and every call to its decision are what make a mixed-level dataset usable.
  • Replay requirements differ by level, so the acceptance test must match the level you will train on.

How do you choose the right action granularity for agent trajectories?

Choose the granularity at which your model will act in production, record the level below it when the source allows, and attach the business decision each step served. A computer-use agent emits clicks and keystrokes, so it trains on interface events. A tool-calling agent emits function calls, so it trains on API calls. A planner, an orchestrator or a grader reasons about decisions, so it needs the decision level.

Getting this wrong is expensive in both directions. A dataset delivered only as business decisions cannot teach an agent where to click in a vendor portal. A dataset delivered only as raw clicks makes a tool-calling model learn screen geometry it will never see, and buries the few steps that mattered among hundreds that did not.

The rest of this page compares the three levels on one task, shows how to link them, and gives a replay test for each, so a buyer can write the granularity into a data request rather than discovering the mismatch after delivery.

What does each level record in systems people already use?

Each level already exists in public agent environments and business software, which is the easiest way to see what it contains and what it leaves out.

Interface events are the finest level. OSWorld defines an action space of typed actions such as MOVE_TO, CLICK, TYPING, PRESS, HOTKEY, WAIT, FAIL and DONE, with x and y coordinates bounded by the screen size. Anthropic's computer use tool exposes comparable actions, such as left_click with a coordinate, mouse_move, type and key. Web environments often raise the level slightly: WebArena's actions refer to element ids from the page's accessibility tree, as in click [id], and Mind2Web records each step as a target element plus an operation, either click, type or select option.

Tool or API calls sit in the middle. In tau-bench's retail environment, cancel_pending_order takes an order id and a reason and changes the order record, and its tool description tells the agent to get explicit user confirmation first. The Model Context Protocol standardizes this level: a client sends tools/call with arguments, and business logic failures come back in the result with isError set to true, so a refusal by the system is itself an observable step.

Business decisions are the coarsest level. Process mining logs show the shape: in the XES standard, an event's concept:name names the executed activity and lifecycle:transition records its phase, such as start or complete. A decision record adds what an event log usually lacks, which is the options considered, the evidence used and the reason for the choice.

How does one task look at all three granularities?

The comparison below records one illustrative accounts payable task at all three levels: a clerk applies a vendor credit memo to an open invoice in a web-based ERP, after checking that the credit matches a returned shipment. Step counts are illustrative, chosen to show proportions, and would differ by system.

Read the table row by row and the trade-off becomes concrete. The interface record is roughly forty steps long, yet only a handful of those steps carry the judgment that made the task correct: opening the return receipt and confirming its lines match the credit. A model trained to imitate all forty steps spends most of its learning on scrolling and field focus.

The call record compresses the same work into six steps, each with a typed response the next step depends on. It is the natural level for an agent that will reach the ERP through an API or an MCP server, and it is where errors such as a credit already applied or an invoice on hold show up as structured results.

The decision record has only two steps, but it is the only level that states why the credit was applied in full instead of partly, and which evidence justified it. That is the information a grader needs to tell a correct application from a lucky one.

How does one task look at all three granularities?
DimensionInterface eventsTool or API callsBusiness decisions
Steps in the episode (illustrative)About 40 clicks, keystrokes and scrolls6 calls: search vendor, get invoice, get credit, get return, apply credit, add note2 decisions: credit is valid for this invoice; apply in full
Action vocabularyA handful of verbs with coordinates, element ids or textNamed functions with typed argumentsBusiness verbs: approve, apply, hold, reject, escalate
Observation the step needsScreenshot or accessibility tree at that momentPrevious tool responses and the requestThe evidence set: invoice, credit memo, return receipt, policy
What it teachesOperating this screen in this applicationChoosing and sequencing calls and handling errorsJudgment: when the credit applies and when to stop
What a grader can checkFinal screen state or the record behind itCalls made, arguments, end state of recordsDecision against an expert's decision and the outcome
What replay needsSame application version, layout and screen sizeSame API version and seeded record stateThe same evidence available to the decision-maker
Typical failure if it is the only levelNoise drowns the decisions; tied to one interfaceMisses the screens a human used to find the informationCannot teach execution in any system
Privacy exposureHighest: screens show unrelated recordsModerate: payloads hold the fields calledLowest per step, but rationale text can name people
Typical source in a businessScreen recordings with event logsIntegration logs, audit trails, API gatewaysApproval records, notes, ticket comments, sign-offs

How do you align the levels step by step?

A mixed-level dataset is only usable if every finer step points to the coarser step it served. The alignment table below shows the same task with each business decision linked to the calls that carried it out and the interface events that produced those calls. Identifiers are illustrative.

Notice that one decision can own several calls, and one call can own a dozen interface events. The links run in one direction, from the finer step to its parent, so a step never needs to know its children. When a supplier cannot produce a link with confidence, the step should carry an explicit unlinked flag rather than a guessed parent, because a wrong link teaches a planner that a decision caused actions it did not cause. The evidence column matters as much as the links: it records what the clerk could see when deciding, which is what a decision-level replay will hand to a second expert.

Checklist · 7 items

  1. Case id and step id at every level, with the parent id of the coarser step.
  2. Actor and timestamp for each step, from the source system rather than the recorder where possible.
  3. The observation at each step: screen or tree for events, prior responses for calls, the evidence set for decisions.
  4. The action as emitted, and the tool or system response, including errors and refusals.
  5. Links to artifacts the step created or changed, such as the note or the updated invoice.
  6. The verified outcome of the case, taken from the system of record, not from the agent's or clerk's last message.
  7. Replay instructions for each level delivered: application version, API version, seed state and screen size.

0 of 7 ticked. Tick each item as you go.

How do you align the levels step by step?
Decision id and decisionCall ids and callsInterface event rangeEvidence linked
D1 identify the documentsC1 search vendor, C2 get invoice, C3 get creditE1 to E14: search box, typing, result clicksVendor record, open invoice, credit memo
D1 check the credit is validC4 get return receiptE15 to E22: open returns tab, filter, open receiptReturn receipt matching the credit memo lines
D2 apply the credit in fullC5 apply credit to invoiceE23 to E33: apply dialog, amount field, confirmPolicy that matched returns allow full application
D2 document the reasonC6 add note to invoiceE34 to E40: notes panel, typing, saveNote text referencing the return receipt

Which level should you train on for each model use?

Match the training level to the model's output, and use the other levels as context or labels. The matrix below covers the common cases; it assumes the dataset includes alignment identifiers.

Two questions settle most borderline cases. First, what will the model output at inference: coordinates, function calls or a choice among business options? Second, what will the grader check: the screen, the records or the judgment? When the answers point to different levels, train on the first and keep the second as labels for evaluation.

Hierarchical agents, where a planner hands subgoals to an executor, need two training levels at once. In that design the decision level trains the planner, the call or interface level trains the executor, and the alignment table is the training signal that connects them. Without the parent identifiers, the two models are trained on data that never shows how a decision became actions.

Which level should you train on for each model use?
Model useTrain onUse as supporting dataWhy
Computer-use agent for a specific applicationInterface eventsDecisions as subgoal labelsThe model must emit clicks and keys on that interface
Tool-calling agent over business APIs or MCP serversTool calls with responsesDecisions as plan labels; events rarelyThe model emits calls; screen detail is noise
Planner or orchestrator that delegates to sub-agentsBusiness decisionsCalls to show how each decision was executedThe model chooses what to do, not how to click
Grader, reward model or process reward modelDecisions and outcomesCalls as evidence of what happenedIt must judge whether the work met the requirement
Policy-following evaluationDecisions with the cited policyCalls to confirm the decision was carried outScoring depends on the rule applied, not the path
Imitation of expert operators across systemsCalls and decisionsEvents for a sample to catch missing callsExperts often use screens that have no API

Can a delivered trajectory be replayed at the level you train on?

Only if the replay requirements of the training level are met, and they differ sharply by level. Run the replay test for the level you train on before accepting a delivery; a dataset that replays at the decision level can still fail completely at the interface level.

At a glance

Can a delivered trajectory be replayed at the level you train on

For interface events, replay needs the same application version, layout and screen geometry, since coordinate actions such as OSWorld's are bounded by the screen size and an element id only means something in the page state where it was read. For tool calls, replay needs the API version and the seeded record state, so that each call returns what it returned the first time, including errors. For decisions, replay means giving a second expert the same evidence and checking whether they reach the same choice; disagreement there is information about the task, not a defect in the data.

  • Interface replay test: run a sample of episodes in the stated application version and screen size; every action should land on the element the original step targeted.
  • Call replay test: reset the seeded state, issue the recorded calls in order, and compare each response and the end state with the recorded ones.
  • Decision replay test: give a second expert the linked evidence without the original decision, and record agreement and the reasons for any difference.
  • Cross-level test: for each decision, confirm its calls exist and its interface events produced those calls, with no orphan steps at any level.

When is the obvious level the wrong one?

Several real situations break the simple rule. The first is a decision with no action: a clerk who reviews a credit and decides not to apply it leaves no call and few events. A dataset recorded only at the call level silently drops these cases, and an agent trained on it learns that every review ends in an action. Record decisions explicitly, including decisions to hold or do nothing.

The second is a call with no interface: integrations, scheduled jobs and other automations generate calls no person made. Mixing them into a human-imitation dataset at the call level teaches the model to act like a script. Label the actor on every step and filter by it.

The third is interface work that matters in itself. When the target system has no API, or the API lacks a field the screen exposes, the interface level is the only faithful record, even for a model that will mostly call tools. In that case keep the interface episodes for those steps and say so in the dataset documentation, rather than inventing calls that do not exist.

A fourth exception is batching. Many business systems let a person apply one action to many records at once, such as approving a page of invoices. At the interface level that is one click; at the call level it may be one bulk call or many single calls; at the decision level it is many separate judgments made quickly. Record the batch as one action with a list of affected records, and record a decision per record only when the source shows the person looked at each one.

What does this comparison not show?

The comparison uses one accounts payable task, and the proportions between levels change by domain. Customer support work has far more text and fewer screens; field service and logistics have more systems per decision. Treat the step counts as illustrative and measure your own source before choosing.

Finer levels also carry more personal and unrelated data, because a screen shows whatever else was open. Choosing a coarser training level does not remove that exposure if the finer level is delivered alongside it; de-identification has to cover every level that leaves the source company.

The three levels are also not the only possible cuts. Some teams add a subtask level between calls and decisions, such as find the documents or verify the return, which suits planners that delegate in small units. The same rules apply: name the level, give every step a parent, and test replay at the level you train on.

Finally, granularity is a recording decision made at the source, and it is hard to change after the fact. Interface events that were never captured cannot be recovered from an API log, and reasons that were never written down cannot be recovered from either. Decide before collection starts, and check a sample before volume is delivered.

How should a trajectory request specify granularity?

Write the training level, the supporting levels and the alignment fields into the request, and ask for a sample that passes the replay test at the training level. Without that, suppliers tend to deliver whatever their recorder captures, which is usually the interface level for screen capture vendors and the call level for log exports.

When a company licenses approval histories, audit trails or ticket notes to AI developers, SourceX runs the rights review, contracting, preparation and delivery. A team writes the granularity it needs, plus scope, volume, language, format and timing, into the request-data form; SourceX then looks for matching trajectories at U.S. businesses rather than selling stock.

Frequently asked questions

What is action granularity in an agent trajectory?

It is the level of abstraction at which each step's action is recorded: a single click or keystroke, a single tool or API call, or a single business decision. The same task can be recorded at any of these levels with very different step counts. Granularity decides what a model trained on the data learns to emit.

Should a computer-use agent ever be trained on API calls?

Not as its main target, because it acts through the interface. API calls and decisions are still useful as subgoal labels that tell the model what each run of clicks was for. They also help a grader check the end state behind the screen.

Can you derive coarser levels from finer ones later?

Partly. Calls can sometimes be reconstructed from interface events plus network logs, but decisions and their reasons usually cannot be inferred from clicks. That is why the decision level should be captured from the source records, such as approvals and notes, rather than derived.

How do you handle steps that belong to no business decision?

Keep them and label them, for example as navigation or recovery after an error. They are part of how real work is done. Orphan steps that cannot be linked even to a navigation goal are a quality signal worth reporting.

Does finer granularity mean higher data quality?

No. Finer data is larger and harder to de-identify, and it is only better when the model must act at that level. A well-linked call-level dataset is usually more useful for a tool-calling agent than a raw screen recording of the same work.

Sources

  • OSWorld's action space defines typed actions such as MOVE_TO, CLICK, TYPING, PRESS, HOTKEY, WAIT, FAIL and DONE, with x and y bounded by the screen size. Source
  • Anthropic's computer use tool exposes interface actions such as left_click with a coordinate, mouse_move, type and key. Source
  • WebArena actions refer to element ids from the page's accessibility tree, as in click [id]. Source
  • Mind2Web records each action as a target element plus an operation (click, type or select option), with a value for type and select. Source
  • tau-bench's cancel_pending_order tool takes an order id and reason, changes the order record, and its description requires explicit user confirmation first. Source
  • MCP clients invoke tools with tools/call, and tool execution errors, including business logic errors, are reported in results with isError true. Source
  • In XES event logs, an event's concept:name names the executed activity and lifecycle:transition records its lifecycle phase. Source

Tell us what data you need

Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.

See if your company qualifies