AI uses for records
Computer use workflow action labels: tie each click to intent and outcome
By SourceX Editorial · Updated
10 sections
Short answer
Give each action a step intent naming the business change it should make, an expected effect stated as record, field and value, and an observed outcome checked against the system of record's change history, not the screen. Add permission, app version and retry or undo links. These computer use workflow action labels let a model learn cause and effect instead of trusting success messages.
Agent run
Computer use workflow action
Key takeaways
- A click log records what was pressed; agent training also needs why it was pressed and what changed.
- Interface checks such as visibility and enabled state say nothing about whether the business record changed.
- Verify each state-changing step against the application's own change history, matched on record, field, value, actor and time.
- Use a closed outcome vocabulary that keeps ui_only, reverted and not_verifiable steps separate from confirmed effects.
- Keep failed, blocked and undone steps in the data, labeled, so models learn recovery.
Which intent labels make business click histories useful for agent training?
Three labels per action do most of the work: a step intent that names the business change the action is meant to make, an expected effect stated as a record, field and target value, and an observed outcome that says whether the system of record actually changed. Add the actor's permission, the application version and any retry or undo link, and a click history becomes a sequence an agent can learn cause and effect from.
Raw interaction logs already capture the action itself. Mind2Web, a widely used web-agent dataset, records each step as an operation (CLICK, TYPE or SELECT), an optional value such as the text typed, and the target element. That tells a model what was pressed. It does not tell it why, or whether the press achieved anything in the business system behind the screen, and that missing half is where computer use workflow action labels earn their cost.
Why isn't a click that worked the same as an action that worked?
Interface success and business success are measured in different places. Playwright, a common browser automation library, checks before a click that the target resolves to exactly one element and is visible, stable, receiving events, enabled and not inert. Every one of those checks is about the page. None says the opportunity was saved, the invoice posted or the ticket moved to Done.
Agent benchmarks learned this early and grade end state, not clicks. WebArena task configs carry an eval block whose types include url_match, string_match and program_html, the last checking required contents on a page after the agent finishes. OSWorld tasks name an evaluator function that reads the resulting state, for example whether a browser's Do Not Track preference ended up enabled, and compares it with an expected value.
Business click histories need the same discipline at step level. A save button can flash a success toast while a validation rule rejects the record, a bulk edit can apply to a filtered subset the user did not intend, and an automation can overwrite the change seconds later. Without a step-level observed outcome, a model trained on those traces learns that the toast was the goal.
Which fields does each labeled step carry?
One record per step, joined to an episode header, with the fields below. Names are a proposal for this article (version 0.1); the example values are invented. Keep raw events and screenshots in the underlying trajectory format and store these labels as a separate layer keyed by step_id, so labels can be audited or regenerated without touching the raw log.
| Field | Required | Definition | Illustrative value |
|---|---|---|---|
| episode_id, task_intent | Yes | The episode and the business goal in one sentence, written from the request that started it | Escalate case 0042 to tier 2 and close the linked bug |
| step_id, timestamp | Yes | Order and time of the action, in UTC with offset | s07, 2026-03-04T14:12:09Z |
| step_intent | Yes | Verb and object from a controlled list, describing the change this step is meant to make | set_field(Case.Priority) |
| action | Yes | Operation, target and value as captured: role and accessible name or selector, plus typed or selected value | select, combobox Priority, High |
| actor_permission and app_version | Yes | Role or permission set of the actor, and the application release or build in use | Support Agent profile; vendor release name |
| expected_effect | When state-changing | Record type, record ID, field and expected new value | Case 0042, Priority, High |
| observed_record | When state-changing | Evidence from the system of record: history entry ID, old and new value, timestamp, user | History row: Priority Medium to High, s07 plus 2 seconds |
| observed_ui | Yes | What the interface showed after the action: screenshot reference or visible message text | Toast: Case saved |
| outcome_label | Yes | One value from the closed outcome vocabulary | effect_confirmed |
| retry_of, undo_of, label_source | Yes | Links to an earlier step this one retries or reverses, and who or what wrote the labels | retry_of s06; annotator A3 from record diff |
Which eight outcome labels belong in the vocabulary, and what does each require?
A closed vocabulary keeps outcome labels comparable across annotators and systems. Each label below states what evidence it needs; a label without that evidence is invalid.
At a glance
Which eight outcome labels belong in the vocabulary, and what does each require
- effect_confirmed: the system-of-record history shows the expected field on the expected record changed to the expected value, by this actor, within the step's time window.
- ui_only: the interface signaled success but no matching history entry exists. Requires a check of the history for the window.
- no_effect: neither the interface nor the record changed, such as a click on a disabled control.
- wrong_effect: a record changed, but not the expected record, field or value. Requires the observed change.
- partial: a multi-record action changed some of its targets. Requires counts of expected and changed records.
- blocked_permission: the system refused the change for lack of rights. Requires the error text or audit entry.
- reverted: the change was confirmed and later reversed by an undo step, an automation or another user. Requires the reversing entry.
- not_verifiable: the system keeps no history for that field or object, so only the interface can be observed. Never merged into effect_confirmed.
Which record shows the business data changed, not just the screen?
A dated entry in the application's own change history, matched to the step on record, field, value, actor and time. Screenshots and DOM text prove what the user saw; only the system of record proves what the business now holds.
Most business systems keep that history, with limits a labeler has to know. Salesforce field history tracking records changes to tracked fields in a related history object, with the date, time and user, and old and new values for most field types; for long text fields it records only that an edit happened. A field that is not tracked leaves no history at all, which is exactly the not_verifiable case. The Jira Cloud REST API returns an issue's changelogs paginated and sorted oldest first, and each changed item carries the field plus from and to values as strings, which makes status transitions straightforward to match.
Use five match rules: same record ID, same field, new value equals the expected value, the history timestamp falls inside a window that starts at the action (a few seconds for interactive saves, longer for queued jobs), and the history user equals the actor or a named automation acting for them. If two entries match, take the earliest and flag the step for review.
Illustrative trace: escalating a support case and closing its linked issue
The trace below is invented for illustration and does not come from any real company's systems. An agent-in-training watches a human escalate a support case in a CRM and close a linked bug in an issue tracker; labels were applied from each system's history.
| Step | Action and step intent | Expected effect | Observed evidence | Outcome label |
|---|---|---|---|---|
| s05 | Select Priority High; set_field(Case.Priority) | Case 0042 Priority to High | Toast shown; no history row because a validation rule blocked save | ui_only |
| s06 | Fill required Escalation Reason; set_field(Case.Escalation_Reason) | Reason filled | Field is long text; history notes an edit only | not_verifiable |
| s07 | Click Save again; retry_of s05 | Case 0042 Priority to High | History row Medium to High, same user, 2 seconds later | effect_confirmed |
| s09 | Drag issue to Done; transition(Issue.Status) | Issue BUG-311 status to Done | Changelog item status, In Review to Done | effect_confirmed |
| s10 | Bulk edit labels on filtered list; set_field(Issue.Labels) | Three issues labeled | Changelog shows two of three changed | partial |
| s12 | Click Undo; undo_of s09 | BUG-311 back to In Review | Changelog item status, Done to In Review | effect_confirmed, and s09 relabeled reverted |
Which checks should a validator run, and which record should it reject?
Run the checks per step, then per episode. A buyer can run all of them on a delivered sample using only the labels, the raw log and the exported histories.
At a glance
Which checks should a validator run, and which record should it reject
- Vocabulary: step_intent and outcome_label come from the declared lists for this schema version.
- Evidence: effect_confirmed, wrong_effect, partial, blocked_permission and reverted each carry the evidence their definition requires; effect_confirmed with only a screenshot fails.
- Matching: for effect_confirmed, the cited history entry agrees with expected_effect on record, field and value, and its timestamp is not earlier than the action.
- Links: retry_of and undo_of point to earlier steps in the same episode, and an undone step is relabeled reverted.
- Coverage: every state-changing step_intent has an expected_effect; read-only intents such as open_record or search do not.
- Provenance: label_source is filled for every step, and model-written labels name the model and prompt version.
- Failing record to keep in the test set: step s08 labeled effect_confirmed, step_intent update_record (not in the vocabulary), expected_effect missing, evidence a screenshot of a success toast, undo_of pointing to s11. It should fail vocabulary, coverage, evidence and links.
Where do step intents come from, and how far can each source be trusted?
Prefer intents fixed before or during the action, and treat intents written afterward as hypotheses checked against the record diff. The source changes both the cost and the bias of a label.
| Intent source | Strength | Risk | Use it for |
|---|---|---|---|
| Work item that started the episode (ticket, email, task) | Written before any action, by the requester | States the goal, not the steps | task_intent |
| Demonstrator think-aloud or a short prompt before each step | Contemporaneous and specific | Slows the work; sparse coverage | step_intent on recorded demonstrations |
| Annotator reading the trace afterward | Consistent vocabulary and coverage | Can use information not visible at that step | step_intent with a no-lookahead rule |
| Derived from the record diff | Objective and cheap at scale | Misses intent behind failed or blocked actions | expected_effect checks and wrong_effect detection |
| Model-generated labels | Fast first pass | Plausible but unverified; terms of the model may limit use | Drafts reviewed by a person |
How should a buyer measure label quality on a delivered sample?
Draw a random sample of state-changing steps, have a second annotator label them blind, and recheck every effect_confirmed label against the raw history export. Agreement on step_intent and the share of outcome labels whose evidence actually matches tell you more about a delivery than its size.
Sample by step, not by episode, so long episodes do not dominate, and stratify by application and by outcome label so rare labels such as wrong_effect and blocked_permission are present in the check. Report the sample size, the strata and the counts behind every rate you compute, and give each rate an interval rather than a single figure, because a few dozen steps per stratum leave real uncertainty.
Three measurements cover most of the risk. Intent agreement: how often two annotators choose the same step_intent from the vocabulary, with disagreements read to see whether the vocabulary or the guidelines are at fault. Evidence accuracy: of the steps labeled effect_confirmed, how many cite a history entry that really matches on record, field, value, actor and time. Coverage: what share of state-changing steps ended up not_verifiable, which reveals how much of the workflow the supplier's systems simply do not record.
Agree the acceptance thresholds with the supplier before labeling starts and write them into the data specification. Setting them after the sample arrives turns a quality check into a negotiation.
What should a request for labeled workflow histories specify?
When the traces come from real business operations rather than staged tasks, ask for the step-level labels and the history exports as part of the scope, along with the permissions model and application versions in use. SourceX keeps no catalog of finished traces; it sources records from U.S. businesses against each buyer's stated scope, volume, format, timing and licensing needs, so the label layer can be written into the request before any preparation begins.
Ask the supplier to state which objects and fields keep history in their systems. That one answer tells you how many steps can ever reach effect_confirmed and how many will stay not_verifiable, before anyone labels a single click.
Frequently asked questions
What is the difference between task intent and step intent?
Task intent is the business goal of the whole episode, such as escalating a case. Step intent is the single change one action is meant to make toward it, such as setting the case priority. Agents need both, because the same click can serve different goals.
Can screenshots alone prove an action succeeded?
No. A screenshot shows what the interface displayed, which can include success messages for saves that failed or were later reverted. Proof comes from the application's own change history matched to the step.
What if a field has no change history?
Label the step not_verifiable and keep it separate from confirmed outcomes. Salesforce, for example, records only that a long text field was edited, not its old and new values, and untracked fields leave no history.
Should failed and undone steps stay in the training data?
Yes, labeled as such. Retries, permission blocks and undos teach recovery, and a model that never sees ui_only or reverted steps learns to trust every success message.
How many labels does each step need?
At minimum a step intent, an observed interface result and an outcome label, plus an expected effect and record evidence for every step that is meant to change data. Read-only steps such as searching need no expected effect.
Sources
- Mind2Web stores each action as an operation of type CLICK, TYPE or SELECT with an optional value, plus the ground-truth target element candidates. Source
- Before a click, Playwright checks that the locator resolves to exactly one element that is visible, stable, receives events, is enabled and is not inert. Source
- WebArena task configs include an intent and an eval block with eval_types such as url_match, a reference_url and program_html checks, plus a reference action sequence. Source
- The WebArena evaluation harness defines string, URL and HTML content evaluators that check answers, URLs and required page contents. Source
- An OSWorld task config names an evaluator function and a result getter, such as reading whether Do Not Track is enabled, and compares it with an expected value. Source
- Salesforce field history tracking records changes to tracked fields in a related history object with date, time and user, storing old and new values for most field types and only the fact of an edit for long text fields. Source
- The Jira Cloud REST API returns a paginated list of an issue's changelogs sorted from oldest, with each changed item giving the field and its from and to values. Source
Tell us what data you need
Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.