Skip to content

AI uses for records

Computer use dataset state evidence: what proves the business state changed

By SourceX Editorial · Updated

10 sections

Short answer

A screenshot proves only what the interface rendered. Proof that a computer-use agent changed business state is a backend read of the system of record after the action, matched field by field to what the screen claimed, plus an audit entry tied to the agent's run and a check that nothing else changed. A computer-use dataset should store that pair and a verdict for every state-changing step.

Agent run

Computer use dataset state

This page covers computer use dataset state evidence: what proves the business state changed.Illustrative

Key takeaways

  • A screenshot shows what was rendered; the system of record shows what was saved.
  • Every state-changing step needs a UI claim, a backend read by record ID, matching audit entries and a verdict.
  • Optimistic updates, silent permission denials, stale reads, duplicate retries, look-alike records and pending workflows all fool screenshots.
  • Tool results and MCP annotations are claims and hints, not proof; verify with a backend read.
  • Label steps that cannot be verified as unverified instead of inferring success.

What proves a computer-use agent changed business state, if not the screenshot?

Proof is a read of the system of record taken after the action, matched field by field to what the interface claimed, plus an audit entry attributed to the agent's session and a check that no other record changed. A screenshot proves only what the screen rendered at one moment, which can be a cached value, an optimistic update or the wrong record that looks right.

For a computer-use dataset, that means every step that is supposed to change something carries two observations, not one: the UI observation the agent saw and a backend observation the agent did not need to see. The pair, and a verdict on whether they agree, is the state evidence. Without it, a trajectory that ends on a green Saved banner is labeled a success even when the record never changed.

This guide gives a UI and backend state alignment task specification: the task-level fields, the step-level record, the verdicts, a worked example, and the cases where backend proof cannot be collected and the dataset should say so instead of guessing.

Where does business state live besides the screen?

State lives in three places, and a dataset should record which one each piece of evidence comes from. The screen is a view; the system of record is the truth for the field; downstream systems show whether the change had its intended effect, such as an email sent or an invoice posted to the ledger.

Where does business state live besides the screen?
LayerWhat it showsEvidence artifactHow it misleads
Rendered UIWhat the agent saw: pixels, DOM or accessibility treeScreenshot plus DOM or accessibility snapshot with a timestampOptimistic updates, stale caches, a toast shown before the save completes
System of recordThe stored field values for the target recordAPI or database read by record ID after the action, with response timeRead too early on an asynchronous system; reading a replica that lags
Audit trailWho changed which field from what to what, and whenField history or audit log entries filtered to the run's user and windowNot enabled for that field, or retention ended before export
Downstream effectsWhether the change triggered what it shouldWorkflow run IDs, outbound message IDs, ledger postingsEffects run hours later or in a system the dataset cannot read
Untouched recordsThat nothing else changedDiff of a declared scope before and after the runScope declared too narrowly, so collateral edits go unseen

Six ways the screen and the record disagree

Each pattern below is common in business software and each produces a screenshot that looks like success. A dataset that only stores screens will label all six as successes, so the alignment record should name the pattern when it occurs.

Six ways the screen and the record disagree
PatternScreen showsSystem of record showsEvidence that exposes it
Optimistic update rejectedNew value in an inline-edited cellOld value; a validation rule rejected the saveBackend read after the save plus the error response the UI swallowed
Silent permission denialForm closes normallyNo change; the role lacked edit rights on that fieldPermission context in the task header and an unchanged audit trail
Stale readOld value after a correct saveNew value, written to the primary storeBackend read with a propagation window and a re-read of the UI
Duplicate on retryOne confirmationTwo records, because a timed-out submit was clicked againCount of records matching the task's natural key before and after
Right-looking wrong recordThe edited record's name matches the instructionA different record ID with the same display name was changedRecord ID captured from the UI or URL, matched to the backend write
Pending workflowStatus reads SubmittedChange is queued for approval and not yet in effectWorkflow or approval object state, read again after the window

Which fields does the alignment task specification need at task level?

The task header fixes everything the steps depend on: what the business intended, who the agent acted as, which software versions were live and where the backend evidence comes from. If any of these is missing, a later reader cannot tell whether a mismatch was the agent's error or the environment's.

The values in the table are illustrative and do not describe a real system or dataset.

Which fields does the alignment task specification need at task level?
FieldPurposeIllustrative valueRequired when
task_id and intentStable ID and the business goal in one sentenceT-0412; move the renewal opportunity to Closed Won and set the close dateAlways
target_recordsRecord IDs the task may change, by object typeopportunity 006-xxxx; no other objectsAlways; drives the collateral check
actor_contextRole, permission set and scopes the agent ran withsales_ops role; edit on Opportunity; no deleteAlways; explains silent denials
versionsApp release, UI locale, tool server and tool versionsCRM release tag; agent tool server 1.4.2Always; UI and tool changes invalidate tasks
initial_state_refSnapshot of target and in-scope records before the runBackend export of the opportunity and its line itemsAlways
evidence_sourcesWhich backend read and which audit trail confirm each fieldRecord API read; field history on Stage and CloseDateAlways; name gaps explicitly
propagation_window_sHow long to wait before a backend read counts30 seconds for this object; 15 minutes for the reporting replicaWhen writes are asynchronous or replicated
expected_end_stateField values that define successStage equals Closed Won; CloseDate equals the run dateAlways

What does one step record, and how is it judged?

Each state-changing step records the UI claim, the backend observation and a verdict. Read-only steps, such as opening a list view, need only the UI observation. The verdict is what turns a trajectory into labeled state evidence: it says whether the screen and the record agreed for that step.

Store the raw evidence alongside the verdict, not just the verdict. A reviewer who disagrees with a label must be able to open the screenshot, the backend response and the audit entry for the same step.

Checklist · 5 items

  1. aligned: the UI claim and the backend read agree and match the expected change.
  2. ui_only: the screen shows the change and the backend does not; the most common false success.
  3. backend_only: the backend changed and the screen did not refresh; usually not an agent error, but it can trigger harmful retries.
  4. neither: no change anywhere; the action failed visibly or invisibly.
  5. collateral: the target is right but a record outside target_records also changed.

0 of 5 ticked. Tick each item as you go.

What does one step record, and how is it judged?
Step fieldWhat it holdsRule
ui_observationScreenshot and accessibility or DOM snapshot references, timestampCaptured after the action and before the next one
actionThe UI action or tool call with argumentsTool calls record tool name, version and returned payload
ui_claimThe field values the UI says now holdRead from the rendered page, not from the agent's intention
backend_observationBackend read of the same fields by record ID, with read timeTaken after propagation_window_s; replica or primary noted
audit_matchAudit entries for this step: field, old value, new value, user, timeUser must equal the agent's run identity
verdictaligned, ui_only, backend_only, neither or collateralComputed by comparing ui_claim, backend_observation and expected delta
recoveryRetry, undo or escalation taken after a bad verdict, with its own evidenceAn undo is verified by a backend read, like any other change

Illustrative walkthrough: a CRM stage change that looked saved

This walkthrough is invented to show the specification in use. It does not describe any real company, record or dataset, and the durations are illustrative.

The task asks the agent to move a renewal opportunity to Closed Won and set the close date to the run date. The agent edits both fields inline in a list view.

  • Step 3, inline edit of Stage: the cell shows Closed Won. The backend read 30 seconds later shows Negotiation, and the save response carried a validation error requiring a close reason. Verdict: ui_only, pattern optimistic update rejected.
  • Step 4, the agent opens the record and sees Negotiation, adds a close reason and saves. Backend read shows Closed Won; field history shows Stage changed from Negotiation to Closed Won by the agent's run user. Verdict: aligned.
  • Step 5, the agent clicks Save again after a slow response. The backend shows no new change, and the record count for the natural key is unchanged. Verdict: aligned, with a retry noted as harmless because the edit set the same values.
  • Final check: CloseDate matches, no record outside target_records changed in the window. The trajectory is labeled a success with one recovered ui_only step, which is exactly the kind of recovery a model should learn from.
  • Without backend reads, step 3 would have been stored as a success and the trajectory would teach that inline edits on this object work without a close reason.

Tool calls, retries and undo: what must the record keep?

Treat a tool's own result as a claim, just like a screen. A tool that returns typed output gives a cleaner claim than a screenshot, and the Model Context Protocol lets tools declare an outputSchema and return structuredContent, but the result is still produced by the server that made the change. The backend read stays separate.

Tool metadata is not evidence either. MCP tool annotations such as readOnlyHint, destructiveHint and idempotentHint are defined as hints that may not describe the tool faithfully, and clients are told not to rely on them from untrusted servers. The current 2026-07-28 revision of the specification repeats that descriptions of tool behavior, such as annotations, should be treated as untrusted unless they come from a trusted server. Record them as context, then verify what the tool did.

Retries need their own evidence because they are where duplicates come from. HTTP semantics define an idempotent method as one where repeating an identical request has the same intended effect as sending it once, which is why only those requests are safe to repeat automatically. A business action like creating an order or posting a payment is usually not idempotent unless the system accepts an idempotency key, so the step record should store the key, if any, and a count of records matching the task's natural key before and after.

Undo is a state change like any other. Store the compensating action, its backend read and its audit entry, and mark actions that cannot be undone, such as a sent email, before the run rather than after.

What do public agent benchmarks verify, and what does a training dataset add?

Public benchmarks already grade final state rather than screenshots, which is the right baseline. OSWorld task files pair setup config for the initial state with an evaluator that reads application state, for example a browser setting, and compares it with an expected value. tau-bench hashes the database after an episode and compares it with the hash produced by replaying the ground-truth actions, setting the reward to zero on a mismatch. WebArena mixes string, URL and page-content checks, and some content checks call the site's own API first, for example to find the latest order, before reading the page.

Those checks answer whether the final state is right. A training or diagnostic dataset needs to know where it went wrong and whether the agent noticed. Per-step alignment verdicts, the permission context and the recovery record give that, and they let a model learn to check the record rather than trust the banner. Final-state grading is covered in depth elsewhere on this site; this specification is the step-level layer underneath it.

When backend proof is out of reach, narrow the claim

Some steps cannot be verified against a system of record, and the dataset should label them unverified rather than infer success from the screen. Narrowing the claim keeps the dataset honest and tells a buyer which part of the success label is evidence and which is inference.

  • No read access to the backend for that object: label the step ui_claim_only and exclude it from success metrics, or replace the task with one on an object that can be read.
  • Asynchronous jobs that finish after the observation window: record the job ID and re-read later; if the window cannot be extended, label the outcome pending.
  • Effects in external systems, such as email delivered to a customer: record the outbound message ID and the system's accepted status, and do not claim delivery or reading.
  • Audit trail not enabled for a field, or history older than the system keeps: record the gap; Salesforce, for example, keeps field history only from the time tracking is enabled and only for fields selected for tracking.
  • Records containing personal data in screenshots or field values: redact before the dataset leaves the source environment, and have counsel review when records involve regulated data.

Sourcing workflow records that already carry state evidence

Operating businesses often already hold the backend half of this evidence: field histories, audit logs, workflow runs and the final records that show what a change achieved. Licensed together with task context, those records let a buyer build alignment checks from real work instead of reconstructing them. Rights still need checking separately: being able to export an audit log does not settle privacy permission, ownership or the licensing rights in the contract, and de-identification does not create a right to license.

Because SourceX sources from U.S. businesses per request instead of keeping stock, a buyer can list the backend evidence fields this guide describes right in the request, next to scope, volume, language, format, timing and licensing needs.

Frequently asked questions

Can a screenshot ever be enough to prove a business state change?

Only for state that exists solely on screen, such as a UI preference stored in the browser. For anything stored in a system of record, a screenshot proves what was rendered, not what was saved. Pair it with a backend read by record ID and, where available, an audit entry.

What should the UI and backend state alignment specification include?

At task level: intent, target records, actor permissions, software and tool versions, an initial state snapshot, evidence sources, a propagation window and the expected end state. At step level: the UI observation and claim, the action, the backend read, matching audit entries, a verdict and any recovery taken.

How long should a dataset wait before reading the backend?

Long enough for the write to reach the store being read, which depends on the system. Declare the window per object in the task header, read the primary store where possible, and record the read time. If a read before the window disagrees, re-read rather than labelling a failure.

Do MCP tool annotations prove what a tool changed?

No. The MCP specification defines annotations like readOnlyHint, destructiveHint and idempotentHint as hints that may not describe a tool faithfully. Record them as context and verify the effect with a backend read.

Is this the same as end-state evaluation for computer-use agents?

It builds on it. End-state evaluation checks whether the final state matches the goal. Alignment evidence records, for each state-changing step, whether the screen and the system of record agreed, which is what training and failure analysis need.

Sources

  • OSWorld task files pair an instruction with setup config for the initial state and an evaluator that reads application state, such as a browser setting, and compares it with an expected value. Source
  • tau-bench scores an episode by hashing the resulting database and comparing it with the hash produced by replaying the ground-truth actions; a mismatch sets the reward to zero. Source
  • WebArena evaluators include string, URL and page-content checks, and some content checks first call the site's API, for example to find the latest order, before reading the page. Source
  • MCP tool annotations such as readOnlyHint, destructiveHint and idempotentHint are hints that may not describe a tool faithfully, and clients should not rely on them from untrusted servers. Source
  • MCP tools can declare an outputSchema and return typed results in a structuredContent field. Source
  • RFC 9110 defines a method as idempotent when multiple identical requests have the same intended effect on the server as a single request, which is why such requests can be retried automatically. Source
  • Salesforce field history tracking records old and new values for selected fields, with the date, time, nature of the change and the user who made it, from the time tracking is enabled. Source
  • The MCP 2026-07-28 specification says descriptions of tool behavior, such as annotations, should be considered untrusted unless obtained from a trusted server. Source

Tell us what data you need

Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.

See if your company qualifies