AI uses for records
Computer use task replay portability: can a buyer rerun it without production?
By SourceX Editorial · Updated
9 sections
Short answer
Package the task so it runs against a reconstructed copy, not the live system: a start-state snapshot of the records the task touches, the configuration and role it ran under, the screen and tool context at each step, and a read-back check that proves the business record changed. Replace external services with stubs that follow their contracts, and mark any step that cannot be reproduced instead of faking it.
Agent run
Computer use task replay
Key takeaways
- A task is portable when someone without your logins can load its start state, run it and confirm the outcome from data.
- Capture what production supplies silently: records, configuration, role, connected services, clock, version and files.
- Agree the replay level first; outcome re-verification needs a target end state and a read-back check.
- Recorded network responses are fine for reads and misleading for writes; stub writes against the reconstructed data.
- Mark steps that cannot be replayed, such as a phone confirmation, and grade the request the agent should make instead.
How do you make a business UI task replayable without production system access?
Treat replay as a packaging job with a pass test: someone with no access to your systems should be able to load a start state, run the task, and confirm the same business outcome from the data rather than from the screen. If any of those three steps needs your production login, the task is not portable yet.
In practice that means shipping four things with each recorded task. The first is the start state: the records, files and configuration the task reads or changes, cut at one point in time. The second is the context of every step: what was on screen or in the page structure, which tool was called and with what arguments. The third is the authority the task ran under, described as a role rather than as a live credential. The fourth is an outcome check that queries the reconstructed system of record after the run.
This page is about that portability test for computer-use and tool-use tasks taken from real business software. It does not cover how to write evaluation tasks from scratch or how to license replay rights; both are linked at the end. The question here is narrower: what has to travel with a recorded task so a buyer can re-execute and verify it in their own sandbox.
What does a recorded task silently borrow from production?
Almost every recorded UI task leans on things the live system supplies without anyone noticing: existing records, tenant settings, the user's role, connected services, the clock and the software version. A replay breaks wherever one of these is missing, and the failure often looks like an agent error when it is really a packaging error.
Public benchmark environments make the same dependencies explicit. An OSWorld task file, for example, lists setup steps that download a specific workbook and open it before the agent starts, then defines an evaluator that saves the file and compares it with a gold workbook. The same file carries a field rating the possibility that the environment changes. That is the shape a portable business task needs too: setup, task, check, and an honest note about drift.
De-identification adds a dependency of its own. If a customer name is replaced in the database snapshot but still visible in the screenshots, or replaced with a different surrogate in each place, the replay no longer lines up: the agent sees one value on screen and finds another in the record. Apply the same substitution to the snapshot, the page captures, the typed text and the tool payloads, and record which fields were changed so the outcome check compares like with like.
| Hidden dependency | How it shows up in a failed replay | What to capture with the task |
|---|---|---|
| Existing records | The agent searches for a customer or invoice that does not exist in the sandbox | A snapshot of every record the task reads or changes, plus the records they join to |
| Tenant configuration | A required field, validation rule or picklist value differs, so a save is rejected | The configuration items the screens and rules depend on, versioned with the snapshot |
| User role and permissions | Buttons are missing or an action is refused because the replay user has a different role | The role, permission set and record-sharing rules, described without live credentials |
| Connected services | A lookup, address check or payment call times out or returns nothing | Each call's request and response contract, and whether the step reads or writes |
| Time and sequence | Due dates, aging buckets or approval deadlines evaluate differently on replay day | The original timestamp and a way to pin the sandbox clock or shift dates consistently |
| Application version | A control moved, a dialog was added or a field was renamed after a release | The product version or release identifier and the date the task was recorded |
| Files and attachments | An upload step cannot find the document the user attached | The files themselves, de-identified as needed, with their original names and types |
Which replay level does the buyer actually need?
Agree the replay level before packaging, because each level needs different material. A buyer who only wants to inspect what happened needs far less than one who wants to let a new agent attempt the task and grade it.
Be explicit about what each level does not support. A level two package proves the recorded path still works; it says nothing about whether a different agent would be graded fairly, because there is no target state to grade against. A level three package grades outcomes but may still penalize an agent that takes a legitimate route the stubs never anticipated, such as calling a lookup the original user skipped. Only level four tolerates that, and it costs the most to build because every tool the agent might reasonably call needs a working stub.
Most requests for agent training and evaluation sit at level three. Level two is useful for imitation data where the recorded path itself is the label. Level four matters when the buyer wants to measure whether an agent can find a different, still correct, route.
| Replay level | What the buyer can do | What the package must include | Typical use |
|---|---|---|---|
| 1. Inspect | Step through the recorded actions and screens | Action log with before and after page snapshots or screenshots per step | Error analysis and labeling |
| 2. Re-execute the recorded path | Run the same actions against a reconstructed system and see them succeed | Level 1 plus start-state snapshot, configuration and role | Demonstration data, regression checks |
| 3. Re-verify the outcome | Run any agent on the task and confirm the business state matches the target | Level 2 plus a target end state and a read-back check | Agent evaluation and reward signals |
| 4. Explore alternatives | Let an agent take a different valid path, including refusing | Level 3 plus full tool contracts, permission rules and side-effect stubs | Robustness and policy testing |
What does the per-task portability checklist test?
Use this checklist per task, not per dataset. Each line has a pass test a reviewer can run without production access; a task passes when every line needed for its replay level passes, and every other line is explicitly marked as not applicable.
At a glance
What does the per-task portability checklist test
Two lines do most of the work. For screen context, browser tooling already shows the standard to aim for: Playwright's trace viewer captures a complete page snapshot at the moment each action is called and at the moment of the input, so a reviewer can see exactly what was acted on. For reset, public environments are blunt but clear; WebArena's instructions return its test sites to their initial state by stopping, removing and re-running the site containers after an evaluation pass. A business sandbox needs the same reset, at the level of one task.
Version the checklist with the task. When the application, the permission model or a connected service changes, rerun the pass tests, because a task that replayed last quarter can fail today for reasons unrelated to the agent.
| Checklist item | What to package | Pass test without production | Common gap |
|---|---|---|---|
| Task intent | The business goal in plain words, the requester and the expected outcome | A reviewer can state the target end state before watching the recording | Intent inferred from clicks after the fact |
| Start state | Records, files and configuration the task touches, cut at one moment | Loading the snapshot recreates every record the first screen shows | Joined records left out, so lookups fail |
| Screen and page context | Screenshot or page snapshot before and after each action, plus the accessibility or DOM view where available | Each action can be matched to the element it acted on | Video only, with no per-step structure |
| Tool contract | Name, input schema, output schema and side effects of every tool or API call | A stub built from the contract returns valid responses for the recorded calls | Responses recorded without the schema they follow |
| Permission | The role, permission set and approval rules in force, with no live secrets | A sandbox user with that role sees the same controls and gets the same refusals | Recorded as an admin account, so no refusals appear |
| Action log | Ordered actions with timestamps, arguments and the element or tool targeted | Re-executing the log on the start state reaches the recorded end screen | Gaps where the user switched windows or applications |
| State verification | A query or file comparison that reads the business record after the run | The check passes on the recorded end state and fails on the start state | Success judged from a confirmation banner |
| Retry and undo | Errors the user hit, how they recovered, and how to reverse the change | The sandbox can be reset to the start state and the task run again | No reset path, so a second run starts dirty |
| Outcome | Final values of the fields that matter and any side effects such as messages queued | The target end state is written down field by field | Outcome left implicit in the recording |
| Environment version | Product version, release date, locale and time zone at recording time | A reviewer can tell whether a newer release could change the task | No version, so drift cannot be diagnosed |
Which read-back check proves the record changed, not just the screen?
Only a read of the system of record after the run proves it: the invoice status in the ledger table, the saved workbook cell values, the ticket's assignee field. A success message, a green banner or a closed dialog proves the interface moved, which is not the same claim.
Benchmark designers converged on this for good reason. The tau2-bench evaluation documentation explains that its reference list of tool calls is one way to solve a task, used to build the target database state, and that scoring depends on whether the agent's database end state matches the target. An agent that takes a different correct path, or correctly refuses, can still get full reward. The OSWorld task above does the file equivalent: it saves the workbook and compares sheet data with a gold copy.
Write the check so it also fails when something extra changed. A task to update one vendor's remit-to address should fail if a second vendor record was edited too, or if a payment run was triggered. Checking only the fields you expected to change misses the most expensive class of agent mistake.
Keep the check outside the agent's reach. If the agent can call the same query the grader uses, or edit the expected values, the check stops measuring anything. Run it from the harness after the agent has finished, against the reconstructed database or files.
When is a recorded network response a safe substitute for the real service?
A recorded response is a safe substitute for a read that does not depend on what the agent did earlier; it is not a substitute for anything that writes. Playwright's documentation describes recording an HTTP Archive file of a page's requests and serving later requests from it, which works well for the lookups and reference data a screen loads.
The trap is a step that changes state on the server. A recorded response to a save request will happily say the save worked on every replay, whatever the agent sent, because nothing on the other side is checking. For any write, use a stub that applies the change to the reconstructed data and returns what the real service would return given that data. Then the state check reads a real change.
Treat tool definitions the same way. The Model Context Protocol specification describes tools by an input schema, an optional output schema and optional annotations about behavior, and warns that clients must consider those annotations untrusted unless they come from trusted servers. A portable package should state the side effects of each tool from your own testing, not copy a hint from the tool description.
How would an illustrative remit-to address change be packaged for replay?
Illustrative only. An accounts payable clerk receives an emailed request to change a vendor's remit-to address, opens the vendor record in the ERP, edits the address, attaches the request, and sets the record to pending approval. The packaged task includes the vendor record, its open invoices and the approval workflow configuration, cut at the moment the email arrived.
The action log has eleven steps; the page snapshots show the vendor form before and after each edit. The tool contract covers one address validation call, which only reads, so a recorded response is acceptable for it. The save is stubbed against the reconstructed vendor table. The outcome check reads the vendor record and confirms three things: the new address is stored as pending, the active address is unchanged until approval, and no other vendor record or invoice changed.
The exception is the step the clerk did by phone. Company policy required a call to the vendor's number on file to confirm the change, and that call left only a note in the record. No sandbox can replay a phone call, so the package marks the step as an external attestation: the agent is expected to request the callback, not to perform it, and the check confirms that a callback task was created with the number on file rather than the number in the email. Writing that down keeps a policy step in the task instead of silently dropping it.
Two further checks come from the same packaging. Rerunning the task twice from a reset start state should give the same end state both times, which confirms the reset path works. And loading the start state with the task unchanged should fail the outcome check, which confirms the check can tell a finished task from an untouched one.
- Replay level: 3, re-verify the outcome.
- Not portable as recorded: the phone confirmation, replaced by a check that a callback task exists.
- Reset path: reload the vendor and invoice snapshot before each run.
- Failure that would count: the active address changing before approval, or any payment field changing.
Which steps should you stub, drop or narrow instead of reproducing?
Stub a step when its contract is clear and its effect can be applied to the reconstructed data. Drop or narrow the task when a step's effect lives entirely outside the systems you can package, or when reproducing it would mean copying data you have no right to share.
Steps that send money, email customers, file with a regulator or call a carrier's live system belong in the second group. Keep the decision to perform them, record the request the agent should produce, and grade that request, but never wire a replay to the real endpoint. Time-dependent steps are often fixable by pinning the sandbox clock; per-user secrets, such as a stored browser login, are not fixable and should never be in a package at all.
Playwright's authentication guide warns that a saved browser state file can contain sensitive cookies and headers that could be used to impersonate the account. The same applies to any session capture from a recording tool. Describe the role instead, and let the buyer create their own sandbox user with it.
Ask for specialist review when a task depends on regulated records, on another company's software in ways its terms may restrict, or on third parties who appear on screen. Those are rights and privacy questions, not packaging questions, and the right reviewer is counsel or the system owner rather than the engineer building the sandbox.
What should travel to the buyer with a replay package?
Send the checklist results with the package, including the lines marked not applicable, so the buyer can see which replay level each task supports. A reset script and a short note on known drift save more evaluation time than extra tasks that cannot be verified.
Real business records are what make the start state and the outcome check credible, and they are also where the rights questions sit. On the supply side, SourceX takes companies that license such records through rights review, contracting, preparation and delivery. On the buying side it keeps no replay-ready stock: records are sourced from U.S. businesses once a buyer's request-data form sets out scope, volume, language, format, timing and licensing needs, so naming the replay level, systems and record types in that form is the useful starting point.
Frequently asked questions
Can I just give a buyer a copy of my production database for replay?
Usually not, and you rarely need to. A replay needs only the records the task touches and the records they join to, cut at one point in time and de-identified as required. A full copy adds privacy and contract exposure without improving the test.
Is a screen recording enough to make a computer-use task replayable?
No. Video supports inspection only. Re-executing and grading a task also needs the start state, per-step action and page context, the role it ran under and a check that reads the business record after the run.
Does the sandbox need the exact same software version?
It needs a known version. Record the product version and date with every task, and rerun the pass tests after a release, because moved controls or new required fields can break a replay for reasons unrelated to the agent.
How do you replay steps that send email or payments?
Do not connect them to real endpoints. Stub them so the request is captured and graded, and check that the agent produced the right request with the right values, rather than letting anything leave the sandbox.
Should grading compare the agent's actions to the recorded actions?
Generally no. Compare end states. A recorded path is one valid solution; other agents may reach the same correct state differently, or correctly refuse, and outcome-based checks credit both.
Sources
- An OSWorld LibreOffice Calc task file defines setup steps that download and open a workbook, an evaluator that saves the file and compares sheet data with a gold workbook, and a field rating the possibility of environment change. Source
- WebArena's environment instructions reset the test sites to their initial state by stopping, removing and re-running the site containers after evaluation. Source
- The tau2-bench evaluation documentation says its reference actions are one solution used to set up the target database state, and that scoring depends on whether the agent's database end state matches the target, so a different correct path or a correct refusal can receive full reward. Source
- Playwright's trace viewer documentation says tracing with snapshots captures complete DOM snapshots for each action, including before and action snapshots. Source
- Playwright's mocking documentation describes recording an HTTP Archive (HAR) file of a page's network requests and serving later requests from it. Source
- Playwright's authentication guide warns that a saved browser state file may contain sensitive cookies and headers that could be used to impersonate the account. Source
- The Model Context Protocol tools specification (2026-07-28 revision) defines tools with an input schema, an optional output schema and optional annotations, and says clients must consider annotations untrusted unless they come from trusted servers. Source
Tell us what data you need
Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.