AI uses for records
Computer use dynamic UI dataset: testing agents after the button moves
By SourceX Editorial · Updated
10 sections
Short answer
Hold the task and its expected business state fixed, change one thing about the interface at a time, and check the system of record after every run. Pair each variant, such as a moved, relabeled or collapsed button, a narrow viewport or an extra dialog, with a base run of the same task, so a failure can be traced to the change rather than to the agent's general skill.
Agent run
Computer use dynamic UI
Key takeaways
- Hold the task and its expected end state fixed, and vary one interface axis at a time.
- A change that alters inputs, permissions or the end state is a new task, not a UI variation.
- Grade every variant with the same backend state check; never grade on what the screen shows.
- Score in pairs against a base run and report the flip rate per variation axis, with repeated runs.
- Real before-and-after release recordings complement synthetic variants and show which axes matter.
How do you test computer-use agents after a button changes location?
Treat the interface as a variable and the business outcome as a constant. Build each test from a base task whose end state you can verify in the system of record, then produce variants that move, relabel, hide or restyle the controls the agent needs while leaving the workflow untouched. Run the base and every variant from the same starting state and grade all of them with the same state check.
The result tells you something an ordinary pass rate cannot: whether the agent learned the workflow or memorized a screen. An agent that approves an expense report only when the Approve button sits in the top right has learned a layout, and the first vendor release will expose it.
This page gives the variation catalog, the case record, the scoring and a worked example, and it marks the point where a changed screen stops being a UI variation and becomes a different task.
Why does a moved button break agents when the workflow is unchanged?
Agents and scripts fail on layout changes for the same reason brittle UI tests do: they anchor on where or how a control is drawn instead of what it is. Playwright's documentation warns that XPath and CSS selectors tied to DOM structure can break when that structure changes, and recommends locating elements by role and accessible name, the way users and assistive technology perceive the page.
Layouts also move on purpose. The WCAG Understanding document for Reflow describes content adapting to a 320 CSS pixel wide viewport, with the example of a site whose navigation collapses into a menu button at that width. The same application can legitimately put the same control in two places depending on window size, so an agent that cannot find a control behind a collapsed menu will fail on a smaller laptop screen even if nothing was released.
People rely on stability too. WCAG's Consistent Navigation guidance asks that repeated navigation keep the same relative order across pages because users rely on spatial memory to find it. When a vendor does move controls, human operators slow down; an agent with no notion of the control's meaning simply clicks the wrong thing or nothing.
Where does a UI variation end and a workflow change begin?
A change is a UI variation only if the same inputs, the same permissions and the same sequence of business decisions still produce the same end state. If any of those changes, you have a new task with its own expected outcome, and scoring it as a variant will penalize an agent for behaving correctly.
Use three questions to classify a change before it enters the test set.
- Does the end state differ? A new required field, a new approval step or a different default value changes the record, so it is a workflow change.
- Does the user's authority differ? If the moved control is now hidden because the user lacks a permission, the correct behavior is to stop and report, which is a different expected outcome.
- Does the agent need information it did not need before? A confirmation dialog that only asks to proceed is a variation; one that asks for a reason code adds an input and is a workflow change.
- If all three answers are no, the change is a variation and the base task's state check applies unchanged.
How do you produce variants without touching the business data?
Change presentation through layers that sit above the data: styles, feature flags, viewport settings, locale settings and injected notices. Never produce a variant by editing records or configuration that the state check reads, or the base and variant stop being comparable.
In a web application you control, a stylesheet override can move or restyle a control, a feature flag can switch between two menu layouts, and the browser window size forces responsive layouts. For a vendor application you cannot modify, use what the vendor exposes: user-level theme and language settings, window size and saved layout preferences, plus a staging tenant on the next release when the vendor offers one.
Record the mechanism with every variant. A variant produced by a stylesheet override may leave the accessibility tree unchanged while a vendor release rebuilds it, and an agent that reads the accessibility tree will behave differently on the two even though both look the same on screen. Knowing which layer changed is what lets you explain a failure instead of guessing.
Which UI axes should vary, and what must stay invariant?
Each row is one variation axis. Vary one axis per variant so a failure points to a cause, and add combined variants only after single-axis results are understood. The invariant column is what must stay identical for the variant to be valid; the check column is how you confirm the run reached the right state.
| Variation axis | How to produce it | Invariant held | State check |
|---|---|---|---|
| Position | Move the target control to another region or reorder a toolbar | Same control, label, role and outcome | Record field values read from the backend after the run |
| Label | Change visible text, for example Approve to Confirm approval | Same role and action; meaning unchanged | Same backend read as the base task |
| Nesting | Move the control into an overflow or More menu | Same action reachable within the same permissions | Backend read plus audit log entry for the action |
| Viewport | Run at a narrow width so navigation collapses | Same app version and data | Backend read; no partial saves |
| Theme and contrast | Switch dark mode or a high-contrast theme | Same DOM and accessibility tree | Backend read |
| Locale | Switch the interface language while data stays the same | Same records; only UI strings change | Backend read with locale-independent values such as IDs and ISO dates |
| Interstitial | Insert a dismissible notice or tour before the target screen | Notice requires no business input | Backend read; notice dismissed, not accepted on the user's behalf |
| Latency | Delay the load of the target control | Same final screen | Backend read; no duplicate submissions |
| Version | Run the same task on the next release of the application | Release notes show no workflow change for this task | Backend read; app build recorded |
What should each test case record?
A variant result is only useful if someone can see exactly what changed, what the agent saw and what the system recorded. Keep the record per run, linked to its base task, so results can be compared pair by pair.
At a glance
What should each test case record
- Task intent: the instruction in plain language and the business outcome it should produce.
- Base environment: snapshot or seed ID, application build and account role.
- Variant: axis, parameters and the mechanism that produced it, such as a CSS override, viewport size or feature flag.
- Screen context: screenshots and, where available, the DOM or accessibility tree at each step.
- Tool contract: the action space and its version, for example click, type and scroll, or named API tools.
- Permissions: the role the agent ran as and any permission the task needs.
- Actions: each step with timestamp, target as described by role and name, and the agent's stated reason if it produces one.
- State verification: the query or file comparison used and its result.
- Retries and undo: every repeated or reversed action, so double submissions are visible.
- Outcome: pass, fail or correct stop, with the reviewer's note when a person adjudicated.
Why must the state check stay identical across every variant?
The checker is the invariant you are testing against, so it must not look at the screen. If the check reads the confirmation banner, a relabeled banner fails a correct run; if it reads the backend record, a cosmetic change cannot affect the grade.
Public benchmarks show the pattern. An OSWorld spreadsheet task, for example, defines an evaluator that saves the file and compares the resulting workbook with a gold workbook, rather than inspecting what was on screen. WebArena's README tells evaluators to reset the environment to its initial state after evaluating its 812 examples, which points to the other half of a fair comparison: base and variant must start from the same data.
For business software the equivalent is a read from the system of record, such as the expense report status and approver in the database or through the vendor API, plus the audit log entry. Write that check once for the base task and reuse it verbatim for every variant. If a variant seems to need a different check, revisit whether it is really a workflow change.
How do you score robustness across variants?
Score in pairs. For each task, compare the base result with each variant result from the same starting state, and report how often a base pass turns into a variant failure. That flip rate, broken down by axis, is the robustness number; an overall pass rate across all variants hides which change caused the damage.
At a glance
How do you score robustness across variants
Run each condition more than once, because agent runs vary, and report counts with an interval rather than a single percentage. Report base success alongside, since a task the agent never solves tells you nothing about robustness.
Generalization has levels, and the design should name the one it tests. Mind2Web, for example, splits its test data into cross-task, cross-website and cross-domain sets so results show how far an agent transfers. AndroidWorld instantiates its 116 tasks with randomly generated parameters to produce many variations of each. A UI variation suite adds another axis to those: same task, same site, different presentation.
- Base success: share of tasks passed in the base condition.
- Flip rate per axis: base passes that fail in the variant, divided by base passes.
- Recovery rate: variant runs that took a wrong step, noticed it and still reached the right state.
- Unsafe outcome count: variant runs that changed the wrong record or submitted twice, reported separately from failures.
Worked example (illustrative): approving expense reports after a redesign
This example is invented to show the design, and its numbers are illustrative counts, not results. A team tests an agent on 40 expense approval tasks in a web expense application. Each task names a report, a policy rule to check and the expected decision; the state check reads the report's status, approver and decision reason from the backend.
The suite runs the base condition plus five single-axis variants: Approve moved from the header to the footer, Approve relabeled Confirm approval, Approve moved into a More menu, a narrow viewport that collapses the sidebar, and a dismissible what's-new notice. Each condition runs 3 times per task, so the plan is 40 tasks by 6 conditions by 3 runs, or 720 runs.
Scoring works per pair. Suppose 30 of 40 tasks pass in the base condition. If 18 of those 30 still pass with Approve inside the More menu, the flip rate for the nesting axis is 12 of 30, or 40 percent, while the relabel axis might flip only 2 of 30. The finding is specific: the agent does not search menus for a missing action.
The exception: a sixth candidate variant ran the task as a user whose role cannot approve reports over a limit, and Approve disappeared. That is not a UI variation. The correct outcome is a stop and an explanation, so it becomes a separate permission task with its own expected end state.
The next step follows from the finding rather than from the overall score. The team adds training demonstrations in which people open overflow menus to find actions, reruns the nesting axis with fresh parameters held back from tuning, and checks that the base and relabel results did not slip while the nesting result improved.
Where does real UI variation data come from, and when do synthetic variants mislead?
Synthetic variants are controlled but can drift from how software really changes. The best complement is real history: the same workflow recorded before and after a vendor release, on different screen sizes, and in different interface languages by people doing actual work. Records like these show which axes matter in practice and give natural variants with real business outcomes attached.
Synthetic suites also invite overfitting. If the same variant set is used to tune and to report, results improve without any gain in robustness. Hold back at least one axis, or one set of parameters per axis, for reporting only.
Recorded sessions carry personal and confidential data on screen, and the people and companies in them have rights that a variant design does not settle. Treat screen recordings and trajectories as records that need consent, redaction and contractual permission before they are used.
Buyers who want such recordings can request them through SourceX, which sources from U.S. businesses on demand and asks each request to cover scope, volume, language, format, timing and licensing needs. A request for workflow records across application releases or screen configurations can name the variation axes from the table above.
Frequently asked questions
What is a dynamic UI dataset for computer-use agents?
It is a set of tasks in which the interface changes while the business task stays the same, such as moved, relabeled or collapsed controls, different viewports or interface languages. Each task keeps one verifiable end state, so you can measure whether an agent completes the work regardless of presentation.
Should a computer-use test check the screen or the database?
Grade on the system of record, such as a database row, an API response or a saved file. Screens change with every variant, so a screen-based check would fail correct runs or pass wrong ones. Screenshots remain useful as evidence of what the agent saw.
How many UI variants should each task have?
Start with one variant per axis that your users actually encounter, such as position, label, menu nesting, viewport and locale, and run each more than once. Add combined variants only after single-axis results are understood, and keep some variants back for reporting only.
Is a new required field a UI variation?
No. A new required field, approval step or permission changes the inputs or the end state, so it is a different task with its own expected outcome. Scoring it as a variation would penalize an agent that correctly stops or asks for the missing input.
Can real business screen recordings be used to build UI variation data?
They can be a strong source because they show how interfaces really change across releases and screens. They also contain personal and confidential information, so they need consent, redaction and contractual permission before use, reviewed for the specific records.
Sources
- Playwright's locator documentation says XPath and CSS selectors tied to DOM structure can break when the DOM changes and recommends role locators and user-facing attributes for resilient tests. Source
- The WCAG Understanding document for Reflow describes content reflowing at a width equivalent to 320 CSS pixels, with an example of navigation collapsed into a menu button at that width. Source
- The WCAG Understanding document for Consistent Navigation explains that presenting repeated content in the same order helps users who rely on spatial memory to locate it. Source
- An OSWorld LibreOffice Calc task defines an evaluator that saves the file and compares the resulting workbook with a gold workbook. Source
- WebArena's README asks evaluators to reset the environment to the initial state after evaluating its 812 examples. Source
- Mind2Web splits its test data into cross-task, cross-website and cross-domain sets. Source
- AndroidWorld contains 116 hand-crafted tasks across 20 apps, dynamically instantiated with randomly generated parameters to create many task variations. Source
Tell us what data you need
Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.