Skip to content

AI uses for records

Computer use dataset accessibility tree or screenshots: how to measure which evidence helps

By SourceX Editorial · Updated

10 sections

Short answer

A screenshot records what was rendered; an accessibility tree records the roles, names, values and states the application exposed to assistive technology. Each misses things the other shows, and neither proves the business record changed. This page is a protocol, not a study: it sets out a matched modality experiment to measure which evidence makes computer-use records more useful. No results are reported.

Agent run

Computer use dataset

This page covers computer use dataset accessibility tree or screenshots: how to measure which evidence helps.Illustrative

Key takeaways

  • Screenshots record pixels; accessibility trees record exposed roles, names, values and states, and each misses things the other shows.
  • Platform rules exclude hidden and aria-hidden elements from the tree, so the two views can disagree by design.
  • Capture both at every step, then derive screenshot-only, tree-only and combined arms from one recording.
  • Judge every arm with the same backend state check; neither observation proves the business record changed.
  • Pre-register the primary metric, paired analysis and minimum meaningful difference before collecting data.

How do screenshots and accessibility trees differ as computer-use task evidence?

They answer different questions. A screenshot shows what a person would have seen: layout, color, images, canvas drawings and any text painted as pixels. An accessibility tree is a structured list of the interface objects the application exposed to assistive technology, each with a role such as button or textbox, an accessible name, often a value, and states such as checked or expanded.

For a computer-use dataset, that difference decides what a reviewer or a model can verify later. A tree can say that the field labeled Invoice total holds a specific value and is disabled; a screenshot can show that a warning banner appeared in red above it. A tree can contain elements that are scrolled out of view; a screenshot can contain content the tree never exposed.

Which one makes a dataset more useful for a given task family is an empirical question. This guide describes how to measure it with a matched experiment. It reports no findings, and any numbers that appear in the design are placeholders for decisions a team makes before collecting data.

What each observation records, and what it silently drops

The accessibility tree is defined by platform rules, not by what is visible. The W3C WAI-ARIA specification says the tree and the DOM are parallel structures, that elements hidden with CSS display none, visibility hidden or the HTML hidden attribute must not be included, and that elements with aria-hidden set to true should not be. A developer can therefore hide something from the tree that is still on screen, or expose a name that differs from the visible label.

On the web, the Chrome DevTools Protocol returns each tree node with an ignored flag and ignored reasons, a role, a name, a description, a value, parent and child IDs and a backend DOM node ID. On Windows, Microsoft's UI Automation documentation describes raw, control and content views of the tree, each a filtered subset of the one before, so which view was captured changes what the record contains.

What each observation records, and what it silently drops
PropertyScreenshotAccessibility treeConsequence for evidence
Visual state such as color, warnings and iconsCaptured as pixelsOnly if exposed as a name, description or stateTree-only records can miss a red validation banner
Field values and checked statesVisible if rendered and legibleExposed as value and state properties when the app supports themTrees give exact strings; screenshots need OCR
Off-screen contentAbsent unless scrolled into viewOften present unless capture is limited to the viewportTree records can reveal data the agent never saw
Canvas, images and remote desktop sessionsCapturedUsually absent or a single opaque nodeScreenshots are the only evidence for some apps
Element identity across stepsNone; must be inferred from positionNode IDs and, on the web, backend DOM node IDsTrees support step-to-step linking of the same control
Hidden or decorative elementsShown if paintedExcluded by ARIA and platform rulesThe two views can disagree by design

How public benchmarks already expose both observations

Benchmark harnesses make the modality a setting, which is what makes a matched comparison practical. OSWorld's run script accepts an observation type of screenshot, a11y_tree, screenshot_a11y_tree or som, and its environment returns a screenshot plus an optional accessibility tree and terminal output with each observation. Each OSWorld task carries setup configuration for the initial state and an evaluator for the outcome.

WebArena's run script offers accessibility_tree, html and image observation types, defaults to the accessibility tree, and has an option to limit the observation to the current viewport. Its processor fetches the tree through the Chrome DevTools Protocol's Accessibility.getFullAXTree command, which the protocol marks experimental and documents as fetching the entire tree for the root document.

These harnesses let you run the same task under several observations. They do not tell you which observation makes recorded business workflows more valuable as training or evaluation data, which is the question the protocol below is built to answer.

The matched modality task utility experiment: question and population

State the question in one sentence and freeze it before collection: for a defined family of business tasks, does adding an accessibility tree to screenshot-only step records change task success, verification accuracy or reviewer effort, and by how much?

At a glance

The matched modality task utility experiment: question and population

An illustrative frame shows the shape, not a recommendation: three applications, two task families each, and tasks split into short and long by step count, giving twelve strata. Every stratum needs enough tasks that a within-stratum estimate is not driven by one odd recording, and strata where an application exposes no usable tree are kept but analyzed apart. The frame, the strata and the planned count per stratum go into the pre-registration before the first recording is made.

  • Population: business tasks in named applications, for example invoice approval in an accounts payable tool or case updates in a service desk, each with a known end state in the system of record.
  • Unit of analysis: one task instance with its starting state, instruction, step records and backend outcome. Steps are nested inside tasks, so analysis must respect that clustering.
  • Sampling frame: a task list drawn from real workflow logs or process documentation, stratified by application, task length and whether the app exposes a usable tree. Record the frame and its date.
  • Inclusion rule: tasks whose outcome can be checked by a backend read or audit entry. Tasks without a state check are logged and excluded, not guessed.
  • Capture rule: every step stores both the screenshot and the tree from the same moment, with timestamps, viewport bounds and the tree view or capture depth used, so any arm can be built from one recording.

Arms, variables and the comparison

Record once, then derive arms. Because each step carries both observations, the screenshot-only arm and the tree-only arm are subsets of the same recording, which removes differences in task content, timing and operator skill from the comparison.

Use a within-task design: every task is presented in every arm, with order randomized for any human reviewers so learning effects do not favor whichever arm comes last. Keep the model, prompt template, action space and step budget identical across arms; WebArena's own harness rejects an accessibility-tree action set with a non-tree observation, a reminder that action and observation choices are coupled and must be fixed together.

Arms, variables and the comparison
ArmObservation givenWhat it isolatesWatch for
AScreenshot onlyBaseline for pixel-based agents and reviewersOCR errors on small text and dense tables
BAccessibility tree onlyValue of structured roles, names and statesMissing canvas content and visual warnings
CScreenshot and tree togetherWhether the two are complementaryLonger inputs and truncation of large trees
DScreenshot plus tree limited to the viewportWhether off-screen nodes help or leakLost context for long scrolled forms
EScreenshot with set-of-marks overlay from the treeGrounding benefit without raw tree textOverlay labels hiding the underlying text

What proves the business state changed rather than only the interface?

Neither observation does. Both are views of the interface, and the interface can show success while the save failed, or show a stale value after a background change. The outcome measure for every arm must be the same backend check: a read of the system of record by record ID, or an audit entry tied to the run, compared with the expected end state.

OSWorld's design is a useful model here, because its evaluators check application state against an expected value rather than grading the final screenshot. Apply the same idea to business tools: for an invoice approval, the check reads the invoice status and approver fields; for a service desk case, it reads the case state, owner and last-modified time.

Score the interface evidence separately as a secondary outcome. For each state-changing step, ask whether a reviewer or a model, given only that arm's observation, would have reached the same verdict as the backend check. This is where the modalities can differ most, and it is the measure that matters when a dataset is used to train verifiers.

Metrics, sample size and uncertainty

Pick a small set of metrics before seeing data, define each in writing, and name one primary metric so the conclusion does not depend on which number looks best afterwards.

At a glance

Metrics, sample size and uncertainty

  • Primary: task success rate by arm, judged by the backend check, compared within tasks.
  • Secondary: step-level verification agreement between each arm's verdict and the backend verdict, reported as a confusion table, not a single accuracy figure.
  • Secondary: reviewer minutes per task, and the share of steps a reviewer marks as undecidable from the observation alone.
  • Cost metrics: input size per step in tokens or bytes for each arm, because large trees can crowd out other context.
  • Paired comparison: because each task appears in every arm, use a paired test on discordant task outcomes and report confidence intervals from a bootstrap that resamples whole tasks, not steps.
  • Sample size: decide the smallest difference that would change a sourcing or modeling decision, then size the task set for that difference with a paired power calculation. Do not set the size after looking at results.
  • Subgroups: report results separately for applications with weak or missing trees; pooling them can hide an effect that runs in opposite directions.

Pre-registration checklist for the modality experiment

Checklist · 8 items

  1. Research question, primary metric and the minimum difference that matters, written and dated.
  2. Sampling frame, strata, inclusion and exclusion rules, and the date range of source workflows.
  3. Applications and versions, operating system or browser version, tree capture method, tree view or depth, and viewport setting.
  4. Arm definitions, the fixed model or reviewer pool, prompt template, action space and step budget.
  5. Backend state check per task family, with the exact fields read and the comparison rule.
  6. Analysis plan: paired test, task-level bootstrap, subgroup splits and how undecidable steps are counted.
  7. Privacy plan for both modalities, including how off-screen tree content and on-screen personal data are masked.
  8. Stopping rules, and a statement that all arms and all metrics will be reported whatever the outcome.

0 of 8 ticked. Tick each item as you go.

Where the comparison breaks down

Some applications cannot take part fairly. Remote desktop sessions, canvas-heavy editors and some legacy terminals expose little or no tree, so arm B is empty by construction; report them as a separate stratum rather than counting them as tree failures.

Tree quality depends on the developer. Custom widgets without accessible names, aria-hidden used on visible content, and names that differ from visible labels all change what arm B shows. Playwright's aria snapshot format, which writes the tree as role, quoted name and attributes, is a convenient way to record and diff tree captures across application versions so this drift is visible.

Privacy runs in both directions. A tree can carry values for fields that are scrolled away or collapsed, which a screenshot would not show, so masking rules written for screenshots are not enough. Results also age: any interface, tree representation or tool version change is a reason to rerun the affected strata, and a dated protocol makes that rerun comparable.

Specifying both modalities when you source computer-use records

If the experiment is planned, ask for recordings that capture both observations at each step, plus the backend outcome, rather than choosing a modality up front. That keeps every arm available later and lets the experiment run on the delivered data.

SourceX does not hold computer-use datasets in stock; it sources records on request from U.S. businesses. Buyers describe scope, volume, language, format, timing and licensing needs through the request form, which is where a dual-capture and backend-check requirement belongs.

Frequently asked questions

What is an accessibility tree in computer-use data?

It is the structured representation of an interface that an application exposes to assistive technology: objects with roles, accessible names, values and states, arranged in a hierarchy. In computer-use records it is captured alongside or instead of screenshots so a model or reviewer can read exact labels and values.

Is an accessibility tree better than a screenshot for agents?

It depends on the application and the task, which is why this page publishes a measurement protocol rather than an answer. Trees give exact strings and stable element identity; screenshots capture visual warnings, images and apps that expose no tree. A matched experiment on your own task families is the defensible way to decide.

Should a computer-use dataset include both screenshots and accessibility trees?

Capturing both at each step is the safest specification when the trade-off is unknown, because any single-modality version can be derived later. The cost is storage, privacy review of two representations and longer model inputs, which the experiment's cost metrics measure.

Can an accessibility tree prove an agent completed a task?

No. The tree reflects the interface, not the system of record. Proof of a completed business task is a backend read or audit entry showing the expected end state, and that check should be identical across every modality being compared.

Why limit the accessibility tree to the viewport?

A full tree can include content the agent could not see, which inflates what a tree-based record appears to show and can expose extra personal data. Limiting capture to the viewport makes the tree comparable with the screenshot; the protocol tests both settings as separate arms.

Sources

  • OSWorld's run script offers screenshot, a11y_tree, screenshot_a11y_tree and som observation types, its environment returns a screenshot with an optional accessibility tree and terminal output, and each task has setup configuration and an evaluator. Source
  • WebArena's run script offers accessibility_tree, html and image observations with a viewport-only option, and fetches the tree through the DevTools Accessibility.getFullAXTree command. Source
  • The Chrome DevTools Protocol defines accessibility nodes with an ignored flag and reasons, role, name, description, value, parent and child IDs and a backend DOM node ID, and marks getFullAXTree as experimental. Source
  • The W3C WAI-ARIA specification describes the accessibility tree as parallel to the DOM and excludes elements hidden by display none, visibility hidden, the hidden attribute or aria-hidden set to true. Source
  • Playwright aria snapshots represent a page's accessibility tree in YAML as role, accessible name and attributes. Source
  • Microsoft UI Automation describes raw, control and content views of the accessibility tree, each a filtered subset of the previous view. Source

Tell us what data you need

Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.

See if your company qualifies