AI agent training data: the workflow, action and decision data agents learn from

Definitions live in the glossary entries for agent trajectory, computer-use data and tool-use data; why AI agents need business data covers the motivation. For data types outside agent work, start from the AI data sourcing guide for buyers.

Six kinds of agent data, sorted by what they teach

Agent data is easiest to scope by the skill it teaches: grounding, action, process, judgment, recovery or environment. Each kind comes from different systems in a different record unit, and most programs need several.

A tool-calling agent behind an API can skip grounding; a computer-use agent for claims intake may need all six. SourceX's capability pages list business records for computer-use agents, tool use and function calling, long-horizon task agents, agentic workflow planning, IT operations agents and email and calendar agents.

Train, build or test: three jobs for the same records

The same business records can train a policy, furnish the environment and grader that policy practices in, or evaluate it on held-out tasks. Each job needs different fields, so fix the job before writing the specification.

Training a policy. Supervised fine-tuning on demonstrations, and reinforcement learning (RL) warm-starts, need step-level observation-action pairs plus a stated goal. Mind2Web is a public example: open-ended tasks paired with crowdsourced action sequences on real websites [3]. Ask for coverage across task variants, not hundreds of repeats of one path; see data for post-training teams.

Building environments and graders. RL and repeatable evaluation need a world the agent can act in and an automatic check of the result. τ-bench builds that world from databases and APIs, domain policy documents and simulated user scenarios, grades the database state at the end of an episode, and reports pass^k, the probability of succeeding on all k repeated trials of a task [4]. OSWorld supports task setup and execution-based evaluation on real operating systems [5].

Business data supplies the realistic parts: seed records, app configuration, the policy in force and the end state that counts as done. See configuration data for enterprise app replicas and the RL environment entry.

Evaluating on held-out tasks. Test tasks must stay out of every training set, including the model vendor's. In a February 2026 post, OpenAI said SWE-bench Verified had become increasingly contaminated and that it had stopped reporting scores on it [6]. SWE-Bench Pro keeps a held-out set private and adds a commercial set from startup codebases whose code is not released [7]: private company data used to keep part of an agent benchmark out of public training corpora. Agent evaluation task suites and human baseline data cover design.

Assign each case to training, environment or test by case ID and time period before anyone uses it. Splitting at the step level leaks, because steps from one case land on both sides.

What public agent benchmarks cover, and what enterprise work adds

Public agent benchmarks cover consumer websites, self-hosted web apps, desktop operating systems and simulated service desks, but say little about configured back-office systems, approval waits measured in days, or the exceptions enterprise agents meet. Use them as baselines and task templates, and source data for the gaps.

Benchmarks are revised, so scores from different releases are not comparable, and public tasks leak into training data, as the SWE-bench Verified case shows [6]. Computer-use evaluation tasks with verifiable end states covers building your own, and legacy desktop and terminal interaction data covers interfaces no benchmark models.

Why business records rarely arrive as clean trajectories

Business systems log state changes, not the steps a person took to cause them. A trajectory has to be reconstructed by joining audit trails, event logs, tickets and messages on shared object IDs and ordering them by time.

An ERP audit trail shows that an invoice's payment block changed at 14:07 and who changed it, not the screens checked first, the value copied from an email or the abandoned attempt. Clock skew between systems reorders steps, and scheduled jobs and RPA bots write under service accounts that look like people. Field-level audit trails, cross-system workflow records and RPA bot logs cover each source.

Process-mining formats help. IEEE 1849-2023 (XES) defines an XML format for event logs and event streams and supersedes the 2016 edition [1]. OCEL 2.0 links events to several objects, records changes to object attributes over time and qualifies object-to-object relationships, with SQLite, XML and JSON exchange formats [2]. That object-centric model fits back-office work, where one order-to-cash case touches an order, deliveries, invoices and payments; see procure-to-pay and order-to-cash records.

Require every step to state its evidence: observed in an audit trail, observed on screen, or inferred. Inferred steps help process supervision but are weak action labels. SourceX's workflow task histories page describes records rebuilt from system audit logs.

Illustrative example: invented to show structure; it does not describe an available dataset.

task:
  task_id: ap-exc-0447
  family: invoice_price_variance_resolution
  goal: "Release an invoice blocked by a purchase-order price variance"
  systems: [erp_accounts_payable, erp_purchasing, email, itsm_queue]
  outcome: {status: posted_after_po_change, label_source: erp_document_status, reopened_within_30d: false}
  timing: {active_minutes: 38, wall_clock_hours: 52}   # most of the elapsed time is waiting for the buyer
steps:   # steps 1-2 (invoice receipt and automatic match) omitted
  - {n: 3, ts: "2025-03-12T14:07:55Z", actor_role: ap_specialist, system: erp_accounts_payable,
     action: {type: field_update, object: invoice_4471, field: payment_block, from: none, to: price_variance},
     evidence: audit_trail}
  - {n: 4, ts: "2025-03-12T14:11:20Z", actor_role: ap_specialist, system: email,
     action: {type: send_message, to_role: purchasing_buyer, template: price_query},
     evidence: mail_log}
  - {n: 5, ts: "2025-03-14T09:40:02Z", actor_role: purchasing_buyer, system: erp_purchasing,
     action: {type: field_update, object: po_88213_line_2, field: net_price, from: 41.20, to: 43.05},
     rationale: "Supplier increase accepted under contract amendment", evidence: audit_trail}
  - {n: 6, ts: "2025-03-14T09:52:10Z", actor_role: ap_specialist, system: erp_accounts_payable,
     action: {type: release_block, object: invoice_4471},
     evidence: inferred_from_status_change}   # no screen capture exists for this step
privacy: {people: role_pseudonyms_consistent_across_systems, supplier_names: tokenized}

Rights and privacy: what a screen or log reveals

Agent data captures more parties than most datasets: customers in the records, employees doing the work, vendors whose software is on screen and, in deployment logs, your own users. Each needs a rights answer before delivery.

Documentation duties follow some agents. From 1 January 2027, Colorado's SB26-189, signed in May 2026, requires developers of automated decision-making technology that materially influences consequential decisions, such as lending, insurance, employment or health care, to give deployers documentation that includes the categories of training data [13]. As of October 2026, the Attorney General's implementing rules are an interim draft [14].

In the EU, providers of general-purpose AI models must publish a sufficiently detailed summary of the content used for training under Article 53(1)(d) of the AI Act [15]. The compliance hub and license terms for agent data go further.

For the data it sources, SourceX runs a rights review that checks the business owns or may share the records and that required consents are in place. Names, emails, phone numbers and account numbers are removed or replaced before delivery and the method is recorded, though no de-identification method is perfect; health records must meet HIPAA Safe Harbor or Expert Determination before they are considered for a license. This section is general information, not legal advice.

Scoping your first agent data request

A workable request names the task family, the systems, the observation and action format, how success is labeled, and every use the license must cover.

1. Task family and goal: for example, "release an invoice blocked by a price variance," with in-scope and out-of-scope variants.
2. Systems and surfaces: application, version, customization level, and whether it runs in a browser, desktop client, virtual desktop or terminal.
3. Observations: pixels, accessibility tree or DOM, API request and response payloads, or event log only.
4. Actions: clicks and keystrokes, tool calls with arguments, or business-level actions such as approve or reassign.
5. Outcome label and its source: document status, reopen flag or QA review; see task success labels.
6. Horizon: steps per task, active versus wall-clock time, and handoffs; see long-horizon task records.
7. Exceptions: a minimum share of cases off the happy path.
8. Privacy: pseudonyms consistent across steps and systems, so the agent can follow one customer through a case.
9. License uses: training, environment replication, derived tasks and benchmark publication, plus split rules.
10. Delivery: JSONL with one episode or step per line (UTF-8, one JSON value per line [16]), OCEL or XES for event logs, and a datasheet covering motivation, composition and collection process [17].

Writing an agent data specification expands each item, evaluating an agent data sample covers acceptance, and cost drivers for trajectory data explains pricing variables. For the sourcing route, compare licensed, commissioned or synthetic trajectories.

SourceX sources operational datasets from US companies and manages the commercial process, including licensing agreements and ongoing purchases. The kinds of data it sources include support and sales histories, engineering records, documents, finance and legal workflows, and new recordings of hands-on work. Datasets are sourced on request rather than held in stock, so a request does not guarantee a match, and every release is approved by the supplying company.

SourceX does not train AI models. To start, describe the workflow records your agent needs.

Agent data guides by interface and decision type

Choose the guide that matches how your agent touches software and what it decides.

• Screens and browsers: browser action logs, screen recordings to action-labeled trajectories, task mining data, spreadsheet task trajectories, contact-center desktop activity with transcripts.
• Tools and APIs: MCP tool-use data, policy-following service agent data.
• Back-office procedures: SOP-to-execution pairs, document-to-system entry pairs, runbook execution records.
• Regulated judgment: underwriting decision rationale, claims adjudication decisions, agent-to-human handoff data.
• Adjacent hubs: coding-agent data sits in the code datasets hub, including developer session recordings; evaluation design sits in the evaluation hub. Enterprise AI agent training data for computer use summarizes record types for that use case.

Mistakes that waste an agent data budget

Each of these gaps is visible in the specification before signing and costs more to fix after delivery.

• Screen video with no input events, task boundaries or outcomes. It is raw material, not a trajectory, and inferring actions from pixels afterwards is harder than logging input events at capture.
• "Closed" taken as success. Auto-close rules, reopened cases and later reversals make status fields poor labels unless checked against what happened next.
• Exceptions filtered out for clean demonstrations. The rework and edge cases removed are the ones agents fail on in production.
• A training-only license. Building an environment replica, deriving tasks or publishing a benchmark may fall outside a grant that names only training, so name each use.

Building agents that need real work histories?

Describe the task family, the systems involved, the outcome labels you need and the uses the license must cover. SourceX looks for US businesses that hold matching records, checks the data and the supplier's licensing permissions, and manages the license and delivery; nothing is contracted until a supplier agrees. Specify your workflow dataset.