AI uses for records
Agent trajectory action provenance: telling human from agent steps
By SourceX Editorial · Updated
9 sections
Short answer
To distinguish human and agent actions in trajectory provenance, label every step with the identity that executed it, its actor class, the principal it acted for, any approver, and a confidence level backed by linked evidence. Do not trust the system's author field alone: many platforms credit the person whose credentials were used, so agent, rule and integration steps can look human.
Agent run
Agent trajectory action
Key takeaways
- A system's author field records whom the platform credited, which can be the person whose credentials an agent or app used.
- Label each step with executor, actor class, principal, approver, initiator and a confidence level tied to evidence inside the package.
- Confirmed labels need direct evidence such as an agent span, a tool call ID, an app token or a rule channel; timing patterns only support inferred labels.
- Approvals and delegated actions are separate steps and separate fields, so credit and responsibility do not collapse into one name.
- The acceptance test is a second team reaching the same labels from the evidence alone, reported by actor class and confidence.
How do you distinguish human and agent actions in a trajectory?
Record the actor of every step as a claim with evidence, not as a single name. Each step needs the identity that executed it, the kind of actor it was, the party it acted for, who approved it if anyone did, and how confident the label is. Business systems often attribute work to a person when software did it, so a bare user field is not proof.
The distinction matters because a trajectory is used for three jobs that all depend on it. Training uses agent steps as targets and human steps as context, or the reverse for a human-demonstration set. Grading credits the agent only for what the agent did. Debugging needs to know whether a bad step came from the model, a rule, an integration or the person who took over.
The W3C provenance model gives a vocabulary that fits this problem well. PROV-DM defines three kinds of agent: a person, an organization, and a software agent that is running software. PROV-O adds delegation, written actedOnBehalfOf, for the case where one agent acts for another that also bears responsibility. A trajectory actor label is a small, step-level application of the same idea.
Mixed episodes are now the norm rather than the exception. A support case may start with a routing rule, pass through an agent that drafts and looks things up, reach a person who approves a refund, and end with a scheduled job that closes it. Each of those steps may be valid training data, but only for the right model and only when the label says which actor produced it.
Where business systems blur who acted
Most systems record an author field that answers a narrower question than the one a trajectory buyer asks. The author is whoever the system decided to credit, which may be the person whose credentials were used rather than the software that made the change.
At a glance
Where business systems blur who acted
Two documented examples show the gap. GitHub says a GitHub App acting on behalf of a user has its API requests attributed to that user, with the app shown only as a badge on the user's avatar; when the app authenticates as an installation instead, the activity is attributed to the app. Zendesk lets an API call set author_id on a comment so another user appears as the author, while the authenticated user making the request is treated as the updater of the ticket. In both cases the visible author and the executing actor can differ on the same record.
The reverse error also happens. When an integration account writes on behalf of many people, a dataset can label genuine human decisions as automation simply because the integration relayed them. A help desk app that posts a phone agent's notes through an API, for example, may show the integration as the author of every note. Treat each of the patterns below as a reason to look for a second piece of evidence before labeling.
- Delegated credentials: an agent or integration calls an API with a person's token, so the record names the person.
- Shared service accounts: one integration user writes for several automations, rules and agents, so the name says almost nothing.
- Business rules: Zendesk records how a change was made in the via object, and a rule channel marks triggers and automations rather than people.
- Suggested then accepted: an assistant drafts a reply and a person sends it unchanged, edited or rewritten, and the system logs only the sender.
- Takeovers: a person steps in mid-episode, and later steps carry the same session or conversation ID as the agent's.
- Backfills and migrations: bulk jobs rewrite fields months later and can overwrite the original author or timestamp.
The actor attribution schema with uncertain labels
The schema below adds eight fields to every step of a trajectory. It keeps the system's own author value untouched and puts the interpretation next to it, so a reviewer can always see what the source said and what the labeler concluded.
Use controlled values for actor_class and confidence; free text in those fields defeats filtering. Keep evidence_refs as pointers to records in the same package, such as an audit row ID, a span ID or a tool call ID, so a second team can check every label without asking the supplier.
| Field | Allowed values or format | What it answers | Validation rule |
|---|---|---|---|
| source_author | Raw user or account value from the system, unchanged | Whom the source system credited | Required; never edited after export |
| executor_id | Stable pseudonymous ID of the identity that performed the step | Which identity actually executed the action | Required; must resolve in the actor registry |
| actor_class | human, ai_agent, rule_automation, integration, batch_job, unknown | What kind of actor executed the step | Required; one value; unknown allowed only with confidence unresolved |
| on_behalf_of | executor_id of the principal, or none | Whose authority the step used | Required when actor_class is ai_agent or integration and credentials belonged to a person |
| approved_by | executor_id of the approver, or none, with approval step ID | Whether a person confirmed the action before it ran | If present, approval timestamp must precede the action timestamp |
| initiated_by | executor_id that proposed or triggered the action | Who decided to act, as distinct from who executed | Required; may equal executor_id |
| confidence | confirmed, inferred, unresolved | How strong the evidence for actor_class is | confirmed needs at least one direct evidence reference |
| evidence_refs | List of record IDs inside the package | Which records support the label | Every ID must resolve; empty only when confidence is unresolved |
Which evidence earns a confirmed label, and when to say unresolved
A label is confirmed only when a record that the actor could not easily forge names it directly. Everything else is inferred or unresolved, and the dataset should say so rather than guess. Buyers can then filter: train only on confirmed agent steps, evaluate on confirmed and inferred, and set unresolved steps aside.
The OpenTelemetry generative AI conventions are a useful direct source when the agent was instrumented: invoke_agent spans can carry gen_ai.agent.id and gen_ai.agent.name, and gen_ai.conversation.id links messages in one session. Those conventions are still marked as in development, so record which version produced the spans.
Write the inference rules down. A rule such as 'twenty or more field changes by one account within a second are non-human' is reasonable, but it is a rule the supplier chose, and a buyer needs to see it to judge how many inferred labels it produced. Each inferred label should name the rule that produced it, so changing a rule later can be traced to the steps it affected.
| Evidence | Typical source | Supports | Label it as |
|---|---|---|---|
| Agent span or tool call ID matching the step | Agent tracing with gen_ai.agent.id, tool call logs | ai_agent | confirmed |
| Installation or app token on the request | App installation audit entries, API gateway logs | integration or ai_agent, by registry | confirmed |
| Rule or automation channel on the change | Zendesk via object with rule channel, workflow rule logs | rule_automation | confirmed |
| Person's token plus an app badge or delegation record | GitHub user-to-server activity, OAuth grant logs | ai_agent or integration on behalf of a person | confirmed for delegation, inferred for which agent |
| Timing pattern only, such as sub-second bursts | Field history, event timestamps | non-human actor | inferred |
| Wording or style of a message | Comment or email body | nothing reliable | unresolved |
| Shared service account with no further trace | Integration user on audit rows | unknown actor_class | unresolved |
Delegation and approvals: when the agent proposes and a person clicks
Split the decision from the execution. Many business agents propose an action and a person approves it, or a person asks for something and the agent carries it out. One actor field cannot describe either case, which is why the schema keeps initiated_by, executor_id and approved_by apart.
The Model Context Protocol specification makes approval a design expectation: it says there should always be a human in the loop with the ability to deny tool invocations, and that applications should present confirmation prompts for operations. When a trajectory comes from such an application, each confirmation is a step of its own, with a person as executor and the agent's proposal as the step it approved.
Three patterns cover most episodes. Agent proposes, person approves, agent executes: initiated_by and executor_id are the agent, approved_by is the person. Person requests, agent executes on the person's credentials: executor_id is the agent, on_behalf_of is the person. Agent drafts, person edits and sends: two steps, an agent draft and a human send, with the edit kept as a diff so a grader can see how much of the final text the agent wrote.
Worked example: a refund episode labeled step by step
The record below is illustrative. It describes an invented support refund handled in a help desk with an AI agent, a business rule and a person; no real customer or system data is used. It shows how the same episode reads when every step carries the schema fields.
Three details are worth noticing. Step s5 shows the team lead as source_author because the agent used the lead's delegated credentials, yet the executor is the agent and the approval at s4 is linked. Step s6 is credited to a person who sent text the agent drafted, so the edit diff decides how much of that reply counts as agent output. Step s7 stays unresolved: the evidence cannot say whether a person, a rule or the agent closed the ticket, and guessing would contaminate the outcome label.
| Step | What happened | source_author | actor_class and confidence | Key evidence |
|---|---|---|---|---|
| s1 | Ticket arrives by email; a rule tags it as a refund request | System | rule_automation, confirmed | Audit event with rule channel and rule ID |
| s2 | Agent reads the order and checks return eligibility through two tool calls | Integration user | ai_agent, confirmed | Agent span ID and two tool call IDs |
| s3 | Agent proposes a refund and requests approval | Integration user | ai_agent, confirmed | Proposal message ID linked to span |
| s4 | Team lead approves in the agent console | Team lead | human, confirmed | Approval event with timestamp before s5 |
| s5 | Refund is issued through the payments tool | Team lead | ai_agent on behalf of team lead, confirmed | Tool call ID, approval reference s4 |
| s6 | Reply sent to the customer after a person edited the agent draft | Support agent | human sender of an ai_agent draft, confirmed | Draft ID, sent comment ID, edit diff |
| s7 | Status set to solved two hours later | Support agent | unknown, unresolved | Audit row only; no span, no rule channel |
A failing record, and the checks a validator should run
Now the same episode as many exports deliver it: every step carries only source_author, so s2, s3 and s5 read as human work by the integration user or the team lead, and s6 reads as fully human. A model trained on it would learn the agent's steps as human demonstrations, and a grader would credit the team lead for a refund the agent executed.
The checks below catch that record and most partial ones. They can run as a script over the package before a buyer accepts a sample, and each failure points to a specific step.
A good validator fails loudly on structure and reports quietly on uncertainty. Missing fields, unresolvable evidence and approvals out of order are errors that block acceptance. A high unresolved share is a finding for the buyer to weigh, not an error, because some systems simply never recorded who acted.
- Every step has actor_class, initiated_by, executor_id, confidence and source_author.
- Every evidence_refs ID resolves to a record inside the package, and confirmed labels have at least one.
- No step is labeled ai_agent with confidence confirmed unless a span, tool call or app token supports it.
- Where on_behalf_of is set, the principal exists in the actor registry and the credential type is recorded.
- Every approval precedes the step it approved, and that step references it.
- Steps whose source_author is a shared integration account are never labeled human.
- The share of unresolved steps is reported per episode, so buyers can drop episodes above their threshold.
- The actor registry maps each pseudonymous executor_id to a class without exposing names.
Can a second team arrive at the same actor labels?
That is the real acceptance test. Give a second team the package without the labels, have them label a sample of episodes from the evidence alone, and compare. Steps where the teams disagree are either missing evidence or ambiguous by nature, and both outcomes belong in the documentation.
Agreement should be reported by actor_class and by confidence level, not as one number. Disagreement on confirmed steps signals a broken evidence chain or a bad registry entry. Disagreement on inferred steps is expected and tells buyers how far to trust the inference rules.
Version the schema and the inference rules alongside the data. If a supplier later adds agent tracing to a system, older episodes keep their original evidence and labels, and new episodes carry stronger ones; mixing the two silently would make the dataset look more certain than it is.
Sample size for the relabeling test depends on how the data will be used. For a training set, a few dozen episodes per source system can show whether the evidence chain holds. For an evaluation set where credit decides a result, every graded step should be checked.
Asking for actor provenance in a trajectory data request
Name the schema in the request rather than asking for trajectories in general. State which actor classes you need as training targets, which confidence levels you will accept, and whether you need the approval and delegation fields. Ask for the actor registry, the evidence types available per source system, and a sample episode with a failing check left in, so you see how gaps are reported.
Actor labels are a technical record, not a rights record. Knowing that an agent or a contractor performed a step does not settle who may license the resulting data; that is a separate question for contracts and counsel.
Trajectory sourcing through SourceX starts from the buyer's request: datasets come from U.S. businesses on request, and the form asks for scope, volume, language, format, timing and licensing needs, which is where an actor attribution requirement belongs. On the supply side, SourceX supports the company licensing the records through sourcing, rights review, contracting, preparation and delivery.
Frequently asked questions
What is action provenance in an agent trajectory?
Action provenance is the record of who or what performed each step, for whom, with whose approval, and on what evidence. It goes beyond the author field a system stores, because that field often names the person whose credentials were used rather than the software that acted.
Why can't I trust the user field in an audit log?
Because systems attribute changes by credential, not by intent. GitHub attributes an app's requests made on behalf of a user to that user, and Zendesk can show a comment author who is not the updater. A user field is evidence, not a label.
Should uncertain steps be dropped from a trajectory dataset?
Keep them in the episode and mark them unresolved. Dropping them breaks the sequence a reviewer needs to replay the task. Buyers can exclude unresolved steps from training targets while keeping them as context.
How should a step that a person approved be labeled?
As two steps: the approval, executed by the person, and the action, executed by the agent with approved_by pointing to the approval. That keeps credit and responsibility separate and lets a grader check that approval came first.
Does knowing the actor tell you who owns the data?
No. Actor labels describe how a record was produced. Ownership and licensing rights depend on contracts, platform terms and law, and counsel reviews them separately.
Sources
- PROV-DM defines three agent types, person, organization and software agent, where a software agent is running software. Source
- PROV-O expresses delegation with actedOnBehalfOf, where the delegate acts for another agent that also bears responsibility. Source
- GitHub attributes API requests a GitHub App makes on behalf of a user to that user, showing the app as a badge on the user's avatar. Source
- API requests made by a GitHub App authenticated as an installation are attributed to the app. Source
- Setting author_id on a Zendesk comment makes that user appear as the author, while the authenticated API user is treated as the ticket updater. Source
- The Zendesk via object records how a ticket or event was created, and the rule via type covers triggers and automations. Source
- Zendesk ticket audits record the author of each audit and a via object describing how the change was made. Source
- OpenTelemetry generative AI conventions define invoke_agent spans with gen_ai.agent.id, gen_ai.agent.name and gen_ai.conversation.id, with Development status. Source
- The Model Context Protocol says there should always be a human in the loop able to deny tool invocations, with confirmation prompts for operations. Source
Tell us what data you need
Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.