AI uses for records
Agent trajectory evidence attachments: proof that an agent finished
By SourceX Editorial · Reviewed by Noah Loul ·
10 sections
Short answer
A read-back from the system of record, fetched by the harness after the episode and matched to the expected end state, is the proof that an agent finished. Tool-call logs with results show how it got there and screenshots show what it saw, but neither proves completion alone. Link every attachment to its step, hash it at capture, record who captured it, and let a second team replay it.
Agent run
Agent trajectory evidence
Key takeaways
- A read-back from the system of record, taken by the harness after the episode, is the primary proof of completion.
- Tool results, snapshots and screenshots explain the path but cannot prove the end state on their own; the agent's own claim proves nothing.
- Link every attachment to a step and a tool call ID, using trace identifiers the systems already log.
- Hash every file at capture and record which component captured it; agent-made files count as claims.
- A second team should restore the initial state, replay the actions and reach the same end state before the label is trusted.
Which receipts, screenshots and logs substantiate an agent's task completion?
The strongest proof is a receipt read back from the system of record after the episode: the order, ledger entry or ticket state, fetched independently of the agent and matched to the expected end state. Tool-call logs with their results show how the agent got there, and screenshots or page snapshots show what it saw. None proves completion alone, and the agent's own statement that it finished proves nothing.
That distinction matters for anyone buying or building agent trajectory data. A trajectory full of plausible screenshots and a confident final message can still describe a task that never completed, because a confirmation page rendered, a form was half submitted or the write went to the wrong record. Evidence that proves completion has to come from somewhere the agent does not control, and it has to be linked to the exact step that caused the change.
This page gives an evidence attachment manifest a team can adopt as is: the fields, how to link files to steps, how to fix their content with digests, a passing and a failing worked example, and the checks a second team runs to verify the outcome.
What each kind of attachment can prove, and what it cannot
Rank attachments by how independent they are of the agent and how directly they show the end state. Use the table to decide which attachments a trajectory must carry for its outcome label to be trusted, and which are supporting context.
| Attachment | What it can prove | What it cannot prove | Weight as completion evidence |
|---|---|---|---|
| System-of-record read-back | The end state exists in the system that owns it | Which step caused it, unless linked by time and ID | Primary |
| Receipt or confirmation record issued by the system | The system accepted a specific transaction | That the transaction was the right one for the task | Primary when matched to the task |
| Tool call with its returned result | What the agent requested and what the tool answered | That the result persisted or was correct | Strong supporting |
| Before, action and after page snapshots | What the interface showed around each action | That the back end recorded the change | Supporting |
| Screenshot or screen recording | What was visible on screen at a moment | Anything off screen, or that the image is unaltered without a digest | Supporting |
| Copy of an outbound message | What was sent and to whom | That it was delivered or read | Supporting for communication tasks |
| The agent's final message | What the agent claims | Anything about the world | None on its own |
The evidence attachment manifest, field by field
Each episode ships with one manifest that lists every attachment and the claim it supports. The fields below are version 1.0 of this page's manifest, dated 2026-10-18. They map onto JSON Lines, a database table or a column in a trajectory file without renaming.
At a glance
The evidence attachment manifest, field by field
| Field | What it records | Format | Required |
|---|---|---|---|
| manifest_version and episode_id | Schema version and the episode this manifest belongs to | Version string plus stable ID | Yes |
| trace_id and step_id | The distributed trace and the step within the episode | Trace ID plus ordered step number | Yes |
| actor | Who acted at this step | agent, user_simulator, harness, human_reviewer | Yes |
| tool_call_id | Identifier of the tool call the attachment relates to | ID supplied by the model or provider, or harness-assigned | When a tool was called |
| artifact_uri, media_type and digest | Where the file is, what it is, and a hash of its bytes | URI, MIME type, sha256 hex | Yes for every file |
| captured_by and captured_at | Which component made the file, and when | harness, system_export, agent; ISO 8601 time with zone | Yes |
| evidence_role | Which row of the ranking table the file belongs to | read_back, receipt, tool_result, snapshot, screenshot, message_copy, agent_claim | Yes |
| supports_claim | The specific outcome statement this file supports | Short text tied to a success criterion | Yes |
| outcome_check | The check that decides completion, and its result | Check name, expected value, observed value, pass or fail | One per success criterion |
| redaction | What was removed before export and by which rule | Rule ID plus digest of the original kept internally | When content was redacted |
Tie every file to the step that produced it
Link at the step level, not the episode level. An episode-level folder of screenshots tells a reviewer that something happened; a step-level link tells them which action produced which change, which is what lets them spot a write to the wrong record or a confirmation shown before the save completed.
Existing tracing standards already provide the keys. W3C Trace Context defines a trace-id that uniquely identifies a distributed trace as it moves through systems, so the harness, the tool server and the system of record can log the same identifier. OpenTelemetry's generative AI conventions define an execute_tool span with the tool name and a gen_ai.tool.call.id set to the identifier the model or provider supplied, with optional attributes for the call's arguments and result. Use those identifiers as foreign keys in the manifest.
For browser agents, Playwright traces show the shape of good step-level capture: for each action they can record a DOM snapshot before the action, at the moment of input and after it, and filter the console and network logs to the requests made during that action. A manifest entry can point to those three snapshots and to the network response that carried the write.
Fix the bytes: digests, capture source and timestamps
Record a cryptographic digest for every file at capture time, and record which component captured it. Without a digest nobody can tell later whether a screenshot was cropped or a log was edited; without the capture source nobody can tell whether the agent produced its own evidence.
NIST's Secure Hash Standard, FIPS 180-4, specifies SHA-256 among its approved hash algorithms and explains that digests are used to detect whether messages have changed since the digests were generated. The in-toto attestation framework uses the same idea in its resource descriptor: a reference to an artifact must carry at least a URI, a digest or the content itself, and a descriptor with a digest is assumed to refer to an immutable artifact. Copy that rule: no attachment without a uri and a sha256.
Capture source decides weight. Files written by the harness or exported from the system of record count as evidence; files the agent produced or chose to save count only as its claim. Timestamps should carry a time zone and come from the harness clock, and any attachment timed after the episode ended, other than the read-back, should be rejected.
Worked manifest (illustrative): an address change and a customer notice
This manifest is invented for the page and describes no real order system or customer. Task: change the shipping address on an open order to the address the customer sent, then confirm the change to the customer. Success criteria: the order's shipping address equals the new address and the customer received one confirmation message.
- Step 1, actor agent: read the customer message. Attachment: tool_result, captured_by harness, sha256 recorded, supports_claim: new address as stated by the customer.
- Step 2, actor agent: read the order. Attachment: tool_result showing the old address and the order status open.
- Step 3, actor agent: update the order address. Attachments: tool_call_id from the provider; tool_result showing the update accepted; before and after page snapshots, each with a digest.
- Step 4, actor agent: send the confirmation. Attachment: message_copy captured by the harness from the outbound mail stub.
- Step 5, actor harness: read back the order from the system of record after the episode ended. Attachment: read_back, captured_by system_export.
- outcome_check 1: order.shipping_address expected equals the new address, observed equals it, pass. outcome_check 2: messages to customer expected one, observed one, pass.
- Verdict: completed, supported by the read-back and the message copy; the snapshots and tool results explain the path.
A manifest that fails, and the verifier's report
The failing manifest below is also illustrative. It describes the same task but reaches a completed label on evidence that cannot carry it. Each line is what an automated verifier should report, and any one of them is enough to withhold the completed label.
At a glance
A manifest that fails, and the verifier's report
- No read_back entry: the only proof of the address change is a screenshot of a page reading address updated.
- The screenshot's captured_by is agent, so it counts as the agent's claim, not as evidence.
- The after snapshot's recomputed sha256 does not match the manifest, so the file changed after capture.
- Step 3 has no tool_call_id, so the update cannot be matched to a tool span in the trace.
- The message_copy is timed after the episode end recorded by the harness.
- outcome_check 1 is missing; only outcome_check 2 is present, so one success criterion was never tested.
- Verifier result: unverified. The episode can stay in a dataset only with an unverified outcome label, never as a completed example.
Can a second team reconstruct the same state and outcome?
A second team should be able to start from the same initial state, replay the recorded actions and observe the same end state, without trusting anything the first team or the agent wrote. If it cannot, the outcome label is an opinion. The checks below are the verification guide that goes with the manifest.
- Restore the initial state named in the episode, for example an environment snapshot and setup configuration of the kind OSWorld attaches to each task.
- Replay the logged tool calls in order against that state, using the recorded arguments.
- Compare the resulting end state with the read-back. tau2-bench takes this approach: it derives a target end state by replaying reference actions on a fresh environment, compares database end states by hash, and lets any trajectory that reaches an equivalent end state pass.
- For tasks that leave no trace in the system, such as a lookup that only answers a question, check the trace instead. AgentDojo lets a task define a utility check from the function-call traces for exactly this case.
- Recompute every digest and confirm every step_id and tool_call_id resolves to one trace span.
- Record the replay's own result as a second outcome_check, with the replaying team as actor, so agreement or disagreement is part of the data.
Attachments carry more personal data than the log does
Screenshots, page snapshots and message copies routinely capture names, addresses, account numbers and other people's data that a text log would never contain. OpenTelemetry's conventions warn that tool call arguments may contain sensitive information; images and snapshots usually contain more.
Redact before export, and record the redaction in the manifest: the rule applied, and a digest of the redacted file that ships. Keep the original digest only inside the system that holds the original, so an auditor with access can confirm the redacted file was derived from it. Never let redaction remove the fields the outcome check reads; if a read-back cannot be shared even redacted, ship the outcome check result with a statement of who verified it.
Rights matter as much as privacy. A screenshot can include third-party content or another customer's records, and permission to view a screen is not permission to license what is on it. An agent data engineer and a task expert should review the attachment types before a dataset is assembled, and counsel should review anything that includes personal or customer data.
Licensing workflow records that already carry completion evidence
Real business systems already produce much of this evidence: audit trails, status histories, confirmation records and outbound message logs. Trajectories built from records like these can carry read-backs instead of screenshots alone.
Teams that want audit trails and confirmation records rather than screenshots can ask SourceX, which sources such records on request from U.S. businesses; the request-data form names the record types, volume, format, timing and licensing needs. Read-back evidence ships only after the contributing business has approved the scope and terms and signed, and any de-identification of attachments is done by SourceX through approved processing routes.
Frequently asked questions
Is a screenshot enough to prove an AI agent completed a task?
No. A screenshot shows what was on screen at one moment and can show a confirmation for a change that never persisted. Treat it as supporting evidence and require a read-back from the system of record for the completed label.
What is a read-back in agent evaluation?
It is an independent query of the system that owns the result, made by the harness after the episode ends, to see whether the expected end state exists. Because the agent does not produce it, it is the strongest single piece of completion evidence.
Why record a hash for every attachment?
A SHA-256 digest taken at capture time lets anyone later confirm the file has not changed. Without it, a cropped screenshot or an edited log cannot be told apart from the original.
How do you prove completion for a task that changes nothing?
Check the trace rather than the state. For a lookup or an answer-only task, verify that the required tool calls were made with the right arguments and that the answer matches what those results contained.
Should failed or unverified episodes stay in a trajectory dataset?
They can, labelled honestly. Unverified and failed episodes are useful for training verifiers and for error analysis, as long as they are never labelled completed.
Sources
- W3C Trace Context defines a trace-id that uniquely identifies a distributed trace through a system. Source
- OpenTelemetry's generative AI span conventions define an execute_tool span with the tool name and a tool call ID supplied by the model or provider, optional call arguments and result attributes, and a warning that arguments may contain sensitive information. Source
- Playwright traces can record DOM snapshots before an action, at the moment of input and after it, plus screenshots, console logs and network requests filterable by action. Source
- FIPS 180-4 specifies SHA-256 among its approved hash algorithms and states that message digests are used to detect whether messages have been changed since the digests were generated. Source
- The in-toto ResourceDescriptor must specify at least a uri, digest or content, and a descriptor with a digest is assumed to refer to an immutable artifact. Source
- OSWorld task examples name an environment snapshot and setup configuration that define each task's initial state, together with an evaluator. Source
- tau2-bench replays reference actions on a fresh environment to derive a target end state, compares database end states by hash, and passes any trajectory that reaches an equivalent end state. Source
- AgentDojo lets a user task define a utility check from function-call traces, for tasks that leave no trace in the environment. Source
Tell us what data you need
Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.