AI uses for records
Human agent collaborative grading: keep a rescue out of the agent's score
By SourceX Editorial · Updated
10 sections
Short answer
Report two numbers, not one. Grade the team's final outcome as usual, then grade the agent only on the work it delivered before and apart from any human change, and log each human action with its type, the step it altered and whether the run would have failed without it. A rescue keeps the team's score but caps the agent's credit at what it finished unaided.
Agent run
Human agent collaborative
Key takeaways
- An end-state grader cannot tell who produced the final state, so a rescued run can look like a perfect agent run.
- Report a team score and an agent-unaided score side by side, plus load-bearing intervention count and person minutes.
- Deduct credit only when the person did the agent's assigned work; designed approvals and information only the person held cost nothing.
- Record the state before every human action so the run can be replayed without it to test whether the rescue was necessary.
- Check constraints, authority and shared state at every return handoff, or the grader will blame the agent for errors the handoff created.
How do you keep a human rescue out of an agent's score in human agent collaborative grading?
Split the grade. The team score answers whether the work got done; the agent score answers how much of it the agent did on its own. Compute the second one from the state the agent left before each human intervention, and record every intervention in a ledger so a reviewer can see why the two numbers differ.
A single outcome grade cannot make that distinction by construction. Collaborative Gym, an open framework for building and evaluating human-agent collaboration, scores the outcome as delivery times task performance, and its maintainers state plainly that a single task score conflates how much of the outcome came from the agent versus the human. Their answer for large studies is a rating model; for a single episode you need records of who changed what.
This page gives that record: a contribution attribution and intervention label guide, a scoring procedure that uses it, an illustrative worked case and the situations where attribution should not be trusted.
Why does an end-state grader pay the agent for work a person did?
Most agent graders compare the final state of a system with a target state, and a final state carries no author. In the tau2-bench evaluation documentation, the database check replays a reference set of tool calls on a fresh environment, compares database hashes, and passes any trajectory that produces an equivalent end state.
At a glance
Why does an end-state grader pay the agent for work a person did
That design is sound for autonomous runs and wrong for shared ones. The same benchmark lets domains give the simulated user its own tools, which is the dual-control setup its title refers to. Once a person can write to the same records, an end-state match only proves that someone on the team got the records right.
Product interfaces widen the gap. Microsoft's MagenticLite, the successor to Magentic-UI, describes letting the user steer, approve or take over at any point and says it stops to check in before critical actions. Each of those moments is a place where a person's work can land in the final state under the agent's name.
- Takeover: the person performs a step the agent was assigned, then hands control back.
- Silent edit: the person corrects a field in the shared workspace without telling the agent.
- Answer in the instruction: the person's reply to a question contains the solution, not just missing facts.
- Environment repair: the person fixes something outside the agent's reach, such as a locked record or an expired session.
Which human actions are rescues and which are part of the designed workflow?
Deduct credit only when the person did work the agent was responsible for. An approval the workflow requires, or an answer to a question only the person could answer, is collaboration working as designed and should not lower the agent's score. The table sorts the common cases by who owned the fix.
| Label | What the person did | Who owned the fix | Effect on agent credit |
|---|---|---|---|
| checkpoint_pass | Approved a proposed action unchanged at a designed approval gate | Workflow | None; count it as a completed step for the agent |
| checkpoint_edit | Approved only after changing an argument, amount or recipient | Agent | The edited step scores as failed for the agent |
| info_supplied | Answered a question about facts only the person held, such as a preference | Person | None; credit the agent if it asked at the right time |
| answer_leak | Replied with the solution or the exact value the agent should have derived | Agent | Steps that depend on the leaked value score as failed |
| correction | Fixed an error the agent had already committed to the shared state | Agent | The corrected item scores as failed for the agent |
| takeover | Performed one or more assigned steps directly, then returned control | Agent | All steps inside the takeover window score as failed |
| halt | Stopped the agent before a harmful or wrong action executed | Agent | The blocked action scores as failed; the stop itself is not penalized further |
| env_repair | Fixed a blocker outside the agent's permissions or tools | Environment | None, if the agent reported the blocker accurately |
What should the contribution attribution and intervention label guide record?
One row per human action, captured at the moment it happens, with enough state to replay the run without it. The fields below are the minimum that lets a grader compute the agent's unaided share and lets a second reviewer audit the call. Example values are illustrative.
At a glance
What should the contribution attribution and intervention label guide record
| Field | What to record | Illustrative value | Why the grader needs it |
|---|---|---|---|
| intervention_id | Stable identifier for the human action | iv-0007 | Links the ledger row to the trajectory and to review notes |
| step_ref | Index or span of the agent step the action touched | step 14 of 31 | Places the action on the task's dependency graph |
| initiator | Agent request, workflow gate or person on their own | person_unprompted | Separates a well-timed ask from a rescue nobody requested |
| label | One value from the label table above | correction | Determines the credit rule that applies |
| state_before | Snapshot or hash of the shared state before the action | hash of the order table | Lets a grader score the agent's pre-rescue state |
| delta | The exact fields, rows or files the person changed | unit_of_measure on 4 lines | Shows which rubric items the person, not the agent, satisfied |
| necessity | Would the final check fail without this action: yes, no or unknown, plus how that was judged | yes, by replay | Separates load-bearing rescues from cosmetic edits |
| authority_change | Any permission, scope or constraint the person granted or altered | none | Catches rescues that only worked because the rules changed |
| person_minutes | Active time the person spent on the action | 9 | Feeds a cost view of the collaboration, not the agent score |
What must be instrumented before the run rather than reconstructed afterward?
Actor identity on every write, and a snapshot before every human turn. Both are cheap to capture live and close to impossible to recover later, because most business systems record that a value changed, not whether a person or an automation acting under that person's login changed it.
Give the agent its own service identity in each system it writes to, even when it acts on a person's behalf. If the agent and the analyst share one user account, the audit trail in the ERP, CRM or ticketing tool will show a single author and the ledger's delta field has to be guessed from timing.
Take the state snapshot at the boundary of each human turn, not on a timer. A snapshot every few minutes can straddle a takeover and mix the agent's work with the person's. For records too large to copy, a hash of the affected rows plus the list of their keys is enough to grade and to replay against.
- Separate credentials for agent and person in every system the task touches.
- A turn boundary event whenever control passes in either direction.
- A state hash or snapshot at each boundary, scoped to the records the task can change.
- The rubric frozen before the run, so the grader cannot adjust it after seeing who did what.
How do you compute the agent's share once the ledger exists?
Grade the same rubric twice and keep both results. The team score uses the final state. The agent score uses each rubric item's status at the last moment the agent alone controlled it, so an item a person later corrected counts against the agent, and an item the agent finished after a harmless clarification counts for it.
- Break the task into checkable sub-goals from its dependency graph, and grade each sub-goal rather than only the final artifact.
- For each sub-goal, find the last agent-only state before any intervention labeled correction, takeover, answer_leak or checkpoint_edit touched it, and grade that state.
- Where necessity is unknown, replay from state_before with the intervention removed. tau2-bench's no-user ablation, which gives the agent all tools and information upfront with no user interaction, is a useful pattern for building that counterfactual run.
- Report four figures per episode: team score, agent-unaided score, the count of load-bearing interventions and total person minutes.
- Across many sessions with many different people, consider a rating model instead of per-episode arithmetic. CollabSkill, from the Co-Gym authors, treats each session score as agent skill plus human skill plus noise and solves for each separately.
Worked example (illustrative): a price-list update with four silent fixes
This case is invented to show the arithmetic; it does not describe a real company or run. An agent is asked to update 40 product lines in an ERP price list from a supplier spreadsheet, keeping the company's rounding rule and units of measure.
The agent asks once whether prices should round to the nearest cent or the nearest five cents, and the purchasing analyst answers. That is info_supplied with initiator agent_request, so it costs nothing. The agent then writes 34 lines correctly, writes 4 lines in the wrong unit of measure and skips 2 lines whose supplier codes changed.
The analyst fixes the 4 unit errors directly in the ERP without telling the agent, then adds the 2 missing lines. Both are logged: one correction row with a delta of 4 lines, one takeover row covering the 2 skipped lines, both with necessity yes, judged by replaying the final validation against state_before. Together they took 9 active minutes.
The final validation passes on all 40 lines, so the team score is 40 of 40, or 100 percent. The agent-unaided score is 34 of 40, or 85 percent, with 2 load-bearing interventions. Reported as a single number, this run would have looked like a perfect agent.
Which requirement or permission gets lost when control passes back to the agent?
The riskiest moment is the return handoff, because the agent resumes from its own memory of the task while the shared state has moved. If a person edited four lines and the agent still holds the old values, a later bulk step can overwrite the fix, and the grader will then blame the agent for an error the handoff created.
At a glance
Which requirement or permission gets lost when control passes back to the agent
Check these items at every handoff in either direction and record any mismatch as part of the ledger row, not as a separate note.
Checklist · 6 items
- Initial constraints: the original rules, such as rounding or approval limits, are still the ones the agent is applying.
- Authority: any permission the person granted for one step was not silently reused for later steps.
- Shared state: the agent re-read the records the person touched before acting on them again.
- Checkpoints: designed approval gates after the handoff still fired.
- Human changes: every edit the person made appears in the delta field.
- Final verification: the closing check runs on the final state, not on the agent's own summary of it.
0 of 6 ticked. Tick each item as you go.
When should a person's action count in the agent's favor, and when is attribution unreliable?
Give the agent credit for asking well. Co-Gym's process metrics count the agent's confirmation questions that actually elicit a human response, and cases where the person verbally halts the agent. A timely, specific question that unlocks information only the person had is a skill, even though the answer came from the human. Score the ask; do not score the answer.
Per-episode attribution breaks down in three situations. When initiative is spread evenly and people edit continuously, the agent rarely holds a sub-goal alone long enough to grade it. When the counterfactual cannot be replayed, for example because a real external system was involved, necessity becomes a judgment call and should be marked unknown. And when the people in a study differ widely in how practiced they are with AI tools, one episode says as much about the person as about the agent; the CollabSkill authors model that variation in AI literacy as a latent human factor rather than as noise.
In those cases report the team score, the intervention counts and the person minutes, and say that an agent-only score was not computed. A missing number is more honest than a guessed one.
Requesting collaboration records that support this kind of grading
Business records already hold many human-agent collaborations: drafted replies a support lead edited before sending, suggested journal entries an accountant changed, automated dispatch plans a coordinator overrode. They support split scoring only if the source system kept the machine's version, the person's version and timestamps for both. Ask for that before asking for volume.
Requests to SourceX for collaboration records are filled case by case from U.S. businesses that hold them, since nothing sits in a ready-made stock; the request names the scope, volume, format, timing and licensing terms, and it should say that the machine draft and the human edit must both survive export. Each supplying business approves what is included and signs its agreement before any record leaves it.
Frequently asked questions
Is a human approval step a rescue?
No, not when the workflow requires it and the person approves the action unchanged. It becomes a rescue only if the person had to change the action before approving it. Label designed approvals separately so they never reduce the agent's score.
Why not just penalize every human interaction?
Because some interactions are the point of collaboration. A person answering a question about their own preferences supplies information the agent could not have, and an agent that asks at the right moment is doing its job. Penalizing those pushes agents to guess instead of ask.
How do you decide whether a correction was necessary?
Replay the run from the state before the correction with the correction removed, and run the same final check. If the check fails, the correction was load-bearing. If a replay is impossible, record the necessity as unknown rather than guessing.
Can a rating model replace the intervention ledger?
It answers a different question. A rating model such as CollabSkill estimates agent and human skill across many sessions, which suits leaderboards. The ledger explains a single episode, which is what you need to debug an agent or audit a specific score.
What should the final report show for each run?
Show the team score, the agent-unaided score, the count of load-bearing interventions and the person minutes spent. Showing both scores side by side makes rescue-inflated results visible at a glance.
Sources
- Collaborative Gym scores outcomes as delivery times task performance, measures initiative and controlled autonomy, and notes that a single task score conflates agent and human contributions; CollabSkill models a session score as agent skill plus human skill plus noise. Source
- Collaborative Gym is a framework for enabling and evaluating human-agent collaboration, with an evaluation suite covering both collaboration outcomes and processes. Source
- CollabSkill uses a Bayesian skill rating system to separate human and agent contributions and treats variation in workers' AI literacy as a latent human factor. Source
- tau2-bench's database check replays reference actions on a fresh environment and passes any trajectory that produces an equivalent database end state. Source
- tau2-bench domains can include a set of user tools for the user simulator, and the benchmark evaluates agents in a dual-control environment. Source
- tau2-bench's telecom domain supports a no-user ablation in which the model receives all tools and information upfront without user interaction. Source
- MagenticLite, the next generation of Magentic-UI, lets users steer, approve or take over at any point and stops to check in before critical actions. Source
Tell us what data you need
Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.