Skip to content

AI uses for records

Human agent collaborative grading: keep a rescue out of the agent's score

By SourceX Editorial · Updated

10 sections

Short answer

Report two numbers, not one. Grade the team's final outcome as usual, then grade the agent only on the work it delivered before and apart from any human change, and log each human action with its type, the step it altered and whether the run would have failed without it. A rescue keeps the team's score but caps the agent's credit at what it finished unaided.

Agent run

Human agent collaborative

This page covers human agent collaborative grading: keep a rescue out of the agent's score.Illustrative

Key takeaways

  • An end-state grader cannot tell who produced the final state, so a rescued run can look like a perfect agent run.
  • Report a team score and an agent-unaided score side by side, plus load-bearing intervention count and person minutes.
  • Deduct credit only when the person did the agent's assigned work; designed approvals and information only the person held cost nothing.
  • Record the state before every human action so the run can be replayed without it to test whether the rescue was necessary.
  • Check constraints, authority and shared state at every return handoff, or the grader will blame the agent for errors the handoff created.

How do you keep a human rescue out of an agent's score in human agent collaborative grading?

Split the grade. The team score answers whether the work got done; the agent score answers how much of it the agent did on its own. Compute the second one from the state the agent left before each human intervention, and record every intervention in a ledger so a reviewer can see why the two numbers differ.

A single outcome grade cannot make that distinction by construction. Collaborative Gym, an open framework for building and evaluating human-agent collaboration, scores the outcome as delivery times task performance, and its maintainers state plainly that a single task score conflates how much of the outcome came from the agent versus the human. Their answer for large studies is a rating model; for a single episode you need records of who changed what.

This page gives that record: a contribution attribution and intervention label guide, a scoring procedure that uses it, an illustrative worked case and the situations where attribution should not be trusted.

Why does an end-state grader pay the agent for work a person did?

Most agent graders compare the final state of a system with a target state, and a final state carries no author. In the tau2-bench evaluation documentation, the database check replays a reference set of tool calls on a fresh environment, compares database hashes, and passes any trajectory that produces an equivalent end state.

At a glance

Why does an end-state grader pay the agent for work a person did

That design is sound for autonomous runs and wrong for shared ones. The same benchmark lets domains give the simulated user its own tools, which is the dual-control setup its title refers to. Once a person can write to the same records, an end-state match only proves that someone on the team got the records right.

Product interfaces widen the gap. Microsoft's MagenticLite, the successor to Magentic-UI, describes letting the user steer, approve or take over at any point and says it stops to check in before critical actions. Each of those moments is a place where a person's work can land in the final state under the agent's name.

  • Takeover: the person performs a step the agent was assigned, then hands control back.
  • Silent edit: the person corrects a field in the shared workspace without telling the agent.
  • Answer in the instruction: the person's reply to a question contains the solution, not just missing facts.
  • Environment repair: the person fixes something outside the agent's reach, such as a locked record or an expired session.

Which human actions are rescues and which are part of the designed workflow?

Deduct credit only when the person did work the agent was responsible for. An approval the workflow requires, or an answer to a question only the person could answer, is collaboration working as designed and should not lower the agent's score. The table sorts the common cases by who owned the fix.

Which human actions are rescues and which are part of the designed workflow?
LabelWhat the person didWho owned the fixEffect on agent credit
checkpoint_passApproved a proposed action unchanged at a designed approval gateWorkflowNone; count it as a completed step for the agent
checkpoint_editApproved only after changing an argument, amount or recipientAgentThe edited step scores as failed for the agent
info_suppliedAnswered a question about facts only the person held, such as a preferencePersonNone; credit the agent if it asked at the right time
answer_leakReplied with the solution or the exact value the agent should have derivedAgentSteps that depend on the leaked value score as failed
correctionFixed an error the agent had already committed to the shared stateAgentThe corrected item scores as failed for the agent
takeoverPerformed one or more assigned steps directly, then returned controlAgentAll steps inside the takeover window score as failed
haltStopped the agent before a harmful or wrong action executedAgentThe blocked action scores as failed; the stop itself is not penalized further
env_repairFixed a blocker outside the agent's permissions or toolsEnvironmentNone, if the agent reported the blocker accurately

What should the contribution attribution and intervention label guide record?

One row per human action, captured at the moment it happens, with enough state to replay the run without it. The fields below are the minimum that lets a grader compute the agent's unaided share and lets a second reviewer audit the call. Example values are illustrative.

At a glance

What should the contribution attribution and intervention label guide record

What should the contribution attribution and intervention label guide record?
FieldWhat to recordIllustrative valueWhy the grader needs it
intervention_idStable identifier for the human actioniv-0007Links the ledger row to the trajectory and to review notes
step_refIndex or span of the agent step the action touchedstep 14 of 31Places the action on the task's dependency graph
initiatorAgent request, workflow gate or person on their ownperson_unpromptedSeparates a well-timed ask from a rescue nobody requested
labelOne value from the label table abovecorrectionDetermines the credit rule that applies
state_beforeSnapshot or hash of the shared state before the actionhash of the order tableLets a grader score the agent's pre-rescue state
deltaThe exact fields, rows or files the person changedunit_of_measure on 4 linesShows which rubric items the person, not the agent, satisfied
necessityWould the final check fail without this action: yes, no or unknown, plus how that was judgedyes, by replaySeparates load-bearing rescues from cosmetic edits
authority_changeAny permission, scope or constraint the person granted or alterednoneCatches rescues that only worked because the rules changed
person_minutesActive time the person spent on the action9Feeds a cost view of the collaboration, not the agent score

What must be instrumented before the run rather than reconstructed afterward?

Actor identity on every write, and a snapshot before every human turn. Both are cheap to capture live and close to impossible to recover later, because most business systems record that a value changed, not whether a person or an automation acting under that person's login changed it.

Give the agent its own service identity in each system it writes to, even when it acts on a person's behalf. If the agent and the analyst share one user account, the audit trail in the ERP, CRM or ticketing tool will show a single author and the ledger's delta field has to be guessed from timing.

Take the state snapshot at the boundary of each human turn, not on a timer. A snapshot every few minutes can straddle a takeover and mix the agent's work with the person's. For records too large to copy, a hash of the affected rows plus the list of their keys is enough to grade and to replay against.

  • Separate credentials for agent and person in every system the task touches.
  • A turn boundary event whenever control passes in either direction.
  • A state hash or snapshot at each boundary, scoped to the records the task can change.
  • The rubric frozen before the run, so the grader cannot adjust it after seeing who did what.

How do you compute the agent's share once the ledger exists?

Grade the same rubric twice and keep both results. The team score uses the final state. The agent score uses each rubric item's status at the last moment the agent alone controlled it, so an item a person later corrected counts against the agent, and an item the agent finished after a harmless clarification counts for it.

  • Break the task into checkable sub-goals from its dependency graph, and grade each sub-goal rather than only the final artifact.
  • For each sub-goal, find the last agent-only state before any intervention labeled correction, takeover, answer_leak or checkpoint_edit touched it, and grade that state.
  • Where necessity is unknown, replay from state_before with the intervention removed. tau2-bench's no-user ablation, which gives the agent all tools and information upfront with no user interaction, is a useful pattern for building that counterfactual run.
  • Report four figures per episode: team score, agent-unaided score, the count of load-bearing interventions and total person minutes.
  • Across many sessions with many different people, consider a rating model instead of per-episode arithmetic. CollabSkill, from the Co-Gym authors, treats each session score as agent skill plus human skill plus noise and solves for each separately.

Worked example (illustrative): a price-list update with four silent fixes

This case is invented to show the arithmetic; it does not describe a real company or run. An agent is asked to update 40 product lines in an ERP price list from a supplier spreadsheet, keeping the company's rounding rule and units of measure.

The agent asks once whether prices should round to the nearest cent or the nearest five cents, and the purchasing analyst answers. That is info_supplied with initiator agent_request, so it costs nothing. The agent then writes 34 lines correctly, writes 4 lines in the wrong unit of measure and skips 2 lines whose supplier codes changed.

The analyst fixes the 4 unit errors directly in the ERP without telling the agent, then adds the 2 missing lines. Both are logged: one correction row with a delta of 4 lines, one takeover row covering the 2 skipped lines, both with necessity yes, judged by replaying the final validation against state_before. Together they took 9 active minutes.

The final validation passes on all 40 lines, so the team score is 40 of 40, or 100 percent. The agent-unaided score is 34 of 40, or 85 percent, with 2 load-bearing interventions. Reported as a single number, this run would have looked like a perfect agent.

Which requirement or permission gets lost when control passes back to the agent?

The riskiest moment is the return handoff, because the agent resumes from its own memory of the task while the shared state has moved. If a person edited four lines and the agent still holds the old values, a later bulk step can overwrite the fix, and the grader will then blame the agent for an error the handoff created.

At a glance

Which requirement or permission gets lost when control passes back to the agent

Check these items at every handoff in either direction and record any mismatch as part of the ledger row, not as a separate note.

Checklist · 6 items

  1. Initial constraints: the original rules, such as rounding or approval limits, are still the ones the agent is applying.
  2. Authority: any permission the person granted for one step was not silently reused for later steps.
  3. Shared state: the agent re-read the records the person touched before acting on them again.
  4. Checkpoints: designed approval gates after the handoff still fired.
  5. Human changes: every edit the person made appears in the delta field.
  6. Final verification: the closing check runs on the final state, not on the agent's own summary of it.

0 of 6 ticked. Tick each item as you go.

When should a person's action count in the agent's favor, and when is attribution unreliable?

Give the agent credit for asking well. Co-Gym's process metrics count the agent's confirmation questions that actually elicit a human response, and cases where the person verbally halts the agent. A timely, specific question that unlocks information only the person had is a skill, even though the answer came from the human. Score the ask; do not score the answer.

Per-episode attribution breaks down in three situations. When initiative is spread evenly and people edit continuously, the agent rarely holds a sub-goal alone long enough to grade it. When the counterfactual cannot be replayed, for example because a real external system was involved, necessity becomes a judgment call and should be marked unknown. And when the people in a study differ widely in how practiced they are with AI tools, one episode says as much about the person as about the agent; the CollabSkill authors model that variation in AI literacy as a latent human factor rather than as noise.

In those cases report the team score, the intervention counts and the person minutes, and say that an agent-only score was not computed. A missing number is more honest than a guessed one.

Requesting collaboration records that support this kind of grading

Business records already hold many human-agent collaborations: drafted replies a support lead edited before sending, suggested journal entries an accountant changed, automated dispatch plans a coordinator overrode. They support split scoring only if the source system kept the machine's version, the person's version and timestamps for both. Ask for that before asking for volume.

Requests to SourceX for collaboration records are filled case by case from U.S. businesses that hold them, since nothing sits in a ready-made stock; the request names the scope, volume, format, timing and licensing terms, and it should say that the machine draft and the human edit must both survive export. Each supplying business approves what is included and signs its agreement before any record leaves it.

Frequently asked questions

Is a human approval step a rescue?

No, not when the workflow requires it and the person approves the action unchanged. It becomes a rescue only if the person had to change the action before approving it. Label designed approvals separately so they never reduce the agent's score.

Why not just penalize every human interaction?

Because some interactions are the point of collaboration. A person answering a question about their own preferences supplies information the agent could not have, and an agent that asks at the right moment is doing its job. Penalizing those pushes agents to guess instead of ask.

How do you decide whether a correction was necessary?

Replay the run from the state before the correction with the correction removed, and run the same final check. If the check fails, the correction was load-bearing. If a replay is impossible, record the necessity as unknown rather than guessing.

Can a rating model replace the intervention ledger?

It answers a different question. A rating model such as CollabSkill estimates agent and human skill across many sessions, which suits leaderboards. The ledger explains a single episode, which is what you need to debug an agent or audit a specific score.

What should the final report show for each run?

Show the team score, the agent-unaided score, the count of load-bearing interventions and the person minutes spent. Showing both scores side by side makes rescue-inflated results visible at a glance.

Sources

  • Collaborative Gym scores outcomes as delivery times task performance, measures initiative and controlled autonomy, and notes that a single task score conflates agent and human contributions; CollabSkill models a session score as agent skill plus human skill plus noise. Source
  • Collaborative Gym is a framework for enabling and evaluating human-agent collaboration, with an evaluation suite covering both collaboration outcomes and processes. Source
  • CollabSkill uses a Bayesian skill rating system to separate human and agent contributions and treats variation in workers' AI literacy as a latent human factor. Source
  • tau2-bench's database check replays reference actions on a fresh environment and passes any trajectory that produces an equivalent database end state. Source
  • tau2-bench domains can include a set of user tools for the user simulator, and the benchmark evaluates agents in a dual-control environment. Source
  • tau2-bench's telecom domain supports a no-user ablation in which the model receives all tools and information upfront without user interaction. Source
  • MagenticLite, the next generation of Magentic-UI, lets users steer, approve or take over at any point and stops to check in before critical actions. Source

Tell us what data you need

Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.

See if your company qualifies