Skip to content

AI uses for records

Business task reconstruction reproducibility: a three-team agreement test

By SourceX Editorial · Updated

10 sections

Short answer

Business task reconstruction is reproducible when independent teams, given only the same approved documentation packet, rebuild equivalent tasks: the same inputs, step sequence, authority and observable outcome. Test it with at least three blinded teams, a frozen packet, a logged clarification channel and blinded adjudication of every discrepancy, then judge agreement against equivalence margins fixed in advance. This page is the protocol; no results are published yet.

License terms

Business task reconstruction

This page covers business task reconstruction reproducibility: a three-team agreement test.Illustrative

Key takeaways

  • A reconstruction is reproducible when independent teams rebuild the same inputs, sequence, authority and observable outcome from one frozen packet.
  • Score the four elements separately; a packet that carries the inputs well can still leave authority to guesswork.
  • Test the outcome by having every team's grader score the same agent runs, and judge pass-rate differences against equivalence margins set in advance.
  • Treat clarification requests as data: each one marks a place where the packet failed to carry the task.
  • Agreement is not correctness, so the challenge complements validation against the operator who did the work.

What makes a business task reconstruction reproducible?

A reconstruction is reproducible when independent teams, working only from the same approved documentation packet, rebuild tasks that are equivalent where it matters: the same inputs, the same step sequence, the same authority for each consequential step and the same observable outcome. Agreement on wording does not count; agreement on what an agent would be graded on does.

The terms come from science, where they are used precisely. The National Academies' 2019 report defines reproducibility as obtaining consistent results using the same input data, computational steps, methods, code and conditions of analysis, and replicability as obtaining consistent results across studies that each collect their own data. A reconstruction challenge sits on the reproducibility side: the packet is the input data and the method is fixed, but the people applying it change. If a task means different things to different careful readers, the packet, not the readers, is the defect.

This page is the protocol for the Task Reconstruction Reproducibility Challenge, a proposed study. It is not an interview guide or a replay check, which validate one reconstruction against the person who did the work; it asks whether the documentation alone carries the task. No results have been collected or published, and all values below are design parameters or labeled Illustrative.

Why one expert reconstruction is not evidence of a well-specified task

Because the author of a task cannot see the ambiguity in it. The author knows the missing context and supplies it without noticing, so only a reader without that context can show whether the written task stands on its own.

Public benchmarks learned this the hard way. When OpenAI ran a human annotation campaign in which professional software developers screened each sample of the SWE-bench test set, it flagged that many samples had underspecified issue descriptions, leaving it ambiguous what the problem was and how it should be solved. The result, SWE-bench Verified, announced on August 13, 2024, is a subset of 500 problems that software engineers confirmed are solvable. Anthropic's guidance on agent evals makes the same point from the builder's side: each task is meant to be one test whose inputs and success criteria are stated, and when tasks turn out ambiguous or valid solutions are penalized, the eval should be revised.

Expert authorship is an input to quality, not a test of it. GDPval builds its tasks from the representative work of industry professionals with an average of 14 years of experience, across 44 occupations in the 9 sectors that contribute most to U.S. GDP. A reconstruction challenge asks a narrower question that expertise alone cannot settle: would a second qualified team, reading the same packet, build the same task?

The reviewer protocol, from frozen packet to second round

Run the challenge in six fixed steps and change nothing between teams except the people. Every deviation from the protocol is logged as a finding, because a challenge that coaches one team has measured the coaching.

Teams never see each other's work, the task author never speaks to a team, and the second round always uses new teams, because round-one reviewers have already learned the case.

The reviewer protocol, from frozen packet to second round
StepWhoInputRecorded output
1. Freeze the packetStudy coordinatorApproved dataset documentation, source record excerpts, system glossaryPacket version and content hash; a list of what the packet deliberately omits
2. Assign teamsCoordinatorAt least three teams of two qualified reviewers with no contact with the author or each otherRoster, relevant experience and conflict declarations
3. Reconstruct blindEach teamThe frozen packet and a fixed reconstruction formOne completed form per task: inputs, sequence, authority, outcome and confidence
4. Ask in writingEach team, to a neutral deskQuestions about the packet; the desk may only cite packet locations or record that the packet is silentClarification log with element, packet location and answer type
5. Adjudicate blindTwo adjudicators on no teamAnonymized reconstructions and a fixed set of agent runsEach discrepancy classed as material, cosmetic or indeterminate
6. Repair and rerunCoordinator, then fresh teamsPacket version 2 fixing every material gapSecond-round agreement on the same tasks

Four elements every team must rebuild

Score agreement on four elements separately, because a packet can carry the inputs perfectly and still leave authority to guesswork. One overall agreement number would average a clean element with a broken one.

Krippendorff's alpha fits the categorical elements because it works for any number of observers and categories and tolerates missing entries; it compares the disagreement observed with the disagreement expected by chance, as alpha = 1 - Do/De. The outcome element gets a stricter test because it is what a buyer's grader will actually apply.

Four elements every team must rebuild
ElementWhat the team writes downHow agreement is measuredIllustrative material discrepancy
InputsEvery record, field and document the task gives the agent at the startKrippendorff's alpha on per-input include or exclude callsOne team includes the customer's prior dispute history; another treats it as out of scope
SequenceThe ordered steps a correct run must take and the dependencies between themShare of team pairs whose required step orders match after optional steps are set asideTeams disagree on whether the credit check must precede the order release
AuthorityWho may approve each consequential step and where that grant is recordedKrippendorff's alpha on the approver role for each consequential stepOne team lets the agent approve within a limit; another requires a manager every time
Observable outcomeThe system state that proves completion and the state that proves failurePass-rate difference when each team's grader scores the same fixed set of agent runsClaim status reopened versus a reserve change as the completion signal

The agreement table: equivalence margins fixed before the first team starts

Decide in advance how close is close enough, and test for equivalence rather than for the absence of a difference. Lakens' primer on equivalence testing describes the two one-sided tests procedure: set lower and upper bounds from the smallest difference that would matter, and declare equivalence only when both one-sided tests reject. Failing to find a difference between teams in a small sample is not evidence that they agree.

The outcome test works like this. Each team's reconstructed grader scores the same fixed set of agent runs, and the pass rates are compared pair by pair. If every pair falls inside the preset margin by the two one-sided tests, the outcome element is reproducible; if not, the runs that flip verdicts show exactly where the packet is unclear.

The thresholds below are illustrative design choices, not findings. A real study would fix its own in the preregistration, with a written justification for each.

The agreement table: equivalence margins fixed before the first team starts
ElementStatisticPreset threshold (Illustrative)Pass ruleA failure points to
InputsKrippendorff's alpha on include or exclude calls0.80 or higherPooled alpha and every team pair meet the thresholdMissing data dictionary or scope note
SequenceShare of team pairs with matching required step order90 percent of pairs or moreNo material sequence discrepancy left after adjudicationUndocumented dependency or system rule
AuthorityKrippendorff's alpha on approver role per step0.80 or higherNo team grants the agent authority the packet does not showMissing delegation table or approval threshold
OutcomePass-rate difference on the fixed run setWithin plus or minus 5 points by two one-sided testsEvery team pair equivalent at the preset marginAmbiguous completion signal or status code
Clarification loadMaterial clarification requests per taskAt most 1 per task, averaged across teamsAt or below the threshold in round twoPacket relies on knowledge only the author has

Which tasks and agent runs belong in the challenge?

Sample tasks by a rule written before anyone looks at them, and build the fixed agent-run set so that disagreements between teams have a chance to show. A challenge run only on the cleanest tasks, or graded only on runs that obviously pass or fail, will report agreement that the full dataset does not have.

At a glance

Which tasks and agent runs belong in the challenge

The run set does most of the work in the outcome test. Two graders that disagree about the completion signal will still agree on a run that did nothing and on a run that did everything; they part ways on runs that changed one field and not the other. Those near-miss runs are where a packet's ambiguity becomes a different grade, so they must be in the set by design rather than by luck.

Record every exclusion with its reason. A task dropped because teams could not reconstruct it is itself a result, and leaving it out of the denominator would flatter the packet set.

  • Task frame: every task in the candidate dataset version, stratified by workflow type and by the number of consequential steps, with the sampling seed recorded.
  • Deliberate hard cases: a preset share of tasks with multiple systems, approval steps or long time gaps, because those are where packets usually fail.
  • Run set: for each task, agent runs that clearly pass, clearly fail and partly complete, produced before the teams start and frozen with a hash.
  • Grader form: each team writes its grader as a checklist of observable states, so that two graders can be compared run by run.
  • Exclusions: tasks or runs removed after the freeze, each with a reason and the step at which it was removed.

Clarification requests are a measurement, not a nuisance

Every question a team has to ask marks a place where the packet failed to carry the task, so the log is one of the study's main outputs. Count the questions, classify them, and keep the answers out of the reconstruction unless every team receives the same answer at the same time.

The neutral desk answers only by pointing to a packet location or by recording that the packet is silent. A silent answer is logged as a gap rather than filled in, so that round two tests the repaired packet instead of the desk's memory. Five categories cover most requests:

  • Missing fact: the packet never states something the task depends on, such as the currency of an amount.
  • Ambiguous term: a field or status means different things in different systems, such as closed in a CRM and closed in a ticketing tool.
  • Conflicting sources: two packet documents disagree and nothing says which one governs.
  • Missing authority: the packet shows the action but not who may approve it.
  • Out-of-packet knowledge: the question can only be answered by someone who worked the case.

Blinded adjudication: is the discrepancy material?

A discrepancy is material when it would change a grade: applied to the fixed set of agent runs, it flips at least one verdict. Everything else, such as different wording for the same step, is cosmetic and is recorded but does not count against agreement.

Adjudicators see anonymized reconstructions in random order and do not know which team produced which, or which version the task author would prefer. Indeterminate discrepancies, where the adjudicators cannot tell whether a verdict would flip, are reported in their own column rather than folded into either side.

Two adjudicators class each discrepancy independently and their own disagreements are reported, because an adjudication only one person can perform is another reproducibility problem.

Before and after: an illustrative repaired task packet

Illustrative example, not a real case or result: a commercial insurance task asks an agent to decide whether a closed claim should be reopened after a late supplier invoice arrives. In round one, teams A, B and C list the same inputs, but each of sequence, authority and outcome has one team reading it differently.

The repair borrows its structure from Datasheets for Datasets, which groups documentation questions by stage of the dataset lifecycle: motivation, composition, collection process, preprocessing, uses, distribution and maintenance. The export note is a collection-process answer and the delegation excerpt is a composition answer. Round two then runs with fresh teams on packet version 2.

Before and after: an illustrative repaired task packet
ElementPacket version 1 (Illustrative)What the teams didVersion 2 repair
InputsClaim file, late supplier invoice and policy scheduleAll three teams listed the same inputsNo change needed
SequenceLists the invoice check and the reopening without an order; export time zone not statedTeam A read a time-zone shift as a late action and put the coverage recheck after reopeningStates the required order and adds a collection-process note on the export time zone
AuthorityMentions an adjuster limit without the figure or the approver above itTeam B let the agent reopen any claimAdds the delegation table excerpt with the limit and the approver role
Observable outcomeSays the claim should be reopenedTeam C graded on a reserve change instead of the reopened statusNames the status code and the reserve field that together prove completion
Two grids of three teams by four task elements: in packet version 1 one team differs on each of sequence, authority and outcome; in the repaired version 2 every team matches.
Illustrative: the insurance packet before and after repair. Teams A, B and C are examples, not study results.

Limits of the challenge, its status, and where packets come from

Agreement is not correctness. Three teams can share one wrong reading, so the challenge complements validation against the operator who did the work rather than replacing it. It says nothing about tasks outside the packet set, and teams drawn from one background may agree for reasons a different buyer would not share.

SourceX has not run this challenge, and no agreement results exist. A preregistration would fix the task sample and selection rule, team qualifications, the four element statistics and their margins, the fixed agent-run set, the adjudication rules and a commitment to publish negative results. NIST's Generative AI Profile lists verifying the provenance of training data and of test, evaluation, verification and validation data among its suggested actions; a frozen, hashed packet with its clarification log is one concrete form of that record.

For buyers, a reconstruction threshold is a usable acceptance test. Because SourceX sources each dataset from U.S. businesses after a buyer asks rather than from stock, a request can name the task types, the packet contents, the agreement thresholds the packets must meet and the intended use. No packet is delivered until the supplying company has signed an agreement whose scope and terms it approved.

Frequently asked questions

How many independent teams does a reconstruction challenge need?

At least three, so that a split shows up as two against one rather than a coin flip between two readings. More teams narrow the intervals on agreement; the study should fix the number in advance and report every team, including any that dropped out.

Is agreement between teams the same as a correct task?

No. Teams can agree on a reading that is wrong, for example when the packet consistently misstates a policy. Agreement shows the packet carries one meaning; validation against the operator or expert who did the work shows it is the right one.

Can reviewers ask questions during the reconstruction?

Yes, in writing to a neutral desk that can only point to packet locations or record that the packet is silent. Every question is logged and classified, and any substantive fix becomes part of the next packet version that fresh teams receive.

Which agreement statistic should a reconstruction challenge use?

Krippendorff's alpha suits the categorical elements because it handles any number of teams and categories and tolerates missing entries. For the outcome, compare the pass rates that each team's grader gives the same agent runs and test them for equivalence with two one-sided tests against a preset margin.

Has SourceX published results from this challenge?

No. This page is the protocol for a proposed study, and its thresholds and insurance example are illustrative. Results would require rights-cleared case packets, independent teams, blinded adjudication and a preregistered analysis.

Sources

  • The National Academies' 2019 report defines reproducibility as consistent results with the same input data, steps, methods, code and conditions of analysis, and replicability as consistent results across studies that obtain their own data. Source
  • OpenAI's SWE-bench Verified announcement describes professional software developers screening each SWE-bench test sample and finding many underspecified issue descriptions. Source
  • The SWE-bench repository announces SWE-bench Verified (August 13, 2024) as a subset of 500 problems that real software engineers confirmed are solvable. Source
  • Anthropic's guidance on agent evals defines a task as a test with defined inputs and success criteria and says evals should be revised when tasks are ambiguous or valid solutions are penalized. Source
  • The GDPval paper builds tasks from the work of industry professionals averaging 14 years of experience, covering 44 occupations across the top 9 sectors contributing to U.S. GDP. Source
  • Krippendorff's explainer defines alpha as 1 minus observed over expected disagreement, applicable to any number of observers and categories and to data with missing values. Source
  • Lakens' equivalence testing primer describes two one-sided tests against bounds set from the smallest effect size of interest, with equivalence declared when both tests reject and bounds prespecified. Source
  • Datasheets for Datasets groups documentation questions by dataset lifecycle stage: motivation, composition, collection process, preprocessing, uses, distribution and maintenance. Source
  • NIST AI 600-1 suggests verifying the provenance of generative AI training data and of test, evaluation, verification and validation data. Source

Tell us what data you need

Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.

See if your company qualifies