AI uses for records
Expert disagreement in business benchmarks: a map and a ranking test
By SourceX Editorial · Updated
9 sections
Short answer
To assess expert disagreement in a business benchmark, have qualified reviewers grade every task blind and independently, then sort each split into missing facts, ambiguous policy, legitimate alternatives or grader inconsistency. Repair the first two and re-grade the last; keep legitimate alternatives as admissible answers. Rescore every model under strict and lenient answer keys and report how often rankings flip. SourceX has not yet run this study.
Evaluation split
Expert disagreement business
Key takeaways
- Expert disagreement is measurement information: the work is deciding which splits are defects to fix and which are valid answers to keep.
- Blind, independent grades frozen before any consensus meeting are the only record of how much experts actually disagreed.
- Four causes cover the splits that matter, missing facts, ambiguous policy, legitimate alternatives and grader inconsistency, and each has a different remedy.
- Scoring a model under the strictest and most lenient admissible answer key gives a label-bound interval; overlapping intervals mean the answer key, not the models, decides the ranking.
- GDPval reports that its expert graders agreed with each other 71 percent of the time, so even strong benchmarks carry answer-key uncertainty worth publishing.
How should an AI team assess expert disagreement in a business benchmark?
Treat disagreement as a measurement to report, not noise to vote away. Collect blind, independent grades from qualified reviewers, classify every split by its cause, repair the splits the benchmark itself created, keep the answers experts can legitimately defend, and then test whether the model ranking survives every defensible answer key.
At a glance
How should an AI team assess expert disagreement in a business benchmark
Business tasks make this unavoidable. Whether a refund exception was justified, whether an invoice line should be capitalized or how firmly a collections notice may be worded depends on facts, policy and judgment, and two experienced people can read the same case differently. A benchmark that hides that split behind one answer key reports more certainty than its own graders had.
Anthropic's January 2026 guide to agent evaluations sets the bar for a well-specified task at agreement between two domain experts who grade it without conferring and arrive at the same pass or fail verdict; whatever ambiguity the specification leaves turns into noise in the metrics. The Expert Disagreement Map finds out, item by item, which side of that bar each task sits on and what to do when it falls short. The process has six steps:
- Select tasks whose records carry the evidence a qualified reviewer needs.
- Grade blind and independently, and freeze the grades before any discussion.
- Classify each split as missing facts, ambiguous policy, a legitimate alternative or grader inconsistency.
- Repair, re-grade or keep, and log every adjudication with its reason.
- Rescore every model under strict and lenient answer keys and under sampled resolutions.
- Publish split rates, label-bound intervals and the rank-flip share beside the headline score.
Which business tasks are complete enough for qualified reviewers to grade?
Only tasks whose record holds everything a qualified reviewer would need to decide: the request, the facts available at the time, the policy version in force and, where one exists, the recorded outcome. If a reviewer has to guess at a missing fact, the resulting disagreement measures a gap in the record, not the difficulty of the task.
Screening for completeness is already standard in serious benchmarks. To build SWE-bench Verified, published in August 2024, OpenAI worked with 93 software developers experienced in Python to screen 1,699 randomly sampled tasks from the original test set, and kept a 500-task subset that its annotators verified as non-problematic. Business tasks need the same screen, with business evidence in place of code.
Write the selection rules before grading starts and apply them to every candidate, including the tasks that end up excluded. The excluded set is itself a finding: if most refund exception cases fail the screen because approval notes were never captured, that describes the available records, and it belongs in the report. The entry criteria:
- Request and context: the customer, vendor or employee request as it arrived, with the attachments a reviewer would see.
- Facts at decision time: records that existed when the decision was made, not ones added later.
- Policy in force: the rule text and its version or effective date for that case.
- Outcome where one exists: what happened afterwards, such as a reversal, a reopened case, a dispute or an audit finding.
- Reviewer qualification: the role and experience a grader needs, written for each task family.
Why must experts grade independently before any consensus meeting?
Because the first discussion destroys the evidence. Once reviewers hear each other's reasoning, later grades converge, and the benchmark loses its only record of how much qualified people disagreed before anyone was persuaded.
The protocol is short and strict. Each reviewer sees the task record and the candidate answers without model names or other reviewers' grades, and records a verdict, a short rationale, the evidence relied on and a confidence rating. Grades are timestamped and frozen. Adjudication happens afterwards and is logged separately, so the pre-consensus split rate can always be recomputed from the frozen grades.
Even strong expert panels split often. GDPval, OpenAI's evaluation of AI deliverables against work by industry professionals, has graders from the same occupations rank unlabeled human and AI deliverables. Its paper reports that the human expert graders agreed with each other 71 percent of the time, while its experimental automated grader agreed with them 66 percent of the time. If experienced professionals agree about seven times in ten, a business benchmark built on a single answer key is claiming a certainty its graders did not share.
Three independent grades per task is a practical floor for this design. With two grades, a split cannot separate a lone outlier from a genuine two-way reading; with three, a single dissent becomes visible and can be checked for inconsistency against the dissenter's own rationale.
The Expert Disagreement Map: four causes and the remedy for each
Every split has a cause, and the cause sets the remedy. The map sorts disagreements with three questions asked in order: did the reviewers rely on the same facts, did they read the policy the same way, and is each answer defensible on those facts and that reading? The answers place each split in one of four categories.
Three of the four are defects of the record, the rubric or the grading, and they are repaired before scoring. Only legitimate alternatives survive as more than one admissible answer, and they are the reason a single answer key overstates confidence. Classify from the frozen rationales, not from what reviewers say after discussion. When a split has two causes, the earlier question wins, because a repaired record often dissolves the later disagreement. The figure draws the triage as a tree.
| Cause | How the split shows up | Test that confirms it | Remedy before scoring | Effect on grading confidence |
|---|---|---|---|---|
| Missing facts | Reviewers assume different facts to fill a gap in the record | Rationales cite facts the record does not contain, or different missing facts | Add the missing record and re-grade, or exclude the task and log why | Low until repaired; the split measures the record, not the model |
| Ambiguous policy | Same facts, different readings of a rule or of which version applied | Rationales quote the same clause and reach opposite conclusions | Get a written ruling from the policy owner, version the rubric, re-grade | Restored once the rubric states the ruling |
| Legitimate alternatives | Same facts and reading, different defensible outcomes | An adjudicator confirms each answer meets every written requirement | Keep each as an admissible answer with its rationale and conditions | Lower by design; report label-bound intervals |
| Grader inconsistency | A grade conflicts with the reviewer's own rationale or earlier grades | A blind duplicate gets a different verdict, or the rationale contradicts the grade | Recalibrate or replace the grader, then re-grade the affected items | Low until re-graded; track the share of grades revised |
How do alternate admissible answers change model rankings?
Score every model twice, then many times. The strict score credits only the majority answer on contested items; the lenient score credits any admissible answer. The gap between them is the label-bound interval, and when two models' intervals overlap, the answer key rather than model quality decides which ranks first.
Then sample resolutions. For each contested item, pick one admissible answer at random, rescore every model, and repeat a few thousand times. The rank-flip share is the proportion of resolutions in which the order of two models reverses, a direct and reproducible measure of how much a ranking rests on disagreement experts could not settle.
Answer keys that move rankings are documented. Northcutt, Athalye and Mueller found label errors averaging at least 3.3 percent across the test sets of 10 widely used datasets, and showed that on corrected ImageNet labels the smaller ResNet-18 can outrank ResNet-50 once mislabeled examples become modestly more prevalent. The MMLU-Redux study (NAACL 2025) estimated that 6.49 percent of MMLU questions contain errors, including 57 percent of the virology questions it analyzed, and reported significant discrepancies with originally reported model performance. Errors and legitimate alternatives are different problems, but both show that a few percent of contested answers can reorder models whose scores sit close together.
The table shows the sensitivity output for four models on an illustrative 200-task benchmark with 24 contested items. The numbers are invented to show the format, not measured. Read it this way: A leads on the strict key and B on the lenient key, their intervals overlap, and B ranks first in 41 percent of sampled resolutions. The defensible headline is that this benchmark does not separate A and B, while C and D are clearly behind.
| Model (illustrative) | Strict score | Lenient score | Label-bound interval | Ranked first across sampled resolutions |
|---|---|---|---|---|
| Model A | 152 of 200 (76.0%) | 160 of 200 (80.0%) | 76.0 to 80.0 | 58% |
| Model B | 150 of 200 (75.0%) | 166 of 200 (83.0%) | 75.0 to 83.0 | 41% |
| Model C | 141 of 200 (70.5%) | 147 of 200 (73.5%) | 70.5 to 73.5 | 1% |
| Model D | 128 of 200 (64.0%) | 131 of 200 (65.5%) | 64.0 to 65.5 | 0% |
What grading-confidence figures should a business benchmark publish?
Publish the disagreement beside the score, with a denominator for every figure. A reader should be able to see how many items were contested, why, how the answer key was settled and how much the ranking depends on it. Seven figures cover it:
- Pre-consensus split rate: contested items over all graded items, from the frozen blind grades, by task family.
- Cause shares: the share of splits in each of the four causes, with the number of splits as the denominator.
- Admissible-answer rate: items that kept more than one answer after adjudication, over all graded items.
- Verdict-invariant share for each model: items where the model's pass or fail is the same under every admissible answer, over all items.
- Label-bound interval for each model: strict and lenient scores, with the contested count beside them.
- Rank-flip share for each adjacent pair of models across sampled resolutions.
- Adjudication completeness: contested items with a logged decision, reason and adjudicator role, over all contested items.
The design rule: keep uncertainty and adjudication history in the benchmark
A benchmark that wants to be trusted on business tasks should ship its disagreement, not just its answer key. The rule below is the commission's main deliverable, and a team can adopt it whether or not the full study runs.
It follows the argument Barbara Plank made at EMNLP 2022: human label variation is an open problem across data, modeling and evaluation, standard pipelines aggregate labels on the assumption that a single ground truth exists, and evaluation should go beyond hard labels. Business benchmarks add a practical reason, because alternative answers often encode real policy discretion that a deployed model will face. Uncertainty from the task sample is a separate matter and still needs its own interval, computed by resampling task families and reported beside the answer-key interval rather than merged into it.
- Keep every blind grade, rationale and confidence rating, frozen and timestamped, in the released benchmark or an auditable escrow.
- Version the rubric and every policy ruling, and tie each item to the versions used to grade it.
- Store admissible answers as a set with conditions, never as one answer plus a note.
- Log each adjudication with its cause category, decision, reason and the adjudicator's role.
- Report strict and lenient scores and the rank-flip share wherever a single headline score appears.
- Re-run the map when the rubric, the policy or the reviewer pool changes, and report what moved.
- Publish negative results, including task families where disagreement was too high to grade.
What the proposed study will measure, and what it will not
This is a proposed study. SourceX has not run it and has no disagreement rates, cause shares or model rankings to report. The protocol is published so the design can be criticized before any data is collected; results, negative ones included, will be released only after the study is actually conducted with rights and disclosure approvals in place.
At a glance
What the proposed study will measure, and what it will not
Limits to state up front: results will describe the task families and reviewer pools sampled, not business tasks in general; reviewer qualifications shape disagreement rates; and a small number of contested items can make rank-flip shares unstable, so they should always be read with the contested count. The design:
- Question: how often do qualified reviewers disagree on business tasks, why, and how much does that change model scores and rankings?
- Population: tasks drawn from licensed operational records in pre-registered task families, such as refund exceptions, invoice exceptions and access requests, with every exclusion logged.
- Graders: at least three qualified reviewers per task, grading blind and independently, with qualifications written for each family.
- Variables: verdict, rationale, evidence cited, confidence and time per grade; cause category, decision and reason per adjudication.
- Metrics: the seven figures above, each with its denominator, reported by task family.
- Uncertainty: intervals from resampling task families, reported separately from label-bound intervals.
- Approvals: the right to use and disclose each record and any published example, settled with the data owner before grading.
Commissioning business tasks that can carry more than one valid answer
Running the map needs business records rich enough to grade and to disagree about: the request, the facts at the time, the policy version and the outcome, under terms that allow evaluation use and, where needed, publication of examples.
SourceX holds no dataset inventory and sources records from U.S. businesses only when a buyer asks. Nothing is delivered until the company holding the records has signed the agreement after approving its scope and terms; that signature is the point at which evaluation use and any publication of examples would need to be agreed. A buyer can submit a task-specific requirement through the request-data form, with the intended use and the acceptance test the records must pass, such as a minimum share of tasks that clear the completeness screen.
Frequently asked questions
What level of expert agreement is good enough for a business benchmark?
There is no universal threshold, because agreement depends on the task, the label type and the reviewers. For reference, GDPval reports that its expert graders agreed with each other 71 percent of the time on blind comparisons. The cause breakdown is more useful than any threshold: many legitimate alternatives can be sound, while many missing-fact splits mean the records need work.
Should a benchmark settle expert disagreements by majority vote?
Only splits caused by grader inconsistency, and only after re-grading. Missing facts and ambiguous policy need repairs to the record or the rubric, and legitimate alternatives should stay as admissible answers. A majority vote on a legitimate alternative simply rewards whichever reading happened to have more graders.
How many expert graders does each benchmark task need?
Three independent grades per task is a practical floor for mapping disagreement, because two grades cannot separate a lone outlier from a genuine two-way split. High-consequence task families justify more. Spend extra grades where a wrong verdict costs most rather than spreading them evenly.
Does a high disagreement rate mean a benchmark is broken?
Not necessarily. A high rate driven by missing facts or ambiguous policy points to fixable defects, while a high rate of legitimate alternatives means the task has more than one right answer and the scoring must reflect that. The problem is hiding the rate, not having it.
Has SourceX published Expert Disagreement Map results?
No. This page publishes the study design only; SourceX has not run the study and has no disagreement rates or model rankings to report. Results, including negative ones, would be published only after the study is conducted with rights and disclosure approvals in place.
Sources
- Anthropic's agent-evaluation guide (January 2026) says a good task is one where two domain experts would independently reach the same pass or fail verdict, and that ambiguity in task specifications becomes noise in metrics. Source
- For SWE-bench Verified (August 2024), OpenAI worked with 93 software developers experienced in Python to screen 1,699 random test samples and released a 500-sample subset verified as non-problematic. Source
- GDPval's graders come from the same occupations as the tasks and rank human and AI deliverables, labeling each AI deliverable better than, as good as or worse than the human one. Source
- The GDPval paper reports 71 percent agreement among human expert graders and 66 percent agreement between its experimental automated grader and human experts, using blind pairwise comparisons. Source
- Northcutt, Athalye and Mueller found label errors averaging at least 3.3 percent across 10 commonly used test sets and showed that on corrected ImageNet labels ResNet-18 can outperform ResNet-50 with a modest rise in mislabeled examples. Source
- MMLU-Redux (NAACL 2025) estimates 6.49 percent of MMLU questions contain errors, 57 percent of analyzed virology questions, and reports significant discrepancies with originally reported model performance. Source
- Plank (EMNLP 2022) argues human label variation is an open problem across data, modeling and evaluation, that standard pipelines assume a single ground truth, and that evaluation should go beyond hard labels. Source
Tell us what data you need
Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.