Privacy and preparation
Expert disagreement in a dataset: how a business keeps the valid kind
By SourceX Editorial · Reviewed by Noah Loul ·
9 sections
Short answer
A dataset should preserve legitimate expert disagreement by storing each expert's answer separately with the evidence seen, reasoning, alternatives, confidence and the conditions under which the answer holds. Classify every disagreement first: slips, missing evidence and unclear rules get fixed, while cases where qualified experts saw the same evidence and still differ stay as multiple valid answers.
License terms
Expert disagreement dataset
Key takeaways
- Store each expert's answer as its own record, never only the adjudicated result.
- Classify every disagreement before deciding whether to keep or fix it.
- Record what each expert saw; many disagreements turn out to be missing evidence.
- Ask each expert what evidence would change the answer and store it as a condition.
- Collect first answers blind, calibrate on settled cases and record reviewer incentives.
- De-identify rationale text and reviewer identities before records leave the business.
How should a dataset preserve legitimate expert disagreement?
Keep every expert's answer as its own record, with the evidence that expert saw, the reasoning, the alternatives considered, a confidence level and the conditions under which the answer holds. Then classify each disagreement. Slips, missing evidence and unclear guidelines get fixed. Disagreements where both answers are defensible stay in the dataset as multiple valid answers, labeled with what would make each one right.
That classification step is the whole difference between a noisy dataset and a valuable one. A business that runs the same decision through two qualified people, such as two claims reviewers, two estimators or two credit analysts, already produces disagreement. Most of it is thrown away when a supervisor picks a winner and the system stores only the final answer.
This guide is for the business that holds those records and wants them documented well enough to be useful for AI training or evaluation. It does not cover the statistics of agreement or the choice between label aggregation methods; linked pages handle those.
Which disagreements are a feature of the data, and which are defects?
Ask one question of every disagreement: if both experts had seen the same evidence and the same current rules, would they still differ? If yes, the disagreement is probably legitimate judgment. If no, it is a defect in capture or guidance, and it should be fixed rather than preserved.
| Disagreement type | How to recognize it | What to record | What to do with it |
|---|---|---|---|
| Slip | One expert misread a field or chose the wrong code, and agrees once shown | Both answers, the correction, who caught it | Correct it; keep the slip only as an error example |
| Missing evidence | Experts saw different documents or versions | The evidence set each expert saw | Re-decide on the full set; fix the capture process |
| Guideline gap | The written policy does not cover the case or is ambiguous | The clause in question and both readings | Escalate to the policy owner; record the ruling and its date |
| Rule change | Answers differ because the policy changed between decisions | Policy version in force for each decision | Keep both, each tied to its policy version |
| Legitimate judgment | Same evidence, same rules, both answers defensible | Each answer with rationale, conditions and confidence | Preserve as multiple valid answers |
| Risk appetite | Experts weigh the same risk differently within allowed discretion | The discretion clause and each expert's weighting | Preserve; tag as discretionary rather than error |
Why does picking one winner throw away the useful part?
Forcing one answer assumes there is a single ground truth for every case. Barbara Plank's 2022 position paper argues that this assumption overlooks legitimate disagreement, subjectivity and cases with several valid answers, and that the problem reaches data, modeling and evaluation alike.
The research community has started to treat the spread of answers as data. Davani, Díaz and Prabhakaran showed that models trained to predict each annotator's rating, then aggregated, matched or beat majority-vote baselines on subjective tasks and gave uncertainty estimates that tracked annotator disagreement. The SemEval-2023 Learning With Disagreements task used soft evaluation, which scores how well a model's probabilities reflect the spread of annotator judgments.
Those studies used crowd annotation of text. Business expert decisions differ in one way that helps: each answer often comes with records that show why. That makes it possible to separate the disagreements worth keeping from the ones worth fixing, which crowd labels rarely allow.
What should the ambiguity and multiple valid answer documentation guide include?
Use one row per expert per case, plus one case-level summary. The fields below are the minimum that lets a reviewer reproduce why each answer was given and whether it still holds. Example values are illustrative.
At a glance
What should the ambiguity and multiple valid answer documentation guide include
Documentation standards already ask for some of this. The Croissant responsible AI vocabulary includes a field for the number of human labels per item and a data annotation analysis field that covers uncertainty or disagreement between annotations as a signal and how final labels relate to individual responses. The Hugging Face dataset card template asks for the annotation process, including guidelines, inter-annotator statistics and validation, and for who the annotators are.
| Field | What it holds | Illustrative value | Why it matters |
|---|---|---|---|
| case_id and decision_question | The case and the exact question decided | Claim 4471: approve, partial goodwill or deny? | Disagreement is only meaningful about one question |
| evidence_set | IDs and versions of every record the expert saw | Claim form v2, service history, policy v7 | Separates missing evidence from judgment |
| expert_ref and qualification | Pseudonymous ID plus role, tenure band and scope | R-12, senior claims reviewer, 5+ years | Shows the answer came from a qualified person |
| answer and rationale | The decision and the reasons in the expert's words | Partial goodwill: long account history | The rationale is often the most useful text |
| alternatives_considered | Other answers the expert weighed and why rejected | Deny: claim filed after expiry | Shows the expert saw the other path |
| confidence | Expert's own rating on a fixed scale | Medium | Low confidence plus disagreement flags real ambiguity |
| valid_if | Conditions under which this answer is right | Valid if the policy discretion clause applies | Turns a disagreement into a conditional rule |
| would_change_if | Evidence that would change the answer | A maintenance lapse in the service log | Tells a model what to look for |
| disagreement_type and status | Type from the taxonomy and whether it is open or settled | Legitimate judgment, preserved | Keeps defects out of the valid-answer set |
| policy_version and decided_at | Rules in force and the decision time | Warranty policy v7, decided during Q2 | Prevents rule changes from looking like disagreement |
Which fact would flip each expert's answer?
Ask each expert directly, at decision time, and store the answer in would_change_if. The question is concrete: which fact, if it were different, would make you choose the other option? Experts usually answer quickly, and the answer turns a vague disagreement into a testable condition.
Then check the answers against each other. If expert one says a maintenance lapse would flip the decision, and the service log shows no lapse, the case moves toward one answer. If both experts name conditions that the evidence cannot settle, such as how much weight a long customer relationship deserves, the case is genuinely ambiguous and both answers stay.
This field is also the honest way to show a buyer the limits of the data. A model trained only on final answers learns that a case like this has one answer. A model trained with the conditions learns what the answer depends on.
How do you know the disagreement is genuine and not an artifact of how you collected it?
Collect first answers independently and blind to each other. If the second expert sees the first answer, agreement rises for reasons that have nothing to do with the case, and the disagreement you keep is the part that survived social pressure rather than the part that reflects the decision.
Calibrate before you trust the spread. Give every expert a small set of cases whose outcome is already settled, such as claims later upheld on appeal, and record how often each one reaches the settled answer. An expert who disagrees with everyone on settled cases is producing errors, not perspectives.
Check incentives. If reviewers are scored on agreement with a supervisor or paid per case closed, they will converge or rush, and both hide legitimate disagreement. Record how reviewers were measured during the period the data covers; it belongs in the documentation as much as the guidelines do.
Finally, keep the raw answers. The Datasheets for Datasets questionnaire asks whether the raw data was saved in addition to the preprocessed or labeled data, and the same logic applies here: without each expert's original answer, nobody can later reconsider how disagreements were handled.
Worked example (illustrative): warranty claims reviewed twice
This example is invented to show the method; the counts are illustrative. A manufacturer's service team has two senior reviewers decide a sample of 40 warranty claims independently. They disagree on 9.
Classifying the 9: 2 are slips, where a reviewer chose the wrong failure code and agreed once shown. 3 are missing evidence, where one reviewer did not have the dealer's service history; re-decided on the full set, all 3 converge. 1 is a guideline gap: the policy does not say whether a replaced part carries the original warranty term, so it goes to the policy owner, whose ruling is recorded with a date. 3 remain.
The 3 that remain are legitimate judgment. In one, a claim arrives shortly after the warranty expired. Reviewer A denies it under the written term. Reviewer B grants partial goodwill under the policy's discretion clause, citing a long service history with no missed maintenance. Both cite the same evidence and the same policy version. The dataset keeps both answers, with valid_if and would_change_if filled in for each.
The exception: one of the 3 looked like judgment at first, but reviewer B's confidence was low and the would_change_if answer named a document neither reviewer had checked. Pulling it settled the case. Low confidence plus a checkable condition is a signal to look again before labeling anything as a valid alternative.
What ships from this sample is therefore not 40 single answers. It is 38 settled cases, each with both original answers kept, plus 2 cases carrying two valid answers with their conditions, the guideline ruling with its date, and a summary that states how many disagreements fell into each type. That summary is often the first thing a careful buyer reads.
What privacy, ownership and regulatory limits apply to disagreement records?
Expert rationale is free text, and free text carries names, account details and opinions about customers. Before any of it leaves the business, it needs de-identification that covers the rationale and alternatives fields, not only the structured columns. Reviewer identities need the same care: a pseudonymous expert reference with a qualification band is usually enough.
Ownership and permission questions come next. Notes written by employees in the course of their work are usually company records, but customer contracts, employment terms and privacy notices can still limit how they are used. De-identification alone does not create a right to license; counsel should review the specific contracts.
Narrow the scope when most disagreements turn out to be missing evidence or guideline gaps. That is a sign the capture process needs repair first. Bring in a domain specialist when the decisions are regulated, such as credit, insurance underwriting or employment, because the reasons experts may legitimately weigh can be limited by law.
Can a business license its documented disagreements?
Disagreement records documented this way are useful for evaluation sets that score whether a model recognizes ambiguity, and for training data where the conditions matter as much as the answer. A business that wants to license them can do so through SourceX, which carries the records through rights review, contracting, preparation and delivery.
SourceX's standing terms provide for de-identification through approved processing routes, and nothing is delivered before the business approves the scope and terms and signs. Eligibility is checked first through a brief questionnaire about the company, with no claim files or rationale sent.
Frequently asked questions
Is expert disagreement a sign of a bad dataset?
Not by itself. Disagreement caused by slips, missing evidence or unclear rules is a defect and should be fixed. Disagreement where qualified experts saw the same evidence and rules and still chose different defensible answers is information that a model or evaluator can use.
Should a business keep only the final adjudicated answer?
Keep the final answer and every expert's original answer, with the evidence each one saw. The final answer alone hides which cases were close calls. The originals let a buyer or reviewer reconsider how disagreements were handled later.
How many experts should review each case?
Two independent reviewers on a sample is a practical start for most businesses, with a third reviewer or a policy owner for unresolved cases. Record the number of reviews per case in the dataset documentation so users know how much of the data was double-checked.
What is a multiple valid answer label?
It is a label set that records more than one acceptable answer for a case, each with the conditions under which it holds and the evidence that would change it. It differs from a single label with a confidence score because it keeps the reasoning behind each path.
Can disagreement records include the reviewers' names?
They usually should not leave the business with names attached. A pseudonymous reviewer reference with a role and tenure band keeps the qualification evidence while protecting the people involved. Counsel should confirm what employment terms and privacy notices allow.
Sources
- The Croissant RAI vocabulary records the number of human labels per item and an annotation analysis covering disagreement between annotations as a signal and how final labels relate to individual responses. Source
- The Hugging Face dataset card template asks for the annotation process, including guidelines, inter-annotator statistics and validation, and who the annotators are. Source
- Plank (EMNLP 2022) argues that assuming a single ground truth overlooks legitimate disagreement and cases with several valid answers across data, modeling and evaluation. Source
- Davani, Díaz and Prabhakaran (TACL 2022) show multi-annotator models that preserve individual perspectives match or beat majority-vote baselines and yield uncertainty that tracks disagreement. Source
- The SemEval-2023 Learning With Disagreements task evaluated systems with soft evaluation of how well they reflect annotator disagreement. Source
- Datasheets for Datasets asks dataset creators whether the raw data was saved in addition to the preprocessed, cleaned or labeled data. Source
See if your company qualifies
A short company assessment. No data uploads are needed.