AI uses for records
Unresolved expert disagreement dataset: what a single label erases
By SourceX Editorial · Updated
10 sections
Short answer
An unresolved expert disagreement dataset keeps every reviewer's judgment, the evidence version behind it, a status that separates contested from insufficient, and the authority that could settle each item. In CLIMATE-FEVER's 7,675 public claim-evidence pairs, 49.4 percent of votes were split, every tie was published as not enough info, and 709 claims with votes on both sides read as settled.
License terms
Unresolved expert
Key takeaways
- A single published label can hide most of a dataset's disagreement: in CLIMATE-FEVER, 49.4 percent of claim-evidence pairs had split votes.
- Ties went to not enough info every time, so 3,290 of 4,930 insufficient labels hid at least one supporting or refuting vote.
- Only 10.0 percent of claims were openly marked DISPUTED, while 709 other claims carried votes on both sides.
- Record who can settle each disagreement: the evidence, a named owner, a later event, or nobody.
- Before buying, recompute labels from votes, split insufficient from contested, and check vote depth and authority fields.
What should an unresolved expert disagreement dataset keep?
Every reviewer's judgment with the evidence it rested on, a status that tells contested apart from insufficient, the authority that could settle the question if one exists, and the rule that produced any single published label. Without those four parts, a dataset can only report a consensus it never had.
The need is sharpest in enterprise knowledge work. Two senior support agents read the same warranty clause and split on whether water damage is covered; a compliance lead and a product manager read a retention policy differently; an analyst's note contradicts a vendor's documentation. An assistant trained or graded on one correct answer for such cases learns to sound certain where the organization is not.
This page asks a measurable version of the question: when a public knowledge dataset publishes one label per item, how often was the underlying judgment split, and what does the label erase? The test case is CLIMATE-FEVER, a claim-verification dataset that, unusually, ships each annotator's vote beside the published label.
How often does a public knowledge dataset leave disagreement unresolved?
Far more often than its labels show. CLIMATE-FEVER pairs 1,535 real-world climate claims with five English Wikipedia evidence sentences each, 7,675 claim-evidence pairs in all, according to its README, and its paper says the claims were drawn in equal numbers from scientifically informed sources and from climate change skeptic or denier sources. In the released file at commit 03de616, the votes on 3,792 pairs (49.4 percent) are split, and 871 pairs (11.3 percent) hold at least one vote that the sentence supports the claim and one that it refutes it.
At claim level the dataset does keep some disagreement in view. Its 154 claims labeled DISPUTED (10.0 percent) each have at least one supporting and one refuting evidence label, and no other claim has both. That explicit label covers a small part of the split judgment: of the 1,381 claims labeled supports, refutes or not enough info, 709 (51.3 percent) carry votes on both sides somewhere in their evidence, and 1,310 of the 1,535 claims (85.3 percent) contain at least one split pair.
Unresolved therefore has two meanings in this file. Openly unresolved disagreement is the 10.0 percent of claims marked DISPUTED; silently unresolved disagreement sits under labels that read as settled, and it is several times larger.
What does collapsing the votes to one label erase?
It erases the minority judgment and, in this dataset, the difference between silence and contradiction. The published evidence label follows the plurality vote whenever one exists, without exception in the file, and all 2,501 ties were published as NOT_ENOUGH_INFO, including 685 ties that held both a supports and a refutes vote.
The label ends up meaning two things. Of 4,930 evidence sentences labeled NOT_ENOUGH_INFO, only 1,640 had every vote agreeing; the other 3,290 (66.7 percent) had at least one reader who judged that the sentence supported or refuted the claim. A retriever trained to treat those sentences as irrelevant, or an evaluator that scores not enough info as the right answer, learns to bury exactly the evidence a domain expert would want surfaced.
Forcing the openly disputed claims into one verdict fares no better. Comparing each one's supporting and refuting evidence labels gives 55 supports, 32 refutes and 67 ties (43.5 percent), so a majority rule cannot decide almost half of them.
The collapse audit sorts every evidence pair, and then every claim, by what its published label kept and what it lost:
| Published situation | Count (share) | What the single label erases | Keep instead |
|---|---|---|---|
| Pair with unanimous votes | 3,883 of 7,675 (50.6 percent) | Nothing, apart from how many readers agreed | The vote count beside the label |
| Pair split, plurality published | 1,291 (16.8 percent) | The minority vote; in 789 pairs a supporting or refuting vote lost to not enough info | Every vote, with the rule that picked the winner |
| Pair tied between a verdict and not enough info | 1,816 (23.7 percent) | That half the readers saw support or refutation | Status contested, both votes, queued for a third read |
| Pair tied with both supports and refutes votes | 685 (8.9 percent) | A direct contradiction, published as not enough info | Status contested, routed to whoever can settle it |
| Claim labeled DISPUTED | 154 of 1,535 (10.0 percent) | Nothing at claim level; a forced verdict would tie on 67 | Keep as contested and never majority-vote it |
| Claim labeled supports, refutes or not enough info with votes on both sides | 709 of 1,381 (51.3 percent) | That some reader saw the opposite verdict in its evidence | A per-claim vote pool and a contested flag |
Who has the authority to settle each disagreement?
Sometimes the evidence does, sometimes a named owner does, and sometimes nobody can yet; the dataset should record which. A knowledge authority map assigns each disagreement to the record or role that could close it, and leaves it open when no such party exists.
At a glance
Who has the authority to settle each disagreement
Public knowledge bases already work this way. Wikidata keeps competing values for one property and ranks them preferred, normal or deprecated, using deprecated for statements known to include errors or to represent outdated knowledge; a query for the best-ranked statement returns the preferred one, falls back to normal when none is preferred, and leaves deprecated statements out unless asked. Wikipedia, the evidence source behind CLIMATE-FEVER, asks articles to represent all significant viewpoints published by reliable sources in proportion to their prominence. A sentence written to report one side of a dispute can read as support to one annotator and as neutral reporting to another, which is a property of the evidence rather than a mistake by either reader.
Five authority types cover the cases a buyer will meet:
- Settled by the evidence: the cited passage states the answer outright. The resolution is a quotation and a version; a split here is a reading slip, corrected with both original votes kept.
- Settled by an owner: a policy, contract or system of record decides and a named role can rule. Record the owner role, the governing document and its version, and the ruling date.
- Settled by time: a later event will decide, such as a claim outcome or a pending policy review. Keep the item open with a review date and link the event when it lands.
- Not settleable: qualified readers weigh the same evidence differently. Keep each position as a valid answer with its rationale, and score answers that state the split instead of picking a side.
- Not yet mapped: nobody has assigned an authority. Treat the item as contested and keep it out of single-answer scoring until it is mapped.
The evidence-grounding test: does every label follow from its votes?
In CLIMATE-FEVER every label does follow from its votes, but that is only the first of six checks worth running on any multi-annotated delivery; together they separate a dataset that preserves disagreement from one that only stores the votes. Each step below was run on CLIMATE-FEVER, with the result after it.
A delivery passes when its labels reproduce from the stored judgments, when ties and insufficient labels carry an explicit contested status, and when contested items name who could settle them. CLIMATE-FEVER passes the first check cleanly and fails the other two, a fair description of many public multi-annotated sets: honest about the votes, silent about what they mean.
The six steps, with the CLIMATE-FEVER result for each:
- Recompute each published label from its votes with the documented rule and count mismatches. Result: none of the 5,174 pairs with a plurality disagrees with it, and the stored entropy matches the votes on all 7,675 pairs.
- Count the ties and record where each was sent. Result: all 2,501 ties became NOT_ENOUGH_INFO.
- Divide every insufficient label into unanimous and contested. Result: 1,640 unanimous, 3,290 contested.
- Pool the votes for each item to find conflicts the item label hides. Result: 709 of the 1,381 claims not marked DISPUTED.
- Check that contested items carry an authority type and an evidence version. Result: no such field; evidence is a sentence plus an article title, with no revision identifier.
- Check vote depth. Result: 5,340 of 7,675 pairs (69.6 percent) have exactly two votes, so a one-to-one split cannot be told apart from a slip.
Is the split real disagreement or a thin annotation design?
Partly design, and the file cannot say how much. With two votes on most pairs, one slip produces a tie, and in this dataset a tie always becomes NOT_ENOUGH_INFO; a third qualified read on the 2,501 tied pairs would settle many of them one way or the other.
The evidence is thin by construction as well. Each pair shows one sentence, in some cases only a quoted article headline (the first claim's evidence includes two), without its surrounding paragraph or page revision, so readers fill in context differently. Text can be outdated, partial or about a neighboring claim, and a reader who marks it refuting may be right about the sentence and wrong about the claim. The CLIMATE-FEVER authors themselves describe labeling real-world climate claims as hard even with renowned climate scientists to consult (paraphrase).
Neither point argues for collapsing the votes. Basile and colleagues argued in 2021 that annotators disagree widely even on tasks that look objective, which makes any single gold label hard to defend (paraphrase). The practical answer is to keep the split, record why it might exist, and pay for a third read or an owner's ruling only where the use case needs one answer.
How should enterprise knowledge records carry an open question?
As a record with separate fields for each position, the evidence version behind it, the item's status and the authority that could close it, never as a final answer with a free-text note. The fields below turn the collapse audit into a schema a supplier can fill from systems it already runs.
At a glance
How should enterprise knowledge records carry an open question
Most of the raw material already exists somewhere: a second-level review in a ticketing tool, a policy exception log, a claims supervisor's override, a legal hold note. What usually goes missing is the link between them and a status saying the question stayed open. Retrieval that must fetch both conflicting records before answering is covered in the guide to retrieval evidence disagreement datasets, and aggregation methods such as majority vote and soft labels are compared in the annotation adjudication guide; this schema sits upstream of both.
Fields for each knowledge item:
- position_id, reviewer role and timestamp for every judgment, never only the winning one
- evidence_ref: document identifier, version or revision, and the passage relied on
- status: agreed, contested, insufficient (evidence silent), superseded or escalated, with contested and insufficient never merged
- authority_type from the map above, plus the owner role for owner-settled items
- resolution: the ruling, its date, its source document and every position it overruled
- aggregation_rule: the rule behind any convenience label, so a buyer can recompute it
Decision rule, limits and the evidence still needed
Use a knowledge dataset for evaluation only if its labels reproduce from stored judgments and it keeps contested apart from insufficient; otherwise budget a re-annotation pass on the contested items first. For training, keep the vote vectors and the contested flag even when a single label is also supplied.
The audit has limits. It measures one public dataset of climate claims checked against Wikipedia, not enterprise records; it cannot tell genuine disagreement from error on two-vote pairs; and it does not test whether models trained on vote distributions do better on any business task.
Three pieces of evidence would sharpen the decision: a third qualified read on a fixed sample of tied pairs, to estimate how many ties are slips; the same audit on an enterprise set where an authority exists, such as policy questions with a named owner; and a held-out evaluation comparing single-label scoring with scoring that rewards stating a split.
Putting disagreement fields into a data request
Write the disagreement requirement into the request itself: the task, the status values needed, the minimum number of independent judgments per item, the authority fields, and the grounding test that will be run on a sample before acceptance. SourceX does not keep datasets in stock; it looks for companies holding matching records after a buyer describes the need, and it licenses the resulting data to the buyer in its own name, so these fields belong in that first description.
Business records that tend to carry real, documented disagreement include second-level support reviews, claim decisions with supervisor overrides, policy exception requests and quality escalations. Whether a given company's records hold those fields, and whether they can be licensed, is settled case by case; nothing here implies that such data is available today.
Methods note: file, rules and what the audit leaves out
Data: dataset/climate-fever.jsonl from github.com/tdiggelm/climate-fever-dataset at commit 03de616 (committed 2020-12-22), read with git show on 2026-10-27 (SHA-256 8a4b9032d861be482ffb49dddfd283ffa6089e654f1e968040011882c5eb6e0b). The script, climate_fever_collapse.py, uses the Python standard library with no randomness and saves a label-only table with no claim or evidence text. Shares are exact fractions rounded once, half up.
Rules as applied: a vote is a non-null entry in a pair's votes list; unanimous means every vote is identical; split means two or more distinct values; a conflict pair has at least one SUPPORTS and one REFUTES vote; the plurality is the most common vote, and a tie for most common means there is none; a contested NOT_ENOUGH_INFO label is a split pair labeled NOT_ENOUGH_INFO with at least one SUPPORTS or REFUTES vote; a hidden conflict is a claim not labeled DISPUTED whose votes, pooled across its five pairs, include both SUPPORTS and REFUTES; the forced verdict for a DISPUTED claim compares its counts of SUPPORTS and REFUTES evidence labels, with equal counts a tie.
Not measured: annotator identity or expertise (the file holds neither), the correctness of any label, agreement coefficients, and model performance. The authority map, the grounding test and the field list are editorial frameworks drawn up for this page on 2026-10-27.
Frequently asked questions
What is an unresolved expert disagreement dataset?
A dataset that stores each qualified reviewer's judgment, the evidence behind it and a status showing whether the question was settled, instead of publishing one label. It records who or what could settle open items. Its value is that a model can learn where an organization is genuinely uncertain.
Should annotator disagreement be resolved by majority vote?
Only for a convenience label, and only with the votes kept. In CLIMATE-FEVER, a majority rule applied to the 154 openly disputed claims would tie on 67 of them. Majority vote also cannot tell a reading slip from a real difference of judgment.
What is the difference between contested and not enough information?
Not enough information means the evidence is silent on the question. Contested means readers found an answer in the evidence and disagreed about it. In CLIMATE-FEVER, 3,290 of 4,930 not-enough-info labels had at least one supporting or refuting vote, so the two were merged.
How many annotators does a disagreement-preserving dataset need?
Two cannot separate a slip from a real split, and CLIMATE-FEVER has exactly two votes on 69.6 percent of its pairs. Three or more qualified reads with a short rationale each let a reviewer tell the cases apart. Owner-settled items need the owner's ruling, not more votes.
Can a company license records that contain internal disagreement?
Possibly, but disagreement records often name employees and reflect confidential deliberation, so rights, privacy and confidentiality review come first. De-identification alone does not create a right to license. Counsel should review the specific records and agreements.
Sources
- CLIMATE-FEVER consists of 1,535 real-world climate claims, each with five manually annotated English Wikipedia evidence sentences (7,675 claim-evidence pairs), and includes disputed claims with both supporting and refuting evidence; the released file stores every annotator vote analyzed here. Source
- The CLIMATE-FEVER paper says claims were collected in equal numbers from scientifically informed and skeptic or denier sources and describes labeling real-world climate claims as difficult. Source
- Wikidata ranks competing statements preferred, normal or deprecated, uses deprecated for erroneous or outdated statements, and returns the best-ranked statement by default. Source
- Wikipedia's neutral point of view policy asks articles to represent all significant viewpoints published by reliable sources in proportion to their prominence. Source
- Basile and colleagues (2021) argue that annotator disagreement is widespread even on tasks that look objective, which makes a single gold label hard to choose (paraphrase). Source
- SourceX sources datasets on request rather than holding them in stock and licenses them to buyers in its own name. Source
Tell us what data you need
Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.