Privacy and preparation
Synthetic data expert disagreement retention: a multi-answer protocol
By SourceX Editorial · Reviewed by Noah Loul ·
10 sections
Short answer
Preserve expert disagreement by capturing it from real cases before generating anything: record each expert's answer and rationale, classify which splits are legitimate, and generate tasks that carry every valid answer with its conditions and observed share. Then score the synthetic set against an untouched real multi-expert holdout on how well it reproduces the spread, not just the majority answer.
License terms
Synthetic data expert
Key takeaways
- Capture and classify expert disagreement on real cases before any synthetic generation begins.
- Keep legitimate and condition-dependent splits; fix unclear questions and correct slips instead of reproducing them.
- Generate every valid position from a real seed, and check that filters do not drop minority answers more often.
- Validate against an untouched real holdout on the spread of answers, with agreed cases as negative controls.
- Synthetic data built from expert records still needs privacy screening and a confirmed right to license.
How do you preserve expert disagreement when generating synthetic answers?
You preserve expert disagreement by capturing it from real cases before any generation, classifying which disagreements are legitimate, and generating synthetic tasks that carry every valid answer with its conditions and its observed share, instead of one consolidated answer. Then you test the synthetic set against a real multi-expert holdout that no generator touched, scoring how well it reproduces the spread of expert answers, not just the most common one.
Synthetic generation tends to do the opposite by default. A generator prompted for the correct answer returns the most probable one, a filter keeps outputs that agree with a reference, and a deduplication step removes near-identical tasks that differ only in the answer. Each step is reasonable alone; together they convert a field where competent experts split into a field where the data says there is one right answer.
This guide gives a multi-answer task construction protocol for teams that build or buy synthetic training and evaluation data in domains where qualified people disagree: accounting judgments, claims handling, contract interpretation, support policy, underwriting notes. It covers what to record, how to construct tasks, how to test the result and where to stop.
What a single synthetic answer erases
A single answer erases the information that a decision is contested, which is often exactly what a model needs to learn: when to ask, when to state an assumption and when two answers are both defensible. Research on annotation has made this point for years. Davani, Diaz and Prabhakaran argue that in many subjective tasks there is no single right answer, and that a majority vote discards how annotators differed.
Disagreement is not limited to subjective tasks. Pavlick and Kwiatkowski examined human judgments on natural language inference, a task usually treated as having gold labels, and found that disagreements are often not dismissible as noise. The ChaosNLI project collected 100 annotations per example for 4,645 existing items and shows cases where the crowd split nearly evenly; one premise and hypothesis pair drew 51 entailment, 3 neutral and 46 contradiction judgments.
Synthetic pipelines add a second pressure. Shumailov and colleagues, writing in Nature, report that indiscriminate training on model-generated content causes models to lose the tails of the original distribution, a process they call model collapse. Minority expert positions are tails. A pipeline that samples only modal answers, round after round, will remove them faster than any annotator would.
Which disagreements deserve to survive generation?
Not every disagreement should be kept. Some are errors, some come from unclear task wording, and some are legitimate differences in professional judgment or in the facts each expert assumed. Classify each disagreement on the real cases first, because only some classes should be reproduced by a generator.
| Disagreement class | How it shows up | Keep in synthetic data? | How to represent it |
|---|---|---|---|
| Legitimate judgment | Experts cite the same facts and rules but weigh them differently | Yes | Multiple valid answers with each rationale and the observed share |
| Condition-dependent | Experts assumed different unstated facts | Yes, split by condition | Separate tasks per condition, or one task with answers keyed to conditions |
| Policy or version difference | Experts applied different editions of a rule or internal policy | Only with the version stated | Tag each answer with the policy and effective date it follows |
| Underspecified task | The question itself is ambiguous | No, fix the task | Rewrite the prompt, then re-collect; keep the original as an ambiguity example only |
| Slip or error | One answer contradicts the cited rule or the record | No | Correct it and log the correction; never weight it as a valid view |
| Perspective difference | Answers vary by reviewer background on a subjective call | Yes, if the use case needs it | Keep per-reviewer answers with role metadata, not just a pooled distribution |
The multi-answer task construction protocol
Run the protocol in order and keep the artifact from each step, because later reviewers will need to see how a synthetic task traces back to real disagreement. Steps one to four happen on real cases only; generation starts at step five.
| Step | What to produce | Acceptance check | Artifact kept |
|---|---|---|---|
| 1. Select real seed cases | Real tasks with at least two independent expert answers each | Experts answered without seeing each other's work | Seed list with source record ids |
| 2. Split off the holdout | An untouched real multi-answer test set | No holdout case or near copy is visible to the generator | Frozen holdout file with a hash |
| 3. Collect rationales | Each expert's answer, reasoning, cited rule and confidence | Every answer has a rationale that names the facts relied on | Per-expert answer table |
| 4. Classify disagreement | A class for each split case from the table above | A second reviewer agrees on the class, or it is escalated | Class label and reviewer ids |
| 5. Write the generation recipe | Prompt, model, grounding sources, policy date, sampling settings | Recipe instructs the generator to produce every valid position | Versioned recipe file |
| 6. Generate per position | Synthetic tasks for each answer and condition, in proportion to the seed shares | Each position appears; no position invented without a seed | Generated set with derivation links |
| 7. Screen and filter | Privacy screen, rule-consistency check, negative controls | Filters do not drop minority answers at a higher rate than majority ones | Filter log with drop counts per position |
| 8. Validate on the holdout | Distribution comparison against real expert splits | Pre-registered thresholds met for both agreed and split cases | Validation report |
| 9. Disclose | Datasheet section on derivation and disagreement handling | A reader can tell real from synthetic and kept from removed views | Published datasheet |
One record per task: the fields that carry disagreement
Store disagreement as structure, not as a comment field. ChaosNLI's released format is a useful model: alongside the majority label it keeps the label counts, the label distribution, the entropy of that distribution and the original individual annotations, so a user can see both the consensus and the spread. A multi-answer record for professional tasks needs the same idea plus the reasons.
At a glance
One record per task: the fields that carry disagreement
Use these fields for each task, real or synthetic. Fields marked by role or date matter most when the record is later licensed or audited.
- Task id and derivation: real, edited from real, or generated; parent seed id for anything not real.
- Grounding sources: the records, policies or documents the task relies on, with versions and dates.
- Expert answers: one entry per expert with role, answer, rationale, cited rule and stated confidence.
- Answer set: the distinct valid positions, each with the conditions under which it holds.
- Observed distribution: counts and shares per position on the real seed, and the entropy of that split.
- Disagreement class: from the classification table, with the reviewer who assigned it.
- Generation recipe id: the prompt, model and settings that produced a synthetic item.
- Screening results: privacy screen outcome, rule-consistency outcome and any correction applied.
Scoring the synthetic set against untouched real cases
Score the synthetic set on whether a model trained or evaluated with it reproduces the real spread of expert answers on the untouched holdout, measured separately for cases where experts agreed and cases where they split. ChaosNLI's scoreboard compares model output distributions with human label distributions using Jensen-Shannon distance and KL divergence, alongside accuracy; the same measures work for multi-answer professional tasks with a small fixed answer set.
Do not assume soft labels always win. In their survey of learning from disagreement, Uma and colleagues report that training on soft labels did better with large datasets and many high-quality judgments per item, and that otherwise combining gold and soft labels generally gave the best results on hard evaluation. Set the training format by how many expert answers each real case actually has.
Include negative controls. Some real holdout cases should be ones where experts agreed. A synthetic mixture that teaches a model to hedge on those has learned false doubt, which is as much a failure as false certainty. The CrowdTruth project's quality metrics for crowdsourcing with disagreement are one open reference for scoring annotators, units and labels without treating every split as noise.
- Agreement cases: accuracy against the agreed answer, and the share of hedged responses.
- Split cases: distribution distance against the expert split, and coverage of every valid position.
- Minority positions: recall of each minority answer with its stated condition.
- Rule consistency: answers that contradict the cited rule, counted separately from disagreement.
Which real exceptions or conditions does the synthetic mixture fail to represent?
The usual gaps are rare conditions, minority positions with few seeds and answers that depend on a policy version the generator did not see. Check each before release by comparing the generated set's coverage with the real seed, position by position and condition by condition.
Look for these signs: a valid position present in the seed but absent or far smaller in the generated set; conditions that appear in real cases but never in synthetic ones; rationales that cite rules the grounding documents do not contain; and agreement cases that the generator turned into splits. Any of these means the mixture does not represent the real field. Gebru and colleagues' Datasheets for Datasets proposes documenting a dataset's motivation, composition and collection process; disclose these gaps there rather than leaving them for a buyer to find.
Worked example (illustrative): capitalize or expense an invoice line
This example is illustrative and uses counts, not real data. A team builds tasks from accounts payable records: given an invoice line and the company's capitalization policy, should the cost be capitalized or expensed? Three staff accountants review each of 400 real lines independently.
On 340 lines all three agree. On 60 they split. Review of rationales shows three classes among the splits: 28 are legitimate judgment calls about whether a repair extends an asset's useful life; 20 are condition-dependent because the invoice does not say whether the item is part of a larger project; 12 are slips where one reviewer missed the policy's threshold. The team corrects the 12, splits the 20 into paired tasks with the condition stated, and keeps the 28 as multi-answer tasks with each rationale.
Before generation, 80 real lines, including 15 split cases, are frozen as the holdout. The generation recipe asks for every valid position on each seed. After filtering, the team finds the rule-consistency filter dropped minority answers twice as often as majority ones, because it was checking against a single reference answer. They change the filter to check against the answer set, regenerate, and only then run the holdout comparison.
| Seed group (illustrative) | Real lines | Handling | Synthetic treatment |
|---|---|---|---|
| All reviewers agree | 340 | Single answer kept | Generated with one answer; used as negative controls |
| Legitimate judgment split | 28 | Both answers kept with rationales and shares | Generated with both positions in seed proportion |
| Condition-dependent split | 20 | Split into paired tasks with the missing fact stated | Generated per condition |
| Reviewer slip | 12 | Corrected and logged | Generated from the corrected answer only |
| Frozen holdout | 80 | Drawn from the 400 lines above (65 agreed, 15 split) and never shown to the generator | None; used only for validation |
Privacy, rights and the cases a generator should not touch
Synthetic data built from real expert records still carries risk from those records. NIST's guidance on de-identifying government datasets lists publishing synthetic data based on identified data as one data-sharing model among several, and recommends re-identification studies to gauge the remaining risk. Expert rationales are a particular concern: they quote names, amounts and circumstances from the underlying case, and a generator can copy them.
Screen grounding records and rationales before they reach a generator, and test generated outputs for copied passages. Separately, confirm that the business supplying the records may license them for this use and that the experts' written rationales are covered; de-identification alone does not create a right to license. These are questions for privacy and legal reviewers with the specific facts, not for the protocol.
Stop and bring in a domain validator when the disagreement involves professional judgment with legal or safety consequences, when experts disagree about what the rule says rather than how to apply it, or when the seed has too few split cases to estimate a distribution. In those cases, collect more real expert answers instead of generating more synthetic ones.
Sourcing real multi-reviewer records for a grounded mixture
A grounded mixture starts with real cases that were reviewed independently by more than one qualified person, with the reasoning written down. Those records exist in many businesses as second reviews, audit samples, quality checks and escalations, but rarely in a form ready to license.
When a team asks, SourceX looks for second reviews, audit samples and escalations at U.S. businesses, then carries the chosen records through rights review, preparation and delivery as licensed datasets for AI. A request can describe the domain, the number of independent reviews per case, whether rationales are needed, the volume, format and timing, and the licensing terms the team requires.
Frequently asked questions
Is expert disagreement just annotation noise?
Often it is not. Studies of natural language inference found that many disagreements persist and are not dismissible as noise, and in professional tasks experts can apply the same rule to the same facts and reach different defensible answers. The work is to separate those cases from slips and unclear questions.
Should a synthetic dataset ever contain only one answer per task?
Yes, when qualified experts agree and the rule is clear. Those tasks are also useful as negative controls, because a model should not learn to hedge on questions that have one answer.
How many expert answers per case are needed to estimate disagreement?
There is no universal number. Two independent answers reveal that a split exists, but estimating its shape needs more; research on learning from disagreement found soft labels most useful when items had many high-quality judgments. Set the target by how the data will be used and record the count per case.
Can a language model generate the minority expert view on its own?
It can be prompted to, but without a real seed it may invent positions no expert holds or miss the one that matters. Generate minority positions only from documented real ones, and validate on a real holdout.
Does synthetic data remove privacy concerns about the source records?
No. Synthetic data built from identified records still carries disclosure risk, and NIST recommends re-identification studies to gauge it. Screen the grounding records and expert rationales, and test outputs for copied passages.
Sources
- Davani, Diaz and Prabhakaran (TACL 2022) argue that many subjective annotation tasks have no single right answer and that majority voting loses information about how annotators differ. Source
- Pavlick and Kwiatkowski (TACL 2019) found that human disagreements on natural language inference are often not dismissible as noise. Source
- ChaosNLI collected 100 annotations per example for 4,645 items, releases label counts, distributions, entropy and original labels, and scores models against human label distributions with JSD and KL. Source
- Shumailov and colleagues (Nature 2024) report that indiscriminate training on model-generated content causes irreversible defects in which the tails of the original content distribution disappear. Source
- Uma and colleagues' JAIR survey reports that soft-label training did better with large datasets and many high-quality judgments per item, and that otherwise combining gold and soft labels generally did best on hard evaluation. Source
- The CrowdTruth project publishes an open-source library implementing quality metrics for crowdsourcing with disagreement. Source
- Gebru and colleagues propose that every dataset be accompanied by a datasheet documenting its motivation, composition, collection process and recommended uses. Source
- NIST SP 800-188 lists publishing synthetic data based on identified data as one data-sharing model and recommends re-identification studies to gauge disclosure risk. Source
Tell us what data you need
Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.