Skip to content

Privacy and preparation

Synthetic data expert disagreement retention: a multi-answer protocol

By SourceX Editorial · Reviewed by Noah Loul ·

10 sections

Short answer

Preserve expert disagreement by capturing it from real cases before generating anything: record each expert's answer and rationale, classify which splits are legitimate, and generate tasks that carry every valid answer with its conditions and observed share. Then score the synthetic set against an untouched real multi-expert holdout on how well it reproduces the spread, not just the majority answer.

License terms

Synthetic data expert

This page covers synthetic data expert disagreement retention: a multi-answer protocol.Illustrative

Key takeaways

  • Capture and classify expert disagreement on real cases before any synthetic generation begins.
  • Keep legitimate and condition-dependent splits; fix unclear questions and correct slips instead of reproducing them.
  • Generate every valid position from a real seed, and check that filters do not drop minority answers more often.
  • Validate against an untouched real holdout on the spread of answers, with agreed cases as negative controls.
  • Synthetic data built from expert records still needs privacy screening and a confirmed right to license.

How do you preserve expert disagreement when generating synthetic answers?

You preserve expert disagreement by capturing it from real cases before any generation, classifying which disagreements are legitimate, and generating synthetic tasks that carry every valid answer with its conditions and its observed share, instead of one consolidated answer. Then you test the synthetic set against a real multi-expert holdout that no generator touched, scoring how well it reproduces the spread of expert answers, not just the most common one.

Synthetic generation tends to do the opposite by default. A generator prompted for the correct answer returns the most probable one, a filter keeps outputs that agree with a reference, and a deduplication step removes near-identical tasks that differ only in the answer. Each step is reasonable alone; together they convert a field where competent experts split into a field where the data says there is one right answer.

This guide gives a multi-answer task construction protocol for teams that build or buy synthetic training and evaluation data in domains where qualified people disagree: accounting judgments, claims handling, contract interpretation, support policy, underwriting notes. It covers what to record, how to construct tasks, how to test the result and where to stop.

What a single synthetic answer erases

A single answer erases the information that a decision is contested, which is often exactly what a model needs to learn: when to ask, when to state an assumption and when two answers are both defensible. Research on annotation has made this point for years. Davani, Diaz and Prabhakaran argue that in many subjective tasks there is no single right answer, and that a majority vote discards how annotators differed.

Disagreement is not limited to subjective tasks. Pavlick and Kwiatkowski examined human judgments on natural language inference, a task usually treated as having gold labels, and found that disagreements are often not dismissible as noise. The ChaosNLI project collected 100 annotations per example for 4,645 existing items and shows cases where the crowd split nearly evenly; one premise and hypothesis pair drew 51 entailment, 3 neutral and 46 contradiction judgments.

Synthetic pipelines add a second pressure. Shumailov and colleagues, writing in Nature, report that indiscriminate training on model-generated content causes models to lose the tails of the original distribution, a process they call model collapse. Minority expert positions are tails. A pipeline that samples only modal answers, round after round, will remove them faster than any annotator would.

Which disagreements deserve to survive generation?

Not every disagreement should be kept. Some are errors, some come from unclear task wording, and some are legitimate differences in professional judgment or in the facts each expert assumed. Classify each disagreement on the real cases first, because only some classes should be reproduced by a generator.

Which disagreements deserve to survive generation?
Disagreement classHow it shows upKeep in synthetic data?How to represent it
Legitimate judgmentExperts cite the same facts and rules but weigh them differentlyYesMultiple valid answers with each rationale and the observed share
Condition-dependentExperts assumed different unstated factsYes, split by conditionSeparate tasks per condition, or one task with answers keyed to conditions
Policy or version differenceExperts applied different editions of a rule or internal policyOnly with the version statedTag each answer with the policy and effective date it follows
Underspecified taskThe question itself is ambiguousNo, fix the taskRewrite the prompt, then re-collect; keep the original as an ambiguity example only
Slip or errorOne answer contradicts the cited rule or the recordNoCorrect it and log the correction; never weight it as a valid view
Perspective differenceAnswers vary by reviewer background on a subjective callYes, if the use case needs itKeep per-reviewer answers with role metadata, not just a pooled distribution

The multi-answer task construction protocol

Run the protocol in order and keep the artifact from each step, because later reviewers will need to see how a synthetic task traces back to real disagreement. Steps one to four happen on real cases only; generation starts at step five.

The multi-answer task construction protocol
StepWhat to produceAcceptance checkArtifact kept
1. Select real seed casesReal tasks with at least two independent expert answers eachExperts answered without seeing each other's workSeed list with source record ids
2. Split off the holdoutAn untouched real multi-answer test setNo holdout case or near copy is visible to the generatorFrozen holdout file with a hash
3. Collect rationalesEach expert's answer, reasoning, cited rule and confidenceEvery answer has a rationale that names the facts relied onPer-expert answer table
4. Classify disagreementA class for each split case from the table aboveA second reviewer agrees on the class, or it is escalatedClass label and reviewer ids
5. Write the generation recipePrompt, model, grounding sources, policy date, sampling settingsRecipe instructs the generator to produce every valid positionVersioned recipe file
6. Generate per positionSynthetic tasks for each answer and condition, in proportion to the seed sharesEach position appears; no position invented without a seedGenerated set with derivation links
7. Screen and filterPrivacy screen, rule-consistency check, negative controlsFilters do not drop minority answers at a higher rate than majority onesFilter log with drop counts per position
8. Validate on the holdoutDistribution comparison against real expert splitsPre-registered thresholds met for both agreed and split casesValidation report
9. DiscloseDatasheet section on derivation and disagreement handlingA reader can tell real from synthetic and kept from removed viewsPublished datasheet

One record per task: the fields that carry disagreement

Store disagreement as structure, not as a comment field. ChaosNLI's released format is a useful model: alongside the majority label it keeps the label counts, the label distribution, the entropy of that distribution and the original individual annotations, so a user can see both the consensus and the spread. A multi-answer record for professional tasks needs the same idea plus the reasons.

At a glance

One record per task: the fields that carry disagreement

Use these fields for each task, real or synthetic. Fields marked by role or date matter most when the record is later licensed or audited.

  • Task id and derivation: real, edited from real, or generated; parent seed id for anything not real.
  • Grounding sources: the records, policies or documents the task relies on, with versions and dates.
  • Expert answers: one entry per expert with role, answer, rationale, cited rule and stated confidence.
  • Answer set: the distinct valid positions, each with the conditions under which it holds.
  • Observed distribution: counts and shares per position on the real seed, and the entropy of that split.
  • Disagreement class: from the classification table, with the reviewer who assigned it.
  • Generation recipe id: the prompt, model and settings that produced a synthetic item.
  • Screening results: privacy screen outcome, rule-consistency outcome and any correction applied.

Scoring the synthetic set against untouched real cases

Score the synthetic set on whether a model trained or evaluated with it reproduces the real spread of expert answers on the untouched holdout, measured separately for cases where experts agreed and cases where they split. ChaosNLI's scoreboard compares model output distributions with human label distributions using Jensen-Shannon distance and KL divergence, alongside accuracy; the same measures work for multi-answer professional tasks with a small fixed answer set.

Do not assume soft labels always win. In their survey of learning from disagreement, Uma and colleagues report that training on soft labels did better with large datasets and many high-quality judgments per item, and that otherwise combining gold and soft labels generally gave the best results on hard evaluation. Set the training format by how many expert answers each real case actually has.

Include negative controls. Some real holdout cases should be ones where experts agreed. A synthetic mixture that teaches a model to hedge on those has learned false doubt, which is as much a failure as false certainty. The CrowdTruth project's quality metrics for crowdsourcing with disagreement are one open reference for scoring annotators, units and labels without treating every split as noise.

  • Agreement cases: accuracy against the agreed answer, and the share of hedged responses.
  • Split cases: distribution distance against the expert split, and coverage of every valid position.
  • Minority positions: recall of each minority answer with its stated condition.
  • Rule consistency: answers that contradict the cited rule, counted separately from disagreement.

Which real exceptions or conditions does the synthetic mixture fail to represent?

The usual gaps are rare conditions, minority positions with few seeds and answers that depend on a policy version the generator did not see. Check each before release by comparing the generated set's coverage with the real seed, position by position and condition by condition.

Look for these signs: a valid position present in the seed but absent or far smaller in the generated set; conditions that appear in real cases but never in synthetic ones; rationales that cite rules the grounding documents do not contain; and agreement cases that the generator turned into splits. Any of these means the mixture does not represent the real field. Gebru and colleagues' Datasheets for Datasets proposes documenting a dataset's motivation, composition and collection process; disclose these gaps there rather than leaving them for a buyer to find.

Worked example (illustrative): capitalize or expense an invoice line

This example is illustrative and uses counts, not real data. A team builds tasks from accounts payable records: given an invoice line and the company's capitalization policy, should the cost be capitalized or expensed? Three staff accountants review each of 400 real lines independently.

On 340 lines all three agree. On 60 they split. Review of rationales shows three classes among the splits: 28 are legitimate judgment calls about whether a repair extends an asset's useful life; 20 are condition-dependent because the invoice does not say whether the item is part of a larger project; 12 are slips where one reviewer missed the policy's threshold. The team corrects the 12, splits the 20 into paired tasks with the condition stated, and keeps the 28 as multi-answer tasks with each rationale.

Before generation, 80 real lines, including 15 split cases, are frozen as the holdout. The generation recipe asks for every valid position on each seed. After filtering, the team finds the rule-consistency filter dropped minority answers twice as often as majority ones, because it was checking against a single reference answer. They change the filter to check against the answer set, regenerate, and only then run the holdout comparison.

Worked example (illustrative): capitalize or expense an invoice line
Seed group (illustrative)Real linesHandlingSynthetic treatment
All reviewers agree340Single answer keptGenerated with one answer; used as negative controls
Legitimate judgment split28Both answers kept with rationales and sharesGenerated with both positions in seed proportion
Condition-dependent split20Split into paired tasks with the missing fact statedGenerated per condition
Reviewer slip12Corrected and loggedGenerated from the corrected answer only
Frozen holdout80Drawn from the 400 lines above (65 agreed, 15 split) and never shown to the generatorNone; used only for validation

Privacy, rights and the cases a generator should not touch

Synthetic data built from real expert records still carries risk from those records. NIST's guidance on de-identifying government datasets lists publishing synthetic data based on identified data as one data-sharing model among several, and recommends re-identification studies to gauge the remaining risk. Expert rationales are a particular concern: they quote names, amounts and circumstances from the underlying case, and a generator can copy them.

Screen grounding records and rationales before they reach a generator, and test generated outputs for copied passages. Separately, confirm that the business supplying the records may license them for this use and that the experts' written rationales are covered; de-identification alone does not create a right to license. These are questions for privacy and legal reviewers with the specific facts, not for the protocol.

Stop and bring in a domain validator when the disagreement involves professional judgment with legal or safety consequences, when experts disagree about what the rule says rather than how to apply it, or when the seed has too few split cases to estimate a distribution. In those cases, collect more real expert answers instead of generating more synthetic ones.

Sourcing real multi-reviewer records for a grounded mixture

A grounded mixture starts with real cases that were reviewed independently by more than one qualified person, with the reasoning written down. Those records exist in many businesses as second reviews, audit samples, quality checks and escalations, but rarely in a form ready to license.

When a team asks, SourceX looks for second reviews, audit samples and escalations at U.S. businesses, then carries the chosen records through rights review, preparation and delivery as licensed datasets for AI. A request can describe the domain, the number of independent reviews per case, whether rationales are needed, the volume, format and timing, and the licensing terms the team requires.

Frequently asked questions

Is expert disagreement just annotation noise?

Often it is not. Studies of natural language inference found that many disagreements persist and are not dismissible as noise, and in professional tasks experts can apply the same rule to the same facts and reach different defensible answers. The work is to separate those cases from slips and unclear questions.

Should a synthetic dataset ever contain only one answer per task?

Yes, when qualified experts agree and the rule is clear. Those tasks are also useful as negative controls, because a model should not learn to hedge on questions that have one answer.

How many expert answers per case are needed to estimate disagreement?

There is no universal number. Two independent answers reveal that a split exists, but estimating its shape needs more; research on learning from disagreement found soft labels most useful when items had many high-quality judgments. Set the target by how the data will be used and record the count per case.

Can a language model generate the minority expert view on its own?

It can be prompted to, but without a real seed it may invent positions no expert holds or miss the one that matters. Generate minority positions only from documented real ones, and validate on a real holdout.

Does synthetic data remove privacy concerns about the source records?

No. Synthetic data built from identified records still carries disclosure risk, and NIST recommends re-identification studies to gauge it. Screen the grounding records and expert rationales, and test outputs for copied passages.

Sources

  • Davani, Diaz and Prabhakaran (TACL 2022) argue that many subjective annotation tasks have no single right answer and that majority voting loses information about how annotators differ. Source
  • Pavlick and Kwiatkowski (TACL 2019) found that human disagreements on natural language inference are often not dismissible as noise. Source
  • ChaosNLI collected 100 annotations per example for 4,645 items, releases label counts, distributions, entropy and original labels, and scores models against human label distributions with JSD and KL. Source
  • Shumailov and colleagues (Nature 2024) report that indiscriminate training on model-generated content causes irreversible defects in which the tails of the original content distribution disappear. Source
  • Uma and colleagues' JAIR survey reports that soft-label training did better with large datasets and many high-quality judgments per item, and that otherwise combining gold and soft labels generally did best on hard evaluation. Source
  • The CrowdTruth project publishes an open-source library implementing quality metrics for crowdsourcing with disagreement. Source
  • Gebru and colleagues propose that every dataset be accompanied by a datasheet documenting its motivation, composition, collection process and recommended uses. Source
  • NIST SP 800-188 lists publishing synthetic data based on identified data as one data-sharing model and recommends re-identification studies to gauge disclosure risk. Source

Tell us what data you need

Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.

See if your company qualifies