Skip to content

Deal economics

Synthetic preference data validation: when AI-judged pairs need experts

By SourceX Editorial · Reviewed by Noah Loul ·

10 sections

Short answer

Validate synthetic preference data where AI judges are weakest, not on average. GPT-4 matched MT-Bench raters 85% of the time, above the 81% rate between humans, yet LLM judges agreed with dietitians and clinical psychologists only 68% and 64% of the time. Generated pairs can replace people on general chat; human calibration earns its cost on expert items the judge cannot solve itself.

Grader rubric

Synthetic preference data

This page covers synthetic preference data validation: when AI-judged pairs need experts.Illustrative

Key takeaways

  • On general chat, AI judges agree with human raters about as often as the raters agree with each other, so bulk preference labeling is exposed.
  • Agreement with experts was 64-68% in two specialist domains, and judges matched finance experts mainly on questions they could answer themselves.
  • Human-written reference answers can lift a small judge above a larger one, so expert demand shifts from votes to references and adjudication.
  • A headline agreement figure needs order-swap, model-family and human-baseline checks before it means anything.
  • The test to ask for scores five matched-budget arms on a sealed expert set split by judge-solvable items; SourceX has published no such result.

How should an investor judge synthetic preference data validation?

Judge it by agreement on the hardest items, split by whether the AI judge could have answered them itself, and never by one average. Generated preference pairs are cheap and often good enough; the investment question is how large the slice is where they are not, and who gets paid to cover it.

Preference data is a set of comparisons: two model responses to the same prompt and a verdict on which is better. Reward models and methods such as DPO learn from those verdicts. In synthetic preference data an AI judge supplies the verdict, often on responses another model generated. Validation asks whether those verdicts match what a qualified person would have chosen.

For a fund looking at data businesses, the answer sets the size of a market. If AI judges match experts nearly everywhere, human preference labeling shrinks to spot checks. If they fail on a definable tail of hard, expert-only items, spending concentrates there, priced by expert time and reference answers rather than by pair volume.

Where AI-judged pairs already stand in for people

On general helpfulness, harmlessness and chat quality, the published head-to-heads favor substitution. Treat that as the base case any thesis about human data has to beat.

Lee and colleagues (arXiv 2309.00267) compared reinforcement learning from AI feedback with learning from human feedback on summarization, helpful dialogue and harmless dialogue and report comparable performance. On harmless dialogue the AI-feedback policy reached a harmless rate of 88%, against 76% for the human-feedback policy and 64% for the supervised baseline.

On general chat questions, judges agree with people about as often as people agree with each other. In the MT-Bench study (Zheng and colleagues, arXiv 2306.05685), GPT-4's verdicts matched human judges 85% of the time, higher than the 81% agreement among the humans themselves. Cost points the same way: the AlpacaFarm authors (arXiv 2305.14387) designed simulated annotators that they report to be about 50 times cheaper than crowdworkers while agreeing well with human judgments.

Those results cover general questions, not specialist ones. None of them says a judge matches a registered dietitian, an underwriter or a tax specialist on the cases those people find hard.

The hard tail: agreement drops when the judge cannot solve the item

Agreement with domain experts is markedly lower than the general-chat figures, and it falls furthest on items the judge cannot solve on its own. That tail is where human calibration earns its cost.

Szymanski and colleagues (arXiv 2410.20266, 2024) had subject matter experts and LLM judges compare domain-specific outputs. Registered dietitians agreed with the LLM judges on overall preference 68% of the time, and clinical psychologists 64% of the time, with agreement varying across the aspect questions within each domain.

Krumdick and colleagues (arXiv 2503.05061) explain why the tail exists. Their finance benchmark has 160 challenging questions written by financial professionals, with experts grading 1,200 model responses. Without a correct reference answer, judges agreed strongly with the experts only on questions the judges could answer themselves. With higher-quality human-written references, a smaller judge, Qwen 2.5 7B, agreed with human annotators more than GPT-4o did with synthetic references.

That second finding changes the shape of human demand. What the judge lacks on hard items is the answer, not the ability to compare. An expert-written reference can be reused across many generated pairs, so a validated pipeline may need far fewer expert inputs than a design that buys a human vote for every pair.

Bias checks that a headline agreement rate hides

A single agreement figure can sit on top of systematic bias, so ask how it was measured before trusting it. Three checks catch most of the problem.

Order effects come first. In the MT-Bench study, Claude-v1 with the default prompt favored the first answer 75.0% of the time, and the authors also found that judges can prefer longer answers. A pipeline that never swaps response order cannot tell judgment from position.

Family resemblance comes second. Panickssery and colleagues (arXiv 2404.13076) found that LLM evaluators can tell their own outputs from others' better than chance out of the box, and that the strength of self-preference rises linearly with that self-recognition ability. When the generator and the judge come from one model family, a share of the verdicts may reward familiarity.

The human baseline comes third. People disagree too: in the InstructGPT paper (Ouyang and colleagues, arXiv 2203.02155) training labelers agreed with each other 72.6% of the time, plus or minus 1.5 points, and held-out labelers 77.3%. A judge that matches one rater 75% of the time may be as good as another rater, so calibration should compare judge-to-expert agreement with expert-to-expert agreement on the same items.

The calibration-need matrix: six preference segments

Split a preference budget into six segments and the human share becomes visible. Generated pairs can carry the first three with light checks; the last three need expert input, priced as references, adjudication or spread rather than as votes.

The evidence column cites the study behind each judgment, and the studies use different tasks, raters and protocols, so their figures are not directly comparable. The examples come from ordinary business work, such as support replies, invoice review and policy questions, and describe no real company.

The six segments and the calibration each one needs:

The calibration-need matrix: six preference segments
Preference segmentBusiness examplePublic evidence on AI judgesCalibration needWhat to measure
General helpfulness and toneWhich password-reset reply is clearerGPT-4 matched MT-Bench raters 85%, above 81% between humans (2023)Low: spot checksAgreement on a random sample with order swapped
Harmlessness and policy complianceAnswer or decline a risky requestAI-feedback policy harmless rate 88% against 76% for human feedback (Lee et al.)Low to medium: audit borderline promptsExpert review of judge disagreements
Tasks with checkable answersInvoice arithmetic, code with testsNo votes needed when a program can verify the answerSwap preference for verificationTest pass rates, not judge votes
Expert items with a reference availableTax treatment covered by a written policySmall judge with human references beat GPT-4o with synthetic ones (2025)Medium: buy references, not votesAgreement with and without the reference
Expert items the judge cannot solveNutrition or clinical advice, hard finance questions64-68% expert agreement (2024); high agreement only where the judge could answer (2025)High: expert adjudicationAgreement split by judge-solvable and unsolvable items
Contested or value-laden callsTone for an angry customer, trade-offs between policiesTraining labelers agreed 72.6% with each other (InstructGPT, 2022)High: keep the spread, not one verdictDistribution of expert answers per item
Column chart of published agreement rates for AI judges and human raters, from 85% on general chat to 64% with clinical psychologists, grouped under the calibration-need segments they inform.
Agreement figures from three studies (2022-2024) placed on the calibration-need matrix. The studies use different tasks and protocols, so the columns locate segments rather than rank results.

A matched-budget calibration experiment

Run five arms at the same spend and score them on one sealed set of hard items adjudicated by experts. The question is whether human calibration beats more generated pairs where it matters, not on average.

Build the sealed set first. Draw items from the target domain, have at least two experts judge each one independently, adjudicate disagreements and record expert-to-expert agreement as the ceiling. Then tag every item by whether the judge model answers the underlying question correctly on its own; that tag creates the solvable and unsolvable strata. No item in the sealed set may appear in any arm's training data, prompts or judge instructions.

Train the same reward model, or the same DPO run, on each arm with three or more seeds. Report pairwise accuracy against expert verdicts per stratum, judge-to-expert agreement next to the expert-to-expert ceiling, and a blinded expert comparison of policies trained on each arm. Fix the decision rule in advance: human calibration earns its cost if arm C or D beats arm A on the unsolvable stratum by more than the seed-to-seed spread at equal spend; otherwise generated pairs substitute for that segment.

SourceX has not run this experiment and no results are published. The arms are a design for a buyer, a vendor or an investor's technical adviser to run:

A matched-budget calibration experiment
ArmWhat the fixed budget buysWhat it isolatesRead-out on the sealed set
A. Generated pairs onlyThe most AI-judged pairs the budget allowsPure substitutionAccuracy by stratum
B. Generated pairs with bias controlsFewer pairs, each judged twice with order swapped and by a second model familyPosition and self-preference effectsB minus A
C. Generated pairs plus expert calibrationPairs plus expert verdicts on items the judge cannot solve, used to filter and reweightValue of targeted human calibrationC minus A on the unsolvable stratum
D. Judge with expert referencesPairs judged against expert-written reference answersReferences versus votesD minus C
E. Expert pairs onlyThe fewest pairs, all judged by expertsHuman-only ceiling at equal costE minus C

The strongest argument against paying for experts

The best case against human calibration is that AI feedback already matches human feedback on the measured tasks, costs a fraction as much and improves with every judge release. A thesis that needs human preference data everywhere does not survive the RLAIF and AlpacaFarm results.

Even where human data clearly helps, the volumes can be small. NVIDIA's HelpSteer2 (arXiv 2406.08673, June 2024) is a permissively licensed (CC-BY-4.0) dataset of about 10,000 human-annotated response pairs, which the authors describe as an order of magnitude fewer than existing preference datasets such as HH-RLHF. A reward model trained on it scored 92.0% on RewardBench's primary dataset as of June 12, 2024. A small, carefully annotated set was enough for a top score at the time.

That points to a defensible conclusion rather than a dramatic one. Synthetic pairs replace most bulk human preference labeling, and human input becomes a concentrated, expert-priced layer: reference answers, adjudication of hard items and the spread of expert opinion. A business selling that layer can hold pricing power without large volumes, and a business selling generic human votes is exposed.

What could break the human-calibration thesis

Five developments would shrink the expert layer. Track each one before underwriting a business that depends on it:

  • Judges learn the hard tail. If newer judges answer expert questions correctly, the unsolvable stratum shrinks, and the finding that agreement follows judge competence turns from a moat into a countdown.
  • References become synthetic. If model-written references reach the quality of expert ones in a domain, the reference advantage reported for human-written answers disappears there.
  • Verifiers spread. Each task that gains a program-checkable outcome moves out of preference data altogether.
  • Buyers stop measuring. If labs accept average agreement figures, demand for stratified calibration never forms, whatever the evidence says.
  • Expert supply is thin or captive. If the qualified raters in a domain work for a few firms, margins accrue to those firms rather than to data intermediaries.

Questions for any preference-data vendor, SourceX included

Ask a preference-data vendor how much of its revenue comes from the expert layer and how it proves its verdicts on hard items; apply the same questions to SourceX. SourceX sits elsewhere in this market. It works with businesses that choose operating records to license and takes them through qualification, rights review, contracting, preparation, delivery and payment on the way to AI developers. Its relevance here is narrower: business records sometimes hold expert decisions made under real consequences, such as a senior adjuster overruling a recommendation, which can become reference answers or calibration items after rights review.

Buyers request datasets by describing scope, volume, format and licensing needs, and SourceX sources them from U.S. businesses on request rather than from a held inventory. It has published no revenue mix, calibration result or buyer data. Company and commercial metrics go to investors directly, and its Rule 506(c) raise accepts only verified accredited investors. Questions to bring to that conversation, or to any vendor:

  • What share of your preference revenue comes from expert references and adjudication, and what share from bulk pair labeling?
  • Do you report judge-to-expert agreement separately for items the judge can and cannot solve?
  • How do you control for response order and for the judge and generator sharing a model family?
  • What expert-to-expert agreement do you observe on the same items, and how do you treat legitimate disagreement?
  • Who are your experts, how are their credentials checked, and can a competitor hire the same people?
  • Has a buyer ever compared your human-calibrated data with a cheaper generated set at equal spend, and on what held-out items?

Methods note: how the calibration-need matrix was assembled

The calibration-need matrix and the five-arm experiment were drawn up for this article on 2026-10-27 as editorial tools. They rest on Lee and colleagues' RLAIF comparison (2023), the MT-Bench study (2023), AlpacaFarm (2023), Szymanski and colleagues on expert-domain judging (2024), Krumdick and colleagues' finance benchmark (2025), Panickssery and colleagues on self-preference (2024), the HelpSteer2 paper (2024) and the InstructGPT paper (2022). Each was checked through a search excerpt on arxiv.org.

Not measured: the share of any lab's preference budget in each segment, prices for expert or crowd labels, agreement figures for current judges, and results for any arm of the experiment. Agreement figures come from different tasks and protocols and are shown side by side only to locate each segment, not to rank studies. Calibration-need levels are judgments from the cited evidence.

Frequently asked questions

What is synthetic preference data?

It is comparison data in which an AI judge, not a person, decides which of two model responses is better. Reward models and methods such as DPO train on those verdicts. Reinforcement learning from AI feedback uses this kind of data in place of, or alongside, human preference labels.

How often do LLM judges agree with human raters?

On general chat, often: GPT-4 matched MT-Bench raters 85% of the time in 2023, above the 81% agreement between humans. In expert domains the figures drop; a 2024 study found 68% agreement with dietitians and 64% with clinical psychologists. Human labelers also disagree, at about 73% agreement in the InstructGPT paper.

When do generated preference pairs need human calibration?

When the items require expertise the judge lacks, especially questions the judge cannot answer correctly on its own, and when the judgment is contested. For general tone and helpfulness, spot checks with order swapped are usually enough, and tasks with checkable answers need verification rather than votes.

Why do reference answers matter for AI judges?

Because a judge's agreement with experts tracks whether it knows the answer. A 2025 finance benchmark found that a small judge given human-written references agreed with annotators more than GPT-4o given synthetic references. A reference written once can be reused across many generated pairs.

Can business records serve as human preference data?

Sometimes, as references or calibration items rather than ready-made pairs. Records can capture an expert's decision with its consequences, such as an approved exception or an overruled recommendation. They need rights review and careful conversion before use, and they do not cover every domain.

Sources

  • Lee et al.: RLAIF achieves comparable performance to RLHF on summarization, helpful dialogue and harmless dialogue; harmless rate 88% for RLAIF, 76% for RLHF, 64% for SFT. Source
  • MT-Bench study (2023): GPT-4 agreement with human raters 85%, above 81% agreement among humans; Claude-v1 favored the first answer 75.0% of the time; judges can show verbosity bias. Source
  • AlpacaFarm (2023): simulated annotators built from LLM prompts are about 50 times cheaper than crowdworkers and agree well with human judgments. Source
  • Szymanski et al. (2024): experts agreed with LLM judges 68% of the time in dietetics and 64% in mental health on overall preference, varying by aspect question. Source
  • Krumdick et al. (2025): on 160 expert-written finance questions, judges agree with experts mainly on questions they can answer; a small judge with human references beat GPT-4o with synthetic references. Source
  • Panickssery et al. (2024): LLM evaluators recognize their own outputs above chance, and self-preference strength correlates linearly with self-recognition. Source
  • HelpSteer2 (June 2024): CC-BY-4.0 dataset of about 10K human-annotated pairs, an order of magnitude fewer than datasets such as HH-RLHF; its reward model scored 92.0% on RewardBench's primary set. Source
  • InstructGPT (2022): training labelers agreed with each other 72.6% plus or minus 1.5% of the time; held-out labelers 77.3%. Source

Request SourceX investor materials

SourceX shares company and commercial metrics with investors directly. Any offering is made only to verified accredited investors.

See if your company qualifies