RLAIF vs RLHF: when to pay for human preference data

The rest of this page treats the choice as a budget split, not a binary. Most post-training programs end up with AI labels for volume and a smaller human or expert set for calibration, hard slices and evaluation. For the format side of the question, see pairwise, ranking or rating formats; for definitions, see the RLAIF and RLHF glossary entries.

What the head-to-head evidence actually shows

The canonical comparison found AI feedback roughly matching human feedback on general tasks, not on every task. Lee et al. (ICML 2024) reported that RLAIF matched RLHF on summarization and helpful dialogue and outperformed it on harmless dialogue, with complementary failure modes: RLHF policies hallucinated more, while RLAIF outputs were sometimes less fluent [1]. The labeler was an off-the-shelf LLM prompted with a preference instruction, which is the same mechanism Constitutional AI style pipelines use with a written set of principles.

Three caveats matter for a buyer. The tasks were general-domain English with crowd-checkable quality, so the result says little about tax memos, radiology impressions or code review in a proprietary stack. The comparison held the human baseline fixed; it did not show that AI labels replace the human data used to build the judges in the first place. And "matches" was measured with human evaluators, which means a human set was still needed to know the AI labels worked.

The judge literature tells the same story from the evaluation side. Zheng et al. found strong LLM judges reach agreement with human preferences comparable to agreement between humans, while documenting position bias, verbosity bias and self-enhancement bias toward the judge's own outputs [4]. Those biases carry straight into DPO and reward-model training if you use judge verdicts as labels [3].

Where AI labels are good enough

AI preference labels hold up when the judgment can be verified from the text, the rubric is explicit and the judge is stronger than the policy being trained. Typical fits:

• Instruction following and format compliance: did the response return valid JSON, respect length limits, cite the provided passage.
• Harmlessness screening against a written policy, where Lee et al. saw AI feedback do well [1].
• Coarse helpfulness on general prompts, especially to bootstrap a first reward model or a first DPO pass [3].
• Pre-filtering: discarding obviously bad candidate responses before humans see the hard pairs.

Open AI-labeled sets such as UltraFeedback, whose ratings came from GPT-4 rather than people, are widely used for exactly this bootstrap role [8]. Check the license of the dataset and the terms of the model that generated the labels separately; the open preference datasets guide covers which ones allow commercial fine-tuning.

Where human or expert preference data earns its cost

Human judgments are worth paying for where the judge model is unreliable, where the target is a human population's taste, or where the labels must stand up in an audit. The pattern is consistent: the judge fails exactly where you most need signal.

The last row is often overlooked. A support ticket reopened within seven days, a pull request reverted, or an edited contract clause is a preference signal produced by real work, not by a paid rater or a judge. Those records need rights review and de-identification before use, but they carry information neither raters nor judges can manufacture.

Cost drivers for the human side (rater qualification, items per hour, multi-rating, rationale length) are covered in what drives the cost of human preference and SFT data.

Human labels are noisy too, so price agreement, not volume

Human preference data is not ground truth by default; it is a measured signal with its own error rate. Work summarized from the "Secrets of RLHF" reward-modeling study reports inter-annotator agreement typically in the 63-72% range, with errors split between mislabeled pairs and genuinely ambiguous ones [6]. InstructGPT and Llama 2 both relied on human rankings or binary comparisons of model outputs to train reward models, and both had to manage rater consistency [2][5].

That changes what a buyer should ask for. Specify multi-rated items on a subset, report Krippendorff's alpha or Cohen's kappa per task slice [10][11], keep "tie" or "both bad" options rather than forcing a choice, and record rater qualifications. The preference data quality guide and the rater calibration guide go deeper on agreement targets and adjudication.

A permissively licensed human-annotated set such as HelpSteer2 shows what documented human preference data looks like: attribute-level ratings, a stated annotation process and a clear license [7]. Use public sets like it as a format reference before you write a collection or licensing spec.

How to calibrate an LLM judge before trusting its labels

Calibrate the judge against a human-labeled subset drawn from your own prompt distribution, using the exact rubric text the judge will receive, and only scale AI labeling on slices where agreement holds. Practitioner guides describe this as a short loop: build the human set, measure agreement, fix the rubric, repeat [11].

Illustrative example: invented to show structure; it does not describe an available dataset.

{
  "pair_id": "cal-0183",
  "slice": "billing_disputes/refund_eligibility",
  "prompt_source": "real_user_request_deidentified",
  "response_a_model": "policy-ckpt-0412",
  "response_b_model": "policy-ckpt-0398",
  "presentation_order": "b_first",
  "human_labels": [
    {"rater_id": "r17", "qualification": "support_tier2_5y", "choice": "a", "confidence": 4, "rationale": "B promises a refund outside the 30-day policy"},
    {"rater_id": "r22", "qualification": "support_tier2_3y", "choice": "a", "confidence": 5, "rationale": "B misstates eligibility"}
  ],
  "judge_label": {"model": "judge-model-x", "rubric_version": "v3", "choice": "b", "swap_consistent": false},
  "adjudicated": "a",
  "disagreement_type": "judge_rewards_fluent_policy_error"
}

Run the checklist below on every slice before using judge labels for DPO or reward-model training.

• Swap test: score each pair in both orders; drop or flag pairs where the verdict flips (position bias [4]).
• Length control: compare judge win rates against response length; a judge that prefers the longer answer in most ties is grading verbosity [4].
• Self-preference check: do not let a judge rank its own family's outputs against a rival's without a human audit [4].
• Agreement threshold per slice: compute judge-versus-adjudicated-human kappa per slice, not one global number [10][11].
• Hard-slice routing: send slices below threshold to humans or experts, and keep the AI labels only where agreement holds.
• Drift re-check: re-run calibration when the policy model, judge model or rubric version changes, since off-policy pairs age quickly; see on-policy vs off-policy preference data.

Terms, provenance and contamination risks of AI-generated labels

AI labels carry rights and provenance questions that human labels do not, so treat them as a licensing input, not just a quality input. Some model-provider terms restrict using outputs to develop competing models; a third-party record of OpenAI's EU terms, for example, tracks a provision of this kind [9]. Read the current terms for the exact judge model and access route you use, and have counsel confirm your use case.

The second risk is undisclosed AI assistance in data sold as human. If a vendor's raters used a chatbot to pick winners or write rationales, you paid human prices for AI labels with unknown biases. Require an annotator agreement that addresses AI-tool use, ask for per-item provenance, and screen deliveries; the human annotation provenance guide and detecting model-generated content in purchased human data cover both controls.

A practical budget split for a post-training program

A defensible default is to spend human budget where it sets the standard and AI budget where it supplies volume. In practice that means four buckets:

1. A multi-rated human gold set per task slice, used only for calibration and evaluation, never for training.
2. Expert or outcome-grounded preference data for the specialist slices where judges failed calibration.
3. AI-labeled pairs for the slices where the judge passed, with swap and length controls applied.
4. A small rolling human audit of AI-labeled training pairs to catch drift.

The ratio between buckets follows from your calibration results, not from a rule of thumb. For allocation across SFT, preference and RL data more broadly, see allocating a post-training data budget and the fine-tuning and post-training data hub. If the expert slice depends on real operational records, such as support resolutions, code review outcomes or document approvals from US companies, SourceX sources that data on request for AI buyers.

Sourcing operational records for the slices AI feedback cannot cover

SourceX sources operational datasets from US companies on request, including support and sales histories, engineering records and new recordings of hands-on work, and manages licensing and ongoing purchases. Every dataset is rights-reviewed, personal details are removed or replaced before delivery, and nothing is contracted until the supplying company agrees. Describe the operational data your expert slices need.