Allocating a post-training data budget across SFT, preference and RL data

This guide is for post-training leads and founders working against one fixed number. It sits next to two related pages: what drives the unit cost of human SFT and preference data explains price per record, and how much data you need to fine-tune an LLM covers volume. This page is about the portfolio split. For definitions, see the post-training glossary entry and the fine-tuning and post-training buyer's guide.

What each post-training data type actually buys

Each data type fixes a different failure, so the first budgeting step is mapping failures to data types. The canonical InstructGPT pipeline used three distinct datasets: human-written demonstrations for supervised fine-tuning, human comparisons of model outputs to train a reward model, and a prompt set for reinforcement learning against that reward model [1]. Modern recipes keep the same three roles even when the algorithms change.

• SFT demonstrations (prompt plus ideal response, usually chat-format JSONL with a messages array) teach output shape: schemas, tool-call syntax, refusal style, domain vocabulary, step order. LIMA showed that 1,000 carefully curated pairs could produce strong instruction following on a 65B base model without RL, while the authors noted that curating such examples is labor-intensive [2]. The budget lesson is that SFT spend scales with curation quality, not row count.
• Preference data (prompt, chosen, rejected, or a single completion with a binary label) teaches judgment between acceptable answers. DPO fits the policy directly to pairs without a separate reward model or on-the-fly sampling [3], and TRL's KTOTrainer accepts unpaired rows with a boolean label [8], which changes what you need to buy. See pairwise, ranking or rating formats.
• RL prompts with verifiers (RLVR) teach correctness on tasks where an answer can be checked: unit tests, numeric answers, schema validation, reconciled totals. You buy prompts, reference answers and the checking code, not responses.

Diagnose the bottleneck before splitting the budget

Run your current checkpoint against a held-out evaluation set and tag every failure before committing any spend. A useful tagging scheme has four buckets: format or protocol failure (invalid JSON, wrong tool name, ignored system prompt), behavior failure (wrong procedure, missing steps), judgment failure (valid but worse answer, verbose, unsafe, off-tone), and correctness failure (wrong number, failing test).

Format and behavior failures point to SFT. Judgment failures point to preference data. Correctness failures on tasks you can check automatically point to RL with verifiers; correctness failures you cannot check point back to expert SFT or domain-expert preference data. If most failures are in one bucket, most of the next tranche of budget belongs there.

A common misallocation is buying large preference sets for a model that still breaks output format. Preference optimization assumes both candidates are in the right space; if the rejected and chosen responses are both malformed, the pair teaches little. Fix the protocol with SFT first.

Budget split by model stage

The right split shifts as the model matures, because each stage removes one class of failure. Treat the table below as a starting hypothesis to test with pilots, not a benchmark.

Illustrative example: invented to show structure; it does not describe an available dataset.

The evaluation reserve is deliberate. Evaluation data is a separate purchase with stricter isolation rules (it must never leak into training), and without it you cannot tell whether the rest of the budget worked. Keep it out of the training tranches entirely.

Price the real cost of each data type, not the row

Each data type carries hidden costs that change the effective split. SFT rows written by domain experts cost far more per record than templated ones, but you may need few of them [2]. Preference rows look cheap per judgment, yet annotator disagreement is high: one review of reward-modeling work reports inter-annotator agreement typically in the 63-72% range [5]. That means you pay for adjudication, redundancy (two or three labels per pair) and filtering of ambiguous pairs, which can multiply effective cost.

RL data looks cheapest because you buy prompts and checks, not written answers, but the verifier is where the money goes. Rule-based verifiers can reject correct answers written in nonstandard forms, and model-based verifiers can be gamed by the policy during training, producing false-positive rewards [6]. Verifier errors behave as reward noise in RLVR [7], so budget for verifier testing against adversarial and equivalent-form answers. Compute matters too: RL consumes far more GPU time per prompt than SFT, which competes with the data line in a fixed budget.

Small, high-quality sets can carry a stage. HelpSteer2 showed that a compact, permissively licensed preference dataset could train strong reward models [4]. Before buying volume, check whether open instruction and preference datasets that allow commercial fine-tuning already cover the generic share, and spend money only on the domain-specific remainder.

Favor assets that survive the next model version

Reusable assets keep their value across checkpoints, so they deserve a larger share than their immediate impact suggests. Prompt pools drawn from real-world user requests, annotation rubrics, verifier code and reference answers can be reused for SFT, preference collection, RL and evaluation on every future model.

Responses and preference labels age faster. Off-policy preference pairs were judged on another model's outputs, and as your policy improves those pairs drift from its distribution; see on-policy vs off-policy preference data. Budget implication: buy durable inputs (prompts, rubrics, verifiers, licensed source records) outright, and buy perishable outputs (comparisons on your model's samples) in smaller recurring tranches.

Licensed operational records, such as resolved support tickets, code review threads or finance workflow histories, are one durable input. They can be converted to instruction-response pairs from business records, mined for prompts, or turned into preference signals from accepted versus rejected outcomes, as in code preference data from review outcomes. Confirm the license covers each of those uses before you plan on reuse; a fine-tuning-only license may not cover evaluation or distillation.

Run a staged pilot before committing the full budget

Spend 10-15% of the budget on small pilots of each candidate data type, then fund the type that moves your evaluation most per dollar. A pilot needs a fixed evaluation set, a frozen training recipe and a single variable: the new data.

Illustrative example: invented to show structure; it does not describe an available dataset.

Worked pilot plan for a fixed budget (units are budget shares, not prices)

Pilot checklist

• Schema agreed in writing: JSONL fields, messages roles, chosen/rejected or label, verifier I/O contract.
• Sample of 50-100 records reviewed by your own team before the full tranche.
• Annotator or writer qualifications documented, with the rubric version attached.
• Provenance and license scope recorded per dataset: source, consents, allowed uses (training, evaluation, distillation), term.
• Contamination check against your eval set and public benchmarks.
• Personal-data handling method stated, with the expectation that no de-identification method is perfect.

For volume decisions after the pilot, use how to evaluate a fine-tuning dataset before buying. For the company-wide picture, the AI training data budget guide and data licensing for AI startups on a budget cover how post-training spend fits alongside pretraining and evaluation.

Where licensed operational data fits in the split

Licensed records from real businesses are most valuable where synthetic or crowd data is weakest: real procedures, real edge cases and real outcomes. SourceX sources operational datasets from US companies on request, such as support and sales histories, engineering records, documents, finance and legal workflows, and new recordings of hands-on work, and manages the licensing and any ongoing purchases. Nothing is held in stock, so a request does not guarantee a match. Buyers who want to scope that slice of the budget can describe the records they need to SourceX.

Every dataset is rights-reviewed for ownership and consents and delivered under a license that defines records, uses, term and delivery. Personal details such as names, emails, phone numbers and account numbers are removed or replaced before delivery, the method is recorded and a sample is checked, though no method is perfect. SourceX does not publish prices; terms, including SourceX's own dataset rights, are agreed per deal.

Sourcing the post-training data your budget calls for

Once your failure tags show which data type deserves the next tranche, describe the records, uses and volumes you need rather than the companies you want. SourceX looks for US businesses that hold the described data, works through Find, Assess, Agree, Transact and Manage, and contracts nothing until a supplier agrees. Start a post-training data request at SourceX for buyers.