AI data market
Post-training data task selection: how to pick data that fixes a specific model failure
By SourceX Editorial · Updated
10 sections
Short answer
To select training data for a specific model failure, start from the failure, not from a catalog. Measure the error cohort on an untouched test slice, confirm that new examples can fix it, choose a selection rule that fits your candidate pool, then compare the selected examples against a random draw of the same size and check other cohorts for regressions. Buy more only where that comparison shows a gain.
Value drivers
Post training data task
Key takeaways
- Spend the first post-training data budget on a failure that is frequent or severe, measurable on held-out cases, and caused by missing examples rather than by policy or tooling.
- A selection rule is a written, repeatable procedure: retrieval by similarity, distribution matching, gradient influence or expert curation, each with different inputs and blind spots.
- The control that matters is a random draw of the same size from the same pool, trained with the same settings, so the result separates selection from volume.
- A fix only counts if the cohorts the model already handled, its safety behavior and its output format hold steady.
- Every selected example should carry its source, the rule version that picked it and its license scope, so the result can be reproduced and the purchase defended.
Which failure should get the first data budget?
Give the first budget to the failure that scores well on four tests at once: it happens often or costs a lot when it happens, you can measure it on cases the model never trained on, new examples plausibly fix it, and a supplier could actually hold records like it. A failure that passes only one of those tests usually belongs in a prompt, a guardrail or a product decision instead.
Write each candidate failure as a cohort, meaning a set of inputs defined by an observable condition, such as support tickets that mention two orders, or invoices with a credit memo attached. A symptom like the model is wrong about refunds cannot be counted, sampled or bought against. Once the cohort is defined, count how many production cases fall into it over a fixed window and how many of those the model got wrong.
Rank the cohorts that pass on a single page and fund one at a time. Running several targeted purchases in parallel feels faster, but when two selected sets land in the same training run, nobody can say which one moved which metric, and the next budget request rests on a guess.
| Triage test | Question to answer | Passes when | Fails when |
|---|---|---|---|
| Frequency or severity | How often does the cohort appear, and what does one error cost the user? | The cohort is a visible share of traffic, or a single error triggers a refund, escalation or compliance review | The cohort is rare and errors are cheap to correct by hand |
| Measurability | Can you assemble held-out cases with a known right answer? | A reviewer can label the correct output for each case in minutes | Correctness depends on judgment nobody has written down |
| Data-fixability | Would more examples of this pattern change the behavior? | Training data has few or no cases with this condition | The model sees the right information but a tool, retrieval step or policy blocks it |
| Sourceability | Do real businesses hold records with this condition and its outcome? | The condition exists in a system of record such as a ticketing, ERP or claims system | Only synthetic or adversarial prompts would contain it |
| Reversibility | If the fix fails, can you roll back cheaply? | Training runs are short and versioned | A failed run would delay a release or a safety review |
Can more data fix this failure at all?
Only some failures are data problems. Missing coverage, where the training set has few cases with the condition, and shallow reasoning on a recurring pattern respond to examples. Conflicting labels, an unwritten policy, a broken retrieval index or a tool returning the wrong field do not, and buying data for them adds cost without moving the metric.
A quick check is to give the model the missing information directly, for example by pasting the policy clause or the correct retrieved document into the prompt for a sample of failing cases. If accuracy on that sample jumps, the gap is in context or tooling. If it stays low, the model lacks the pattern and examples may help.
Research on image classifiers points the same way. Singla and colleagues started from a handful of failure samples, pulled visually similar examples from a large, noisily labeled pool and trained on those, aiming to improve the failure distribution while keeping performance on the original test set. The method works because the failure is a coverage gap that similar examples can fill, which is the condition to confirm before you buy.
The failure-to-dataset requirements mapping worksheet
Fill in one worksheet per failure cohort before you draw a single training example. The right-hand column is an illustrative entry for an insurance intake assistant; it shows the level of detail each field needs, not a real result. Fields marked as decided before training must not change after the first run, or the comparison stops meaning anything.
At a glance
The failure-to-dataset requirements mapping worksheet
| Field | What to write | Illustrative entry | Owner |
|---|---|---|---|
| Cohort definition | Observable input condition plus the wrong behavior | Submissions listing two or more policies; model extracts only the first policy number | Evaluation lead |
| Measured error, untouched slice | Cases, errors and the date range of the held-out slice, decided before training | Illustrative: 400 held-out cases from the last quarter, 31 percent extraction errors | Evaluation lead |
| Root-cause test | Result of the context-injection check from the previous section | Pasting the full document still leaves most errors, so coverage is the likely gap | Applied researcher |
| Record type and candidate pool | System, record and outcome fields a supplier must hold, and how many candidates exist before selection | Closed intake records with every policy number keyed by a human processor; illustrative 20,000 candidates from own logs plus a supplier sample | Data sourcing lead |
| Selection rule and version | The procedure and its parameters, frozen before training | Embedding similarity to 50 seed failures, top 1,000 by cosine score, rule v1 | Applied researcher |
| Matched-budget control | Same count and token budget drawn at random from the same pool | 1,000 random records, same training settings, three seeds per arm | Applied researcher |
| Success threshold | The gain on the cohort that would justify buying more, set in advance | Illustrative: error rate falls by at least a third against the random arm on every seed | Evaluation lead |
| Regression guards | Cohorts, safety tests and format checks that must not worsen | Single-policy submissions, refusal tests, JSON schema validity | Evaluation lead |
| Lineage fields | Per-example source, rule version and license scope | Record ID, source system, rule v1, license allows fine-tuning only | Data governance |
| Buy decision and review trigger | Buy, narrow, change rule or stop, with the reason, plus what reopens the worksheet | Buy more of the selected kind if the threshold holds on all seeds; reopen on new policy forms or a model upgrade | Budget owner |
Which selection rule suits the candidate pool you have?
Pick the rule by what you already have: a few failure seeds, a target sample that looks like the cohort, a model you can compute gradients on, or expert reviewers with time. Each rule is a bet about what makes an example useful, so write the bet down next to the rule.
The published methods differ mainly in their inputs. LESS selects influential instruction-tuning data for a target capability by running a short warmup training, building a gradient datastore and scoring candidates against target examples; its example scripts keep 5 percent of the pool. DSIR selects raw text distributed like a target dataset by matching feature distributions such as n-gram frequencies, and is built to run on a single CPU node at very large scale. LIMA showed that 1,000 carefully curated prompts and responses could fine-tune a 65B LLaMa model, which makes the case for expert curation when the pool is small and quality varies.
Whatever rule you choose, freeze it before training: the seed set, the embedding or scoring model, the cutoff and any filters for length, language or duplicates. Then deduplicate the selected set against the held-out slice. Similarity retrieval in particular tends to pull near-copies of the very cases you plan to test on, which inflates the measured fix without improving the model on new inputs.
| Selection rule | Inputs it needs | Good fit when | Watch for |
|---|---|---|---|
| Similarity retrieval | A few dozen seed failures and an embedding model | The cohort has a recognizable surface pattern | Near-duplicates of the seeds and of the test slice |
| Distribution matching (DSIR-style) | A target sample that resembles the cohort | The pool is huge and mostly irrelevant | Matching surface features while missing the outcome field |
| Gradient influence (LESS-style) | Model access, a warmup run and target examples | You can afford the compute and the pool is instruction data | Scores that depend on the warmup checkpoint |
| Expert curation | Reviewers with domain judgment and written criteria | The pool is small and quality varies widely | Reviewer drift; keep criteria and a second-rater sample |
| Random draw | Nothing beyond the pool | Always, as the control arm | Treating it as a baseline once and never rerunning it |
How do you show that selection, not volume, fixed the failure?
Run at least two arms with the same number of examples and the same token budget: the selected set and a random draw from the same pool. Train both with identical settings, repeat each over several random seeds, and score both on the untouched slice. If the selected arm beats the random arm on the cohort and the gap holds across seeds, selection did the work.
Small comparisons mislead in a specific way. Card and colleagues found underpowered experiments common in NLP and showed that significant results from small samples tend to overstate, or even reverse, the true effect. So size the held-out slice for the effect you set as your threshold, report the per-seed results rather than only the mean, and show an interval around the difference.
Two optional arms sharpen the answer. A no-new-data arm, which retrains with the same settings on the existing mix, tells you how much of any gain comes from the run itself. A larger random arm tells you whether simply buying more of everything would have reached the same place at a different volume. The pre-purchase evaluation guide linked below covers arm design for one supplier sample in more depth.
Which regressions must the comparison rule out?
A targeted set fixes nothing if it breaks what already worked. Define the regression guards in the worksheet before training and treat any guard that worsens beyond its own threshold as a failed run, even when the target cohort improved.
At a glance
Which regressions must the comparison rule out
Safety behavior is the guard teams forget most often. Qi and colleagues showed that fine-tuning an aligned model even on a benign, utility-oriented dataset can erode its safety alignment, so a narrow business fine-tune needs its own refusal and harmful-request checks rather than an assumption that clean data is safe.
- Neighbor cohorts: the inputs that look like the target but should be handled differently, such as single-policy submissions next to multi-policy ones.
- Previously fixed failures: every cohort that an earlier data purchase repaired, rerun with its original slice.
- Safety and refusal: the harmful-request and over-refusal sets your release process already uses.
- Format and schema: the share of outputs that pass your JSON schema or field validator.
- Length and tone: response length and style drift, which selected sets with a uniform style can introduce.
- General capability: a small fixed sample of broad tasks, so a narrow gain does not hide a broad loss.
Would the gain survive a different supplier's records?
The assumption that most often flips the decision is that the candidate pool looks like what a supplier will deliver at volume. If the pilot pool came from your own logs and the purchase comes from another company's system, field names, outcome codes and writing style can differ enough to erase the gain. Test the rule on a supplier sample before committing.
Three other assumptions deserve a line each in the decision note: that the cohort's share of traffic will stay roughly where it is, that the held-out slice represents production rather than the easiest cases, and that a gain at one volume continues at a larger one. Show uncertainty as a range for the cohort gain across seeds, the regression guard results with their own ranges, and a plain statement of which assumption would reverse the recommendation if it proved wrong.
Write the decision note so that someone outside the team could repeat it: the cohort definition, the rule version, the arms, the seeds, the threshold set in advance, the result per seed and the guards. A note that only reports the best seed or the mean gain invites a second purchase that the evidence does not support.
Worked example (illustrative): multi-policy insurance submissions
All figures here are illustrative and describe no real model or dataset. An intake assistant extracts policy numbers from broker submissions. The team defines the cohort as submissions listing two or more policies and assembles 400 held-out cases from one quarter, where the model misses at least one policy number in 31 percent of them.
Pasting the full submission into the prompt barely helps, so the team treats the gap as coverage. It builds a pool of 20,000 candidate intake records, selects 1,000 by similarity to 50 seed failures, and draws 1,000 at random as the control. Both arms train with the same settings over three seeds.
The selected arm cuts cohort errors by more than a third on every seed; the random arm moves them only slightly. Single-policy accuracy and schema validity hold, but one seed shows a small rise in refusals on routine requests, so the team adds 100 ordinary single-policy examples to the selected set and reruns that guard before approving more purchases of multi-policy records.
The exception is worth noting. Had the random arm matched the selected arm, the conclusion would have been that volume, not selection, drove the gain, and the cheaper decision would have been to buy general intake records instead of a narrowly specified slice.
What lineage has to travel with each selected example?
Each selected example should carry enough lineage to rebuild the training set and to show where it came from. Without it, a later reviewer cannot tell which rule version picked which record, whether the license allows the use, or whether a test case leaked into training.
Dataset documentation standards already name most of these fields. The Croissant RAI vocabulary from MLCommons covers who created a dataset and why, which sources were used, versioning with timestamps, how the data was processed and the provenance and lineage of derived datasets. Record the same facts per example: source system and record ID, collection window, selection rule and version, selection score, license scope and the deduplication check against the held-out slice.
Lineage also protects the next purchase. When a supplier later offers more records of the same kind, the team can apply the identical rule version to the new pool, compare the selection scores with the first batch and spot a drift in record type or outcome coding before any training run is spent on it.
From a measured cohort to a sourcing request
A filled-in worksheet is also the clearest data request you can send. It states the record type, the outcome field, the volume band suggested by your matched-budget run, and the license scope you need, which is what lets a supplier say yes or no quickly.
Datasets for a targeted fix like this are not held in stock: SourceX sources them for AI teams from U.S. businesses on request, and its request form asks for the scope, volume, language, format, timing and licensing details your worksheet already records. On the supply side it works with U.S., Canadian and EU companies that had at least 50 full-time employees at their peak, not counting contractors, and it can consider companies in the UK, Australia, New Zealand, India, Singapore, South Africa, Latin America, the UAE and Saudi Arabia when the records on offer are at least 90 percent English, measured on the dataset itself. Delivery waits until the supplying business has approved the scope and terms and signed its agreement.
Which of six causes is behind the failure, and what should you try before buying data?
Buy data only when the failure traces to learned judgment, or to domain knowledge that cannot be supplied at run time. The other causes have cheaper fixes, and each can be tested in about a day on the same failing cases the cohort was built from.
The context check earlier on this page separates missing context from missing coverage. Two more tests close the remaining gaps. First, give a qualified person exactly the inputs, tools and access the model had: if they cannot finish the task either, the gap sits in context, tooling or permissions, and no training record will close it. Second, inspect the task itself. Anthropic's engineering team wrote on January 9, 2026 that a 0 percent pass rate across many trials, such as 0 percent at pass@100, is most often a sign of a broken task rather than an incapable agent.
OpenAI's guide to optimizing LLM accuracy draws the remaining line: knowledge that was never in training, is out of date or is proprietary is a context problem for retrieval or prompting, while fine-tuning on examples is a lever for consistency. Write the outcome of each test into the worksheet's root-cause field. A request that cannot name its cause in one row of the table below is not ready to send.
| Cause | One-day test on the failing cases | Points to this cause when | Try first | Data that could help |
|---|---|---|---|---|
| Broken task or grader | Two reviewers regrade 20 failures against the written criteria and reference answer | Reviewers disagree, or mark many failures as correct | Rewrite the criteria and reference answers | None until the test itself is fixed |
| Missing context | Paste the needed document or record into the prompt, as in the context check above | Accuracy recovers once the information is present | Retrieval, input or prompt change | None for training; at most a reference corpus to retrieve from |
| Tooling | Replay the failures with the correct tool output substituted | Errors disappear when the tool returns the right field | Fix the tool, schema or parser | None |
| Permissions | Give a qualified person the same access the model had, and read the access logs | The person fails too, or the logs show denied or missing access | Change the access, or route the step to a person | None; this is an access decision, not a data gap |
| Missing knowledge | Ask the domain questions directly, without the task around them | The model lacks facts a professional in the role would know | Retrieval over reference material | Reference documents, only if they cannot be reached at run time |
| Learned judgment | Run the cases with correct context, tools and access all in place | Errors persist although the model had everything a professional had | A few worked examples in the prompt, as a cheap probe | Records of how professionals decided similar cases, with the outcome |
How do you price being wrong and rank failures when the rates are uncertain?
Rank each cohort that passed triage by expected weekly cost: errors per week, times a severity weight, times the chance a data remedy works, each carried as a low and a high value. The weights belong to whoever absorbs the errors; the ones below are placeholders, not industry values.
Cost per error is rarely one number. scikit-learn's cost-sensitive learning example, checked on October 26, 2026, uses a credit cost matrix in which accepting a bad credit costs, on average, five times as much as rejecting a good one, and a fraud case in which the cost of an error depends on each transaction's amount. Anthropic's guide to success criteria sets a target that 90 percent of errors cause inconvenience rather than egregious error, notes that a team still has to define those words, and adds that citation accuracy matters more in a medical app than in a casual chatbot.
Have the owner of each consequence, such as claims operations or compliance, sign the four classes below. In the illustrative table every figure is invented, and ranges multiply low by low and high by high. The multi-policy cohort ranks first under the placeholder weights, but its range overlaps the endorsement cohort's, and a class 4 weight of 400 reverses the order.
That sensitivity is the finding: the purchase should rest on one signed weight, or on another month of counts until the ranges separate, never on midpoints alone. A broad category such as more insurance documents never enters the ranking, because it has no error count, severity class or remedy chance with which to beat a repeated, costly, data-fixable failure.
- Class 1, cosmetic: corrected in under a minute, and nothing downstream changes. Placeholder weight 1.
- Class 2, rework: a step is redone or a ticket reopens before a customer is affected. Placeholder weight 5.
- Class 3, customer-visible: a wrong answer, delay or refund reaches a customer. Placeholder weight 25.
- Class 4, financial or regulated: money moves to the wrong place, a regulated decision is affected or a safety rule breaks. Placeholder weight 100, and every class 4 cohort also gets its own review.
| Cohort (illustrative) | Errors per week and main class | Data-remedy chance | Weekly cost units, placeholder weights | Weekly cost units, class 4 weight 400 |
|---|---|---|---|---|
| Submissions listing two or more policies | 50 to 80, class 2 rework | 0.6 to 0.8, coverage gap confirmed | 150 to 320, rank 1 | 150 to 320, rank 2 |
| Mid-term endorsements that change the policy number | 2 to 4, class 4 misrouted claim payment | 0.3 to 0.5, judgment with few examples | 60 to 200, rank 2 | 240 to 800, rank 1 |
| Broker name formatting | 100 to 150, class 1 cosmetic | 0.4 to 0.6, inconsistent conventions | 40 to 90, rank 3 | 40 to 90, rank 3 |
| Low-quality scanned attachments | 30 to 60, class 2 rework | Near zero, the OCR step fails | Not ranked: fix the OCR step | Not ranked: fix the OCR step |
What belongs in a dataset specification written from a product incident?
Copy seven lines from the incident record into the specification: the task inputs, the behavior a professional produced, the condition that defines repeats, the cases deliberately excluded and why, what makes a delivered record unusable, the acceptance test, and what the incident review ruled out. Exclusions, rejection grounds and ruled-out causes are the lines most requests drop, and without them a supplier can only send look-alikes.
Anthropic's eval guidance of January 9, 2026 recommends testing both the cases where a behavior should occur and the cases where it should not, and keeping a reference solution for each task. Both habits carry over: exclusions are the should-not cases, and the processor's corrected output is the reference solution. The table continues the endorsement cohort from the ranking above.
Keep the first evidence packet small. The same Anthropic post calls 20 to 50 tasks drawn from real failures a good start for an eval set; that is enough to show the failure repeats and the reference answers can be graded, not to measure a modest gain, which needs the sizing described in the matched-budget comparison above.
Write every rejection line so two reviewers reach the same verdict. On OpenAI's GDPval benchmark, whose tasks come from professionals averaging 14 years of experience and are graded blind by experts in the same occupations, human expert graders agreed with each other 71 percent of the time (Patwardhan and colleagues, 2025). Narrow yes-or-no record tests leave less room for disagreement than those open-ended deliverables; if two reviewers split on a sample, rewrite the line before a supplier sees it.
Send the finished specification with the data request. SourceX keeps no inventory and goes looking among U.S. businesses only once a buyer asks, so the specification is the only thing a candidate supplier can screen its records against.
| Specification line | Copy from the incident record | Illustrative entry | Rejection or test it enables |
|---|---|---|---|
| Task inputs | The exact documents and fields the model received | Broker submission plus an endorsement form that changes the policy number mid-term | Records without the endorsement form are rejected |
| Desired behavior | The output the processor produced when correcting the case | Current policy number extracted, superseded number flagged | Gives the acceptance test a reference answer |
| Inclusion condition | The observable condition shared by the incident and its repeats | Endorsement effective before the claim or renewal date | Defines what the supplier searches for |
| Exclusions, with reasons | Cases the review decided this fix should not cover | Cancellations and reinstatements, which follow a separate rule; lines of business the product does not write | Stops look-alike records from padding the volume |
| Record-level rejection | What makes a delivered record unusable | No final policy number keyed by a person; missing effective date; near-duplicate of a held-out case | Lets the buyer reject records on stated grounds instead of disputing a whole delivery |
| Acceptance test | Held-out repeats with reference outputs and two graders | 30 endorsement cases from after the incident, never shared with any supplier | Measures the fix on cases no supplier saw |
| Negative result kept | Causes the review tested and ruled out | Retrieval did return the endorsement; the model read it and still chose the older number | Shows why data is the remedy, and reopens the specification if that stops being true |
Is a rare case worth buying, or only unusual?
A rare case is worth buying when it occurs in the population the model will actually serve, the model fails on it, and an error there sits in a high severity class. An unusual record that the deployment never sees is irrelevant, however scarce it is.
Anthropic's testing documentation tells teams to design evaluations that mirror the real-world task distribution, and lists irrelevant or nonexistent input data among the edge cases to test. An irrelevant record belongs in a test of how the model declines or routes it, not in training. Run this coverage test on your own logs before asking for a supplier sample:
- Define the deployment population: the channels, products, form versions and regions the model will serve next quarter.
- Count the rare condition there over a fixed window. Zero cases in n still allows a rate of about 3 in n at 95 percent confidence, since solving (1 - p) to the power n = 0.05 gives roughly 3 divided by n; zero in 2,000 leaves room for about 0.15 percent, a reason to keep watching rather than to buy.
- Confirm the model fails on the cases that do appear, and record their severity class from the rubric above.
- Check that candidate records share the deployment's condition, not only its rarity: same product, same form version, a process still in use.
- Rejection example (illustrative): a supplier sample offers thousands of handwritten, faxed submissions from a broker channel the insurer closed two years ago. The model fails on them, but current traffic contains none and the form is retired, so the sample is rejected with the reason recorded. Mid-term endorsements occur every week in current traffic and sit in class 4, so they pass.
Sources for this section
- Anthropic's engineering post (January 9, 2026) says 20 to 50 tasks drawn from real failures is a good start, that a 0 percent pass rate across many trials most often signals a broken task, and recommends testing cases where a behavior should and should not occur and keeping a reference solution per task. Source
- Anthropic's success-criteria guide gives a target that 90 percent of errors cause inconvenience rather than egregious error, notes that a team would define those terms, and says citation accuracy may be critical for medical apps but less so for casual chatbots. Source
- Anthropic's testing guide says to design evals that mirror the real-world task distribution and lists irrelevant or nonexistent input data among example edge cases. Source
- OpenAI's accuracy guide treats knowledge missing from training, out of date or proprietary as a context problem, and describes fine-tuning on examples as a way to improve consistency. Source
- scikit-learn's cost-sensitive learning example uses a credit cost matrix where misclassifying a bad credit as good is five times as costly as the reverse, and a fraud case where the business cost depends on each transaction's amount. Source
- GDPval tasks were built by industry professionals averaging 14 years of experience and are graded by experts from the same occupations who blindly compare model and human deliverables. Source
- The GDPval paper reports human expert inter-rater agreement of 71 percent on its gold subset, with an experimental automated grader at 66 percent. Source
Frequently asked questions
What is post-training data task selection?
It is the practice of choosing which tasks and examples go into fine-tuning or preference training based on a measured need, usually a specific failure. It covers which failure to target, which rule picks the examples, and how the choice is tested against a same-size random draw.
How many examples does a failure-targeted fine-tune need?
There is no fixed number. LIMA showed that 1,000 carefully curated examples could shape a large model's behavior, but a narrow business failure may need fewer or more depending on how varied the cohort is. Run the matched-budget comparison at two sizes and let the difference guide the purchase.
Should the failing cases themselves go into training?
Keep the measured failures out of training and use them as the held-out slice or as seeds for retrieval. If they enter training, the test can no longer show whether the model learned the pattern or memorized the cases.
Can an LLM judge select the training examples instead of gradients or embeddings?
Yes, a judge prompted with written criteria is a form of curation and can be the selection rule. Treat the judge prompt and model version as part of the rule, check a sample against human ratings, and still compare against a random draw of the same size.
What if the selected set fixes the failure but hurts another cohort?
Treat it as a failed run. Common repairs are mixing in examples of the neighbor cohort, lowering the share of selected data in the mix, or tightening the selection rule, then rerunning every guard before you buy more.
Sources
- LESS selects influential instruction-tuning data for a target capability using warmup training and a gradient datastore; its example scripts keep 5 percent of the pool. Source
- DSIR selects data distributed like a target dataset by matching a feature space such as n-gram frequencies. Source
- LIMA fine-tuned a 65B LLaMa model on 1,000 carefully curated prompts and responses. Source
- Data-centric debugging retrieves examples similar to a few failure samples from a noisy pool to improve the failure distribution while keeping original test performance. Source
- Underpowered experiments are common in NLP, and significant results from small samples tend to overstate or reverse the true effect. Source
- Fine-tuning an aligned model on a benign, utility-oriented dataset can degrade its safety alignment. Source
- The Croissant RAI vocabulary documents data sources, versioning with timestamps, processing, and the provenance and lineage of derived datasets. Source
Tell us what data you need
Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.