AI data market
Post-training data ablation plan: isolate which part of a dataset helped
By SourceX Editorial · Updated
10 sections
Short answer
To isolate which part of a dataset improved a model, retrain the same checkpoint in ablation arms that each remove or mask one component (a modality, field, source system or period) at a matched token budget and fixed seeds, then score every arm on a frozen failure cohort and a regression suite with paired tests. This is a protocol for running that experiment; it reports no results.
Value drivers
Post training data ablation
Key takeaways
- Credit a dataset component only when removing it erases the gain on a frozen failure cohort across several seeds, with an interval that excludes zero.
- Field ablations need two arms: delete the field from the prompt, and keep it but exclude it from the loss; they answer different questions.
- Match arms on supervised tokens and refill removal arms with random candidate examples so they lose the component, not the volume.
- Score sub-cohorts and a regression suite in every arm; a component can carry a gain and a cost at the same time.
- Use datamodels, Data Shapley or gradient-similarity selection to screen components cheaply, then confirm on a split the screen never saw.
- Pre-register the question, population, sampling, comparisons, metrics and limits before the first run, and publish the protocol apart from the results.
How do you isolate which part of a dataset improved a model?
You isolate it with ablation arms: train the same starting checkpoint several times, each time removing or masking exactly one component of the dataset (a modality, a field, a source system or a period) while holding the training budget, hyperparameters and seeds fixed, then score every arm on the same untouched failure cohort and regression suite. The component whose removal erases the gain is the one that produced it.
What follows is an experimental design for a team to run, with no findings attached. It sets out the question, population, sampling, variables, comparisons, metrics, uncertainty handling, limits and a pre-registration checklist for a modality and field ablation experiment. No results are reported here, and every number in the examples is a placeholder for a team to replace with its own measurements.
The design assumes a team has already seen that a candidate dataset helps in aggregate. The open question is which part of that dataset deserves the credit, because that answer decides what to license more of, what to drop and what to ask a supplier to keep in the next delivery.
Freeze the failure cohort and the test set before the first arm trains
The outcome variable has to exist before any training run, or the analysis will drift toward whatever arm looks best. Write down one specific model failure, the cohort of held-out business tasks where it shows up, and the scoring rule for each task. A useful cohort is narrow: refund-policy answers that cite the wrong clause, invoice line items mapped to the wrong cost code, or field service notes summarized without the part that failed.
Keep the test items out of every training arm and check for overlap mechanically. The EleutherAI lm-evaluation-harness documentation notes that evaluations are less useful when test data also exists in the training set, and describes flagging contaminated test documents by n-gram overlap (it uses 13-grams) to produce a clean version of a benchmark. Run the same kind of overlap check between each arm's training file and the frozen cohort, and record the count of flagged items per arm.
Add a regression suite next to the target cohort: tasks the model already handled well before post-training. An ablation that fixes one failure while breaking another capability is not a win, and the only way to see that is to score both in every arm.
Which dataset components can be removed cleanly?
A component is ablatable only if it can be removed or masked without changing anything else in the arm. Before writing the plan, list the candidate components and test each one against that rule. Operational records usually split along a handful of axes, and each axis behaves differently when you take it away.
Illustrative case, with no measured values: a support assistant improved on warranty-claim questions after training on licensed help desk records. Each training example combined the customer message, an attached product photo in some threads, the agent reply, an internal comment and a resolution code. The plan would tag the photo as a modality component, the internal comment and the resolution code as field components, and the two help desk systems the records came from as source components, then build one removal arm per tag.
- Tag every training example with its components before any arm is built; tagging after results invites selective tagging.
- Keep a lineage column (source system, export date, record identifier) on every example so arms can be rebuilt exactly.
- Drop a candidate component from the plan if removing it would also remove more than a stated share of examples; that share is a pre-registered threshold, not a judgment call after the fact.
| Component axis | Examples in business records | How to remove it | What can go wrong |
|---|---|---|---|
| Modality | Ticket text, attached screenshots, scanned invoices, call transcripts, table extracts | Drop the modality from every example that has it and keep the rest of the example | Examples that were only that modality disappear entirely, which shrinks the arm |
| Field | Resolution note, internal comment, product code, outcome label, customer tier | Delete the field from the prompt, or keep it in the prompt but exclude it from the loss | Other fields may quote the removed field, so traces leak back in |
| Source system | Zendesk tickets versus Salesforce cases versus a legacy help desk export | Exclude all examples whose lineage tag names that system | Systems differ in period and volume, so the arm changes in more than one way |
| Period | Records before and after a policy change, a migration or a new product line | Exclude examples by creation date window | Period often correlates with label quality and field fill rates |
| Example type | Single-turn answers versus multi-turn threads versus preference pairs | Exclude by structural tag set during preparation | Example types differ in token length, so budget matching matters most here |
The modality and field ablation experiment plan
The plan below is the asset: a set of arms, each defined by one change from the full candidate set, with the question each arm answers. Run every arm from the same checkpoint, with the same optimizer settings and the same number of supervised tokens. The comparison of interest is always the full arm against one removal arm, never two removal arms against each other unless that contrast was pre-registered.
- Run at least three seeds per arm and report every seed, not only the mean.
- Refill removal arms with randomly drawn candidate examples that lack the component, so the arm loses the component but not the volume.
- Score each arm on the target cohort, each named sub-cohort and the regression suite with the identical scorer and prompt template.
- Store per-item scores, not just aggregates, so paired tests can be run later.
| Arm | Training data | Held fixed | Question it answers |
|---|---|---|---|
| A0 Baseline | Current post-training mix, no candidate data | Checkpoint, hyperparameters, seeds, evaluation | How the model performs on the cohort before anything is added |
| A1 Full candidate | Current mix plus the full candidate set | Same as A0, token budget set here | Whether the candidate set helps at all on the frozen cohort |
| A2 Minus modality | A1 with one modality removed, refilled to the A1 token budget with matched random examples | Budget, seeds, evaluation | Whether that modality carries the gain |
| A3 Field removed | A1 with one field deleted from every prompt | Budget, seeds, evaluation | Whether the model needed to see that field as input |
| A4 Field masked | A1 with the field kept in the prompt but excluded from the loss | Budget, seeds, evaluation | Whether the gain came from learning to produce that field rather than read it |
| A5 Component only | Current mix plus only the examples containing the component | Seeds, evaluation; budget reported separately | Whether the component works alone or only alongside the rest |
| A6 Random control | Current mix plus a random subset of the candidate set the same size as A5 | Budget equal to A5, seeds, evaluation | Whether A5 beats an equally sized slice chosen without the component rule |
Delete a field from the prompt, or keep it and stop training on it?
Deleting a field and masking it from the loss answer different questions, so a field ablation usually needs both arms. Deleting the field (arm A3) tests whether the model needs that information as context. Masking it (arm A4) keeps the context but stops the model from being trained to generate it, which tests whether the improvement came from imitation of that field's text.
Common training libraries already expose the masking half. The Hugging Face TRL documentation for its SFT trainer says that for prompt-completion data the loss is computed on completion tokens only by default, and that setting assistant_only_loss computes loss only on assistant responses in conversational data. Field-level masking goes one step further than either switch: the team marks the token span of one field inside the completion and sets those labels to the ignore index, which the same documentation describes as the mechanism used for padding tokens.
Watch for quotation. In support and case records, an agent's resolution note often repeats the internal comment, and a summary field often paraphrases the description. If the removed field survives inside another field, the ablation understates its effect. Search each arm for long substrings of the removed field and report how many examples still contain them.
Matching budgets and seeds so the comparison is fair
Arms must differ in one component only, which means matching on supervised tokens rather than example counts. A modality removal can drop long examples and short ones unevenly, so an arm that looks the same size by count can receive less training signal. Count the tokens that receive loss in each arm and refill to the A1 figure.
Seed variance is large enough to fake a component effect. Dodge and colleagues report that, when fine-tuning BERT on GLUE tasks, changing only the random seed (which sets the weight initialization of the new layer and the training data order) produced substantial variance in out-of-sample performance. The protocol therefore treats a single-seed difference between arms as unreadable and pre-registers a seed count before any run.
Test differences on paired per-item scores. Every arm is scored on the same items, so the right comparison is item by item, using a paired bootstrap or permutation test. Dror and colleagues surveyed ACL and TACL papers from 2017 and found that significance testing was often ignored or misused; their guide is a practical reference for choosing a test that matches the metric. Report the interval for each contrast, not only whether it crossed a threshold.
Which failure cohort improved, and did an existing capability regress?
Read every contrast at the cohort level, because a component can fix the target failure while quietly harming another group of tasks. Mitchell and colleagues' Model Cards proposal argues for evaluation results disaggregated across relevant groups rather than reported as a single average; the same logic applies to error cohorts in a post-training ablation.
At a glance
Which failure cohort improved, and did an existing capability regress
Set the reading rules before results exist. A component is credited with the gain only if removing it lowers the target cohort score by more than the pre-registered minimum effect, with an interval that excludes zero across seeds, and if the regression suite in the full arm stays within the pre-registered tolerance of the baseline. If removing a component helps the regression suite, record that as a separate finding: the component may carry both the gain and a cost.
- Target cohort: score change from A1 to each removal arm, per seed and pooled, with an interval.
- Sub-cohorts: the same change for each named sub-cohort, for example by product line, record period or customer tier.
- Regression suite: score change from A0 to A1 and from A1 to each removal arm.
- Interaction check: if A5 is close to A1 but A2 is also close to A1, the component is sufficient but not necessary, and another component carries overlapping signal.
- Lineage: the record identifiers and source systems behind any example cluster that moves the result, so a supplier can be asked for more of the same.
When the ablation grid is too expensive, screen with attribution first
Attribution methods can rank candidate components cheaply, but the ranking is a hypothesis for the ablation, not a result. A team with dozens of fields and several modalities cannot afford a full grid, so it screens first and confirms the top candidates with the arms above.
Three published families are useful screens. Datamodels, from Ilyas and colleagues, fit a model that predicts a target example's outcome from which training examples were included, and the authors list predicting the effect of dataset counterfactuals among the uses. Data Shapley, from Ghorbani and Zou, values each training point by its contribution to model performance using the Shapley value, and improves on a simple leave-one-out baseline, at the cost of many retraining runs. LESS, from Xia and colleagues, selects instruction-tuning data by low-rank gradient similarity to a few examples of the target capability.
Each screen has a blind spot. Gradient similarity favors examples that look like the target cohort, which can miss a field whose value lies in generalization. Shapley-style values depend on the evaluation set used to compute them, so computing them on the frozen cohort and then confirming on the same cohort double-counts. Hold back a confirmation split of the cohort that the screen never sees.
Pre-registration checklist and how to report without overclaiming
Pre-register the design in a dated document before the first arm trains, and publish the protocol and the results as separate artifacts. A report should state the population of tasks, the collection dates of the training records, how examples were sampled into each arm, the denominator for every metric, the comparison conditions and the uncertainty, and it should list limits plainly.
At a glance
Pre-registration checklist and how to report without overclaiming
Checklist · 10 items
- Research question: one sentence naming the failure, the cohort and the components under test.
- Population: the source of held-out tasks, how they were drawn and the date range of the underlying records.
- Sampling: the rule that assigns candidate examples to each arm, including the refill rule and its random seed.
- Variables: component tags, token budget per arm, seed list, checkpoint identifier and hyperparameters.
- Comparisons: the exact contrasts to be tested (A1 against each removal arm, A5 against A6) and nothing else as confirmatory.
- Metrics: the per-item scorer for the target cohort, each sub-cohort and the regression suite.
- Uncertainty: the paired test, the number of resamples and the minimum effect that counts.
- Contamination: the overlap check between each arm and the test set, with the flagged count reported.
- Limits: one model family, one checkpoint and one training recipe; effects may not transfer to a different base model or a larger budget.
- Change log: any deviation from the plan, dated, with the reason.
0 of 10 ticked. Tick each item as you go.
Turning a credited component into a sourcing request
Once an ablation credits a specific modality, field or period, the next delivery can be specified around it rather than around raw volume. That request is concrete: for example, case records from a named type of system with the resolution note and outcome label populated, created within a defined window, with lineage identifiers kept.
Since SourceX locates each dataset at U.S. businesses only after a buyer asks, rather than selling from a catalog, the request form can name the credited component directly, along with volume, format, timing and licensing needs. The supplying company approves the scope and terms and signs before any data is delivered, which keeps lineage for the next round of arms documented from the start.
What can an investor conclude from a company's dataset ablation table?
Only as much as the comparison held fixed. Read each reported data gain against four questions: what else changed between the arms, how many runs stand behind each cell, whether the gain survives another training recipe or scale, and whether it exceeds what settings alone move. The protocol above tells a team how to build the arms; the table tells an investor how far each kind of result carries.
| What the company shows | What it supports | What it does not support | What to ask for |
|---|---|---|---|
| One with-and-without pair at one recipe, one run each | That the data moved this model under this recipe once | An effect size, or that the effect repeats | Repeat runs per arm and their spread |
| A gain larger than the recipe-to-recipe spread, with the same ranking at every recipe | That tuning did not manufacture the effect | Transfer to another base model or to post-training | The same arms at a second scale or model family |
| A gain no larger than learning rate or weight decay alone moves one dataset | Nothing about the data yet | Any credit to the data | A full recipe sweep for every arm |
| A gain quoted at the best of several recipes | The upper end of the plausible effect | The typical effect | The gain at every recipe tried |
| Arms that differ in tokens, epochs or repeats | A budget comparison | A data comparison | Matched supervised tokens and repeat counts per arm |
| An aggregate score whose definition has changed | Comparisons within one version of the metric | Comparisons with older reports | Scores under both definitions |
Does a data ranking survive a change of recipe? DCLM's public grid, recounted
In a public grid that crosses training sets with recipes, the ranking survives, but only the large gain clearly exceeds recipe noise. DataComp-LM (DCLM) publishes evaluation files for the models it trained. At the 1B scale, three training sets were each trained under the same four learning-rate and weight-decay settings, one run per cell, and scored on DCLM's CORE average, which rescales each task against its random-guess baseline.
The ranking held at all four settings, and again at 7B under the one setting all three share. A fastText-filtered web pool beat C4 by 4.1 to 6.6 CORE points depending on the recipe, more than the 1.1 to 3.6 points that recipe changes alone moved any single set. A mixed-source set beat C4 by only 0.7 to 3.2 points, inside the 3.6-point spread recipes produced on that set itself, and by 0.6 points at 7B. A company quoting its best recipe would report 6.6 points where another gives 4.1. The grid, in CORE points:
| Scale and recipe (learning rate, weight decay) | C4 | Mixed-source set | Filtered web pool | Filtered pool minus C4 |
|---|---|---|---|---|
| 1B: 0.0003, 0.33 | 19.49 | 20.50 | 23.64 | 4.14 |
| 1B: 0.003, 0.0033 | 19.22 | 20.38 | 25.83 | 6.61 |
| 1B: 0.003, 0.33 | 19.23 | 19.89 | 24.44 | 5.21 |
| 1B: 0.03, 0.0033 | 20.28 | 23.49 | 26.59 | 6.30 |
| 7B: 0.002, 0.05 | 34.58 | 35.22 | 43.04 | 8.46 |
Was it the data, or the recipe, the repeats or the scoring?
Rule out each before crediting data; the same files show why. Recipe: choosing among four settings moved the mixed-source set by 3.6 points, more than its lead over C4 at any of them. Repeats: none of the 92 files that name an open_lm model and a full recipe repeats a training set and setting under a second seed, so run-to-run noise is unmeasured in this grid, while the protocol above asks for at least three seeds per arm.
Scoring: DCLM's README says a September 2025 fix to task baselines makes earlier CORE scores not directly comparable with newer ones, and the files keep both versions. The ranking holds under each, but the gap moves: at 7B the filtered pool leads C4 by 8.46 points on the new version and 7.72 on the old. Coverage: that 7B file lists four chain-of-thought math tasks as missing, so its broader EXTENDED score is stored as not available while CORE, which excludes them, is intact. Read the missing-task list before trusting any aggregate a company reports.
How far does a marginal data gain go in an investment case?
Not far on its own. A pretraining gain on public web text does not show that business records improve post-training, that the gain holds on a buyer's base model, or that anyone will pay for it, and the grid above has one run per cell from one research team. The thesis that licensed business records add marginal value needs ablations on those records at a buyer's scale, with repeats; this page reports no ablation of any SourceX dataset.
For records licensed through SourceX, the supplying company approves scope and terms and signs before delivery, so a buyer's arms can draw only on the approved scope. SourceX's investor materials are available on request through its investors page.
Methods note: DCLM repository at commit 361714b (September 8, 2025), files under exp_data, read October 27, 2026. A 1B cell is a file for model open_lm_1b_swiglutorch with warmup 5,000, cooldown 3e-05, batch 256, multiplier 1, seed 124 and 28,795,904,000 tokens; the training sets are the three present at four or more learning-rate and weight-decay settings, and the recipes are the four all three share. The 7B row uses learning rate 0.002 and weight decay 0.05. Values are stored Core_v2 numbers times 100, with differences computed before rounding. File labels: c4_original; rpj_original, whose dataset record lists Common Crawl, C4, GitHub, Wikipedia, books and arXiv among its sources; and rw_v2_cc_v3_f0.15_resiliparse_fasttext_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train_0.1. Not measured: seed variance, post-training, business records. Script: dclm_recipe_grid.py, held for reviewers.
This article is for information only and is not an offer to sell or a solicitation of an offer to buy securities. Any offering by SourceX is made only to verified accredited investors through its offering documents.
Sources for this section
- DataComp-LM's published experiment records at commit 361714b hold evaluation files for trained models, with CORE scores before and after the September 2025 fix, and dataset records listing each training set's sources; the 1B recipe grid, the 7B check and the seed-repeat count here were recounted from them. Source
- DCLM's README says bug fixes to task baselines mean CORE and EXTENDED scores from before September 2025 are not directly comparable with newer ones, and that both versions are kept in the evaluation files. Source
- DCLM's aggregation script rescales each task score against its random baseline before averaging, and records CORE as not available when a CORE task is missing and EXTENDED as not available when any task is missing. Source
Frequently asked questions
What is a data ablation in post-training?
A data ablation retrains the same starting model with one part of the training data removed or masked and compares the result with a run that includes it. Everything else, including the token budget, hyperparameters, seeds and evaluation set, stays fixed. The difference between the two runs is the evidence that the removed part mattered.
How many seeds does a dataset ablation need?
There is no universal number, but a single seed per arm is not enough because seed choice alone can move fine-tuning results substantially. Pre-register a seed count, run it for every arm and report each seed alongside the pooled estimate. If seed-to-seed spread is as large as the effect, add seeds or treat the result as inconclusive.
Should I match ablation arms by example count or by tokens?
Match by supervised tokens, the tokens that actually receive loss. Removing a modality or field often removes long and short examples unevenly, so two arms with the same example count can receive different amounts of training signal. Refill removal arms with randomly drawn examples until the token budgets match.
Can influence scores replace an ablation?
Influence and valuation methods such as datamodels, Data Shapley or gradient-similarity selection are good screens for ranking candidate components. They are estimates with their own assumptions, so confirm the top-ranked components with real ablation arms on a split the screen never saw before acting on them.
What if removing a component makes the model better?
Record it as a finding, not a failure. A component can carry both the target gain and a cost on another cohort, or it can add noise. Check the regression suite and the sub-cohorts, inspect the examples behind the change, and decide whether to clean, narrow or drop that component in the next delivery.
Sources
- The lm-evaluation-harness documentation says evaluations are less useful when test data also exists in training and describes flagging contaminated test documents by 13-gram overlap. Source
- TRL's SFT trainer documentation describes completion-only loss by default for prompt-completion data, an assistant_only_loss option, and an ignore index for masked tokens. Source
- Dodge et al. (2020) report that weight initialization and data order alone produce substantial variance when fine-tuning BERT on GLUE tasks. Source
- Dror et al. (ACL 2018) found statistical significance testing often ignored or misused in ACL and TACL 2017 papers and give guidance on choosing tests. Source
- Mitchell et al. propose model cards with evaluation results disaggregated across relevant groups. Source
- Ilyas et al. introduce datamodels, which predict a model's output on a target example from the training subset, with uses including predicting the effect of dataset counterfactuals. Source
- Ghorbani and Zou propose Data Shapley, valuing each training point by its contribution to performance via the Shapley value, improving on leave-one-out. Source
- Xia et al. propose LESS, selecting instruction-tuning data by low-rank gradient similarity to a few examples showing the target capability. Source
Tell us what data you need
Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.