AI uses for records
Dataset linkage utility ablation: how to measure linked records
By SourceX Editorial · Updated
10 sections
Short answer
Measure the value of linked records with a pre-registered ablation: derive linked and unlinked training sets from the same cases, change only the joins, train the same model on each with an equal budget and several seeds, and compare paired scores on a frozen holdout with a confidence interval. This page gives the protocol and a comparison template; it reports no results.
Agent run
Dataset linkage utility
Key takeaways
- Measure linkage value with a pre-registered ablation that changes only the joins between records, never their content.
- Derive every arm from one de-identified, deduplicated corpus by severing or scrambling keys, and add a mislinked control to rule out format effects.
- Split by case and customer so no case appears in both training and test data, and watch for outcome fields that leak the target.
- Report paired differences across several seeds with a confidence interval and a pre-specified smallest effect of interest.
- Single-record tasks act as negative controls; a gain there points to a confound, not to linkage.
- This is a method only; no results are reported here.
How do you measure the training value of keeping records linked?
You measure it with a pre-registered ablation: build two training sets from the same source cases, identical in every respect except whether the records that belong to one case stay joined, train the same model on each with the same budget, and score both on an untouched holdout of tasks. The paired difference, with an uncertainty interval, is the estimate of what linkage adds.
This page is a protocol, not a study. It reports no results and contains no measured effect sizes. It sets out the question, population, sampling, variables, comparison conditions, metrics, uncertainty method, known confounders and a pre-registration checklist, so that a data buyer, a supplier or an independent evaluator can run the experiment and publish the outcome with its limits.
The reason a protocol is needed at all is that the intuition behind linked records is easy to state and easy to fool yourself about. A linked quote, order, job, invoice and callback chain looks richer than the same documents scattered across folders. Whether a model actually learns more from the chain, for a specific task, is an empirical question, and a sloppy comparison will usually overstate the answer.
What exactly is the treatment variable in a linkage ablation?
The treatment is the presence of stable joins between records that describe one business case; everything else is held constant. Define linkage before any data is touched, because a vague definition lets the linked arm quietly gain content the unlinked arm lacks.
In business records a case is usually a chain across systems: a request in a CRM such as Salesforce or HubSpot, a quote or order in an ERP such as NetSuite, a work order in a field service tool such as ServiceTitan, an invoice, and an outcome such as a callback, return, credit memo or closed ticket in Zendesk or Jira. Linkage is the set of keys (order number, job number, ticket ID, customer account) that let a reader follow that chain.
Write the linkage levels you will test into the pre-registration. The four levels below are a reasonable default; most studies compare the top and bottom levels first and add intermediate levels only if the first comparison is informative.
| Level | What the model sees | Typical construction | What it isolates |
|---|---|---|---|
| L0 unlinked | Each record as an independent example, keys removed | Strip join keys and shuffle record order across cases | Content of individual records only |
| L1 within-system | Records from one system grouped by case | Group CRM or ticket history by case ID, no cross-system joins | Value of history inside one application |
| L2 cross-system | Full chain from request to invoice | Join CRM, ERP and job records on shared keys | Value of joins across systems |
| L3 chain with outcome | Full chain plus the recorded result | Add callback, return, credit or resolution record | Value of connecting decisions to outcomes |
| Mislinked control | Chains joined to the wrong case of the same type and period | Re-pair records at random within strata | Whether gains come from true joins or from format |
What should the linked versus unlinked case comparison include?
The comparison table is the core asset: every row is a variable that must be identical across arms, with the one exception being the linkage itself. If any row other than linkage differs, the result measures that difference instead.
Fill this in before training and attach it to the pre-registration. The three illustrative entries listed next show how one fictional job appears in each arm; the table after them lists what must match.
- Illustrative case, linked arm: one training example contains the service request text, the dispatcher's priority choice, the technician's work notes and parts list, the invoice lines, and a callback record dated later that names the same job number.
- Illustrative case, unlinked arm: the same five records appear as five separate examples, keys removed, in shuffled positions in the training stream, so the model never sees that the callback belongs to that job.
- Illustrative case, mislinked control: the same request, notes and invoice are joined into one example, but the callback comes from a different job of the same type and month.
| Variable | Linked arm | Unlinked arm | Why it must match |
|---|---|---|---|
| Source cases | Same case IDs | Same case IDs | Different cases would change content, not linkage |
| Record content | Every field and free-text note | Every field and free-text note | Missing fields would be a content ablation |
| Token or example budget | Fixed total tokens | Same total tokens | More tokens alone can raise scores |
| Formatting and templates | One serialization template | Same template per record | Format cues can be learned as shortcuts |
| De-identification | Same method and pass | Same method and pass | Different redaction changes what is learnable |
| Deduplication | Applied before arm split | Applied before arm split | Duplicates inflate apparent learning |
| Model, seeds, hyperparameters | Same base model, same seed set | Same base model, same seed set | Tuning one arm harder biases the result |
| Evaluation items | Same frozen holdout | Same frozen holdout | Paired scoring is what makes the test sensitive |
How should you build the unlinked arm without changing anything else?
Build the unlinked arm by severing joins in the linked arm, not by assembling a second dataset from scratch. Start from one fully linked, de-identified and deduplicated corpus, then derive every arm from it with a script that only removes or scrambles keys and grouping. That way the content is identical by construction.
Severing has two parts. First, remove or randomize identifiers that let a model reconnect records on its own, including order numbers repeated inside free text, customer names that survived de-identification, and dates that make a chain obvious. Second, break the grouping, so records from one case are not adjacent in the training stream. If you only drop the key column and keep records in case order, the model can still learn the chain from position.
Add the mislinked control whenever you can afford a third arm. It holds format and example length equal to the linked arm while destroying the true relationship. If the linked arm beats the unlinked arm but not the mislinked control, the gain probably came from longer, multi-record examples rather than from the real joins, which is a very different finding for anyone deciding what records to license.
Partial linkage is a natural extension. Real archives are rarely all or nothing: some jobs have invoices and some do not. If you want a dose-response view, pre-register a small number of linkage rates and build each by severing a random subset of cases, stratified by case type and period.
Which population, sampling and splits keep the comparison honest?
The sampling unit is the case, not the record, and every split must keep all records of a case, and ideally all cases of a customer or site, on one side. If records from one job land in both training and test sets, the linked arm can score well by memorizing the job rather than learning anything general.
Scikit-learn's documentation describes exactly this risk for grouped data: when samples within a group are dependent, a model can overfit to group-specific features, and group-aware splitters such as GroupKFold ensure the same group is not represented in both training and test sets. For business records, the group is the case, customer account, site or technician, depending on what the model could memorize. Where cases evolve over time, hold out the most recent period as well, in the spirit of a time-ordered split, so the evaluation also reflects how the data would be used going forward.
Kapoor and Narayanan's survey of leakage in machine-learning-based science is a useful warning list here: train-test overlap, features that would not be available at prediction time, and preprocessing fitted on the full dataset all produce optimistic results. In a linkage study the classic trap is an outcome field (for example a credit memo amount) that leaks the target the model is asked to predict.
Define the population in writing: which systems, which record types, which date range, which business units, what share of cases had a full chain before any filtering, and which cases were excluded and why. Readers need the denominator to judge whether linked cases are representative of the whole archive.
Which tasks and metrics show whether linkage helped?
Use two families of evaluation tasks: tasks that require information from more than one record in a chain, where linkage should matter, and single-record tasks, where it should not. The second family is a negative control; a large gain there signals a confound rather than a linkage effect.
Pick metrics that a business reader understands, such as the accuracy of a predicted outcome or the share of correct next steps, and pre-specify one primary metric. Secondary metrics are fine, but report them as secondary.
| Task family | Needs linkage? | Example metric | How to read a difference |
|---|---|---|---|
| Predict the outcome of a case from its request and decisions | Yes | Accuracy or calibration of predicted callback, return or credit | Primary test of linkage value |
| Recommend the next step in a case | Yes | Exact-match or expert-rated correctness | Tests whether chains teach process order |
| Reconcile records across systems | Yes | Precision and recall of correct matches | Direct measure of cross-record reasoning |
| Classify a single record by type | No | Accuracy on record category | Negative control; should not move |
| Extract fields from one document | No | Field-level exact match | Negative control; movement suggests a confound |
| General capability check | No | Score on a fixed public or internal regression set | Detects damage from either arm |
What sample, baseline and uncertainty would make a result defensible?
A defensible result reports a paired difference on the same frozen evaluation items, across several training seeds, with a confidence interval and a pre-specified smallest effect worth acting on. A single run per arm with a point score is not evidence of anything.
Bouthillier and colleagues show that sources of randomness such as data sampling, parameter initialization and hyperparameter choice noticeably affect benchmark results, and that robust comparisons generally require repeating the full training pipeline. Budget for multiple seeds per arm from the start; if compute is tight, reduce model size rather than seeds.
Because both arms are scored on the same items, use paired methods. SciPy's bootstrap function computes a confidence interval of a statistic by resampling and supports paired data, and its permutation_test function supports a samples permutation type that its documentation describes as appropriate for paired-sample tests. Dror and colleagues found that significance testing in NLP papers was often skipped or misapplied, and their guide is a good reference for choosing a test that matches the metric.
Size the evaluation set from a pilot. Run a small version of both arms, measure the variance of the paired difference, and pick the number of evaluation items and seeds that would detect your smallest effect of interest. Write that number down before the main run. The baseline is the unlinked arm, not zero; also report the base model with no fine-tuning, so readers can see how much either arm moved it.
What alternative explanations could account for an apparent improvement?
The most likely alternative explanations are more effective training signal per example, leakage of identifiers or outcomes, and selection bias in which cases happen to be linked. Each has a specific check, and the report should state the result of every check, including the ones that found nothing.
At a glance
What alternative explanations could account for an apparent improvement
- Longer examples: linked examples are longer and may simply carry more context per gradient step. Check: the mislinked control holds length equal.
- Token budget drift: severing can change tokenization or template overhead. Check: count tokens per arm and equalize.
- Identifier leakage: case numbers or customer names inside free text let the model reconnect records or memorize cases. Check: search the unlinked arm for surviving keys.
- Target leakage: an outcome field present in the input. Check: list every input field and confirm it existed at the time the prediction would be made.
- Duplicate and near-duplicate records across splits: Lee and colleagues found that common language-model datasets contain many near-duplicates and that deduplication reduced train-test overlap. Check: deduplicate before splitting and report the method.
- Selection bias: well-run jobs may be the ones that got fully linked, so linked cases are easier. Check: compare the same case IDs in both arms, never linked cases against a different set of unlinked cases.
- Tuning asymmetry: more hyperparameter search on one arm. Check: one shared search, or none, fixed before the main run.
- Evaluation contamination: holdout items written by someone who saw the linked arm. Check: freeze and hash the evaluation set first.
Pre-registration checklist for a linkage ablation
File the plan before training any arm. The Center for Open Science describes preregistration as specifying the research plan in advance and submitting it to a registry, which separates planned confirmatory analysis from exploratory analysis. For private studies, a dated internal document hashed and shared with the counterparty serves the same purpose.
Document the dataset alongside the plan. The Datasheets for Datasets template asks about motivation, composition, collection process, preprocessing, uses, distribution and maintenance, and those answers are what a reader needs to judge whether your linked corpus resembles theirs.
Checklist · 12 items
- Research question in one sentence, naming the task family and the linkage levels compared.
- Population: systems, record types, date range, business units, and the pre-filter share of fully linked cases.
- Sampling unit (case) and grouping variable for splits (case, customer, site or technician).
- Arm construction script, with the severing and mislinking rules and a check that content is identical.
- De-identification and deduplication methods, applied once before the arm split.
- Base model, training budget in tokens or steps, hyperparameters and the full seed list.
- Frozen evaluation set: size, tasks, how items were written, and its hash.
- One primary metric, secondary metrics, and the smallest effect of interest.
- Statistical method: paired bootstrap interval or permutation test, and how multiple comparisons are handled.
- Negative controls and the planned confounder checks listed above.
- Stopping rules and what will be reported if the result is null.
- Limits stated in advance: domain, record types and model scale the result will not generalize beyond.
0 of 12 ticked. Tick each item as you go.
How to use this protocol when requesting or licensing records
A buyer can use this protocol to turn a vague preference for linked records into a request with testable terms: which keys must survive de-identification, which record types must join, and what evaluation rights are needed to run the comparison on a sample. Suppliers can use the same table to describe linkage honestly, including the share of cases that are only partially linked.
Linked case histories are not something SourceX keeps on hand; when a team asks, it looks for them at U.S. businesses, and the request-data form records the scope, volume, language, format, timing and licensing needs of that one search. If linkage is part of what you need, say which joins and outcome records matter and whether you plan an ablation, so suppliers can scope a sample that supports it. No results from this protocol are reported here; anyone who runs it should publish the method, the denominator and the uncertainty with the finding.
Frequently asked questions
What is a dataset linkage utility ablation?
It is an experiment that removes or scrambles the joins between related records while keeping their content, then compares models trained with and without those joins. The difference on a fixed evaluation set estimates how much linkage itself contributes for that task. It says nothing about other tasks unless they are tested too.
Is a linkage ablation the same as a record linkage quality audit?
No. A quality audit checks whether joins are correct and complete, for example how many invoices match a job. An ablation asks whether correct joins improve a trained model. Run the audit first, because a corpus with many wrong joins will understate the value of linkage.
What alternative explanation most often fakes a linkage effect?
Longer, richer examples and leaked identifiers are the usual culprits. A mislinked control, which keeps length and format but joins the wrong records, separates real linkage value from format effects. Searching the unlinked arm for surviving case numbers catches most identifier leakage.
How many seeds and evaluation items do I need?
There is no universal number; size it from a pilot. Measure the variance of the paired difference on a small run, then choose seeds and items that can detect the smallest effect you would act on. Report that calculation with the result.
Can a supplier run this before licensing any data?
A supplier can run it internally on its own records, but the evaluation must still be independent of whoever built the arms, and the rights to train test models on the records must be confirmed first. Buyers usually run it on a licensed sample under evaluation terms.
Does this page report any results?
No. It is a method only. Results depend on the records, the model and the task, and should be published with their population, sampling, uncertainty and limits by whoever runs the study.
Sources
- Scikit-learn's cross-validation guide says dependent samples within groups break the i.i.d. assumption and that GroupKFold ensures the same group is not represented in both testing and training sets. Source
- Kapoor and Narayanan survey data leakage in machine-learning-based science, offer a taxonomy of leakage types and show it leads to overoptimistic, irreproducible results. Source
- Bouthillier et al. find that randomness from data sampling, parameter initialization and hyperparameter choice noticeably affects ML benchmark results and recommend repeated training for robust comparisons. Source
- SciPy's bootstrap function computes a bootstrap confidence interval of a statistic with a paired option, and permutation_test offers a samples permutation type for paired-sample tests. Source
- Dror et al. (ACL 2018) found statistical significance testing in NLP papers was often skipped or misapplied and give guidance on choosing appropriate tests. Source
- Lee et al. found common language-model datasets contain many near-duplicate examples and that deduplication reduced memorized output and train-test overlap. Source
- The Center for Open Science describes preregistration as specifying a research plan in advance and submitting it to a registry, separating confirmatory from exploratory research. Source
- Datasheets for Datasets proposes that each dataset be documented with its motivation, composition, collection process, recommended uses and related questions. Source
Tell us what data you need
Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.