Skip to content

AI uses for records

Dataset linkage utility ablation: how to measure linked records

By SourceX Editorial · Updated

10 sections

Short answer

Measure the value of linked records with a pre-registered ablation: derive linked and unlinked training sets from the same cases, change only the joins, train the same model on each with an equal budget and several seeds, and compare paired scores on a frozen holdout with a confidence interval. This page gives the protocol and a comparison template; it reports no results.

Agent run

Dataset linkage utility

This page covers dataset linkage utility ablation: how to measure linked records.Illustrative

Key takeaways

  • Measure linkage value with a pre-registered ablation that changes only the joins between records, never their content.
  • Derive every arm from one de-identified, deduplicated corpus by severing or scrambling keys, and add a mislinked control to rule out format effects.
  • Split by case and customer so no case appears in both training and test data, and watch for outcome fields that leak the target.
  • Report paired differences across several seeds with a confidence interval and a pre-specified smallest effect of interest.
  • Single-record tasks act as negative controls; a gain there points to a confound, not to linkage.
  • This is a method only; no results are reported here.

How do you measure the training value of keeping records linked?

You measure it with a pre-registered ablation: build two training sets from the same source cases, identical in every respect except whether the records that belong to one case stay joined, train the same model on each with the same budget, and score both on an untouched holdout of tasks. The paired difference, with an uncertainty interval, is the estimate of what linkage adds.

This page is a protocol, not a study. It reports no results and contains no measured effect sizes. It sets out the question, population, sampling, variables, comparison conditions, metrics, uncertainty method, known confounders and a pre-registration checklist, so that a data buyer, a supplier or an independent evaluator can run the experiment and publish the outcome with its limits.

The reason a protocol is needed at all is that the intuition behind linked records is easy to state and easy to fool yourself about. A linked quote, order, job, invoice and callback chain looks richer than the same documents scattered across folders. Whether a model actually learns more from the chain, for a specific task, is an empirical question, and a sloppy comparison will usually overstate the answer.

What exactly is the treatment variable in a linkage ablation?

The treatment is the presence of stable joins between records that describe one business case; everything else is held constant. Define linkage before any data is touched, because a vague definition lets the linked arm quietly gain content the unlinked arm lacks.

In business records a case is usually a chain across systems: a request in a CRM such as Salesforce or HubSpot, a quote or order in an ERP such as NetSuite, a work order in a field service tool such as ServiceTitan, an invoice, and an outcome such as a callback, return, credit memo or closed ticket in Zendesk or Jira. Linkage is the set of keys (order number, job number, ticket ID, customer account) that let a reader follow that chain.

Write the linkage levels you will test into the pre-registration. The four levels below are a reasonable default; most studies compare the top and bottom levels first and add intermediate levels only if the first comparison is informative.

What exactly is the treatment variable in a linkage ablation?
LevelWhat the model seesTypical constructionWhat it isolates
L0 unlinkedEach record as an independent example, keys removedStrip join keys and shuffle record order across casesContent of individual records only
L1 within-systemRecords from one system grouped by caseGroup CRM or ticket history by case ID, no cross-system joinsValue of history inside one application
L2 cross-systemFull chain from request to invoiceJoin CRM, ERP and job records on shared keysValue of joins across systems
L3 chain with outcomeFull chain plus the recorded resultAdd callback, return, credit or resolution recordValue of connecting decisions to outcomes
Mislinked controlChains joined to the wrong case of the same type and periodRe-pair records at random within strataWhether gains come from true joins or from format

What should the linked versus unlinked case comparison include?

The comparison table is the core asset: every row is a variable that must be identical across arms, with the one exception being the linkage itself. If any row other than linkage differs, the result measures that difference instead.

Fill this in before training and attach it to the pre-registration. The three illustrative entries listed next show how one fictional job appears in each arm; the table after them lists what must match.

  • Illustrative case, linked arm: one training example contains the service request text, the dispatcher's priority choice, the technician's work notes and parts list, the invoice lines, and a callback record dated later that names the same job number.
  • Illustrative case, unlinked arm: the same five records appear as five separate examples, keys removed, in shuffled positions in the training stream, so the model never sees that the callback belongs to that job.
  • Illustrative case, mislinked control: the same request, notes and invoice are joined into one example, but the callback comes from a different job of the same type and month.
What should the linked versus unlinked case comparison include?
VariableLinked armUnlinked armWhy it must match
Source casesSame case IDsSame case IDsDifferent cases would change content, not linkage
Record contentEvery field and free-text noteEvery field and free-text noteMissing fields would be a content ablation
Token or example budgetFixed total tokensSame total tokensMore tokens alone can raise scores
Formatting and templatesOne serialization templateSame template per recordFormat cues can be learned as shortcuts
De-identificationSame method and passSame method and passDifferent redaction changes what is learnable
DeduplicationApplied before arm splitApplied before arm splitDuplicates inflate apparent learning
Model, seeds, hyperparametersSame base model, same seed setSame base model, same seed setTuning one arm harder biases the result
Evaluation itemsSame frozen holdoutSame frozen holdoutPaired scoring is what makes the test sensitive

How should you build the unlinked arm without changing anything else?

Build the unlinked arm by severing joins in the linked arm, not by assembling a second dataset from scratch. Start from one fully linked, de-identified and deduplicated corpus, then derive every arm from it with a script that only removes or scrambles keys and grouping. That way the content is identical by construction.

Severing has two parts. First, remove or randomize identifiers that let a model reconnect records on its own, including order numbers repeated inside free text, customer names that survived de-identification, and dates that make a chain obvious. Second, break the grouping, so records from one case are not adjacent in the training stream. If you only drop the key column and keep records in case order, the model can still learn the chain from position.

Add the mislinked control whenever you can afford a third arm. It holds format and example length equal to the linked arm while destroying the true relationship. If the linked arm beats the unlinked arm but not the mislinked control, the gain probably came from longer, multi-record examples rather than from the real joins, which is a very different finding for anyone deciding what records to license.

Partial linkage is a natural extension. Real archives are rarely all or nothing: some jobs have invoices and some do not. If you want a dose-response view, pre-register a small number of linkage rates and build each by severing a random subset of cases, stratified by case type and period.

Which population, sampling and splits keep the comparison honest?

The sampling unit is the case, not the record, and every split must keep all records of a case, and ideally all cases of a customer or site, on one side. If records from one job land in both training and test sets, the linked arm can score well by memorizing the job rather than learning anything general.

Scikit-learn's documentation describes exactly this risk for grouped data: when samples within a group are dependent, a model can overfit to group-specific features, and group-aware splitters such as GroupKFold ensure the same group is not represented in both training and test sets. For business records, the group is the case, customer account, site or technician, depending on what the model could memorize. Where cases evolve over time, hold out the most recent period as well, in the spirit of a time-ordered split, so the evaluation also reflects how the data would be used going forward.

Kapoor and Narayanan's survey of leakage in machine-learning-based science is a useful warning list here: train-test overlap, features that would not be available at prediction time, and preprocessing fitted on the full dataset all produce optimistic results. In a linkage study the classic trap is an outcome field (for example a credit memo amount) that leaks the target the model is asked to predict.

Define the population in writing: which systems, which record types, which date range, which business units, what share of cases had a full chain before any filtering, and which cases were excluded and why. Readers need the denominator to judge whether linked cases are representative of the whole archive.

Which tasks and metrics show whether linkage helped?

Use two families of evaluation tasks: tasks that require information from more than one record in a chain, where linkage should matter, and single-record tasks, where it should not. The second family is a negative control; a large gain there signals a confound rather than a linkage effect.

Pick metrics that a business reader understands, such as the accuracy of a predicted outcome or the share of correct next steps, and pre-specify one primary metric. Secondary metrics are fine, but report them as secondary.

Which tasks and metrics show whether linkage helped?
Task familyNeeds linkage?Example metricHow to read a difference
Predict the outcome of a case from its request and decisionsYesAccuracy or calibration of predicted callback, return or creditPrimary test of linkage value
Recommend the next step in a caseYesExact-match or expert-rated correctnessTests whether chains teach process order
Reconcile records across systemsYesPrecision and recall of correct matchesDirect measure of cross-record reasoning
Classify a single record by typeNoAccuracy on record categoryNegative control; should not move
Extract fields from one documentNoField-level exact matchNegative control; movement suggests a confound
General capability checkNoScore on a fixed public or internal regression setDetects damage from either arm

What sample, baseline and uncertainty would make a result defensible?

A defensible result reports a paired difference on the same frozen evaluation items, across several training seeds, with a confidence interval and a pre-specified smallest effect worth acting on. A single run per arm with a point score is not evidence of anything.

Bouthillier and colleagues show that sources of randomness such as data sampling, parameter initialization and hyperparameter choice noticeably affect benchmark results, and that robust comparisons generally require repeating the full training pipeline. Budget for multiple seeds per arm from the start; if compute is tight, reduce model size rather than seeds.

Because both arms are scored on the same items, use paired methods. SciPy's bootstrap function computes a confidence interval of a statistic by resampling and supports paired data, and its permutation_test function supports a samples permutation type that its documentation describes as appropriate for paired-sample tests. Dror and colleagues found that significance testing in NLP papers was often skipped or misapplied, and their guide is a good reference for choosing a test that matches the metric.

Size the evaluation set from a pilot. Run a small version of both arms, measure the variance of the paired difference, and pick the number of evaluation items and seeds that would detect your smallest effect of interest. Write that number down before the main run. The baseline is the unlinked arm, not zero; also report the base model with no fine-tuning, so readers can see how much either arm moved it.

What alternative explanations could account for an apparent improvement?

The most likely alternative explanations are more effective training signal per example, leakage of identifiers or outcomes, and selection bias in which cases happen to be linked. Each has a specific check, and the report should state the result of every check, including the ones that found nothing.

At a glance

What alternative explanations could account for an apparent improvement

  • Longer examples: linked examples are longer and may simply carry more context per gradient step. Check: the mislinked control holds length equal.
  • Token budget drift: severing can change tokenization or template overhead. Check: count tokens per arm and equalize.
  • Identifier leakage: case numbers or customer names inside free text let the model reconnect records or memorize cases. Check: search the unlinked arm for surviving keys.
  • Target leakage: an outcome field present in the input. Check: list every input field and confirm it existed at the time the prediction would be made.
  • Duplicate and near-duplicate records across splits: Lee and colleagues found that common language-model datasets contain many near-duplicates and that deduplication reduced train-test overlap. Check: deduplicate before splitting and report the method.
  • Selection bias: well-run jobs may be the ones that got fully linked, so linked cases are easier. Check: compare the same case IDs in both arms, never linked cases against a different set of unlinked cases.
  • Tuning asymmetry: more hyperparameter search on one arm. Check: one shared search, or none, fixed before the main run.
  • Evaluation contamination: holdout items written by someone who saw the linked arm. Check: freeze and hash the evaluation set first.

Pre-registration checklist for a linkage ablation

File the plan before training any arm. The Center for Open Science describes preregistration as specifying the research plan in advance and submitting it to a registry, which separates planned confirmatory analysis from exploratory analysis. For private studies, a dated internal document hashed and shared with the counterparty serves the same purpose.

Document the dataset alongside the plan. The Datasheets for Datasets template asks about motivation, composition, collection process, preprocessing, uses, distribution and maintenance, and those answers are what a reader needs to judge whether your linked corpus resembles theirs.

Checklist · 12 items

  1. Research question in one sentence, naming the task family and the linkage levels compared.
  2. Population: systems, record types, date range, business units, and the pre-filter share of fully linked cases.
  3. Sampling unit (case) and grouping variable for splits (case, customer, site or technician).
  4. Arm construction script, with the severing and mislinking rules and a check that content is identical.
  5. De-identification and deduplication methods, applied once before the arm split.
  6. Base model, training budget in tokens or steps, hyperparameters and the full seed list.
  7. Frozen evaluation set: size, tasks, how items were written, and its hash.
  8. One primary metric, secondary metrics, and the smallest effect of interest.
  9. Statistical method: paired bootstrap interval or permutation test, and how multiple comparisons are handled.
  10. Negative controls and the planned confounder checks listed above.
  11. Stopping rules and what will be reported if the result is null.
  12. Limits stated in advance: domain, record types and model scale the result will not generalize beyond.

0 of 12 ticked. Tick each item as you go.

How to use this protocol when requesting or licensing records

A buyer can use this protocol to turn a vague preference for linked records into a request with testable terms: which keys must survive de-identification, which record types must join, and what evaluation rights are needed to run the comparison on a sample. Suppliers can use the same table to describe linkage honestly, including the share of cases that are only partially linked.

Linked case histories are not something SourceX keeps on hand; when a team asks, it looks for them at U.S. businesses, and the request-data form records the scope, volume, language, format, timing and licensing needs of that one search. If linkage is part of what you need, say which joins and outcome records matter and whether you plan an ablation, so suppliers can scope a sample that supports it. No results from this protocol are reported here; anyone who runs it should publish the method, the denominator and the uncertainty with the finding.

Frequently asked questions

What is a dataset linkage utility ablation?

It is an experiment that removes or scrambles the joins between related records while keeping their content, then compares models trained with and without those joins. The difference on a fixed evaluation set estimates how much linkage itself contributes for that task. It says nothing about other tasks unless they are tested too.

Is a linkage ablation the same as a record linkage quality audit?

No. A quality audit checks whether joins are correct and complete, for example how many invoices match a job. An ablation asks whether correct joins improve a trained model. Run the audit first, because a corpus with many wrong joins will understate the value of linkage.

What alternative explanation most often fakes a linkage effect?

Longer, richer examples and leaked identifiers are the usual culprits. A mislinked control, which keeps length and format but joins the wrong records, separates real linkage value from format effects. Searching the unlinked arm for surviving case numbers catches most identifier leakage.

How many seeds and evaluation items do I need?

There is no universal number; size it from a pilot. Measure the variance of the paired difference on a small run, then choose seeds and items that can detect the smallest effect you would act on. Report that calculation with the result.

Can a supplier run this before licensing any data?

A supplier can run it internally on its own records, but the evaluation must still be independent of whoever built the arms, and the rights to train test models on the records must be confirmed first. Buyers usually run it on a licensed sample under evaluation terms.

Does this page report any results?

No. It is a method only. Results depend on the records, the model and the task, and should be published with their population, sampling, uncertainty and limits by whoever runs the study.

Sources

  • Scikit-learn's cross-validation guide says dependent samples within groups break the i.i.d. assumption and that GroupKFold ensures the same group is not represented in both testing and training sets. Source
  • Kapoor and Narayanan survey data leakage in machine-learning-based science, offer a taxonomy of leakage types and show it leads to overoptimistic, irreproducible results. Source
  • Bouthillier et al. find that randomness from data sampling, parameter initialization and hyperparameter choice noticeably affects ML benchmark results and recommend repeated training for robust comparisons. Source
  • SciPy's bootstrap function computes a bootstrap confidence interval of a statistic with a paired option, and permutation_test offers a samples permutation type for paired-sample tests. Source
  • Dror et al. (ACL 2018) found statistical significance testing in NLP papers was often skipped or misapplied and give guidance on choosing appropriate tests. Source
  • Lee et al. found common language-model datasets contain many near-duplicate examples and that deduplication reduced memorized output and train-test overlap. Source
  • The Center for Open Science describes preregistration as specifying a research plan in advance and submitting it to a registry, separating confirmatory from exploratory research. Source
  • Datasheets for Datasets proposes that each dataset be documented with its motivation, composition, collection process, recommended uses and related questions. Source

Tell us what data you need

Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.

See if your company qualifies