AI uses for records
Enterprise data split lineage: trace each example to its source record
By SourceX Editorial · Updated
9 sections
Short answer
To trace derived training examples to original business records, give every example a manifest row naming its root record, source system, company, entities, task family, time window and transformation. Assign train and test splits by group on those fields rather than by example, then run collision checks that block the release whenever a root record, entity, time window or near-duplicate cluster appears on both sides.
Evaluation split
Enterprise data split
Key takeaways
- One business record often yields several examples, so random example-level splits leak answers between train and test.
- Choose the split unit from the claim: record, entity, company, task family or time, often nested.
- A ten-field manifest written at creation time makes every split decision checkable.
- Collision checks for roots, ancestry, entities, time, exact and near duplicates and answer leaks should block a release, not warn.
- Attachments, knowledge articles, follow-up records and outcome fields can leak answers without sharing an ID.
How do you trace derived training examples to original business records?
Give every derived example a manifest row that names its root business record, the source system and record ID, the entities and company it involves, the task family, the time window it covers and the transformation that produced it. Assign splits by group on those fields, not by example, then run collision checks that fail the release whenever one group, root record or near-duplicate appears on both sides of the split.
Enterprise data split lineage matters because one business record rarely yields one example. A single support ticket can become a classification item, a summarization pair, a retrieval query and an agent task; a single contract can feed extraction, question answering and redlining examples. If those siblings land in both training and evaluation, the evaluation measures recall of a record the model has already seen.
This page gives a minimum manifest schema, a set of collision checks with the action each one triggers, a worked valid and failing entry, and the limits of what automated scans can catch. It is written for dataset engineers and evaluation leads assembling train and test sets from licensed business records.
Why a random split of derived examples leaks answers
Random splitting assumes examples are independent, and examples derived from business records are not. The scikit-learn documentation states the general rule: the independence assumption breaks when the generating process yields groups of dependent samples, and when you want to know whether a model generalizes to unseen groups, every validation sample must come from groups absent from the paired training fold.
Business data has several overlapping group structures at once. Examples share a root record, a customer or vendor, a company, an employee who wrote them, a template or macro, a product, and a period. Any one of these can carry the answer across a split: the same customer's tickets resolve the same way, the same macro produces the same reply, the same quarter's prices repeat across invoices.
Duplication compounds the problem. Lee and colleagues found that deduplicating standard language model datasets reduced train and test overlap that, by their measurement, affected over 4% of the validation set of standard datasets. Kapoor and Narayanan's survey of leakage in ML-based science found errors across 17 fields and 329 papers and organized them into a taxonomy of eight types of leakage. Two routes matter most for business-record datasets: the same record or near-duplicate text on both sides of the split, and later information shaping earlier examples.
Choosing the split unit: company, entity, task family or time
Pick the split unit from the claim the evaluation is meant to support, because each unit answers a different question. If the claim is that the model works for a new customer of the same company, split by customer. If it is that the model works at a company it has never seen, split by company. If it is that the model handles next quarter's work, split by time.
- Most business evaluations need nested units: split by company or time first, then keep root records and entities whole inside each side.
- scikit-learn's TimeSeriesSplit accepts a gap parameter; leaving a gap between training and test windows is the simplest guard against records that close weeks after they open.
- Report which unit was used. A score from a record-level split and a score from a company-level split describe different capabilities and should not be compared.
| Split unit | Use when the claim is | Leak it prevents | Cost |
|---|---|---|---|
| Root record | The model handles unseen cases | Sibling examples from one ticket, contract or order | Minimal; this is the floor for any split |
| Entity (customer, vendor, employee) | The model generalizes to new counterparties | Shared history, names and recurring resolutions | Large entities can unbalance split sizes |
| Company | The model transfers to a new business | Shared templates, policies, product names and style | Needs many contributing companies to estimate well |
| Task family | The model handles a task type it was not trained on | Learning the task format rather than the skill | Answers a narrower question; report separately |
| Time window | The model works on future records | Later records informing earlier labels, and drift | Older training data; leave a gap between windows |
| Template or macro | The model writes, not retrieves, the answer | Memorized boilerplate replies scoring as correct | Requires template IDs or clustering to recover them |
The split lineage manifest, field by field
One row per derived example, written when the example is created, not reconstructed later. The fields below are the minimum; identifiers should be pseudonymous keys that survive de-identification, so the manifest can travel with a licensed dataset without carrying personal data.
At a glance
The split lineage manifest, field by field
| Field | Purpose | Illustrative value | Validation rule |
|---|---|---|---|
| example_id | Stable ID of the derived example | ex-000412 | Unique across all splits and versions |
| root_record | System and record ID of the original business record | helpdesk:T-88213 | Required; must resolve in the source register |
| derived_from | Parent example or record, for multi-step derivations | ex-000398 | Must exist; chain must end at root_record |
| transform | How the example was made, with tool and version | summarize-v3 | From a controlled list; versioned |
| company_key | Pseudonymous key of the contributing business | co-17 | Required when more than one company contributes |
| entity_keys | Pseudonymous keys of customers, vendors or staff involved | cust-4471; agent-022 | Every key on one side of the split only, per split unit |
| task_family | Task type the example tests | refund-eligibility | From the task taxonomy |
| event_window | Earliest and latest dates of the information used | 2024-03-02 to 2024-03-19 | Ends before the split cutoff for training rows |
| split | Assigned split and the unit used to assign it | test by company_key | Set by the assigner, never by hand |
| content_hash and signature | Exact hash and a near-duplicate signature of the text | hash plus MinHash signature | Recomputed on every release |
Collision checks to run before any score is reported
Run every check on every release and treat a failure as a blocker, not a warning. Each check should output the colliding rows so a reviewer can see the cause.
- Near-duplicate detection usually relies on MinHash; the open-source datasketch library, for example, provides MinHash for estimating Jaccard similarity and MinHash LSH for threshold queries over large collections.
- Set the threshold on a labeled sample of known templates and true distinct records from your own data, and keep the sample so the threshold can be rechecked.
| Check | What it detects | Method | Action on failure |
|---|---|---|---|
| Root collision | Two examples from one root record in different splits | Group by root_record, count distinct splits | Move all siblings to one side |
| Ancestry collision | A derived example whose parent sits in the other split | Walk derived_from to the root | Reassign by root, then rerun |
| Entity collision | An entity key appearing in train and test under an entity or company split | Group by each key in entity_keys | Reassign or drop the smaller group |
| Temporal overlap | Training rows whose event window ends after the cutoff | Compare event_window end with cutoff and gap | Drop or move the row |
| Exact duplicate | Identical text across splits under different IDs | Match content hashes | Keep one, record the duplicate |
| Near duplicate | Templated or lightly edited text across splits | MinHash LSH at a Jaccard threshold, then human review of hits | Group the cluster and reassign it whole |
| Answer leak | A training row that contains a test row's label or outcome | Search training text for test answers and outcome codes | Remove the leaking row or the test item |
| Unresolved lineage | Rows whose root_record does not resolve | Join with the source register | Exclude from all splits until fixed |
A valid manifest entry and a failing one (illustrative)
The records below are invented to show the manifest and checks working; they come from no real business.
Valid entry: example ex-000412 is a refund-eligibility question derived by summarize-v3 from helpdesk ticket T-88213 at company co-17, involving customer cust-4471 and agent agent-022, with an event window of March 2 to 19, 2024. The split is test by company_key. All three other examples derived from T-88213 are also in test, no other co-17 row is in train, the event window ends before the test period starts, and its MinHash signature has no hit in train above the threshold. Every check passes.
Failing entry: example ex-000977 is a reply-drafting example derived from ticket T-90102 at company co-09, assigned to train. Three checks fail. Its root ticket has a sibling, ex-000981, in test. Its customer key appears in four test rows because co-09 was split by record, not by company. And its text is a near duplicate of a test reply because both came from the same support macro. The fix is to reassign co-09 as a whole, group the macro cluster, and rerun all checks before the release is cut.
A meaningful exception: shared reference material such as a published returns policy appears in both training and test prompts by design. Mark such documents as shared context in the manifest with their own ID so the answer-leak check does not flag them, and confirm that the policy text alone does not reveal the test answers.
Could an attachment, summary or later record reveal the evaluation answer?
Yes, and these leaks pass every ID-based check because they arrive through a different record. Look for them in five places before you trust a split.
Attachments: an invoice PDF attached to a training ticket can contain the amounts a test question asks about. Hash and link attachments as their own root records. Summaries and knowledge articles: a help article written from a set of tickets restates their resolutions; trace each article to its source tickets and keep them on one side. Later records: a follow-up ticket, a credit memo or a renewal often quotes the earlier case and its outcome, so link records that reference each other into one group.
Outcome fields: when a training row carries the final status, refund amount or win or loss, any test item from the same group that asks the agent to predict that outcome is compromised. System metadata: derived fields such as a resolution code set after the fact can encode the label in a training feature.
The practical rule is to grow groups along every reference link the source systems record, such as ticket links, order numbers, contract IDs and email threads, before splitting. W3C's PROV-O models this with wasDerivedFrom, a derivation being the transformation of one entity into another; chains of such links are exactly what the manifest's derived_from field follows.
Versions, access history and what the scans cannot see
Version the manifest with the dataset, because moving one record across a split changes every score computed on it. The Croissant metadata specification treats edits, removals or records shuffled across splits as changes that call for a major version increment, which is a sensible rule for business-derived sets as well.
Keep an access history for evaluation splits: who and which pipelines read the test rows, and when. A test set that was used to tune prompts, select checkpoints or write few-shot examples has leaked through people rather than files, and no collision check will find it.
Be explicit about scan limits. Hash and MinHash checks catch exact and lexical near duplicates; they miss paraphrases, translations and the same facts restated in a different record type. Embedding-based checks catch more at the cost of false positives that need review. Record which checks ran, their thresholds and the number of hits reviewed, and state in the evaluation report what the scans could not cover.
Asking a data supplier for split-ready lineage
Split-ready lineage is far cheaper to capture at the source than to reconstruct after delivery, so ask for it in the data request: pseudonymous root record, entity and company keys, link fields between records, event dates and attachment linkage, all preserved through de-identification.
Through SourceX's request form, an AI team can spell out the scope, volume, format, language, timing and licensing it requires, and SourceX looks for U.S. businesses whose records meet it rather than offering a stocked catalog. Because SourceX also runs the rights review, contracting, preparation and delivery with each contributing company, lineage fields and pseudonymous keys for group-level splits can be written into the specification before extraction starts.
Frequently asked questions
What is split lineage in an enterprise dataset?
It is the record of how each training or evaluation example traces back to its original business record, the entities and company involved, the period covered and the split it was assigned to. It lets you prove that no group crosses the split.
Is deduplication enough to prevent leakage between splits?
No. Deduplication removes repeated text, but sibling examples from one record, the same customer's history and later records that quote an outcome can leak answers without being duplicates. Group-level splits and lineage checks are needed as well.
Should I split business data by record, customer or company?
Split by the unit that matches the claim you want to make. Record-level splits are the minimum; customer splits test new counterparties; company splits test transfer to a new business; time splits test future records.
How do I handle a policy document that appears in both training and test prompts?
Mark it as shared context with its own ID so the checks treat it as intentional, and confirm the document alone does not reveal test answers. Shared context should be the same for every model compared.
Can lineage keys survive de-identification?
Yes, if the supplier replaces identifiers with consistent pseudonymous keys rather than deleting them. The same customer or ticket should map to the same key everywhere in the delivery so groups can still be formed.
Sources
- scikit-learn's cross-validation guide explains that the i.i.d. assumption breaks when samples come in dependent groups, that group splitters keep validation groups out of the training fold, and that TimeSeriesSplit supports a gap parameter. Source
- The Croissant specification supports dataset splits and calls for a major version increment when existing data is edited, removed or shuffled across splits. Source
- W3C PROV-O defines wasDerivedFrom for chains of entities, describing a derivation as the transformation of one entity into another. Source
- The datasketch library provides MinHash to estimate Jaccard similarity and MinHash LSH for Jaccard threshold queries. Source
- Lee and colleagues found train and test overlap affecting over 4% of the validation set of standard language model datasets and reduced it by deduplication using exact substring matching and MinHash. Source
- Kapoor and Narayanan surveyed leakage in ML-based science, finding errors in 17 fields affecting 329 papers, and proposed a taxonomy of eight leakage types. Source
Tell us what data you need
Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.