Skip to content

AI uses for records

Enterprise data split lineage: trace each example to its source record

By SourceX Editorial · Updated

9 sections

Short answer

To trace derived training examples to original business records, give every example a manifest row naming its root record, source system, company, entities, task family, time window and transformation. Assign train and test splits by group on those fields rather than by example, then run collision checks that block the release whenever a root record, entity, time window or near-duplicate cluster appears on both sides.

Evaluation split

Enterprise data split

This page covers enterprise data split lineage: trace each example to its source record.Illustrative

Key takeaways

  • One business record often yields several examples, so random example-level splits leak answers between train and test.
  • Choose the split unit from the claim: record, entity, company, task family or time, often nested.
  • A ten-field manifest written at creation time makes every split decision checkable.
  • Collision checks for roots, ancestry, entities, time, exact and near duplicates and answer leaks should block a release, not warn.
  • Attachments, knowledge articles, follow-up records and outcome fields can leak answers without sharing an ID.

How do you trace derived training examples to original business records?

Give every derived example a manifest row that names its root business record, the source system and record ID, the entities and company it involves, the task family, the time window it covers and the transformation that produced it. Assign splits by group on those fields, not by example, then run collision checks that fail the release whenever one group, root record or near-duplicate appears on both sides of the split.

Enterprise data split lineage matters because one business record rarely yields one example. A single support ticket can become a classification item, a summarization pair, a retrieval query and an agent task; a single contract can feed extraction, question answering and redlining examples. If those siblings land in both training and evaluation, the evaluation measures recall of a record the model has already seen.

This page gives a minimum manifest schema, a set of collision checks with the action each one triggers, a worked valid and failing entry, and the limits of what automated scans can catch. It is written for dataset engineers and evaluation leads assembling train and test sets from licensed business records.

Why a random split of derived examples leaks answers

Random splitting assumes examples are independent, and examples derived from business records are not. The scikit-learn documentation states the general rule: the independence assumption breaks when the generating process yields groups of dependent samples, and when you want to know whether a model generalizes to unseen groups, every validation sample must come from groups absent from the paired training fold.

Business data has several overlapping group structures at once. Examples share a root record, a customer or vendor, a company, an employee who wrote them, a template or macro, a product, and a period. Any one of these can carry the answer across a split: the same customer's tickets resolve the same way, the same macro produces the same reply, the same quarter's prices repeat across invoices.

Duplication compounds the problem. Lee and colleagues found that deduplicating standard language model datasets reduced train and test overlap that, by their measurement, affected over 4% of the validation set of standard datasets. Kapoor and Narayanan's survey of leakage in ML-based science found errors across 17 fields and 329 papers and organized them into a taxonomy of eight types of leakage. Two routes matter most for business-record datasets: the same record or near-duplicate text on both sides of the split, and later information shaping earlier examples.

Choosing the split unit: company, entity, task family or time

Pick the split unit from the claim the evaluation is meant to support, because each unit answers a different question. If the claim is that the model works for a new customer of the same company, split by customer. If it is that the model works at a company it has never seen, split by company. If it is that the model handles next quarter's work, split by time.

  • Most business evaluations need nested units: split by company or time first, then keep root records and entities whole inside each side.
  • scikit-learn's TimeSeriesSplit accepts a gap parameter; leaving a gap between training and test windows is the simplest guard against records that close weeks after they open.
  • Report which unit was used. A score from a record-level split and a score from a company-level split describe different capabilities and should not be compared.
Choosing the split unit: company, entity, task family or time
Split unitUse when the claim isLeak it preventsCost
Root recordThe model handles unseen casesSibling examples from one ticket, contract or orderMinimal; this is the floor for any split
Entity (customer, vendor, employee)The model generalizes to new counterpartiesShared history, names and recurring resolutionsLarge entities can unbalance split sizes
CompanyThe model transfers to a new businessShared templates, policies, product names and styleNeeds many contributing companies to estimate well
Task familyThe model handles a task type it was not trained onLearning the task format rather than the skillAnswers a narrower question; report separately
Time windowThe model works on future recordsLater records informing earlier labels, and driftOlder training data; leave a gap between windows
Template or macroThe model writes, not retrieves, the answerMemorized boilerplate replies scoring as correctRequires template IDs or clustering to recover them

The split lineage manifest, field by field

One row per derived example, written when the example is created, not reconstructed later. The fields below are the minimum; identifiers should be pseudonymous keys that survive de-identification, so the manifest can travel with a licensed dataset without carrying personal data.

At a glance

The split lineage manifest, field by field

The split lineage manifest, field by field
FieldPurposeIllustrative valueValidation rule
example_idStable ID of the derived exampleex-000412Unique across all splits and versions
root_recordSystem and record ID of the original business recordhelpdesk:T-88213Required; must resolve in the source register
derived_fromParent example or record, for multi-step derivationsex-000398Must exist; chain must end at root_record
transformHow the example was made, with tool and versionsummarize-v3From a controlled list; versioned
company_keyPseudonymous key of the contributing businessco-17Required when more than one company contributes
entity_keysPseudonymous keys of customers, vendors or staff involvedcust-4471; agent-022Every key on one side of the split only, per split unit
task_familyTask type the example testsrefund-eligibilityFrom the task taxonomy
event_windowEarliest and latest dates of the information used2024-03-02 to 2024-03-19Ends before the split cutoff for training rows
splitAssigned split and the unit used to assign ittest by company_keySet by the assigner, never by hand
content_hash and signatureExact hash and a near-duplicate signature of the texthash plus MinHash signatureRecomputed on every release

Collision checks to run before any score is reported

Run every check on every release and treat a failure as a blocker, not a warning. Each check should output the colliding rows so a reviewer can see the cause.

  • Near-duplicate detection usually relies on MinHash; the open-source datasketch library, for example, provides MinHash for estimating Jaccard similarity and MinHash LSH for threshold queries over large collections.
  • Set the threshold on a labeled sample of known templates and true distinct records from your own data, and keep the sample so the threshold can be rechecked.
Collision checks to run before any score is reported
CheckWhat it detectsMethodAction on failure
Root collisionTwo examples from one root record in different splitsGroup by root_record, count distinct splitsMove all siblings to one side
Ancestry collisionA derived example whose parent sits in the other splitWalk derived_from to the rootReassign by root, then rerun
Entity collisionAn entity key appearing in train and test under an entity or company splitGroup by each key in entity_keysReassign or drop the smaller group
Temporal overlapTraining rows whose event window ends after the cutoffCompare event_window end with cutoff and gapDrop or move the row
Exact duplicateIdentical text across splits under different IDsMatch content hashesKeep one, record the duplicate
Near duplicateTemplated or lightly edited text across splitsMinHash LSH at a Jaccard threshold, then human review of hitsGroup the cluster and reassign it whole
Answer leakA training row that contains a test row's label or outcomeSearch training text for test answers and outcome codesRemove the leaking row or the test item
Unresolved lineageRows whose root_record does not resolveJoin with the source registerExclude from all splits until fixed

A valid manifest entry and a failing one (illustrative)

The records below are invented to show the manifest and checks working; they come from no real business.

Valid entry: example ex-000412 is a refund-eligibility question derived by summarize-v3 from helpdesk ticket T-88213 at company co-17, involving customer cust-4471 and agent agent-022, with an event window of March 2 to 19, 2024. The split is test by company_key. All three other examples derived from T-88213 are also in test, no other co-17 row is in train, the event window ends before the test period starts, and its MinHash signature has no hit in train above the threshold. Every check passes.

Failing entry: example ex-000977 is a reply-drafting example derived from ticket T-90102 at company co-09, assigned to train. Three checks fail. Its root ticket has a sibling, ex-000981, in test. Its customer key appears in four test rows because co-09 was split by record, not by company. And its text is a near duplicate of a test reply because both came from the same support macro. The fix is to reassign co-09 as a whole, group the macro cluster, and rerun all checks before the release is cut.

A meaningful exception: shared reference material such as a published returns policy appears in both training and test prompts by design. Mark such documents as shared context in the manifest with their own ID so the answer-leak check does not flag them, and confirm that the policy text alone does not reveal the test answers.

Could an attachment, summary or later record reveal the evaluation answer?

Yes, and these leaks pass every ID-based check because they arrive through a different record. Look for them in five places before you trust a split.

Attachments: an invoice PDF attached to a training ticket can contain the amounts a test question asks about. Hash and link attachments as their own root records. Summaries and knowledge articles: a help article written from a set of tickets restates their resolutions; trace each article to its source tickets and keep them on one side. Later records: a follow-up ticket, a credit memo or a renewal often quotes the earlier case and its outcome, so link records that reference each other into one group.

Outcome fields: when a training row carries the final status, refund amount or win or loss, any test item from the same group that asks the agent to predict that outcome is compromised. System metadata: derived fields such as a resolution code set after the fact can encode the label in a training feature.

The practical rule is to grow groups along every reference link the source systems record, such as ticket links, order numbers, contract IDs and email threads, before splitting. W3C's PROV-O models this with wasDerivedFrom, a derivation being the transformation of one entity into another; chains of such links are exactly what the manifest's derived_from field follows.

Versions, access history and what the scans cannot see

Version the manifest with the dataset, because moving one record across a split changes every score computed on it. The Croissant metadata specification treats edits, removals or records shuffled across splits as changes that call for a major version increment, which is a sensible rule for business-derived sets as well.

Keep an access history for evaluation splits: who and which pipelines read the test rows, and when. A test set that was used to tune prompts, select checkpoints or write few-shot examples has leaked through people rather than files, and no collision check will find it.

Be explicit about scan limits. Hash and MinHash checks catch exact and lexical near duplicates; they miss paraphrases, translations and the same facts restated in a different record type. Embedding-based checks catch more at the cost of false positives that need review. Record which checks ran, their thresholds and the number of hits reviewed, and state in the evaluation report what the scans could not cover.

Asking a data supplier for split-ready lineage

Split-ready lineage is far cheaper to capture at the source than to reconstruct after delivery, so ask for it in the data request: pseudonymous root record, entity and company keys, link fields between records, event dates and attachment linkage, all preserved through de-identification.

Through SourceX's request form, an AI team can spell out the scope, volume, format, language, timing and licensing it requires, and SourceX looks for U.S. businesses whose records meet it rather than offering a stocked catalog. Because SourceX also runs the rights review, contracting, preparation and delivery with each contributing company, lineage fields and pseudonymous keys for group-level splits can be written into the specification before extraction starts.

Frequently asked questions

What is split lineage in an enterprise dataset?

It is the record of how each training or evaluation example traces back to its original business record, the entities and company involved, the period covered and the split it was assigned to. It lets you prove that no group crosses the split.

Is deduplication enough to prevent leakage between splits?

No. Deduplication removes repeated text, but sibling examples from one record, the same customer's history and later records that quote an outcome can leak answers without being duplicates. Group-level splits and lineage checks are needed as well.

Should I split business data by record, customer or company?

Split by the unit that matches the claim you want to make. Record-level splits are the minimum; customer splits test new counterparties; company splits test transfer to a new business; time splits test future records.

How do I handle a policy document that appears in both training and test prompts?

Mark it as shared context with its own ID so the checks treat it as intentional, and confirm the document alone does not reveal test answers. Shared context should be the same for every model compared.

Can lineage keys survive de-identification?

Yes, if the supplier replaces identifiers with consistent pseudonymous keys rather than deleting them. The same customer or ticket should map to the same key everywhere in the delivery so groups can still be formed.

Sources

  • scikit-learn's cross-validation guide explains that the i.i.d. assumption breaks when samples come in dependent groups, that group splitters keep validation groups out of the training fold, and that TimeSeriesSplit supports a gap parameter. Source
  • The Croissant specification supports dataset splits and calls for a major version increment when existing data is edited, removed or shuffled across splits. Source
  • W3C PROV-O defines wasDerivedFrom for chains of entities, describing a derivation as the transformation of one entity into another. Source
  • The datasketch library provides MinHash to estimate Jaccard similarity and MinHash LSH for Jaccard threshold queries. Source
  • Lee and colleagues found train and test overlap affecting over 4% of the validation set of standard language model datasets and reduced it by deduplication using exact substring matching and MinHash. Source
  • Kapoor and Narayanan surveyed leakage in ML-based science, finding errors in 17 fields affecting 329 papers, and proposed a taxonomy of eight leakage types. Source

Tell us what data you need

Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.

See if your company qualifies