AI uses for records
Company-level dataset splits: why random row splits fail for workflow data
By SourceX Editorial · Updated
8 sections
Short answer
Random row splits mislead whenever records from one company, customer or case land on both sides, because shared templates, policies, phrasing and linked later records let a model recognize context instead of solving the task. For enterprise workflow datasets, hold out whole companies, add a time cutoff and task-family boundaries, then scan for near-duplicates, shared artifacts and outcome records that cross the split.
Evaluation split
Company level dataset splits
Key takeaways
- Pick the grouping key that matches the claim: new companies need a company held-out test, the same companies next quarter need a time cutoff.
- Case id is the minimum grouping key for any workflow data; it does not stop company-level leakage.
- Derivatives such as quoted replies, summaries and forwarded attachments are the leakage path most scans miss.
- Labels come from later linked records, so no record created after a test decision may sit in training.
- Report random-split scores only as an in-distribution reference, with company-level intervals for the held-out score.
When do random row splits mislead workflow dataset evaluations?
Random row splits mislead whenever rows are not independent, and in enterprise workflow data they almost never are. Tickets, invoices and tasks from one company share templates, customer names, staff phrasing, macros and policies, so a model can score well on the test rows by recognizing the company rather than by solving the task. Split by company first, then by time and task family, and audit what still crosses the boundary.
The question to ask is what claim the evaluation must support. If the claim is that a model will work for a business it has never seen, every row from that business has to sit on one side of the split. If the claim is that it will keep working for the same businesses next quarter, the split has to respect time. A random split supports neither claim; it measures performance on a mix of memorized context and genuine generalization, in unknown proportions.
This page gives a decision rule for the grouping key, a company held-out split specification with concrete leakage examples, the checks that catch derivatives and near-duplicates, and the one situation where a random split is still defensible.
How one company's habits leak across a random split
Leakage through company identity is subtle because no single row looks like a duplicate. The signal is spread across shared vocabulary: product SKUs, internal team names, the wording of a refund macro, the approval limits in a policy, even the order in which a particular support agent fills in fields.
The scikit-learn documentation states the general principle plainly: the independence assumption behind ordinary cross-validation is broken when the generating process yields groups of dependent samples, and to measure whether a model generalizes to unseen groups, every sample in a validation fold must come from groups that are not represented at all in the paired training fold. Its example is several samples from each medical patient; in workflow data, the patient is the company, the customer account or the case.
Kapoor and Narayanan's taxonomy of leakage in machine-learning-based science names this case directly: non-independence between training and test samples counts as leakage unless the claim is about a population with the same dependence structure, and the extreme case is samples drawn from the same people or units. Their survey found leakage-related errors across many scientific fields, so this is not a niche mistake.
Public benchmarks show what the inflation looks like. Lewis, Stenetorp and Riedel found that in several open-domain question answering datasets a large share of test answers also appeared in the training data, and that overlapping items accounted for much of the measured performance. Company-level overlap in workflow data produces the same effect with less visible duplication.
Picking the grouping key: company, entity, time or task family
Choose the coarsest key that matches the claim you need to make, then add time and task-family boundaries inside it. Splitting by a fine key, such as ticket id, when the claim is about new companies is the most common error; splitting by a coarse key you do not need throws away data and can make test sets too small to read.
| Grouping key | Leakage it blocks | Claim it supports | Cost |
|---|---|---|---|
| Company or tenant | Shared templates, macros, policies, product names, staff phrasing | Works for a business not seen in training | Fewer groups; test set size depends on company count |
| Customer account or entity | Repeat requests from the same account, account-specific history | Works for new customers of known businesses | Needs a reliable entity id across systems |
| Time window | Future policies, later outcomes, records edited after the event | Keeps working as the business changes | Test reflects one period; seasonality can confound |
| Task family or template | Memorized solution patterns for one task type | Handles a task type it has not practiced | Requires a family taxonomy agreed with the process owner |
| Case or thread | Replies and attachments from the same case on both sides | Minimum for any workflow data | Does not block company-level leakage |
The company held-out split, with leakage examples
The specification below is the asset: a split you can implement as written, plus the leakage examples it is designed to catch. It assumes a dataset of workflow histories from several companies, each record carrying a company id, an entity id, a case id, a task family and event timestamps.
Assign whole companies to test first, stratified so the test companies span the industries, sizes and systems in the dataset. Inside the remaining companies, hold out the most recent time window as a second test set for the same-business claim. Then run the leakage checks against both test sets before any model is scored, and write the results into a split manifest that ships with the data.
- Group ids: company id, entity id and case id on every record, with a mapping table for entities that appear in more than one system.
- Company held-out test: a fixed list of company ids that never appear in training or validation, stratified by industry and system.
- Temporal test: for in-distribution companies, every case that opened after a cutoff date, with the cutoff recorded in the manifest.
- Task-family holdout (optional): one or more task families withheld entirely to test transfer to a new task type.
- Boundary rule for cases spanning the cutoff: assign by the date the decision was made, and drop events after the cutoff from training.
- Near-duplicate scan: text similarity between every test record and the training set, including attachments and quoted replies, with the threshold and tool recorded.
- Shared-artifact scan: macros, templates, policy documents and knowledge base articles that appear on both sides, listed by id.
- Manifest: group assignments, cutoff dates, scan thresholds, counts per split and a content hash of each split file.
| Leakage example | How it crosses a random split | Check that catches it | Fix |
|---|---|---|---|
| Refund macro text quoted in agent replies | Same macro appears in train and test rows from one company | Shared-artifact scan on macro ids and long common substrings | Company held-out test |
| Thread with an escalation and its resolution | Early messages in train, resolution in test, or the reverse | Case id must not appear on both sides | Split by case at minimum |
| Invoice and its later credit note | Credit note in train reveals the dispute outcome for a test invoice | Entity and document link check across splits | Group by entity or by linked document chain |
| Policy update effective mid-year | Training rows apply the new rule to cases graded under the old one | Policy version tag versus decision date | Temporal test with decisions dated before the cutoff |
| Account manager's personal phrasing | Model learns to match a writer, not the task | Author id overlap report | Company or team held-out |
| Forwarded attachment summarized in a later ticket | Summary in train states the answer to a test question | Near-duplicate scan including derived text | Remove or move the derivative to the test side |
Could a training example, attachment or derivative reveal the answer?
Yes, and derivatives are the leakage path most scans miss. Workflow systems copy content constantly: replies quote earlier messages, tickets carry forwarded emails, summaries and handoff notes restate decisions, and the same PDF is attached to several records. Any of these on the training side can state the answer to a test case outright.
Exact-match deduplication is not enough. Lee and colleagues found that standard language model datasets contain many near-duplicate examples and that removing train-test overlap affected a noticeable share of validation sets in common benchmarks; their approach keeps the shared text in the evaluation set and removes it from training. Yang and colleagues showed that paraphrased or translated test items can pass n-gram decontamination checks while still inflating scores, which is exactly what a summary or a handoff note is: a paraphrase.
So scan at three levels: identical text, near-duplicate text above a similarity threshold, and semantic restatements, where a model or a person checks whether a training record would let a reader answer the test case. Scan attachments and quoted blocks separately from the message body, because they are where copied content concentrates.
Which linked record proves the decision and its later outcome?
The record that proves the outcome usually sits later in time and often in another system, and that is what makes it dangerous for a split. A support case is resolved when the refund posts in billing; an invoice dispute ends with a credit note in the ERP; a sales exception is confirmed by a signed amendment. Those later records are the right labels and the wrong training inputs.
Kapoor and Narayanan describe temporal leakage as test data drawn from before the training data when the model predicts a future outcome, so the model effectively learns from the future. In workflow data the version to watch is the label record: if the model can see, anywhere in training, a record created after the decision it is asked to make, the evaluation is measuring lookup, not judgment.
Write the timestamp rule into the manifest. Features for a test case may only come from records created before its decision time; labels come from linked records created after it; and no later record linked to a test case may appear in training. For time-ordered evaluation within one company, scikit-learn's time series splitter keeps every test fold later than its training data, but it does not know about cross-system links, so the link check still has to be run separately.
Illustrative case: 40 companies' support and billing histories
Every number in this case is illustrative. A team builds an evaluation of a model that drafts dispositions for billing disputes, using histories from 40 companies: tickets from a help desk, invoices and credit notes from an ERP, and account notes from a CRM. The first version used a random 80/20 split of 30,000 tickets.
Applying the specification, the team moves 8 companies, stratified by industry, into a company held-out test set and holds out the last quarter of the other 32 companies as a temporal test. The leakage scan on the original random split had found that about one in five test tickets shared a macro or a quoted thread with a training ticket, and about one in twelve had a linked credit note on the training side that stated the outcome.
The team now reports three numbers rather than one: the original random-split score labeled as an in-distribution reference only, the temporal test score, and the company held-out score with an interval computed across companies rather than tickets, since the companies are the independent units. The distance between the first and third numbers is the honest measure of how much the model relied on knowing the company.
The counterexample matters too. For a dispute type that occurred at only three companies, a company held-out test would leave too few cases to read, so the team reports that family as not evaluated for new companies rather than quietly mixing it back into a random split.
When a random split is the honest choice
A random split is defensible only when the deployment claim matches it: the model will serve the same companies, over the same period, on the same kinds of tasks, and you have removed shared cases and derivatives. Even then, treat it as an upper bound and say so in the report.
Dataset publishers have learned to fix splits rather than leave them to users. The DocLayNet authors, for example, ship predefined train, validation and test sets designed to keep class labels proportional and to avoid leakage of unique layout styles across sets. A workflow dataset should arrive the same way, with company, time and family boundaries already drawn and a manifest that lets an evaluator verify them.
When you acquire workflow histories for evaluation, ask for the group identifiers and timestamps before anything else. SourceX sources such datasets from U.S. businesses on request, so a buyer can specify the split it needs, such as the number of distinct companies, the time range and the linked systems, alongside volume, format, timing and licensing needs. Each supplying company agrees the scope and terms before any data moves, and that approval belongs in the provenance record next to the split manifest.
Frequently asked questions
What should a company held-out split with leakage examples include?
A fixed list of test companies that never appear in training, a temporal test for in-distribution companies, case and entity ids on every record, near-duplicate and shared-artifact scans, and a manifest with cutoff dates, thresholds and split hashes. Pair it with concrete leakage examples so reviewers know what the scans are for.
Why do random splits overestimate model performance on business data?
Rows from the same company share templates, policies, names and phrasing, so a model can recognize the company instead of solving the task. Linked later records can also reveal outcomes. Both inflate test scores without improving real-world performance on new businesses.
Is GroupKFold enough to prevent leakage?
Group-wise splitters such as scikit-learn's GroupKFold keep one group on one side of each fold, which handles company or entity grouping. They do not detect derivatives, shared attachments or cross-system links, and they ignore time, so you still need near-duplicate, link and timestamp checks.
How many companies should be held out for testing?
Enough that the test score has a usable interval when computed across companies, since companies are the independent units. Very few held-out companies make the score depend on which ones were picked; stratify by industry and system and report the interval with the score.
Should the split be decided before or after cleaning the data?
Draw the group and time boundaries before tuning any preprocessing, and fit cleaning or normalization steps on training data only. Otherwise information from test records can shape the pipeline that the model is judged with.
Sources
- The scikit-learn user guide says grouped data breaks the i.i.d. assumption, that group-wise splits must keep validation groups out of training, and that time series splits evaluate on later observations. Source
- Kapoor and Narayanan classify non-independence between training and test samples, and test data that predates training data for future predictions, as forms of leakage. Their survey found leakage-related errors in papers across many scientific fields. Source
- Lewis, Stenetorp and Riedel found substantial test-train overlap in open-domain question answering datasets and that overlapping items account for much of measured performance. Source
- Lee and colleagues found many near-duplicate examples in language model datasets and that removing train-test overlap affects part of common validation sets. Source
- Yang and colleagues showed that paraphrased or translated test items can bypass n-gram decontamination and still inflate benchmark scores. Source
- DocLayNet provides predefined train, validation and test sets designed to keep class labels proportional and avoid leakage of unique layout styles across sets. Source
Tell us what data you need
Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.