Privacy and preparation
Enterprise benchmark task family IDs: assign them before you split
By SourceX Editorial · Updated
10 sections
Short answer
Assign task family IDs from the fields that decide how a task is solved: task type, source system, the procedure or policy version the answer follows, and the template it came from. Hash that normalized key, merge families joined by near-duplicates, freeze the IDs, then split by family, plus entity, company and time where needed, so no sibling task sits on both sides.
Evaluation split
Enterprise benchmark task
Key takeaways
- Workflow tasks cluster around shared procedures, so a row-level split puts copies of the same answer in training and evaluation.
- Derive the family from how a task is solved: task type, source system, procedure version and template, never the subject line.
- Merge families joined by near-duplicate clusters, then freeze IDs and record the derivation version before any split.
- Validate family, entity, company and time isolation, and check comments, attachments and knowledge articles for exposed answers.
- Ask suppliers for the key fields and registers so family IDs can be recomputed after delivery.
How do you assign task family identifiers before splitting a dataset?
Derive each task's family identifier from the fields that decide how the task is solved, not from how it looks: the task type, the source system, the procedure or policy version the correct answer follows, and the template or form it was created from. Normalize those fields, join them into a canonical key, hash the key into a stable ID, merge families whose tasks are near-duplicates, and freeze the IDs before any split is drawn.
Then split by family. Every task in a family goes to training or to evaluation, never both, and the same rule applies to company and entity groups and to a time cutoff when those matter. A split drawn at the row level on workflow data almost always places siblings of the same task on both sides, and the model is rewarded for remembering a sibling rather than for doing the work.
This page gives the schema for that identifier, the derivation steps, a worked record that passes and one that fails, and the validator checks to run before the split is frozen.
Why a random row split leaks answers in workflow data
Workflow tasks come in clusters that share their answer, so a random split puts the answer key on both sides. Thirty purchase-order exceptions resolved under the same tolerance rule, or two hundred access requests closed by the same runbook step, are not thirty or two hundred independent tests. They are one test with many copies.
The scikit-learn documentation states the underlying assumption plainly: ordinary cross-validation assumes independent, identically distributed samples, and that assumption breaks when the generating process yields groups of dependent samples. Its grouped splitters exist so that every sample in a validation fold comes from groups not represented at all in the paired training fold.
The cost of ignoring this is measurable. Lee and colleagues found that every one of the four common language-model datasets they studied contained duplicates and that train-test overlap affected more than 4 percent of the validation set of standard datasets. Kapoor and Narayanan surveyed leakage in machine-learning-based science and found published leakage errors in 17 fields affecting hundreds of papers, and their taxonomy counts non-independence between training and test samples as leakage unless the claim is about data with the same dependence structure.
Enterprise records are worse than web text on this point, because business work is deliberately repetitive. Templates, macros, standard operating procedures and approval matrices exist precisely so that similar requests get the same answer.
Task family, entity group, company group: which unit does each split protect?
A task family protects against memorized procedures, an entity group protects against memorized facts about one customer, vendor or asset, and a company group protects against memorized conventions of one organization. Most enterprise benchmarks need at least two of these, recorded as separate fields so each can be checked on its own.
| Grouping unit | What it holds together | Leak it prevents | Example key fields |
|---|---|---|---|
| Task instance | One request and its resolution | None on its own; it is the row being split | Ticket ID, order line, case number |
| Task family | Instances solved by the same procedure, rule version and template | The model recalls a sibling's answer instead of applying the procedure | Task type, source system, procedure reference and version, template ID |
| Entity group | Instances about the same customer, vendor, asset or employee | The model recalls facts about a specific entity seen in training | Pseudonymous customer, vendor or asset key |
| Company group | Instances from the same contributing organization | The model learns one company's naming, codes and habits and scores well only on that company | Contributing organization key |
| Time slice | Instances created in the same period | Training on events that happened after the evaluation tasks | Created timestamp, policy effective date |
The task family identifier and derivation schema
Every task record carries the fields below. The family ID is computed, never typed, and the derivation version travels with it so a buyer can recompute the ID and get the same answer. Field names are suggestions; the derivation rules and checks are the part to keep.
At a glance
The task family identifier and derivation schema
| Field | Type | Derivation rule | Validation check |
|---|---|---|---|
| task_id | string | Stable ID of the instance in the delivered dataset, never the raw source key if that key is sensitive | Unique across the dataset |
| task_type | enum | Controlled vocabulary for the kind of work, such as invoice_exception or access_request | Value is in the published vocabulary |
| source_system | enum | System family the task came from, such as ERP, ITSM or CRM, plus its configuration profile | Present on every row |
| procedure_ref | string | ID and version of the runbook, policy, macro or approval rule the correct resolution follows; none_found when no procedure applies | Resolves to an entry in the procedure register |
| template_ref | string | Form, ticket template or document template the request was created from; none when free-form | Resolves to an entry in the template register |
| family_key | string | Lowercased, trimmed join of task_type, source_system, procedure_ref and template_ref in a fixed order | Recomputes byte for byte from the four fields |
| family_id | string | First 16 hex characters of the SHA-256 hash of family_key, then replaced by the cluster root after near-duplicate merging | Every task with the same family_key has the same family_id |
| near_dup_cluster | string | Cluster ID from text similarity over request and resolution text after placeholders are normalized | No cluster spans two family_id values after merging |
| entity_group and company_group | string | Pseudonymous keys for the main entity and the contributing organization | Present when the split plan names them as split units |
| derivation_version and split | string | Version of these rules, and the split label assigned only after all IDs are frozen | Split assigned by group, never by row; version recorded in the dataset card |
Deriving the family key, step by step
The derivation has to be deterministic and boring. Anyone holding the delivered records and the procedure register should get the same family IDs, and a change in the rules should produce a new derivation version rather than silently new IDs.
Checklist · 7 items
- Build the procedure register first: every runbook, policy, macro and approval rule cited in resolutions, each with an ID and version. Unknown procedures are recorded as none_found, not guessed.
- Map each task to its procedure from the resolution itself, such as a macro applied in Zendesk, a knowledge article linked in ServiceNow, or the approval rule recorded on a NetSuite transaction, rather than from the request subject line.
- Normalize the four key fields: fixed order, lowercase, trimmed, controlled vocabulary for task_type and source_system.
- Hash the joined key and keep a short, fixed-length prefix as the provisional family_id; check the prefix for collisions across the dataset.
- Normalize placeholders in request and resolution text so de-identified names and numbers do not hide duplicates, then cluster near-duplicates with a hash-based method such as MinHash, which Lee and colleagues used for approximate document matching.
- Merge any families joined by a near-duplicate cluster into one, using the lowest family_id as the root, and log every merge with its reason.
- Freeze the IDs, publish the derivation version, and only then assign splits by family, entity, company and time as the split plan requires.
0 of 7 ticked. Tick each item as you go.
Worked record (illustrative): one task that passes, one that fails
Both records are invented to show the structure; they are not drawn from any company's data. The passing record is an accounts payable exception from an ERP, keyed to the tolerance rule its resolution actually applied.
At a glance
Worked record (illustrative): one task that passes, one that fails
- Passing record: task_id ap-000412; task_type invoice_exception; source_system erp/three-way-match; procedure_ref price-tolerance-rule v3; template_ref vendor-invoice-standard; family_key invoice_exception|erp/three-way-match|price-tolerance-rule v3|vendor-invoice-standard; family_id taken from the hash of that key; near_dup_cluster c-0917, which contains only tasks from this family; split evaluation, with all 41 sibling tasks in the family also in evaluation.
- Failing record: task_id it-002230; task_type access_request; family_id derived from the ticket subject line, Request access to finance share. A second subject line, Need permissions for finance folder, produced a different family_id. Both tickets were resolved by the same runbook step, runbook access-finance v2, so the validator finds near_dup_cluster c-1204 spanning two family_ids, one in training and one in evaluation.
- Fix for the failing record: recompute family_key from procedure_ref instead of the subject line, merge the two families, and move the whole merged family to one side of the split. The evaluation set loses a handful of tasks and gains a score that means something.
Which validator checks should run before the split is frozen?
Run the checks below on every release, keep the output with the dataset, and treat any failure as blocking. Each one is a simple query over the fields in the schema, so a buyer can rerun them without access to the source systems.
| Check | Rule | Fails when | Action on failure |
|---|---|---|---|
| Family isolation | Each family_id appears in exactly one split | Any family_id has rows in two splits | Move the whole family; never move single rows |
| Key reproducibility | family_id recomputes from family_key under the stated derivation version | A recomputed ID differs from the delivered one | Block release and fix the derivation |
| Cluster containment | Each near_dup_cluster maps to one family_id | A cluster spans families | Merge the families, then recheck isolation |
| Entity and company isolation | Named split units do not cross splits | An entity or company key appears on both sides | Reassign the group or drop the crossing rows |
| Temporal cutoff | Evaluation tasks were created after the stated cutoff | An evaluation task predates the cutoff | Move it to training or drop it |
| Answer exposure | No training text contains an evaluation resolution or a near copy of it | A comment, attachment, summary or knowledge article repeats the answer | Remove the derivative or move its family |
| Coverage balance | Each family type appears in evaluation at the planned share | Families concentrate in one split after merging | Rebalance with a stratified group split |
Could a comment, attachment or derivative still reveal the evaluation answer?
Yes, and family IDs alone will not catch it, because the leak travels in a different record. A resolution that becomes a knowledge article, a ticket comment that pastes the fix, an email thread attached to an order, or a summary written for a monthly report can all carry the answer into training while the task itself sits safely in evaluation.
Public benchmarks show the pattern. The SWE-Bench+ authors examined passing patches on SWE-bench and reported that 32.67 percent of them involved solutions provided directly in the issue report or its comments. Enterprise records hold the same trap in a different shape: the fix is often written into the ticket long before the ticket is closed.
Trace every derivative back to the task it came from. Give knowledge articles, macros and report extracts a source_task_ids field where possible, run the answer-exposure check against them, and when the link cannot be recovered, keep that derivative out of training for the evaluated families.
When does a family split need a time split or a stratified group split as well?
Add a time split whenever the procedures themselves change, and add stratification whenever merging leaves a few very large families. A family split stops a model from recalling a sibling, but it does not stop a model trained on next year's records from being tested on last year's.
LiveCodeBench handles the time side by tagging each problem with its release date and evaluating newer models only on problems released after their training cutoff, and its authors report sharp drops on problems released after that date for some models. The same discipline applies to workflow data: record a cutoff, keep evaluation tasks after it, and split the analysis when a policy version changes inside the window.
For balance, scikit-learn documents StratifiedGroupKFold, which keeps groups intact while trying to preserve the class distribution across folds, and TimeSeriesSplit for ordered data. Record the result in the dataset card. The Croissant metadata format lets a dataset declare its splits through a cr:Split record set, and its versioning guidance treats records shuffled across splits as a major version change, which is the right signal for a re-derived family ID too.
Requesting workflow records that already carry family identifiers
The fields this schema depends on, especially procedure_ref and the links from derivatives to tasks, are easiest to capture while records are prepared, and much harder to rebuild after delivery. Put them in the request rather than planning to infer them later.
Because SourceX fills workflow requests from U.S. businesses rather than from inventory, an AI team can state scope, volume, language, format, timing and licensing needs and attach a family-ID schema like this one. That tells the supplying company and the preparation team which procedure keys to preserve and which derivatives to trace before anything is exported.
Frequently asked questions
What is a task family ID in an enterprise benchmark?
It is a computed identifier shared by every task that is solved by the same procedure, rule version and template in the same kind of system. Splitting by it keeps all siblings of a task on one side, so evaluation scores reflect applying the procedure rather than recalling a sibling's answer.
Why not use the ticket subject or category as the family?
Subject lines and categories describe how a request was phrased or filed, not how it was solved. Two differently worded requests resolved by the same runbook step belong to one family, and a key built from subjects will split them across training and evaluation.
Is GroupKFold enough to prevent leakage?
GroupKFold keeps each group out of the paired training fold, but it is only as good as the group labels you give it. It also does nothing about answers that leak through comments, knowledge articles or later records, so the answer-exposure and temporal checks are still needed.
How many family IDs should a dataset have?
There is no target count; the number follows from the procedures in the source systems. Watch the distribution instead: if near-duplicate merging collapses most tasks into a few families, the evaluation set will be thin, and a stratified group split or more source variety is needed.
What should a buyer ask a supplier to deliver with the split?
The family key fields, the derivation version, the procedure and template registers, the near-duplicate method and threshold, and the validator output for the delivered release. With those, the buyer can recompute every family ID and confirm isolation without access to the source systems.
Sources
- scikit-learn's cross-validation guide says the i.i.d. assumption breaks for groups of dependent samples, describes GroupKFold, StratifiedGroupKFold and GroupShuffleSplit for grouped data, and TimeSeriesSplit for ordered data. Source
- The Croissant 1.1 specification lets a dataset declare training, validation and test splits through cr:Split and treats records shuffled across splits as a MAJOR version change. Source
- Lee and colleagues found duplicates in all four language-model datasets studied, train-test overlap affecting over 4 percent of standard validation sets, and used hash-based approximate document matching. Source
- Kapoor and Narayanan found leakage errors in 17 fields affecting hundreds of papers and presented an eight-type taxonomy that treats non-independence between training and test samples as leakage. Source
- LiveCodeBench tags problems with release dates, evaluates newer models on problems released after their cutoff, and reports performance drops on post-cutoff problems for some models. Source
- The SWE-Bench+ study reports that 32.67 percent of successful SWE-bench patches involved solutions provided directly in the issue report or comments. Source
Tell us what data you need
Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.