AI uses for records
Business dataset utility benchmark: a protocol for shared-task tests
By SourceX Editorial · Updated
10 sections
Short answer
To compare datasets on the same downstream business tasks, freeze one model, recipe and training budget, train on each candidate, and score every model on a shared suite of task families from companies none of the datasets came from. Grade blind with expert rubrics, repeat across seeds, report per family with company-clustered intervals, and trust only rankings that survive held-out families. Method only, no measured results.
Evaluation split
Business dataset utility
Key takeaways
- The task suite is the yardstick: every dataset is judged on the same tasks, graders and metrics.
- Freeze the model, recipe and training budget so the dataset is the only thing that varies.
- Hold out whole companies and whole task families, because rows from a seen company leak its habits.
- Report a utility profile per task family with intervals, not one leaderboard number.
- This page publishes the method only; it reports no results.
How do you compare datasets using the same downstream business tasks?
Treat each dataset as a treatment and the task suite as a fixed test. Train the same base model with the same recipe and the same training budget on each candidate, then score every resulting model on one shared set of business tasks drawn from companies that contributed to none of the datasets. The comparison is the difference in task scores, task family by task family, with its uncertainty.
Three choices make or break the result. The task suite has to represent the work the buyer actually needs done, the holdout has to be separated by company and by task family rather than by row, and the grading has to be blind to which dataset produced the model. Everything else in the protocol below protects those three.
Read what follows as a study design. It sets out the controls, the records to keep and the tests a ranking has to pass; it contains no measurements, and every example is illustrative.
Why one shared task suite beats each supplier's own demo
Suppliers naturally show their data on tasks where it shines, scored by metrics they chose. Those demos cannot be compared with each other, because the task, the model and the metric all change at once. A shared suite holds them still.
Public data benchmarks were built on the same reasoning. DataComp fixes the training code and compute, forbids participants from changing hyperparameters, architecture, compute budget or training steps, and evaluates each proposed dataset on 38 downstream test sets. DataPerf, a community suite for data-centric work, freezes model architectures, training hyperparameters and task metrics so that entries differ only in their data. HELM, which evaluates models rather than datasets, makes the parallel point that the adaptation strategy has to be controlled for any comparison to mean something, and it reports several metrics rather than one.
A business benchmark adds what those public suites lack: tasks from real operations, graded the way the business judges the work, and a holdout built from companies rather than from web pages. That makes it smaller and more expensive per item, which is why its design has to be fixed before any dataset is trained on.
Building the task suite: families, cases and rights
Group tasks into families that match distinct pieces of work, such as ticket triage, invoice exception handling, contract clause extraction or scheduling a field visit, and draw several cases per family from each source company. Every case is a record with the fields below, so that any grader or harness can run it without asking the author what was meant.
At a glance
Building the task suite: families, cases and rights
Rights come first. Each case must be cleared for evaluation use by the company it came from, including the right to keep it as a private test and the right to publish aggregate scores. A suite with one uncleared family is a suite that cannot be released.
Size the families evenly where you can. A suite dominated by one family will rank datasets by their fit to that family alone, and an overall average will hide it. If some families can only supply a few cases, keep them, report them separately with wide intervals, and do not fold them into a headline figure.
Have a second domain expert solve a sample of cases cold, without the reference outcome. Cases the expert cannot solve from the inputs given are underspecified, and they test the grader's patience rather than the model. Fix or drop them before the suite is frozen.
- Case ID, task family and source company code (a code, never the company's name in the released file).
- Inputs exactly as the model receives them: the records, documents or conversation history, with personal data removed or replaced.
- The instruction, written as the business would give it to a new employee.
- Reference outcome: what the business actually did or accepted, plus acceptable alternatives where several answers are valid.
- Grading rubric for the family, with the rubric version and the criteria each score level requires.
- Difficulty notes and known traps, recorded by the domain expert who prepared the case.
- Provenance: source system, extraction date and the rights clearance reference.
The matched training budget and held-out evaluation protocol
This is the asset. Fix every row before any candidate dataset is trained on, date the document and store it read-only. The right-hand column is what must be logged so that someone else could repeat the comparison.
Run the work in a fixed order so that no later step can influence an earlier one. Freeze and hash the test suite first, then deduplicate every candidate against it, then tune the recipe once on the validation split, then train all arms, and only then unblind graders to the full set of outputs. A team that tunes after seeing test scores, even informally, has turned the holdout into a second validation set and should say so in the report.
| Component | What is fixed | Why it is fixed | Record for every run |
|---|---|---|---|
| Base model | One model and checkpoint for all arms, plus a no-fine-tune arm | Model capacity must not vary with the dataset | Model name, version and checkpoint hash |
| Training recipe | Code, hyperparameters, prompt format and stopping rule, tuned once on a separate validation split | Retuning per dataset rewards tuning effort, not data | Code commit, config file and validation split ID |
| Training budget | Same training tokens per arm; smaller datasets repeat for more epochs, larger ones are subsampled by a fixed rule | More compute can look like better data | Tokens, steps, epochs and subsampling seed |
| Candidate datasets | Each deduplicated against every test case and against the other source companies' test material | Overlap with the test inflates scores | Dedup method, matches removed and dataset version |
| Holdout companies | Companies whose cases form the test, contributing nothing to any candidate | Rows from a seen company carry its habits into the test | Company codes per fold |
| Holdout families | At least one task family kept out of every candidate's training and tuning | Tests whether the gain carries to work nobody prepared for | Family list per split |
| Seeds | Same seed list in every arm | Training randomness can exceed the data effect | Seed list and per-seed scores |
| Grading | Blinded experts using versioned rubrics, with a share of cases double-graded | Graders who know the source may favor it | Grader IDs, rubric version and agreement on double-graded cases |
| Decision rule | Primary metric per family, interval method and what counts as a win, written in advance | Choosing the metric after seeing scores manufactures winners | Pre-registration link and any deviation with its reason |
Holding out whole companies and task families, not just rows
Split by company, because records from one company share templates, codes, staff habits and customers. If one company's tickets are in a candidate dataset and other tickets from the same company are in the test, the model can score well by recognizing the house style. The scikit-learn documentation describes the same problem for grouped data and the GroupKFold splitter as the remedy: every sample in the test fold comes from groups not represented at all in the paired training fold.
Hold out whole task families too. A dataset that helps on families resembling its own content may do nothing for a family it never touched, and a buyer usually needs the second answer. Keep at least one family out of all training and tuning, report it separately, and rotate which family is held out if the suite is large enough.
Leakage can enter in quieter ways. Kapoor and Narayanan documented leakage in published machine learning work across many scientific fields, which is a reminder to check for shared documents, near-duplicate records and test material that appears in public sources a candidate was built from. Run those checks before training and log what they removed.
Graders, baselines and what counts as a win
Use domain experts as graders, give them the rubric for the family, and hide which arm produced each answer. Mix the arms' outputs in random order. Double-grade a share of cases fixed in advance, report agreement, and adjudicate disagreements with a third expert who also does not know the source.
Every comparison needs two baselines. The base model with no fine-tuning shows whether any dataset helps at all. A reference dataset, such as a public dataset for the same kind of work, shows whether the candidates help more than something freely available. A candidate wins on a family only if it beats both baselines and the other candidates by more than the pre-registered interval allows.
Report a profile, not a single number. A dataset can lead on invoice exceptions, tie on triage and trail on the held-out family. That pattern is more useful to a buyer than an average that hides it.
Watch for graders drifting over a long grading period. Seed the queue with a small set of calibration cases whose scores were agreed at the start, check each grader against them at intervals, and record any rubric clarification as a new rubric version rather than an informal note.
Seeds, items and companies: three sources of uncertainty
Scores move for reasons that have nothing to do with the data. Bouthillier and colleagues showed that variation from data sampling, parameter initialization and hyperparameter choice can change the conclusions drawn from machine learning benchmarks, so a single run per arm is not evidence. Run every arm on the same seed list and report the spread.
Items and companies add their own noise. A suite drawn from a handful of companies can reward the dataset that happens to resemble them. Compute intervals by resampling whole companies, not individual cases, so the uncertainty reflects how many independent sources stand behind the result.
Choose the test before the data. Dror and colleagues found that significance testing in natural language processing is frequently neglected or misapplied and proposed a protocol for selecting the right test. Name the paired test or resampling method, the correction for comparing several datasets across several families, and the smallest difference that would matter to the buyer.
Write the uncertainty into the result itself. A sentence such as dataset B led on two of five families, tied on two and trailed on the held-out family, each with its interval, tells a buyer far more than a rank. If the intervals for two candidates overlap on a family, say the comparison was inconclusive there; that is a legitimate outcome of a well-run benchmark, not a failure of it.
Which results would stay valid on held-out companies and task families?
Only the ones that pass validity tests written in advance. A ranking that holds on the seen families but reverses on the held-out family is a finding about narrow fit, not a general advantage. The tests below decide how far a result can be stated.
At a glance
Which results would stay valid on held-out companies and task families
- Company rotation: the ranking of candidates is stable when each holdout company is dropped in turn.
- Family rotation: the leading dataset still beats the baselines on the held-out family.
- Seed stability: the ranking holds across seeds, not only on the mean.
- Grader stability: conclusions do not change when only double-graded, adjudicated cases are scored.
- Leakage control: scores do not drop sharply after a stricter deduplication pass.
- Scope: the result is stated only for the base model, budget and task families tested.
Pre-registration and release checklist
Fix these items before the first candidate is trained, and treat anything decided afterwards as exploratory in the report. The release conditions matter as much as the method, because a business task suite is built from other companies' records.
Checklist · 8 items
- Task families, cases per family and the source companies by code, with rights clearance for evaluation use and for publishing aggregate scores.
- Holdout companies and held-out families, and the rotation plan.
- Base model, recipe, validation split and the matched training budget.
- Candidate datasets with versions, and the deduplication method against all test material.
- Grader pool, blinding, rubric versions and the double-grading share.
- Primary metric per family, interval method, multiple-comparison correction and the decision rule.
- Validity tests and the wording rules for stating a result.
- What is published: aggregate scores and methods only, never test cases, company names or anything that would reveal a case.
0 of 8 ticked. Tick each item as you go.
Sourcing candidate datasets and task cases for the comparison
A comparison like this needs two kinds of material from businesses: candidate training data, and separate, rights-cleared task cases from companies that supplied none of it. Keep the agreements apart, so that evaluation cases can never drift into a training set.
Ask each supplier the same questions about how its data was collected, which companies it covers and whether any of it overlaps public sources, and record the answers next to the dataset version. Those answers explain results later: a candidate that wins on one family and loses on the held-out one is often a dataset built from a few similar companies.
SourceX can source candidate datasets on request from U.S. businesses rather than from stock, and the buyer's request form records scope, volume, language, format, timing and licensing needs for each candidate. Keep holdout material under its own evaluation-only terms so it never joins a training set. Every supplying company, whether it provides training data or test cases, approves the scope and terms and signs before delivery.
Frequently asked questions
What is a business dataset utility benchmark?
It is a fixed suite of downstream business tasks used to compare training datasets: each dataset trains the same model under the same budget, and the resulting models are scored on the same tasks. The utility of a dataset is how much it improves task performance over baselines. This page publishes a protocol for such a benchmark, with no results.
How is this different from a model evaluation benchmark?
A model benchmark holds the test fixed and varies the model. A dataset utility benchmark holds both the test and the model recipe fixed and varies only the training data. The task suite can look similar, but the controls and the conclusions differ.
Why hold out entire companies instead of a random sample of records?
Records from one company share formats, codes and habits, so a model trained on some of them can recognize the rest without learning the task. Holding out whole companies tests whether a dataset helps on organizations the model has never seen, which is the situation a buyer deploys into.
Does a matched training budget mean the same number of records?
No. Match training tokens or steps for the same model, because records vary in length and a dataset with longer records would otherwise receive more compute. Smaller datasets are repeated for more epochs and larger ones subsampled by a rule fixed in advance, and both are logged.
What sample size and baselines make a result defensible?
Enough source companies that resampling by company gives intervals narrow enough to separate the differences the buyer cares about, set with a statistician before any training. Every comparison needs the untuned base model and a reference dataset as baselines. Without both, a win only shows that training helped, not that the dataset did.
Sources
- DataComp fixes training code and compute, lets participants change only the dataset and evaluates each dataset on 38 downstream test sets. Source
- DataPerf freezes model architectures, training hyperparameters and task metrics so entries are compared by their data. Source
- HELM argues the adaptation strategy must be controlled for meaningful model comparison and reports multiple metrics. Source
- The scikit-learn documentation describes GroupKFold, which keeps every test group out of the paired training fold to measure generalization to unseen groups. Source
- Kapoor and Narayanan document data leakage in machine learning based studies across many scientific fields. Source
- Bouthillier and colleagues show that data sampling, initialization and hyperparameter choice add variance that changes benchmark conclusions. Source
- Dror and colleagues report that significance testing in NLP is often neglected or misused and propose a protocol for choosing tests. Source
Tell us what data you need
Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.