Skip to content

AI uses for records

Dataset diversity task coverage: why 12 industries can be one workflow

By SourceX Editorial · Updated

10 sections

Short answer

Dataset diversity task coverage asks whether cases vary in workflow, decision and context, not how many industry labels the sources carry. Run three checks: collapse labels into distinct workflow profiles, confirm each workflow appears under more than one context, and confirm each decision shows both outcomes. A 12-industry dataset can fail all three while a four-industry dataset passes.

Agent run

Dataset diversity task

This page covers dataset diversity task coverage: why 12 industries can be one workflow.Illustrative

Key takeaways

  • Industry counts depend on classification depth: NAICS 2022 has 20 sectors but 1,012 six-digit codes.
  • The same work recurs across labels, so report distinct workflow profiles instead of industries.
  • A dataset whose workflows never cross contexts invites shortcut learning on sector vocabulary.
  • Each decision needs both outcomes present at a minimum support, cell by cell.
  • Set floors in cases and distinct companies per cell before the first sample arrives.

What does dataset diversity task coverage measure that an industry count does not?

It measures whether the cases in a dataset vary along the dimensions that change what a model has to do: the workflow being run, the decision taken at its choice points and the context that alters the right answer. An industry count measures how many labels the sources carry, a number that can climb while none of those three moves.

The gap shows the moment a model meets a case outside its training mix. A dataset from 12 industries in which every source contributed the same invoice-dispute workflow, through one channel, under one policy, teaches a single job twelve times. A dataset from four industries that covers seven workflows, each seen through several channels and policy versions, with approvals and denials both present, teaches far more.

This guide gives a curator or research lead a reproducible diversity test for task coverage built on three checks, called collapse, crossing and contrast, plus a sampling worksheet that records how cells were defined and filled and a table of blind spots the test still cannot see. The example datasets and their numbers are illustrative.

Why industry labels make a weak unit of diversity

Industry labels are coarse, arbitrary in depth and only loosely tied to the work inside the records. Their count depends first on the classification level you pick: the 2022 North American Industry Classification System divides the economy into 20 sectors, while a Census Bureau working paper counts 1,012 codes at the six-digit level of the same vintage. Twelve industries can mean twelve sectors or twelve neighboring six-digit codes inside one sector.

The work itself crosses label boundaries. O*NET, the U.S. occupational database, links each detailed work activity to task statements in two or more occupations, which is another way of saying the same activity turns up under many job titles. Processing a refund, scheduling a technician or disputing an invoice looks much the same at a utility, a clinic and a distributor.

Researchers have named the wider problem. In an ICML 2024 position paper, Zhao and colleagues examined how diversity is described across 135 image and text datasets, found the word often used without a definition or any validation, and drew on measurement theory from the social sciences to argue that curators should conceptualize, operationalize and evaluate diversity explicitly. An industry count skips all three steps.

Three axes that change what a model must do

Define diversity on workflow, decision and context, because each one changes the correct output in its own way. Every axis needs a written list of levels before any counting, taken from a process map, a policy manual or the buyer's task description rather than read off the data:

  • Workflow: the sequence of steps and systems a case passes through, such as a refund in Zendesk that triggers a credit memo in NetSuite, or a ServiceTitan job that moves from booking to dispatch to invoice. Two cases share a workflow when they follow the same path, whatever their sector.
  • Decision: the choice point inside the workflow and its possible outcomes, such as approve, partly approve or deny, or escalate versus resolve. A dataset that shows only one outcome at a choice point teaches the model that the choice does not exist.
  • Context: the conditions that change which outcome is right, such as channel, policy version, customer tier, contract terms or jurisdiction. Most costly business errors sit here, because the same request deserves a different answer under a different rule.

The collapse check: how many distinct profiles hide behind the labels?

Collapse the labels into workflow profiles and count what remains. For each industry label, tabulate its cases by workflow and decision outcome; two labels whose distributions sit within a distance fixed in advance merge into one profile. The number of profiles, not the number of labels, is the first diversity figure worth reporting.

Choose the merge rule before seeing the result, and publish it. A simple rule merges labels whose workflow shares differ by less than ten percentage points on every workflow; a stricter one uses a divergence measure with a stated threshold. Either way the rule, the threshold and the merged pairs go into the worksheet, so a second reviewer can reproduce the count.

The check also runs in reverse. One label can hide several profiles, such as a healthcare source whose records hold both appointment scheduling and claims appeals, and splitting it is as informative as merging two others.

An illustration of the merge in practice: suppose the labels dental clinics, veterinary practices and physical therapy each show more than 80 percent appointment rescheduling, the same two outcome codes and the same booking system. Under the ten-point rule they become one profile, and the worksheet records the three labels against it. The dataset has not lost any cases; it has lost two claims of breadth it could not support.

The crossing check: is one axis confounded with another?

Check that each workflow and decision appears under more than one context, and each context under more than one workflow. When every industry contributes exactly one workflow through one channel under one policy, the axes are perfectly confounded, and a model can learn the label instead of the job.

Machine learning has a name for that failure. Geirhos and colleagues describe shortcut learning: decision rules that perform well on standard benchmarks but fail to transfer to harder conditions such as real-world use. In a confounded business dataset the shortcut is cheap, because sector vocabulary such as clinic or freight predicts the workflow and the outcome without the model reading the case.

Report the crossed share: the fraction of workflow-outcome cells observed under at least two contexts, each at the minimum support the worksheet sets. A crossed share near zero means the context axis exists on paper only, whatever the industry count says.

The contrast check: does each decision show both outcomes?

Within every workflow and context, look for cases that went each way: approved and denied, escalated and resolved, acted on and left alone. A cell that only ever shows one outcome teaches a one-sided policy, however many cases it holds.

Google's Machine Learning Crash Course makes the mechanical version of the argument: training batches need enough examples of both classes, and in its example of 200 majority and 2 minority examples, most batches of 20 contain no minority case at all. An illustrative archive holding 4,000 approved refunds and 40 denials has the same problem at the level of one decision, and adding more approvals does not fix it. The balance has to hold at each decision, not only in aggregate.

Report the contrast share: the fraction of workflow-context cells that hold both outcomes above the minimum support. It is usually the lowest of the three numbers, because archives keep more of whatever outcome the business produced most often.

Worked example: 12 industries against 4

Two illustrative datasets show how far the headline and the test can diverge. Dataset A carries 12 industry labels and 30,000 cases from nine companies; Dataset B carries four labels and 9,000 cases from 14 companies. On an industry count, A looks three times as diverse. Both are scored against the same buyer design: seven workflows, two outcomes at each choice point and three contexts, giving 14 workflow-outcome cells and 21 workflow-context cells.

The three checks reverse the order. A's labels collapse into three workflow profiles, all seen in a single context, so none of the 14 workflow-outcome cells appears under a second context and only one of the 21 workflow-context cells holds both outcomes. B keeps four distinct profiles across all seven workflows, sees 11 of the 14 cells under two or more contexts and shows both outcomes in 14 of the 21 workflow-context cells. For a model that must handle refunds, disputes and scheduling across channels and policy changes, B is the more diverse dataset with less than a third of the cases. The figure draws the collapse and the fan-out; the table gives each measure:

Worked example: 12 industries against 4
MeasureDataset A (illustrative)Dataset B (illustrative)What it tells a buyer
Industry labels124The figure a sales sheet leads with
Distinct profiles after the collapse check34How many genuinely different jobs the labels hold
Design workflows with 30 or more cases from 3 or more companies3 of 77 of 7Breadth that clears the support floor
Workflow-outcome cells seen under two or more contexts0 of 1411 of 14Whether the model can learn that context changes the answer
Workflow-context cells holding both outcomes1 of 2114 of 21Whether each decision is shown going both ways
Largest single company's share of cases41 percent18 percentHow far one source's habits dominate
Two panels: twelve industry labels in Dataset A converge on three workflows seen in one context each, while four labels in Dataset B fan out to seven workflows, most seen in two or three contexts.
Illustrative: industry labels versus workflow and context coverage in two example datasets.

The diversity sampling worksheet

Record how cells were defined and filled before any coverage number leaves the team. Industry labels still belong in the worksheet, as background with a stated source and depth, not as the measure. The eight lines below are the minimum, each with an illustrative entry and the condition that should send the worksheet back:

The diversity sampling worksheet
Worksheet lineWhat to recordIllustrative entrySend it back when
Unit of diversityWhat one independent case is, as distinct from a row or a messageOne resolved support case with its messages and status changes joinedCounts are given in rows or messages
Workflow list and sourceThe workflows the target task needs and the document that lists themSeven workflows from the buyer's process map dated September 2026Workflows are inferred from industry labels
Decision and outcome codesThe choice point in each workflow and its possible outcomesRefund: approve, partly approve or deny; escalate or resolveOnly one outcome appears per workflow
Context factorsConditions that change the right answer, with their levelsChannel (email, chat), policy version (2024, 2025), customer tier (two levels)No context field exists in the records
Industry label source and depthWho assigned each label and at which classification levelSupplier self-report mapped to NAICS two-digit sectorsLabels mix sector and six-digit levels
Minimum support per cellCases and distinct companies a cell needs to count as covered30 cases from at least three companiesA single case or company fills a cell
Selection rule per sourceHow each company's records were filtered before exportClosed cases from 2024 and 2025, all channelsOne source exported escalations only
Empty cells and reasonsEach empty cell marked impossible, absent by selection or not yet sourcedChat under the 2024 policy: chat launched in 2025, impossibleEmpty cells carry no reason

Could a large record count stand in for the test?

No, because volume and variety are separate properties, and large archives are where the two drift furthest apart. The strongest alternative explanation for an impressive-looking dataset is that its size hides repetition: one company's templates, one queue's habits or one policy period copied thousands of times.

Three effects drive it. Repeated cases fill a cell from a single source, so 5,000 cases from one company are weaker evidence of coverage than 60 cases from five, which is why the worksheet sets a company minimum. Selection bias leaves cells empty for reasons unrelated to the work, such as a source that exported only escalated cases. Unobserved failures hide in outcome codes nobody checked, which flatters the contrast check.

Breadth has to be allocated on purpose rather than hoped for from volume. Fix the cells first, set a floor in cases and companies for each, and fill toward the floors; surplus cases in a full cell add less than the first few in an empty one. Even a complete worksheet leaves the blind spots below:

Could a large record count stand in for the test?
Blind spotHow it fools the three checksNext evidence
Work no system recordedPhone fixes and hallway approvals never become cases, so no check can see themInterview process owners and compare case volume with staffing hours
Context missing from the exportWithout policy version or customer tier fields, crossings cannot be tested at allJoin policy change logs or account tiers by date or account key
Inferred outcomesOutcome codes set by a classifier inflate the contrast shareHand-check a random sample of outcome codes against later events
Wrong or coarse industry codesSelf-assigned or outdated labels make the collapse check merge or split wronglyMap every label to a registry code at one stated depth
Contexts that do not exist yetA rule or product change after the export date has no cellSchedule a refresh tied to the next policy or product change

Putting the three checks into a data request

Decide on the three numbers, not the label count: accept a dataset when its crossed and contrast shares clear the floors your task needs, narrow the claimed scope when only some workflows pass, and source more widely or walk away when the collapse check leaves one or two profiles. Write the floors down before the first sample arrives.

Training and evaluation deserve different floors. A training set can tolerate a few thin cells if the crossed share is high, because the model still sees context change the answer; an evaluation set should report each thin cell separately rather than fold it into an average, since a single pooled score hides exactly the cells the test was meant to probe.

A request built this way names the workflows, decision outcomes and context factors that matter, the minimum cases and companies per cell, and the acceptance test, such as a crossed share of at least two thirds across the workflows in scope. That turns a loose ask for diverse data into terms a supplier can answer cell by cell.

SourceX assembles each buyer's dataset to order from U.S. businesses rather than drawing on a catalog, and its request form asks what scope, volume, language, format, timing and licensing a buyer needs. Attach the worksheet to that request, and candidate records can be described against your cells, with no promise about what will be available or whether a given cell can be filled.

Frequently asked questions

Is industry count a good measure of dataset diversity?

Rarely on its own. The count depends on the classification depth chosen, since NAICS 2022 has 20 sectors but 1,012 six-digit codes, and the same workflows recur across industries. Count distinct workflow profiles, crossed contexts and two-sided decisions instead, and keep industry labels as background.

How is task coverage different from embedding-based diversity metrics?

Embedding and lexical metrics measure how varied the text is. Task coverage measures whether the cases vary in workflow, decision and context, which can stay flat while the wording varies a lot. Use both: text metrics catch templated records, and the three checks catch missing jobs and confounded contexts.

How many cases does a cell need before it counts as covered?

Set the floor from the task and state it in the worksheet in both cases and distinct companies. An illustrative floor of 30 cases from at least three companies stops one source from filling a cell alone. Evaluation cells usually need higher floors than training cells, because their results get reported.

What is a confounded business dataset?

One where two axes move together so closely that a model cannot separate them, such as every clinic case arriving by chat under one policy while every freight case arrives by email under another. A model trained on it can learn sector vocabulary instead of the decision rule, the shortcut learning that Geirhos and colleagues describe.

Can synthetic data fill empty cells?

Sometimes, for cells that are feasible but rare, provided the generated cases follow the business rules and are judged against real cases held out from generation. It cannot fill cells nobody has defined, and it can make a confounded dataset look crossed without teaching the real distinction. Keep generated cases as their own stratum in the worksheet.

Sources

  • Zhao et al. (ICML 2024 position paper) analyzed how diversity is described across 135 image and text datasets and recommend conceptualizing, operationalizing and evaluating it using measurement theory. Source
  • Geirhos et al. define shortcuts as decision rules that perform well on standard benchmarks but fail to transfer to more challenging testing conditions, such as real-world scenarios. Source
  • The U.S. Census Bureau explains that NAICS divides the economy into 20 sectors, with levels running from two-digit sector to six-digit national industry. Source
  • A 2024 Census Bureau working paper states that the 2022 NAICS vintage identifies 1,012 codes at the six-digit level (1,057 in 2017). Source
  • O*NET links each detailed work activity to one parent intermediate work activity and to multiple task statements across two or more occupations. Source
  • Google's Machine Learning Crash Course explains that training batches need enough examples of both classes; with 200 majority and 2 minority examples, most batches of 20 contain no minority example. Source

Tell us what data you need

Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.

See if your company qualifies