AI data market
Enterprise workflow rarity index: how to measure rare business decisions
By SourceX Editorial · Updated
10 sections
Short answer
An enterprise workflow rarity index scores how seldom complete examples of each business decision type, with inputs, options, choice and outcome, appear in a named public text corpus. This page publishes the protocol, not results: a frame built from O*NET work activities and NAICS industries, a four-element codebook, double coding, recall checks, calibration items and interval rules, plus a pre-registration checklist.
Retrieval
Enterprise workflow rarity
Key takeaways
- This is a protocol page; it reports no rarity scores and no results.
- The unit is a recurring decision type, drawn from O*NET work activities crossed with NAICS industries.
- A complete example needs inputs, options, the choice and a case-linked outcome; partial examples are coded too.
- Components are reported before any composite, each with a stated denominator and interval.
- Calibration items, recall checks and double coding keep the index from confusing missing queries with rare decisions.
- Rarity in public text is not value and not a right to license records.
What is an enterprise workflow rarity index?
An enterprise workflow rarity index is a measured score, per type of business decision, of how seldom complete examples of that decision appear in openly available text that AI models are commonly trained on. This page publishes the study protocol only. No results are reported here, and none should be inferred from the examples.
The question it answers is narrow on purpose: which business decisions are hardest to find, in usable form, in public training data? Usable form means the example shows the situation, the options, the choice and what happened next. A forum post that mentions credit holds is not a usable example of a credit-hold decision; a case record that shows the account state, the policy, the hold and the payment that followed is.
The index exists because claims about rare data are easy to make and hard to check. A protocol that fixes the population, the corpus, the coding rules and the uncertainty in advance lets anyone rerun the study, dispute a score or extend it to a new industry.
Which business decisions does the study cover?
The population is recurring decision types inside business workflows, not companies, documents or occupations. A decision type is a choice that a role makes repeatedly under a policy, such as approving a refund exception, routing a field-service job, or holding a shipment for a credit check.
The sampling frame is built from two public classifications so that anyone can rebuild it. The O*NET database lists task statements and detailed work activities for U.S. occupations and links each detailed work activity to the tasks that require it. NAICS, the standard federal statistical agencies use to classify business establishments by production process, supplies the industry axis. Crossing the two gives a frame of activity-by-industry cells; each cell is then turned into one or more concrete decision types by two analysts working from the task statements.
Inclusion rules: the decision recurs, has at least two real options, is recorded in a business system somewhere in the workflow, and its outcome is observable later. Exclusions: clinical judgments, legal advice to a client, and decisions made only in conversation with no system record. Excluded cells are listed, with the reason, so readers can see what the frame leaves out.
| Frame layer | Source | What it contributes | Decision the team records |
|---|---|---|---|
| Work activity | O*NET detailed work activities and task statements | What people in a role actually do | Which activities contain a recurring choice |
| Industry | NAICS sectors and industries | Where the activity happens | Which industry level to sample (sector or finer) |
| Decision type | Analyst drafting from task statements | A concrete, codable decision | Name, options, policy, outcome signal |
| System of record | Analyst judgment, checked by a second analyst | Where the decision would be logged | Whether a system record plausibly exists |
| Exclusion log | Protocol rules above | Transparency about gaps | Reason each excluded cell was dropped |
What counts as a complete decision record?
A text passage counts as a complete example only when a coder can find four elements in it: the inputs available at the time, the options considered or available, the choice made, and an outcome or consequence. Partial examples are still coded, because the pattern of which element is missing is one of the study's most useful outputs.
Coders work from a fixed codebook. Each element is scored present, partial or absent, with a short quote as evidence. Coders do not guess at missing elements and do not use outside knowledge of the industry to fill gaps.
| Element | Present means | Partial means | Absent means |
|---|---|---|---|
| Inputs | Specific facts the decider had, such as account status, amounts or dates | Generic description of the situation | No situation described |
| Options | At least two named alternatives | One option implied by context | Only the outcome is stated |
| Choice | The action taken and who took it | Action stated without the actor or timing | No action stated |
| Outcome | What happened after, tied to this case | A general claim about results | No consequence stated |
| Rationale (secondary) | A stated reason tied to a rule or tradeoff | A vague reason | No reason given |
How is rarity measured? The index design
Rarity is reported as separate components first and a composite second. Components are easier to argue with than a single number, and a composite hides which part of the record is scarce. The composite is an equal-weighted combination fixed before data collection, with alternative weightings shown as a sensitivity check.
The corpus is a named, dated snapshot of openly available web text. Common Crawl publishes free, open crawl archives that anyone can use, and documented corpora derived from it, such as C4, make the filtering steps visible. The protocol names the exact archive or corpus version and never claims that any particular model was trained on it.
| Component | Definition | Denominator | Reported with |
|---|---|---|---|
| Presence | Estimated count of passages that are real instances of the decision type | Passages in the named corpus snapshot | Count estimate and interval |
| Completeness | Share of true instances with all four elements present | True instances found in the coded sample | Proportion and interval |
| Outcome visibility | Share of true instances with a case-linked outcome | True instances found in the coded sample | Proportion and interval |
| Source concentration | Share of true instances coming from the most common few domains | True instances found in the coded sample | Proportion and domain list |
| Composite rarity | Prespecified combination of the four components, higher means rarer | Decision types in the frame | Rank with bootstrap interval |
How are passages sampled and coded?
Each decision type gets a written query set: a core phrase, synonyms used in the industry, and system-specific terms. Queries are frozen before retrieval. Retrieval runs over the named snapshot, and a simple random sample of hits is drawn for coding; the sample size per decision type is set from a precision target, not from convenience.
Recall matters as much as precision. Before the main run, the team builds a small gold set of known true instances for a subset of decision types, then measures what share of them the query set retrieves. That recall estimate is used to adjust presence counts and is reported next to every score.
Two trained coders label every sampled passage independently. Disagreements go to a third adjudicator, and agreement is reported per element using a chance-corrected statistic such as Cohen's kappa. Coders see the passage and the codebook, not the decision type's current rarity score.
- Step 1: freeze the frame, decision types and query sets, and record the corpus version.
- Step 2: build the recall gold set and measure query recall.
- Step 3: retrieve hits and draw a simple random sample per decision type.
- Step 4: double-code every sampled passage against the codebook.
- Step 5: adjudicate disagreements and compute agreement statistics.
- Step 6: estimate components with intervals, then the prespecified composite.
- Step 7: run the sensitivity checks named in the pre-registration and publish all tables.
Illustrative: how one passage is coded
This example is invented to show the codebook in use; it is not drawn from the study corpus and implies no result. Decision type: a distributor's credit team decides whether to release an order for a customer that is over its credit limit.
Passage A is a vendor blog post explaining that credit teams should review orders over the limit and consider payment history. A coder marks inputs as partial (a generic situation), options as partial (release or hold is implied), choice as absent and outcome as absent. It is a mention of the decision, not an example of it, and it counts toward presence only if the codebook's instance rule is met, which here it is not.
Passage B is a public case write-up in which a fictional account is past due, the order would exceed the limit, the analyst holds the order, the customer pays the open invoice later that week, and the order ships. A coder marks all four elements present and the rationale as present, because the write-up cites the policy threshold. This passage counts as a complete instance.
The contrast is the point of the index. A query for credit holds may return many passages like A and very few like B, and only passages like B help a model learn the decision itself. If two coders split on Passage A, for example one marking options as absent, the adjudicator applies the written rule and the disagreement is counted in the agreement statistic for that element.
How do calibration items and intervals keep a rarity score honest?
A rarity score means little without a comparison. The protocol includes calibration decision types expected to be common, such as a standard retail return, so readers can see whether the method separates common from scarce at all. If calibration items score as rare, the method or the query sets are broken, and that is reported rather than hidden.
Proportions are reported with intervals suited to small samples, such as Wilson score intervals, and the composite rank is reported with a bootstrap interval that resamples coded passages within each decision type. Ranks whose intervals overlap heavily are reported as ties, not as an ordering.
Three sources of error are tracked separately: sampling error from coding a sample instead of every hit, retrieval error from imperfect query recall, and coder error measured by agreement. A defensible score is one whose interval stays clear of the calibration items after all three are accounted for.
Sample size follows from the precision the team needs for the completeness proportion, which is usually the component with the fewest observations because only true instances enter its denominator. If retrieval for a decision type yields too few true instances to estimate completeness with the planned precision, the protocol reports presence alone and marks completeness as not estimable, rather than widening the coding rules after the fact.
What can a rarity score not tell you?
The design measures openly available text, so it cannot show what any specific model saw in training; most training sets are not fully disclosed. It also cannot show what exists in private or licensed collections. A decision type that scores as rare in public text may be common inside companies that never publish their records.
Rarity is not value. A decision can be scarce in public text because nobody needs examples of it. The index is an input to judgments about dataset demand, not a price signal, and it says nothing about who owns a record or whether it can be licensed.
The corpus choice also shapes the answer. Web text can over-represent industries that publish a lot about their own work online and under-represent industries whose work lives mostly in private systems; the study treats that skew as a hypothesis to check with the domain mix, not as a known fact. Documentation of C4 found text from unexpected sources, including patents and U.S. military websites, which is a reminder that any snapshot has its own quirks. The protocol therefore reports the domain mix of true instances for every decision type, so a reader can see when a score rests on a handful of sites.
- Not supported: a claim that a named model has never seen a decision type.
- Not supported: a dollar value or market size for any decision type.
- Not supported: conclusions about non-English text unless a non-English corpus and codebook are added.
- Supported: a reproducible, dated comparison of how often complete examples appear in a named public corpus.
Pre-registration checklist for a rarity index study
Register the plan before retrieval starts, on a public registry or a dated page that cannot be edited silently. The point of pre-registration is to separate the confirmatory analysis from exploration, so any change after data collection is reported as a deviation with its reason.
At a glance
Pre-registration checklist for a rarity index study
Disclosure follows survey research practice: state the dates of collection and of the content analyzed, the frame and where it came from, and what the frame does not cover. Publish the dataset documentation alongside, using a datasheet that covers motivation, composition, collection process and recommended uses.
Checklist · 10 items
- Research question and the decision a reader should be able to make from the answer.
- Frame sources with versions: O*NET release, NAICS edition, and the analyst rules for drafting decision types.
- Corpus name, snapshot identifier and access date.
- Codebook, coder training materials and the adjudication rule.
- Query sets per decision type, frozen and versioned.
- Sample size rule and the precision target behind it.
- Component formulas, composite weights and the sensitivity weightings.
- Calibration decision types and the expected direction of their scores.
- Interval methods for proportions, counts and ranks.
- Planned exclusions, deviation log and the date results will be published.
0 of 10 ticked. Tick each item as you go.
How a business can use this protocol, and where SourceX fits
A company does not need to run the full study to use it. The codebook works on internal records too: pull a sample of your own cases for one decision type and score whether each one carries inputs, options, choice and outcome. That tells you whether your records hold the complete examples that public text lacks, which is the practical question behind any licensing conversation.
If that sample turns up complete decision records, SourceX is one route to licensing them: it takes selected business records through rights review, contracting, preparation and delivery as datasets for AI. Its published guide is that buyers favor two or more years of consistent history and tens of thousands of records in one system, while rare decision types can qualify on smaller counts, judged deal by deal. Qualifying starts with a brief company assessment, and nothing has to be uploaded for it.
Frequently asked questions
Has this study published rarity scores yet?
No. This page is the protocol: population, corpus, codebook, measures and uncertainty methods. Results will only be published after data collection under the pre-registered plan, with every deviation disclosed.
Why measure public web text rather than model training sets?
Most training sets are not fully disclosed, so they cannot be sampled reproducibly. A named, dated public corpus can be. The protocol is explicit that this is a proxy and never claims to describe what a specific model saw.
What makes a decision example complete?
A coder must find the inputs available at the time, the options, the choice made and an outcome tied to that case. Examples missing any element are coded as partial, and the pattern of missing elements is reported.
Does a high rarity score mean a dataset is valuable?
Not by itself. Rarity is one input to judgments about demand. A decision can be rare in public text because nobody needs examples of it, and rarity says nothing about ownership or the right to license records.
Can a company apply the codebook to its own records?
Yes. Sample cases for one decision type from your system of record and score each one for inputs, options, choice and outcome. Do this on de-identified extracts or inside your own environment, and keep the results internal unless you have reviewed the rights to share them.
Sources
- Preregistration separates hypothesis-generating (exploratory) research from hypothesis-testing (confirmatory) research, and planning the analysis in advance improves quality and transparency. Source
- Datasheets for Datasets proposes that every dataset be documented with its motivation, composition, collection process and recommended uses. Source
- The O*NET database publishes task statements and detailed work activities for U.S. occupations, with a file linking each detailed work activity to the task statements that require it. Source
- NAICS is the standard U.S. federal statistical agencies use to classify business establishments into industries, grouped by similarity of production processes. Source
- Common Crawl publishes a free, open repository of web crawl data that anyone can use, released as dated crawl archives. Source
- Documenting the C4 corpus, built from a Common Crawl snapshot, found text from unexpected sources such as patents and U.S. military websites, plus machine-generated text and benchmark examples, and argued that large webtext corpora are under-documented. Source
- AAPOR's disclosure standards ask researchers to state dates of data collection (and of content analyzed), the sample frame and its supplier, and coverage gaps. Source
See if your company qualifies
A short company assessment. No data uploads are needed.