Leadership and readiness
Operational data outcome completeness study: how to measure it
By SourceX Editorial · Updated
10 sections
Short answer
How to measure how often business records include verifiable outcomes: randomly sample closed cases from defined workflows at disclosed companies, code each on a five-level ladder from no outcome to program-verifiable, and report each level's share per workflow with denominators and intervals. This page is a study protocol with a pre-registration checklist. The study has not been run, and no results are reported.
Checklist
Operational data outcome
Key takeaways
- This page is a study protocol; the study has not been run and it reports no results.
- Outcomes are coded on a five-level ladder, from no outcome to program-verifiable, rather than as yes or no.
- Cases are sampled at random within each company and workflow after a finality window, so later reversals are seen.
- Every share is reported with its denominator and an interval that accounts for clustering by company.
- The design describes participating companies only and says nothing about rights to share or license records.
How could a study measure the share of business cases with verifiable outcomes?
Sample closed cases at random from defined workflows inside a disclosed set of companies, code each case on a fixed ladder from 'no outcome recorded' to 'outcome a program can verify', and report the share at each level per workflow with intervals. That is the whole design in one sentence. This page publishes it as a study protocol; the study has not been run and no results are reported here.
The question matters because training methods built on checkable answers depend on it. The Tulu 3 report from Ai2 describes reinforcement learning with verifiable rewards, in which a completion earns a reward only when it is verified as correct. Business records can play that role only when the case carries a result a program can check, and we are not aware of a published, careful measurement of how often that is true across ordinary workflows.
Opinions about this circulate freely, so the protocol is built to resist them. Definitions, sampling, coding rules and the analysis plan are fixed and registered before any case is read, and the report has to state its denominator for every number it gives.
What counts as a verifiable outcome? A five-level coding ladder
The outcome variable is ordinal, not yes or no. Each sampled case gets exactly one level, the highest it fully meets, and coders record the evidence that supports the level.
| Level | Definition | Example (illustrative) | Evidence the coder must cite |
|---|---|---|---|
| 0. No outcome | The case closed with no recorded result | A work order closed with blank resolution and no linked invoice | The empty or missing result fields |
| 1. Status only | A terminal status exists but says nothing about whether the goal was met | A ticket marked solved, which the requester can still reopen | The status field and its definition in the system |
| 2. Recorded result | A result field states what happened | An opportunity closed as lost with a reason code | The result field and its value |
| 3. Corroborated result | The result is confirmed by an independent record | A repair outcome confirmed by no return visit and a paid invoice | The corroborating record, its source and date |
| 4. Program-verifiable | A program can compare an answer to the recorded result deterministically, and the result is final after the workflow's window | An invoice coding checked against the posted ledger entry after period close | The check rule and the finality window |
Which companies, workflows and cases the study samples
The unit of analysis is the case: one ticket, work order, invoice, opportunity, purchase order or claim, inside one workflow, inside one company. The study population is operating companies above a stated size floor in stated countries and sectors, and the frame is disclosed with its supplier and its known gaps, as AAPOR's disclosure standard asks for any sample.
Workflows are fixed in advance and defined by their closing event, so that a support ticket, a field service job and an accounts payable invoice are each counted from comparable records. Within every company and workflow, cases are drawn by simple random sampling from all cases that closed inside a fixed period, after a finality window has passed, so that late reversals are visible.
Recruitment is the weak point. Companies that agree to let researchers read records are likely to keep better records than companies that decline, so the protocol records refusals and reasons and treats the result as describing participating companies only.
Case dispositions: keeping unreadable and restricted cases in the denominator
Not every sampled case can be coded, and the way those cases are handled can move a share more than any coding decision. A case whose attachments were purged, whose record is restricted by a legal hold, or whose system was replaced mid-period is still part of the sample, because dropping it quietly assumes it looks like the readable cases.
The protocol gives every sampled case one disposition before coding starts, keeps all of them in the reported denominator, and publishes the counts per company and workflow. Shares are reported two ways: over all sampled cases, with unreadable cases shown as their own category, and over codable cases only, labeled as such.
| Disposition | Meaning | In the denominator? | How it is reported |
|---|---|---|---|
| Coded | Records were readable and a ladder level was assigned | Yes | By ladder level |
| Unreadable | Records exist but key fields or linked records are missing or corrupted | Yes | As its own category, with the reason |
| Restricted | The company withheld the case, for example for a legal hold or confidentiality | Yes | As its own category; never imputed |
| Out of scope | On reading, the case does not meet the workflow's closing-event definition | No, but counted | Listed with the count, and the sampling frame is checked |
| Duplicate | The same case was drawn twice through a merged or migrated record | No, but counted | Listed with the count and the dedupe rule |
The cross-workflow completeness study protocol, element by element
The table is the protocol's core. Every element is set and registered before data collection begins; later changes are reported as deviations.
At a glance
The cross-workflow completeness study protocol, element by element
| Element | Specification | Recorded as | Reason |
|---|---|---|---|
| Research question | Among closed cases in each workflow at participating companies, what share reach each level of the ladder? | Registered text | Fixes the estimand before anyone looks |
| Population and frame | Named company list, size floor, countries, sectors, plus excluded groups | Frame name, supplier, date, coverage notes | Lets readers see who could not be selected |
| Workflows | A fixed list, each defined by its closing event and source system | Workflow code and closing-event definition | Makes levels comparable across companies |
| Case sampling | Simple random sample of closed cases per company-workflow cell from a stated period | Sampling seed, period, cell size | Gives every closed case a known chance of selection |
| Finality window | A per-workflow wait after close before sampling, such as a reopen or chargeback period | Window length and its source | Prevents counting results later reversed |
| Outcome level | Ladder level 0 to 4 with cited evidence | Level, evidence fields, coder IDs | The main variable |
| Covariates | Company size band, sector, system vendor, record age, workflow volume | Coded fields | Supports pre-registered comparisons only |
| Denominator | All sampled cases in the cell, including unreadable ones | Counts by disposition | Stops silent exclusion from inflating shares |
| Uncertainty | Interval method and how clustering by company is handled | Registered analysis plan | Every share is reported with an interval |
| Disclosure | Frame, dates, sizes, refusals, deviations and the coding manual | Published appendix | Allows outside checking |
How two coders classify each case, and how disagreements are settled
Two trained coders classify every sampled case independently from the records, using a written manual with examples of each level per workflow. Neither sees the other's codes until both are done.
Agreement is reported before adjudication. Because the ladder is ordinal, a weighted Cohen's kappa is a natural statistic; scikit-learn's implementation describes kappa as a score that expresses agreement between two annotators and supports linear or quadratic weights. Disagreements are settled by a third coder who reads both evidence notes, and the final level, not either original code, enters the analysis.
Status definitions need special care. Zendesk, for example, separates solved, which agents set and requesters can reopen by replying, from closed, which the system applies later. Dynamics 365 lets a user close an opportunity as won or lost and notes it can be reopened. Coders record which state a case was sampled in, and the finality window is set so a reopen would already be visible.
Interval, clustering and baseline rules a published share must meet
A result is defensible when every share comes with its denominator, an interval that reflects how the sample was drawn, and a clear statement of what it is being compared with. The protocol requires all three.
Within one company and workflow, the share at each level is a binomial proportion. Intervals such as Wilson or Clopper-Pearson, both available in statsmodels' proportion_confint function, behave better than the simple normal approximation when shares are near zero or one, which is likely for the top ladder level. Pooled estimates across companies must account for clustering, since cases from one company resemble each other; the registered plan names the method, for example company-level resampling, and reports the number of companies alongside the number of cases.
The baseline is internal, not a market figure: each workflow is compared with the others under the same definitions, and each company's share is shown next to the pooled one. Cell sizes are chosen in advance from the interval width the study needs to separate workflows, and the report says when a cell is too small to support a comparison.
Conclusions this design would not support
Saying in advance what a design cannot show protects the result from being stretched. This protocol supports statements about participating companies and the listed workflows during the sampled period, and nothing wider.
- It cannot estimate the share for all businesses, because participating companies are a self-selected group.
- It cannot say that verifiable cases are valuable to any particular AI developer; value depends on the task, volume and rights.
- It cannot say that recorded results are true; level 3 adds corroboration, but no level proves the record matches what physically happened.
- It cannot rank software vendors, because system choice is tangled with company size and sector.
- It says nothing about whether records may be shared or licensed; rights, privacy and contract review are separate questions.
- It reports no result on this page: the study has not been run.
Pre-registration checklist before the first company is enrolled
The Center for Open Science describes preregistration as separating hypothesis-generating, exploratory research from hypothesis-testing, confirmatory research. For this study that means the items below are written and time-stamped in a public registry before the first company is enrolled, and anything decided later is labeled exploratory.
At a glance
Pre-registration checklist before the first company is enrolled
Checklist · 12 items
- Research question and the exact estimand per workflow.
- Frame, inclusion and exclusion rules, and the recruitment script.
- Workflow list with closing-event definitions and finality windows.
- Ladder definitions and the coding manual, with worked examples per level.
- Sampling period, cell sizes and the random seed procedure.
- Treatment of unreadable, missing or restricted cases in the denominator.
- Agreement statistic, adjudication rule and coder training plan.
- Interval method and the approach to clustering by company.
- Pre-specified comparisons, and a statement that all others are exploratory.
- Privacy handling: records are read where they are held and only codes leave the company.
- Disclosure items to publish, following the AAPOR standard.
- How deviations from the plan will be reported.
0 of 12 ticked. Tick each item as you go.
Should you apply the ladder to your own records first?
A company does not need to wait for the study to use the ladder. Draw a random sample of recently closed cases from one workflow, code each on the five levels with the evidence noted, and the distribution shows which records could grade an AI system and which only describe activity.
Keep the internal version small and honest. Pick one workflow with a clear closing event, wait out its reversal window, sample at random rather than choosing familiar cases, and have two people code the same cases so disagreements expose fuzzy definitions. Record the dispositions as well as the levels; a high share of unreadable cases is itself a finding about the archive. The result describes that workflow in that period, and nothing more.
That self-audit is also a sensible step before a licensing conversation. For companies that later license selected records directly to SourceX, SourceX starts with a brief assessment of fit that involves no data upload, and its process has the company approve scope and terms and sign before any delivery.
Frequently asked questions
Has this outcome completeness study been run?
No. This page publishes the protocol only and reports no results. Any figure about how often business records include verifiable outcomes should come with its population, sampling, dates and denominator; without them it is an opinion.
What is a verifiable outcome in a business record?
A recorded result that a program can check an answer against deterministically, and that is final once the workflow's reversal window has passed. A posted ledger entry after period close qualifies; a ticket marked solved, which can still be reopened, does not on its own.
Why code outcomes on a ladder instead of yes or no?
Because the useful distinction is graded. A status, a recorded result, a corroborated result and a program-checkable result support different uses, from analytics to evaluation to reward signals, and a yes or no code would hide which one a workflow reaches.
Why is the study limited to participating companies?
Reading case records requires a company's consent, and companies that agree probably keep better records than those that decline. The protocol records refusals and reports results only for participants rather than claiming to describe all businesses.
How large should the sample be?
Large enough per company and workflow to separate workflows at the interval width the study pre-registers. Cell sizes are set from that precision target before data collection, and the report flags any cell too small to compare.
Sources
- The Tulu 3 report describes reinforcement learning with verifiable rewards, in which a model receives a reward only when its completion is verified as correct. Source
- AAPOR's disclosure standard asks researchers to state whether the sample is probability-based, describe the frame, its supplier and coverage, and disclose collection dates and sample sizes. Source
- statsmodels' proportion_confint computes binomial proportion intervals by several methods, including Wilson score and Clopper-Pearson. Source
- scikit-learn's cohen_kappa_score computes Cohen's kappa for agreement between two annotators, with optional linear or quadratic weights. Source
- The Center for Open Science describes preregistration as separating exploratory, hypothesis-generating research from confirmatory, hypothesis-testing research. Source
- Zendesk agents set tickets to solved, requesters can reopen a solved ticket by replying, and closed status is applied by the system rather than by agents. Source
- Dynamics 365 Sales lets users close an opportunity as won or lost and reopen it later. Source
See if your company qualifies
A short company assessment. No data uploads are needed.