Skip to content

AI uses for records

Outcome label maturity experiment: how to measure if later labels help

By SourceX Editorial · Updated

10 sections

Short answer

This is a protocol, not a finding. To test whether later verified outcomes improve training labels, freeze one record set, label it twice (as of a short window after closure and after a long maturity window), train identical models on each at equal budget, and score both on a later holdout verified from independent evidence. Add drop-the-uncertain, matched-subset and placebo arms, and report paired differences with intervals. No results are reported here.

Grader rubric

Dataset outcome label

This page covers outcome label maturity experiment: how to measure if later labels help.Illustrative

Key takeaways

  • This page sets out a method for an outcome maturity comparison study and reports no results.
  • The only planned difference between the main arms is the as-of date of the labels; records, features, model and tuning budget stay fixed.
  • A case still open at the cut is censored, not a negative, and each arm must treat it the same written way.
  • Label the holdout from independent evidence, not from the matured field, or the matured arm wins by construction.
  • Matched-subset and placebo arms separate the effect of maturity from dropped hard cases and from random label changes.

How do you test whether later verified outcomes improve training labels?

Take one fixed set of records, give it two label snapshots, and train the same model on each. The early snapshot holds labels as they stood shortly after each case closed; the matured snapshot holds labels for the same cases after a verification window in which reversals, reopenings and corrections had time to appear. Everything else stays equal: records, features, model, tuning budget and random seeds.

Both models are then scored on an untouched holdout from a later period whose labels were verified independently, and the difference is reported with a paired interval. This page is a protocol for that comparison, an outcome maturity comparison study. It reports no results, because none have been collected for it.

The design matters because the obvious version of this test is easy to fool. If the matured labels also define the holdout, if the matured snapshot quietly drops hard cases, or if the second run gets more tuning, the matured arm can look better for reasons that have nothing to do with label quality. The sections below set out the arms, the controls that rule those explanations out, and the checklist to file before training starts.

Provisional label or matured label: what actually changes

A provisional label records what a system said about a case at a cut time; a matured label records what was true once the outcome had time to settle. In operational systems the gap is common. A Zendesk or Jira ticket marked solved can be reopened, an insurance claim approved at first review can be reversed on appeal, a Salesforce opportunity marked closed won can be canceled before delivery, and an invoice recorded as paid in NetSuite can be followed by a chargeback.

Advertising research has studied the same timing problem for years. Chapelle's KDD paper on delayed feedback starts from the observation that conversions can arrive long after the click, so an example that has not converted yet is not the same as a negative; the paper models the delay rather than treating every unconverted click as a failure.

Survival analysis gives the vocabulary. The lifelines documentation calls cases whose event has not been observed by the analysis date right-censored, and warns that simply dropping them biases the estimate. A case still open at the cut is censored, not failed. The maturity experiment has to decide, in writing, how censored cases appear in each arm, because that choice can produce a difference on its own.

The question and hypotheses to fix before any training run

Write the research question as a comparison with a named estimand: on records from the training window, does a model trained on labels matured for the long window score higher on independently verified holdout outcomes than the same model trained on labels as of the short window, at equal budget? The estimand is the difference in the primary metric between those two arms on the same holdout items.

Name one primary hypothesis and keep secondary ones visibly secondary. A reasonable primary hypothesis is directional: the matured arm scores higher on the primary metric. A secondary hypothesis can ask where the gain sits, for example on the outcome classes most often reversed. Decide also what result would count against maturity: an interval that excludes any gain larger than the smallest difference the team would act on.

  • Population: which systems, record types and business units the records come from, and which are excluded.
  • Short window and long window: the two cut points, measured from each case's closure date, with the reason for each.
  • Label definition: the exact field or rule that turns a record into a label, identical in both arms except for the as-of date.
  • Primary metric and the smallest difference worth acting on, both fixed before training.
  • Decision rule: what the team will do if the interval includes zero, excludes zero, or sits entirely below the threshold.

The outcome maturity comparison study: arms and what each rules out

The core comparison needs two arms, but two arms alone cannot say why a difference appears. The study below adds three controls. Each arm uses the same record IDs unless the row says otherwise, the same feature snapshot taken as of decision time, the same model family, and the same tuning budget counted in trials and compute hours.

  • Read A versus B as the headline estimate, and only alongside the controls.
  • If C matches B, waiting may not be worth it: removing unresolved cases does the work.
  • If D beats C, maturity helps beyond the exclusion effect.
  • If E moves the metric about as much as B, the gain comes from perturbation or regularization, not from the corrections themselves.
The outcome maturity comparison study: arms and what each rules out
ArmLabels usedRecordsWhat it isolates
A: provisionalLabels as of the short window after closureFull study setBaseline: what a dataset cut early would train on
B: maturedLabels as of the long window, after reversals and correctionsSame IDs as arm APrimary treatment: the effect of letting outcomes settle
C: drop the uncertainProvisional labels with every case still open or disputed at the short cut removedSubset of arm AWhether a simple exclusion gets the same gain without waiting
D: matured, matched subsetMatured labels on exactly the record IDs in arm CSame IDs as arm CSeparates the effect of maturity from the effect of losing hard cases
E: placebo flipsProvisional labels with the same number of labels changed as A to B, but on randomly chosen recordsSame IDs as arm AWhether any change of that size helps, or only the verified corrections do

Sampling, cut dates and a holdout that does not hand arm B the answer

Sample from a training window that ended long enough ago for every case to have passed the long maturity window; otherwise arm B silently inherits censored cases too. Stratify the sample by outcome class and by source system, record the denominator at every filtering step, and keep the record ID list frozen before any label snapshot is pulled.

The holdout must come from a later period than the training window, never from a random split of the same months. The scikit-learn cross-validation guide describes TimeSeriesSplit as a scheme in which the test fold always follows the training folds in time; the same principle applies to a single holdout. A later holdout catches drift, keeps reopened cases from leaking across the boundary, and mirrors how a model trained today will be used tomorrow.

Features need their own cut. Any field written after the decision point, such as a final resolution code, an appeal date or a refund line, can carry the outcome into the inputs. Build the feature table as of the moment the original decision was made and use that same table in every arm.

Do not label the holdout with the same matured-snapshot process that produces arm B's training labels. If you do, arm B learns the exact quirks the scorer rewards, and the comparison measures agreement with a process rather than closeness to the truth.

Instead, verify holdout outcomes from evidence outside the label field: a payment settlement record for a billing dispute, a returned-goods receipt for a refund, a second inspection for a field repair, or an expert review of the case file using a written rubric. Keep the evidence reference with each holdout label so a reviewer can audit it.

Report how many holdout items could not be verified and keep them out of the primary metric, but list them. A large unverifiable share is itself a finding about the dataset, and hiding it makes the comparison look more certain than it is.

Metrics, paired comparison and uncertainty

Score every arm on the same holdout items and compare arms pair by pair, because the item-level pairing removes a large source of noise. For a binary label, a two-by-two table of items each model got right or wrong supports McNemar's test, which statsmodels implements as a test of equal marginal distributions on paired data, with an exact binomial option for small disagreement counts.

At a glance

Metrics, paired comparison and uncertainty

Retrain each arm with several seeds and report the spread, so a single lucky initialization cannot carry the result. For proportions such as accuracy on a class, the NIST Engineering Statistics Handbook describes the Wilson score interval, which behaves better than the simple normal interval when counts are small or the proportion sits near zero or one.

Report the result as an effect with an interval, not a single score: the paired difference between arms, the interval, the denominators, and the per-class breakdown fixed in advance. Exploratory slices are welcome, but they go in a separate section labeled exploratory.

  • Primary: paired difference in the chosen metric between arms B and A on verified holdout items.
  • Secondary: the same difference for C versus D and for E versus A.
  • Calibration: whether predicted probabilities match observed rates in each arm, since corrected labels often shift class balance.
  • Stability: spread across seeds and across two or more holdout periods if the data allows.

Rival explanations for an apparent gain, and the control for each

The study is only as convincing as its answer to one question: what else could make the matured arm look better? Write each rival explanation into the plan with the control that addresses it, so a reader can see it was considered before the data was seen.

Rival explanations for an apparent gain, and the control for each
Rival explanationHow it would show upDesign control
Holdout labeled by the matured processArm B wins by agreeing with the scorer's methodVerify holdout from independent evidence, not the matured field
Hard cases dropped as they matureArm B has fewer ambiguous recordsFreeze record IDs; compare C and D on matched subsets
Class balance shiftGains vanish after reweightingReport class counts per arm; add a reweighted run as a secondary
Unequal tuning effortArm B had more trials or a later code versionFix trials, compute and code commit for all arms
Outcome fields leaking into featuresVery high scores in every armBuild features as of decision time; audit columns by write date
Label definition changed between cutsGain concentrated after a policy dateHold the label rule constant; log any code-set change
Random label changes act as regularizationPlacebo arm gains as much as arm BRun arm E with the same number of flipped labels

Pre-registration checklist for the maturity study

The Center for Open Science describes preregistration as specifying the design and analysis plan before observing research outcomes, in a timestamped record that cannot be altered without trace. For this study, file the items below before the first label snapshot is pulled, and record any later change with its date and reason.

At a glance

Pre-registration checklist for the maturity study

Checklist · 8 items

  1. Research question, estimand and primary hypothesis, with the smallest difference worth acting on.
  2. Population, source systems, training window, holdout period and exclusion rules, with expected denominators.
  3. Short and long maturity windows, the label rule, and how open or disputed cases are treated in each arm.
  4. Feature cut time and the list of fields excluded because they are written after the decision.
  5. Holdout verification method, the evidence each label must cite, and how unverifiable items are reported.
  6. Model family, tuning budget, seeds and the code commit to be used for all five arms.
  7. Primary metric, paired test, interval method and the per-class breakdown.
  8. What will be published regardless of outcome, including a null or negative result.

0 of 8 ticked. Tick each item as you go.

Illustrative design: a support desk where solved tickets reopen

Illustrative only; no data has been collected and no outcome is implied. Suppose a software company wants a model that predicts whether a first reply will resolve a support ticket. Arm A takes the solved or not-solved field as of 7 days after each ticket closed; arm B takes the same field as of 60 days, by which point reopened tickets have been reopened. Both windows are illustrative choices that the plan would justify from the desk's own reopen timing.

The study set would be tickets closed in one calendar year that ended at least 60 days before extraction, with the record ID list frozen first. The holdout would be tickets from the following quarter, each verified by a reviewer who reads the full thread and the later account history against a written rubric, not by reading the solved field.

Arm C drops tickets that had a pending customer reply at the 7-day cut, arm D applies the 60-day labels to that same subset, and arm E flips as many randomly chosen labels as changed between 7 and 60 days. The team would publish the paired differences, intervals and class counts for each comparison, whatever they show.

What the protocol cannot settle, and what data it needs

Even a clean result covers one population, one pair of windows and one model family. A gain on support tickets says little about claims or invoices, and a longer window can cost more in stale records than it returns in label accuracy. Treat the study as evidence about this dataset and this decision, and repeat it when the outcome definition, source export or model changes.

Running it also needs records that rarely leave a business: case histories with timestamps for every status change, not just the final value, and enough history that a long window has fully passed. Through SourceX, a business can license selected case histories for AI, with the rights review, contracting and preparation handled for it. A research team that needs status changes with timestamps across several years can say so in a data request, alongside volume, language, format, timing and licensing needs, and SourceX then sources the records from U.S. businesses for that request rather than from an existing collection.

Frequently asked questions

What should the outcome maturity comparison study include?

At minimum it needs a frozen record list, two label snapshots taken at stated windows, identical features and tuning for each arm, and a later holdout verified from independent evidence. Add the drop-the-uncertain, matched-subset and placebo arms so a difference can be attributed to maturity rather than to sample changes or random perturbation. File the plan before pulling labels.

Is a case that is still open at the cut a negative label?

No. A case whose outcome has not happened yet is censored, and treating it as a failure biases the labels toward the negative class. Either wait until the window has passed for every case in the training set, or record the open status explicitly and treat it the same way in every arm.

How do you choose the short and long maturity windows?

Base it on how long reversals actually take in the source system: measure the time from closure to reopening, appeal or reversal on past cases and choose a window that covers nearly all of them. A window much longer than that adds staleness without adding corrections. State the reason for the choice in the plan.

What sample, baseline and uncertainty make a result defensible?

The baseline is the provisional arm trained at the same budget on the same records. Defensibility comes from a later, independently verified holdout, paired comparisons with stated intervals, several seeds per arm, and denominators reported at every filtering step. Size the holdout so the interval is narrow enough to rule in or rule out the smallest difference you would act on.

Can the matured labels themselves serve as the test set?

Only if you accept that the test then measures agreement with the maturation process. That design favors the matured arm by construction. Verify the holdout from evidence outside the label field, such as settlement records, return receipts or a reviewer reading the full case file.

Sources

  • Chapelle's KDD paper on delayed feedback notes that conversions can arrive long after the click and models the delay instead of treating every unconverted example as a negative. Source
  • The lifelines documentation defines right-censored individuals as those whose event has not been observed and warns that ignoring them biases estimates. Source
  • The scikit-learn cross-validation guide describes TimeSeriesSplit, in which each test fold follows its training folds in time. Source
  • statsmodels implements McNemar's test for a 2x2 table of paired outcomes, with an exact binomial option. Source
  • The NIST Engineering Statistics Handbook describes the Wilson score interval for a proportion and its better behavior for small samples or proportions near 0 or 1. Source
  • The Center for Open Science describes preregistration as specifying designs and analysis plans before observing research outcomes, in a timestamped record. Source

Tell us what data you need

Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.

See if your company qualifies