AI uses for records
Delayed outcome dataset labels: late failures are a different kind
By SourceX Editorial · Updated
10 sections
Short answer
Late-arriving outcomes need separate handling because they differ in kind, not only in timing. In SourceX's analysis of 402 reverted changes in the Go repository, each followed for two years, 26% of reverts landed after day 7, and about 9 in 10 of those late reverts cited no test, builder or failure. Freeze labels early and a dataset learns mainly the failures automated checks catch.
Grader rubric
Delayed outcome dataset
Key takeaways
- In Go's history, 54.5% of eventual reverts landed within a day of the change, 73.6% within a week and 95.3% within 90 days.
- Reverts whose stated reason named a builder, a test or a failure were 92.8% complete by day 7; every other revert was 63.5% complete.
- A short window shifts the failure mix toward what automated checks detect, so it changes what a model learns, not only how many failures it sees.
- Give each detection channel its own observation window, chosen from a cohort with full follow-up, and keep cases in an open window out of both pass and fail.
- Each delivery should state its freeze date and the share of eventual outcomes still expected, so a refresh can tell late arrivals from rule changes.
Is a late-arriving outcome just a slow label?
No. A late-arriving outcome is a result recorded after a dataset froze its labels, and the cases that produce one tend to fail for different reasons than cases that fail fast. Waiting longer recovers the count; putting every outcome on one clock still distorts which kinds of failure the dataset contains.
Every delayed label has four parts: the event that starts its clock, the success or failure event it waits for, the observation window, and the freeze date on which the dataset was cut. A support reply starts a reopen clock, a shipped change starts a rollback clock, a hospital discharge starts a readmission clock. A label is mature only when its window closed before the freeze date.
How to label cases still inside an open window, as neither pass nor fail, is covered in the guide to unresolved case censoring linked below. This page takes the next step. Once open cases are kept out of the failure column, how long should the window be, and can one window serve every kind of outcome? Public records answer that better than intuition.
How long did reverted Go changes take to fail?
Most failed within a day, but a long tail took months. In the Go project's history, 54.5% of reverts landed within one day of the change they undid, 73.6% within 7 days, 85.8% within 30 days and 95.3% within 90 days. These are SourceX's counts of 402 reverts of changes merged between December 2014 and October 2024, each with at least two years of follow-up.
Go makes a good public test bed because a merged change has already passed an approval gate. Its contribution guide says maintainers trigger trybots that run the full test suite on several architectures, and a change needs a Code-Review +2 from a maintainer before it can be submitted. Under that rule, approved and merged is a provisional outcome, and a revert overturns a change that cleared review.
Reverts are also unusually well documented. Git's reference for git revert says the command records new commits that reverse the effect of earlier ones, starts the message body by naming the reverted commit, and strongly recommends explaining why. Go's reverts follow that pattern, many with a Reason for revert line, which makes the cause of a late failure countable.
Measured from each original change's merge time, the share of eventual reverts already visible at each window:
| Window after merge | Reverts visible | Share of eventual reverts | Still to come |
|---|---|---|---|
| 1 day | 219 of 402 | 54.5% | 45.5% |
| 7 days | 296 of 402 | 73.6% | 26.4% |
| 30 days | 345 of 402 | 85.8% | 14.2% |
| 90 days | 383 of 402 | 95.3% | 4.7% |
| 180 days | 401 of 402 | 99.8% | 0.2% |
Which failures arrive late, and why does the mix matter?
The late ones are mostly failures that no automated check reported. Reverts whose stated reason mentioned a builder, a test or a failure were 92.8% complete by day 7 and 97.8% by day 30. Every other revert, whether its reason described something else or gave none, was 63.5% complete by day 7 and 79.5% by day 30.
The 106 reverts that arrived after day 7 were therefore not a random slice. Only 10 of them, about 9%, cited a builder, test or failure, against 129 of the 296 seen by day 7, about 44%. The late reasons read differently. One change was reverted after 146 days because it "causes long compile times on certain functions"; another after 175 days because it "can cause child exit_group to hang"; a new archive interface was withdrawn after 171 days when the proposal behind it was un-accepted.
For a dataset, that changes the lesson as well as the size. Freeze labels at day 7 and the failure examples lean toward problems a test run surfaces within hours, while slower failures, found by users, downstream projects or later analysis, are thin in training data and absent from evaluation. A model or agent scored on such a set can look safe on exactly the failures that are most expensive to find.
Expect the same pattern in business records until a measurement says otherwise. A refund a payment system rejects fails at once; a refund that ends in a chargeback, a complaint or a lost account fails weeks later and for other reasons.
Where business rules already set the observation window
Many business outcomes carry their window inside the rule that defines them, and a dataset should copy that window rather than invent one. CMS's Hospital Readmissions Reduction Program measures unplanned readmissions within 30 days of discharge, so no discharge can be labeled not readmitted until 30 days have passed.
Rules can add reporting lag on top of the window. CMS released its fiscal year 2027 hospital-specific reports on September 1, 2026, with payment reductions effective October 1, 2026, based on a performance period of July 1, 2023 to June 30, 2025. The official outcome for a June 2025 discharge therefore arrived more than a year after the patient went home.
Credit labels add a second clock for recovery. The Basel Committee's 2017 guidelines on problem assets treat an exposure as non-performing when it is more than 90 days past due or the borrower is unlikely to pay, and let it return to performing only after on-time repayment over a continuous period of at least three months. A loan current today cannot yet be called healthy, and a defaulted loan that resumed payment cannot yet be called cured.
Where no rule sets the window, measure it the way the Go count does: take a cohort old enough for its outcomes to have arrived in full, and read how long they took.
One window per detection channel, set by a capture target
Give each way a failure can be detected its own label track and its own window, instead of one status read on one clock. The detection channel predicts arrival time better than the outcome name does: in Go, test-detected failures settled within days and other failures took months, though both ended in the same kind of revert.
At a glance
One window per detection channel, set by a capture target
Set each window as the shortest one that captures a stated share of eventual outcomes on its track, and publish that share. The capture target is a business choice; the window that meets it is a measurement. Reading Go's curves at a 90% target, the fast track needs 7 days (92.8%) and the slow track 90 days (92.8%), while a single 7-day window would catch only 63.5% of slow-track failures. At a 95% target the fast track needs 30 days and the slow track 180.
The price is freshness. A record younger than its slow-track window can carry only fast-track labels at delivery, so with a 90-day slow window the newest quarter arrives half-labeled. That is worth paying when the slow failures are the ones a buyer cares about, and wasted when the task only concerns failures an automated check would catch.
Check that the curve holds before relying on it. Splitting Go's cohort by merge year, 74.0% of reverts arrived within 7 days for changes merged in 2014-2019 and 73.3% for 2020-2024, and the 90-day share stayed near 95%, but the 30-day share moved from 81.4% to 89.3%. A curve that shifts that much at one point should be re-estimated for each period a dataset spans.
The tracks, and the rule each follows:
- Fast track: outcomes raised by automated checks, validators or system rules. Short window from its own curve; in Go, 7 days captured 92.8%.
- Slow track: outcomes raised by people or downstream systems, such as customer reopenings, chargebacks, readmissions or user bug reports. Window from its own curve; in Go, 90 days captured 92.8%.
- Reversal ledger: events that overturn an earlier outcome, kept apart from both tracks, because a failure label can itself be undone; eight Go reverts were later reverted.
- Status on every track: open, observed failure or observed clean, with the date its window closes. A case open on a track counts in no rate that uses that track.
The time-aware data card for delayed outcomes
Add these fields to the dataset card and to each record, so a buyer can see which labels are final and which are still arriving. The example values come from the Go analysis on this page.
| Field | What it records | Example (Go reverts) | Check before use |
|---|---|---|---|
| Outcome event | The success or failure event and the system that records it | A revert commit naming the change it undoes | Every label points to a dated event or to an open window |
| Clock start | The event that opens each case's window | Merge time of the original change | One clock and one time zone for every case |
| Detection channel | Who or what raised the outcome | Whether the stated reason names a builder, test or failure | The rule that assigns a channel is written down |
| Window per track | Observation window and the capture it achieves | 7 days fast and 90 days slow, each capturing 92.8% | Capture measured on a cohort with full follow-up |
| Maturation evidence | Cohort, follow-up and dates behind each window | 402 reverts of changes merged December 2014 to October 2024, two years of follow-up | Re-estimated when process or tooling changes |
| Freeze date | When labels were cut | October 27, 2026, at Go commit 6c49ac3 | Stated on the card and in each record |
| Open share at freeze | Cases still inside a window, per track | Changes merged after July 29, 2026 still open on the slow track | Open cases left out of every rate on that track |
| Expected late arrivals | Share of eventual outcomes not yet visible at delivery | 14.2% of reverts arrive after day 30 | Used as the revision budget for the next refresh |
Change-impact checklist when a refresh rewrites old labels
A refresh of a delayed-outcome dataset will flip some old labels, and this checklist separates the flips the maturation curve predicted from flips that signal a changed rule or pipeline. Run it before refreshed labels reach training or evaluation.
Checklist · 7 items
- Compute expected late arrivals for each delivered cohort from its age at the last freeze and its track's curve; on Go's curve, a cohort frozen at day 30 still had 14.2% of its eventual reverts to come.
- Compare observed flips with that budget, track by track. Far more flips than expected points to a rule, definition or extraction change; far fewer suggests outcome events have stopped being captured.
- Look for dated changes to the detection machinery. Go's commit messages began carrying a LUCI-TryBot-Result trailer in August 2023, a sign of new test infrastructure that could move the fast-track curve.
- Confirm that no case went from open to clean only because a field was blanked or the system that records outcomes was retired.
- Store the old and new label side by side with both freeze dates instead of overwriting.
- Re-score evaluations built on the earlier freeze only on cases whose windows closed in both versions, so scores stay comparable.
- Log the refresh on the data card: date, flips per track, expected against observed, and any window that was re-estimated.
0 of 7 ticked. Tick each item as you go.
Could newer records just be noisier rather than late?
Sometimes, and the way to tell is to compare cohorts at equal age before blaming the clock. Newer records can differ because the process changed, because the case mix changed, or because their outcomes have not arrived; only the last is a maturity problem, and only it shrinks as records age.
Three checks separate the explanations. Compare a recent cohort's outcome rate at age t with older cohorts at the same age t, not with their final rates; a match means the gap is maturity. Look for dated process changes, such as the new trybot trailer that appears in Go's history from August 2023, and estimate curves on each side. Then check whether the mix moved, since a quarter heavy with complex cases fails later and more often whatever the window.
If a recent cohort differs from older ones even at equal age, the records describe a changed task and waiting will not fix it. Treat that as distribution shift, and decide separately whether those records belong in the dataset.
Putting maturation requirements into a data request
Ask for outcome events with their own timestamps and full follow-up history, not a status field read on extraction day. Name each outcome type, the event that starts its clock, the channels that can raise it, the capture target per track and the freeze date, and add an acceptance test a reviewer can run: no case labeled clean while one of its windows was open at the freeze, and maturation evidence attached to every window.
SourceX looks for matching records among U.S. businesses once a buyer has described the task, so clock and window rules can shape what gets extracted instead of being rebuilt afterward. The request form collects scope, volume, language, format, timing and licensing needs; the outcome definitions and acceptance test belong in that description. Whether a given company's systems kept dated outcome events, rather than overwriting a status, is established when its records are reviewed.
Methods: repository, commit, definitions and limits
Data: the commit history of github.com/golang/go reachable from master commit 6c49ac311e5642fa5399b46c736295a9dc4eeec9, read on October 27, 2026 from a commit-only clone. A revert is a commit whose subject begins with Revert, or with a package name followed by revert, and whose message says This reverts commit or This reverts CL; CL numbers were matched to commits through their Reviewed-on lines. Lag runs from the original's committer time to the revert's. The cohort holds first reverts, not reverts of reverts, of changes merged on or before October 27, 2024, counting reverts that landed within 730 days: 402. Eight revert messages whose original was not on master and one negative lag were excluded.
Reason classes come from a keyword rule on the Reason for revert paragraph: builder, trybot, longtest, test or fail, in any case. Of the 402, 139 matched, 133 gave another reason and 130 gave none. A keyword rule will misfile some reverts, so the split is a strong signal, not an exact count. The analysis script is kept with this article's working files for reviewers.
Not measured: reverts written without a recognizable message, failures fixed forward instead of reverted, and any business dataset. Go is one well-run public project chosen because its outcomes are dated and explained; its curves are evidence that late outcomes can differ in kind, not a window to borrow for other records. SourceX has published no maturation results on licensed data.
Frequently asked questions
How long should you wait before labeling a delayed outcome?
Long enough to capture a stated share of eventual outcomes on each detection track, measured on a cohort whose outcomes have fully arrived. In Go's history, 7 days captured 92.8% of reverts that cited a test, builder or failure but only 63.5% of the rest, which needed 90 days to reach the same share. Publish the window together with the capture it achieves.
Should unresolved cases be labeled as failures?
No, unless the task defines failure by a deadline that has already passed. A case inside an open window has no outcome yet: counting it as a failure inflates failure rates for recent records, and counting it as a success hides late failures. Keep it out of every rate that depends on that window.
What is a maturation curve for dataset labels?
It is the cumulative share of eventual outcomes that have appeared by each age, measured on cases followed long enough for nearly all outcomes to arrive. It shows how many outcomes a given window will miss and whether different kinds of outcome arrive at different speeds. Re-estimate it when the process or tooling that detects outcomes changes.
Do late outcomes matter if they are a small share?
They matter when they differ in kind. In the Go count, reverts after day 7 were 26% of the total, yet about 9 in 10 of them cited no test, builder or failure, against about 56% of the reverts seen earlier. A short window changes which failures a model learns from, not only how many.
How should a dataset refresh handle labels that changed?
Keep both versions with their freeze dates, compare the number of flipped labels with what the maturation curve predicted, and investigate a gap in either direction. Compare scores from evaluation sets built on different freezes only on cases whose windows closed in both versions.
Sources
- The Go repository's commit history, from which SourceX counted 402 reverts with two years of follow-up and their stated reasons. Source
- Go's contribution guide: maintainers trigger trybots that run the full test suite on several architectures, and a change needs a Code-Review +2 from a maintainer to be submitted. Source
- Git's git revert reference: the command records new commits that reverse earlier ones, names the reverted commit in the message and strongly recommends explaining why. Source
- CMS's Hospital Readmissions Reduction Program measures unplanned readmissions within 30 days of discharge. Source
- CMS's FY 2027 hospital-specific reports, released September 1, 2026, set payment reductions effective October 1, 2026 from a July 1, 2023 to June 30, 2025 performance period. Source
- The Basel Committee's 2017 problem-asset guidelines treat exposures more than 90 days past due as non-performing and require at least three months of on-time repayment before a return to performing. Source
Tell us what data you need
Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.