Skip to content

AI uses for records

Dataset utility decay refresh decision: run a knee-and-ceiling test

By SourceX Editorial · Updated

10 sections

Short answer

Settle the dataset utility decay refresh decision by replaying your task on dated history: score a model frozen on an old window against one retrained on the latest window, at every step of staleness. The curve has a knee, where loss stops growing, and a ceiling, the most a refresh can recover. Replaying 5,307 Kubernetes release notes, SourceX found a 4-point accuracy gap within four releases and small, uneven gains after that.

Agent run

Dataset utility decay

This page covers dataset utility decay refresh decision: run a knee-and-ceiling test.Illustrative

Key takeaways

  • Automatic refreshes pay only when the utility lost between deliveries is worth more than a delivery costs, and a decay curve measures that loss directly.
  • In SourceX's replay of Kubernetes release notes 1.19 to 1.37, a frozen classifier trailed a refreshed one by 2.3 points at two releases stale and 4.1 points at four; beyond that the gap moved unevenly between 3.2 and 5.4 points.
  • The ceiling caps what any refresh plan can buy back; the knee shows how quickly it is lost, and the two together price every cadence.
  • Adding new records to the full history beat training on the newest window alone, 60.4% against 58.2% mean accuracy, so refreshes should be priced as additions.
  • Check balanced accuracy and class shares before calling a gap decay: the bug share in the notes moved from 45% to 20% and back to 40%.

What should decide whether you buy automatic refreshes?

Buy automatic refreshes only when a frozen copy of your dataset loses more task performance between refreshes than each refresh costs to license, check and retrain on. That loss is utility decay: the drop in a model's score on current work as its training records age, measured on your task and metric rather than guessed from how old the records look.

The decision has three possible outcomes. A fixed cadence fits when decay is steady and large. A refresh triggered by a known business change fits when decay is small most of the time and jumps after events. A single purchase with no refresh plan fits when the curve stays flat across your planning horizon.

The evidence behind any of the three is a decay curve: performance plotted against staleness, measured on dated history you hold or can license, read against a minimum gain below which a refresh is not worth paying for. Fix that minimum before anyone looks at results, or the curve will be read to fit the contract already on the table.

Why a single before-and-after comparison cannot set a cadence

A single comparison tells you what one refresh added; a cadence depends on how fast the loss grows with time, which takes comparisons at many gaps. The existing SourceX protocol for measuring what one refresh contributed answers the first question. This page is about the second: how stale a dataset can get before the shortfall matters.

Research on temporal misalignment treats decay as a rate. Luu and colleagues (NAACL 2022, arXiv 2111.07408) built eight tasks spanning five years or more, defined a temporal degradation score as the average drop in performance per step of gap between training and test periods, and found that finetuning on data from the target period helped far more than continued pretraining. Vela and colleagues (Scientific Reports, July 2022) studied 32 datasets from four industries with four standard model types and reported that model quality can degrade over time even when the data drifts only slightly.

The working method comes from forecasting. Hyndman and Athanasopoulos describe evaluation on a rolling forecasting origin: train up to a point, test on what follows, move the origin forward and average over the test sets (Forecasting: Principles and Practice). A decay replay changes one thing. It holds the training window still while the test period moves away from it, so the gap itself becomes the variable being measured.

One measured decay curve: 5,307 Kubernetes release notes, 1.19 to 1.37

To show what a measured curve looks like, SourceX replayed a ticket-triage-style task on public records: sorting Kubernetes release notes into the four main kinds the project files them under, Bug or Regression, Feature, API Change, and Other (Cleanup or Flake). The notes come from the changelog of every minor release from 1.19 to 1.37 in the kubernetes/kubernetes repository at commit dd410c4 (read October 27, 2026). The project's contributor guide requires a release note for pull requests with user-visible changes, so the population is user-facing change, not all engineering work.

A deliberately simple classifier, naive Bayes on words and word pairs, was trained on three consecutive releases, subsampled to 600 notes and rerun under 20 fixed seeds, then scored on each later release. For every test release, the frozen model was compared on the same notes with a model trained on the three releases immediately before it. SIG Release's KEP-2572 moved the project to a planned cadence of three releases a year starting with 1.22, so one step of staleness is roughly four months.

Because each row averages a different set of test releases, compare the paired gains, not the raw accuracies, across rows. Each figure is the mean of the 20 seeds, with the lowest and highest seed in brackets, and the interval is a 95% bootstrap on the seed-averaged gain. Frozen-model accuracy by staleness, and the paired gain a refresh would have bought back, came out as follows:

One measured decay curve: 5,307 Kubernetes release notes, 1.19 to 1.37
Releases staleTest releasesFrozen accuracy, mean (seed range)Refresh gain, points, mean (seed range)95% interval of the mean gain
1 (refreshed baseline)1657.2% (56.5 to 58.1)0, referenceReference
21555.1% (54.1 to 56.2)+2.3 (+1.5 to +4.1)+1.5 to +3.2
31454.3% (53.5 to 54.9)+2.8 (+1.8 to +3.8)+1.8 to +3.9
41353.2% (52.4 to 54.0)+4.1 (+2.8 to +5.4)+2.8 to +5.4
51253.2% (52.2 to 53.9)+3.9 (+2.4 to +5.2)+2.4 to +5.3
61153.9% (53.0 to 54.6)+3.2 (+1.4 to +5.1)+1.8 to +4.7
71052.1% (51.1 to 53.2)+5.4 (+3.4 to +6.9)+3.9 to +7.0
8952.7% (51.8 to 53.8)+5.0 (+3.8 to +6.1)+3.2 to +6.7
Line chart of the paired accuracy gain a refresh would have bought back at two to eight releases of staleness, rising from 2.3 points at two releases to 4.1 at four and then moving unevenly between 3.2 and 5.4, beside bars pricing four refresh cadences at 0, 1.7, 2.7 and 4.3 points lost.
Utility decay in SourceX's replay of 5,307 Kubernetes release notes (1.19 to 1.37, commit dd410c4); mean of 20 seeds, shaded band is the 95% bootstrap interval. One public task, not a forecast for business data.

Knee, ceiling and the price of each refresh cadence

Two numbers on the curve carry the decision. The knee is where the gap stops growing quickly: in this replay, about four releases, or roughly sixteen months at the planned cadence, when the gap reached 4.1 points, about three-quarters of the largest gap measured. The ceiling is the level the gap settles at, and here it did not settle cleanly: from five to eight releases stale the gap moved between 3.2 and 5.4 points, and the seven-release gap sat 1.3 points above the four-release gap on average, from 0.1 to 2.4 across seeds, though the two rows average different test releases. Read the ceiling as soft, roughly 4 to 5 points: most of the gain arrives by about four releases, and later gains are small and uneven. No refresh program, however frequent, can recover more than the ceiling.

Averaging the gap across one refresh cycle prices each cadence in lost points. These figures are averages of the paired gains in the previous table, not separate experiments, and they apply to this task only.

A second result matters for contract terms. Training on every release before the test release, 877 to 4,939 notes, averaged 60.4% accuracy over the 16 test releases, against 58.2% for the newest three releases alone at full size. Older notes still carried signal, so a refresh that replaced history would have discarded some of it. Price refreshes as additions to what you keep, and ask whether a delivery adds, revises or replaces.

Read together, the replay says an annual refresh would have bought back about 2.6 points over never refreshing, and a release-by-release refresh about 1.7 more. Whether either is worth paying for depends on what a point of triage accuracy is worth to the buyer, which is why the minimum gain comes first. Average loss by cadence, against a model refreshed every release:

Knee, ceiling and the price of each refresh cadence
Refresh cadenceRelease steps per cycleAverage points lost, mean (seed range)Reading
Every release10Pays only if a 1.7-point edge over yearly is worth its cost
About yearly31.7 (1.1 to 2.6)About 60% of the never-refresh loss avoided at a third of the deliveries
About every two years62.7 (1.7 to 3.8)Avoids only about a third of the never-refresh loss
Never, after the knee4 or more4.3 (2.9 to 5.5)The soft ceiling: still edging up at seven and eight releases

Decay, or did the labels and the class mix move instead?

Test the strongest alternative before accepting a curve: newer records can carry a different task, a different labeling habit or plain noise, and each can pass for decay. The Kubernetes notes show why. The share filed as Bug or Regression was 45% in 1.19, 20% in 1.34 and 40% in 1.37, a swing in class mix large enough to move accuracy on its own.

Three checks separate the explanations here. Balanced accuracy, which weights each kind equally, fell from 50.4% at one release stale to between 46.0% and 48.3% beyond it, so the gap is not only a prevalence effect. Subsampling every window to 600 notes removes window size as a cause. And single releases are noisy: for 1.27, the model frozen on 1.19 to 1.21 scored 4.2 points above the refreshed one on average, with single seeds ranging from 1.9 points below it to 8.9 above, so the pooled curve, not any one release, carries the evidence.

Labeling drift is the hardest to exclude. The project's release-note guide asks writers to say whether a change added, changed, fixed, removed or deprecated something, and writing habits can shift between releases. A buyer's replay should keep a sample labeled under one guide across all periods; if the gap shrinks on that sample, part of the decay was labeling change, and a refresh would be buying relabeled history rather than new knowledge.

Running the same replay on records you might license

The replay needs dated records, a fixed task and a held budget, and it can run on a supplier's sample or your own archive before any subscription is signed. The steps, in order:

  • Fix a decision time for each record, the moment the task was decided rather than the export date, and cut periods on it.
  • Match the window length to what one refresh would deliver, and the period length to the cadence you are pricing.
  • Hold the training budget constant: subsample every window to the same count and average several seeds.
  • Score frozen and refreshed models on identical test records and bootstrap the paired difference.
  • Convert the refresh's full cost into the smallest gain worth buying, and write it down before the first run.
  • Report the knee, the ceiling and the average loss for each candidate cadence, with intervals.
  • Repeat with balanced accuracy and with a sample labeled under one guide, to split decay from mix and labeling change.
  • Keep one frozen holdout outside every window so later deliveries are judged against the same yardstick.

A time-aware data card that makes decay measurable

A dataset can be replayed only if its documentation says when each record belongs and which rules shaped it. Data Cards, proposed by Pushkarna, Zaldivar and Kjartansson at FAccT 2022, are structured summaries of dataset facts for the people who use them. For refresh decisions the card needs time fields that most summaries leave out:

A time-aware data card that makes decay measurable
FieldWhat it recordsWhy the decay replay needs it
Decision time keyWhich timestamp places a record in a period, and its time zonePeriods cut on export or edit dates blend eras
Period counts and class sharesRecords and label shares per period, with denominatorsSeparates decay from a shifting class mix
Source scope versionSystems, business units and filters in scope, each datedA widened scope can pass for new knowledge
Rule change logDated changes to business rules that decide outcomesExplains steps the curve shows
Label guide versionWhich labeling instructions applied to which recordsIsolates labeling drift from decay
Frozen holdoutRecords withheld from every delivery, with the freeze dateKeeps refresh comparisons on one yardstick
Delivery relationWhether a delivery adds, revises or replaces earlier recordsAdditions and replacements are worth different amounts

Change-impact checklist before a refresh enters training

Every delivery should clear these checks before it reaches a model, so a refresh earns its place on evidence rather than on its date:

Checklist · 7 items

  1. Did the source scope, filters or business units change since the last delivery? If so, score the new slice on its own.
  2. Did a rule that decides outcomes change inside the new period? Log it with its effective date.
  3. Did the labeling guide change? Relabel a small cross-period sample under one guide.
  4. Did class shares move beyond the tolerance you set? Report balanced metrics next to accuracy.
  5. Is the gain on the frozen holdout above the preset minimum, with an interval that excludes zero?
  6. Does the delivery add to history or replace it, and does the agreement say which?
  7. Has any record in the new delivery leaked into the holdout?

0 of 7 ticked. Tick each item as you go.

Specifying a decay replay when you request records

A buyer can ask for the inputs of this replay at the start. SourceX does not hold datasets in stock: it finds them at U.S. businesses against a buyer's written request, so the request is the place to name the decision time key, per-period counts, the rule change log and enough history to locate a knee.

As a general guide, buyers look for two or more years of consistent history and tens of thousands of records in one system, with smaller sets considered when the work is rare. That depth is also roughly what a replay needs: enough periods to find where the gap levels off, and enough records per period to put an interval around it. Newer records do not imply any particular improvement; the replay is how a buyer finds out before committing to a cadence.

Methods note: data, commit, model and what was left out

Data: CHANGELOG-1.19.md to CHANGELOG-1.37.md in the CHANGELOG folder of github.com/kubernetes/kubernetes at commit dd410c4864ad52093ab74be9f7aa39a8cbc4dc06, read October 27, 2026. Unit: each bullet under Changes by Kind in the section for release 1.Y.0, which lists changes since the previous minor release. Of 5,560 bullets, 14 were dropped because they file 7 notes under two kinds each (matched by identical text or a shared pull request number), and 239 in smaller kinds (Deprecation, Documentation, Failing Test, Dependency, Uncategorized) were excluded, leaving 5,307, from 197 to 368 per release. Pull request links, author handles and SIG tags were stripped from the text.

Model: multinomial naive Bayes with add-one smoothing on lowercase words and adjacent word pairs. Tokenizer: runs of letters and digits ([a-z0-9]+) after lower-casing; word features a window never saw in training are ignored at prediction. Windows: three consecutive releases, each subsampled to 600 notes. Seeds: 0 to 19, fixed before the run; under seed s, the window ending at release 1.e draws its notes with random seed 1,000 times e plus s, and each seed is one complete replay. Figures are the mean over the 20 seeds with the lowest and highest seed. Refreshed baseline: the window ending one release before the test release. Gain: paired accuracy difference on the same test notes, averaged over test releases, with 95% intervals from 1,000 bootstrap resamples of test notes within each release, applied to per-note results averaged over the seeds, models held fixed. History arm: all releases before the test release, without subsampling. The script, xs0638_refresh.py, and its output are filed with this article's review materials and rerun in under a minute.

Limits: each staleness row averages a different set of test releases, 16 at one release stale and 9 at eight, so later rows lean on newer releases and rest on fewer of them. The bootstrap interval covers the choice of test notes, not the 600-note training draw, which the seed range shows, and it treats notes within a release as independent. Fixed choices move the level: scoring unseen word features with the add-one floor instead of ignoring them gives gains of 2.1 to 4.5 points with the same rise to four releases and the same uneven tail, balanced accuracy 3 to 4 points higher, and a 1.27 frozen lead of 6.3 points. Not measured: stronger models, other tasks, the cost of refreshing, and calendar dates for each release, which the changelog does not carry. This is one public task chosen because its records are dated and labeled; it is not a forecast for business records, and SourceX has published no decay results on licensed data.

Frequently asked questions

How quickly does training data lose its value?

It depends on the task, which is why it has to be measured rather than assumed. In SourceX's October 2026 replay of Kubernetes release notes, most of the loss, about 4 accuracy points, arrived within four releases; later gains were small and uneven, between 3.2 and 5.4 points. Vela and colleagues (2022) found quality degradation across industry datasets even when the data drifted only slightly.

Is a monthly data refresh better than a quarterly one?

Only if your decay curve says so. Average the measured gap over each candidate cycle and compare the difference with what the extra deliveries cost. If the curve is flat inside a quarter, monthly deliveries buy little, so set the minimum gain worth paying for before running the test.

Should refreshed records replace older ones or be added to them?

Test both, and expect additions to hold up well. In the Kubernetes replay, training on all earlier releases averaged 60.4% accuracy against 58.2% for the newest three releases alone. Agreements should state whether each delivery adds to, revises or replaces what came before.

What is the difference between data drift and utility decay?

Drift is a change in the data's distribution; utility decay is a loss of task performance as training data ages. They often move together but not always: a dataset can drift in ways a model ignores, and performance can slip with little visible drift. A refresh decision should rest on decay measured on the task itself.

Can a vendor's freshness promise stand in for a decay test?

No. A promise about how recent the records are says nothing about whether recency improves your model. Ask for dated records and per-period counts, then run the replay on a sample before committing to a cadence.

Sources

  • Kubernetes release changelogs for 1.19 to 1.37 list each release's notes under Changes by Kind headings; SourceX's replay used 5,307 of them. Source
  • Kubernetes requires release notes for pull requests with user-visible changes and asks writers to indicate whether a change added, changed, fixed, removed or deprecated functionality. Source
  • KEP-2572 moved Kubernetes to three releases per calendar year beginning with the 1.22 release. Source
  • Luu et al. (NAACL 2022) built eight tasks spanning five or more years, defined a per-step temporal degradation score, and found target-period finetuning helped more than continued pretraining. Source
  • Vela et al. (Scientific Reports, 2022) studied temporal degradation on 32 datasets from four industries and found it can develop even under minimal data drift. Source
  • Forecasting: Principles and Practice describes evaluation on a rolling forecasting origin, averaging accuracy over test sets as the origin moves forward. Source
  • Data Cards (Pushkarna, Zaldivar and Kjartansson, FAccT 2022) are a framework for structured summaries of dataset facts. Source

Tell us what data you need

Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.

See if your company qualifies