Skip to content

AI uses for records

Business rule change label validity: a replay test before relabeling

By SourceX Editorial · Reviewed by Noah Loul ·

9 sections

Short answer

A changed business rule does one of four things to a historical training label: leaves it valid, flips it to a value you can recompute from recorded fields, makes it undecidable because the new rule reads a fact nobody recorded, or strands the later steps of a multi-step case. Encode both rule versions, replay them over every record, sort labels into those outcomes, and keep the old labels beside the new.

Agent run

Business rule change label

This page covers business rule change label validity: a replay test before relabeling.Illustrative

Key takeaways

  • Tagging a record with the rule in force says which rule made the label; replaying the new rule says what the label should be now and whether that can be computed.
  • The 2024 U.S. overtime salary level took effect on July 1, 2024 and was vacated on November 15, 2024, so overwriting old labels would have erased the version that returned.
  • Parametric changes such as a new threshold are usually replayable from recorded fields; structural changes often read inputs the old process never captured.
  • Replaying the old rule on old records first exposes labels that came from discretion or error rather than from the rule.
  • In multi-step cases, keep the steps before the first divergence and turn the divergence into a corrected next-action label.

What does a changed business rule do to a training label?

It changes the correct answer for some inputs and leaves the rest alone, so the job is to find which records moved and whether their new answer can be computed. Every affected label ends in one of four states: still valid, flipped but recomputable, undecidable from what was recorded, or stranded partway through a case whose later steps would not have happened under the new rule.

In machine learning terms this is concept drift with a known cause. The review Learning under Concept Drift (arXiv 2004.05785) defines drift as a change in the joint distribution of inputs and labels from one time to the next. A rule change moves one part of that distribution, the label given the input, and only for the cases its changed clause touches. Unlike drift discovered through model errors, the change point and the clause are known in advance, so the effect can be computed instead of estimated.

Tagging each record with the policy version in force tells you which rule produced a label. It does not tell you what the label would be under the current rule, or whether that answer can be computed at all. That second question sets the cost of relabeling, and it is what this page answers.

A real rule that changed twice in under five months

The 2024 federal overtime rule shows that rule changes are not one-way events. The U.S. Department of Labor's 2024 final rule for the executive, administrative and professional exemptions raised the standard salary level to $844 a week, about $43,888 a year, and the total annual compensation threshold for highly compensated employees to $132,964 on July 1, 2024, with a new calculation method set to follow on January 1, 2025.

On November 15, 2024, the U.S. District Court for the Eastern District of Texas vacated the 2024 rule. The Department's overtime pages say it now enforces the 2019 rule's standard salary level of $684 a week and its $107,432 threshold for highly compensated employees, and they note that the government appealed the decision.

For a dataset of employee classification decisions, the salary-level part of the label moved twice in about four and a half months. A team that rewrote its labels to the new level in August 2024 would have been encoding a level that was no longer enforced by December. Which version governs a decision made inside that window is a question for counsel; the data lesson is that the current rule is itself a dated fact, and overwriting the old labels would have destroyed the version that came back.

The same rule shows the replay outcomes. Records with weekly salaries at or above $844, or below $684, keep their salary-level result under both versions. Records between $684 and $844 flip, and the flip can be computed from the salary field. Highly compensated status between $107,432 and $132,964 depends on total annual compensation, so a record holding only base salary cannot settle it.

Rule replay: run both versions over every record

Replay means writing the old and the new rule as executable decision logic and running both over the records in scope. The comparison, rather than a reviewer's impression, shows which labels the change touched and why.

Business-rule tooling already supports this. The Object Management Group's Decision Model and Notation standard expresses decisions as decision tables whose hit policy, such as Unique, First or Collect, fixes how matching rules produce a result, so two versions of a rule can be executed and compared row by row. Rules-as-code projects make the dating explicit: OpenFisca's documentation, written for tax and benefit law, separates a parametric change, a new parameter value with a start date that existing formulas pick up for the right period, from a structural change that needs a new dated formula, and it applies a reform to a copy of the reference system instead of modifying it.

That distinction predicts replay cost. A parametric change, such as a new salary level or a shorter refund window, can usually be replayed from fields already on the record. A structural change, such as a newly required document or a new eligibility condition, often reads an input the old process never captured.

Programmatic labeling research supplies the mechanics. Snorkel (arXiv 1711.10160) showed that training labels can come from labeling functions that users write, heuristics applied to data in code, instead of hand annotation. A rule replay uses the same machinery with deterministic functions, one per rule version, each allowed to return cannot decide when a field it needs is missing.

Run one check before trusting any replay: apply the old rule to the old records and compare its output with the labels people actually assigned. Where the two disagree, the rule did not produce the label. It may record discretion, an exception or a data-entry error, and replaying the new rule over it would replace a human judgment with a formula.

Five replay outcomes and what each does to a label

Each record lands in one of five outcomes, and each outcome gets its own treatment. Four describe how the rule change touched a label; the fifth flags labels the old rule never explained.

The outcomes, with the overtime rule as the running example where it applies:

Five replay outcomes and what each does to a label
Replay outcomeTestWhat happens to the labelTraining useExample
UnchangedOld and new rule return the same resultKeep it as recordedTrain and evaluate under the current ruleWeekly salary at or above $844 meets both levels
Flipped, recomputableResults differ and every input the new rule reads is recordedWrite the new result to a separate current-rule columnTrain on the current-rule column; keep the original for historyWeekly salary between $684 and $844
UndecidableThe new rule reads an input the record lacksMark it cannot decide; never guessHold out, or collect the missing field for a sampleHighly compensated status when only base salary is recorded
Path-divergedThe new rule would have changed an earlier step of a multi-step caseValid only up to the divergence stepTrain on the prefix and add the corrected next actionIllustrative: a refund approved before a newly required photo
Not rule-drivenThe old rule's replay disagrees with the recorded labelTreat it as discretion or error, not a rule outputReview before any use; do not overwrite with the new ruleIllustrative: a manual override recorded without a reason

Where a multi-step case stops being valid

At the first step where the new rule would have acted differently. Everything before that step remains a correct example of the work; everything after it records a path the new rule would not have taken.

An illustrative refund workflow shows the point. Under the old policy an agent approves a damaged-goods refund at step two, ships a replacement at step three and closes the case at step five. A new policy requires a photo of the damage before any refund above a set amount. Replayed under the new policy, the case diverges at step two, where the correct action is to request the photo, and the recorded ending, a satisfied customer and a closed ticket, belongs to a path that should no longer exist.

Truncating at the divergence step keeps the valid prefix and turns the divergence itself into a new labeled decision: given this state, the correct next action under the current rule is to ask for evidence. The steps after the divergence still serve history-faithful tasks, such as auditing past handling, but they should not train an agent that will act under the new rule.

Step-level replay needs a timestamp on each action and the rule inputs available at that moment. Without them, the whole case falls into the undecidable outcome.

How large is the change for the task you will deploy?

Measure the share of records in each outcome, weighted to the mix of cases the deployed model will actually see, before spending anything on relabeling. A change that flips a large share of an archive may matter little if it touches a segment the agent will never handle, while a small share can matter a great deal if it sits where the agent spends most of its time.

An illustrative replay of 1,000 historical labels might sort into 712 unchanged, 164 flipped and recomputable, 71 undecidable, 38 path-diverged and 15 not rule-driven. Relabeling cost sits almost entirely in the 71 and the 38: the 164 flips are a column write, while undecidable and diverged cases need new data or expert judgment. The 15 not rule-driven labels deserve attention out of proportion to their count, because they mark where the rule never described the work.

Report each share per task segment with its denominator, and write decision thresholds before the replay runs, for example the undecidable share above which a segment is held out of training rather than partly relabeled.

An alluvial diagram splitting 1,000 illustrative historical labels into five replay outcomes: 712 unchanged, 164 flipped and recomputable, 71 undecidable, 38 path-diverged and 15 not rule-driven, each with its treatment.
Illustrative counts from a rule replay; not measured data.

The relabel feasibility card and change-impact checklist

Record each rule change on one card per rule and segment, so the next change starts from a documented state. Every field carries a date, because the card describes the dataset as of a rule version rather than for all time.

The checklist sets the order of work, and the table lists the card's fields with illustrative entries drawn from the overtime example.

Checklist · 8 items

  1. Write the changed clause as code or a decision table for both versions, citing the policy text and its effective date.
  2. Classify the change as parametric or structural, and list every input the new version reads.
  3. Run the old version over the old records and set aside each label it fails to reproduce.
  4. Run the new version over every record in scope and sort each one into the five outcomes.
  5. For multi-step cases, replay step by step and mark the first divergence.
  6. Weight the outcome shares to the deployment mix and compare them with thresholds written in advance.
  7. Write current-rule results to a new column and leave the original labels untouched, since rules can revert.
  8. Rerun the replay whenever the rule changes again, including when a change is withdrawn or vacated.

0 of 8 ticked. Tick each item as you go.

The relabel feasibility card and change-impact checklist
Card fieldWhat it recordsIllustrative entry
Rule and versionsRule name, old and new version identifiers, and the clause that changedOvertime salary level: 2019 rule and 2024 rule, standard level clause
DatesAnnounced, effective and any later reversal, each with its sourceEffective July 1, 2024; vacated November 15, 2024
Change typeParametric (new value) or structural (new logic or input)Parametric for the standard salary level
Replay fidelityShare of recorded labels the old rule reproduces98.5 percent: 15 of 1,000 not rule-driven
Outcome countsRecords in each of the five outcomes, by segment, with the denominator712, 164, 71, 38 and 15 of 1,000
Missing inputsFields the new rule reads that the records lackTotal annual compensation
Label columns keptOriginal label, old-rule replay and current-rule replay, none overwrittenThree label columns per record
Sign-offWho approved the replay logic and the thresholds, and whenData owner and policy owner, with dates

Would training only on post-change records be simpler?

Sometimes, but it swaps one bias for another. Records created after the change follow the new rule by design, yet there are usually few of them, their outcomes are less mature, and decisions taken in the first weeks often still follow the old habit.

Replay measures that adoption lag directly. Post-change records whose labels match the old rule and not the new one are non-compliant decisions, not valid examples of the current rule, and they belong outside the training set or in a set that teaches correction. Newer records can also reflect a changed task, a new form or a different customer mix, which makes comparisons with older scores unreliable whatever the rule did.

Use post-change records alone when they are plentiful, mature and cover the deployment mix. Otherwise combine them with replayed older records and keep the outcome label on each, so the model owner can weight or exclude any group. The overtime example adds a final reason: a dataset built only from records made under a rule that is later vacated has nothing to fall back on.

Asking a supplier for records you can replay

Ask for the rule inputs, not just the decisions. A dataset that holds only final labels can be tagged by date but not replayed; one that holds the fields each rule version reads, with a timestamp per step, can be relabeled the next time the rules move.

Name these in the request: the rules that govern the task, with version history and effective dates; the input fields each version reads; step-level timestamps for multi-step cases; any field showing which rule clause or override applied; and an acceptance test stating the largest undecidable share you will accept per segment.

SourceX does not keep datasets in stock. It looks for matching records at U.S. businesses once a buyer describes the need, and its request form collects scope, volume, language, format, timing and licensing requirements, so a replay requirement can be written into the request from the first message. Whether a given company's records carry the inputs a rule reads is established in its review, not assumed.

The method has limits. It needs rules that can be written as logic, so it helps little with judgment-heavy decisions; it cannot recover inputs that were never recorded; and it says nothing about whether a model trained on replayed labels performs better, which needs its own held-out test after the change. SourceX has not run this replay on any dataset and publishes no results from it; the counts on this page are illustrative.

Frequently asked questions

Should I relabel historical data when a business rule changes?

Only the records the change touches, and only where the new label can be computed. Replay both rule versions first: unchanged labels stay, recomputable flips go into a separate current-rule column, undecidable and path-diverged cases are held out or reviewed, and the original labels are never overwritten.

What is the difference between a parametric and a structural rule change?

A parametric change alters a value inside existing logic, such as a salary level or a refund window, and can usually be replayed from recorded fields. A structural change adds logic or a new input, such as a required document, and often needs data the old process never captured. OpenFisca's documentation draws the same line for tax and benefit law.

How can I tell whether a label was produced by the rule at all?

Replay the old rule on the old records and compare. Labels the old rule reproduces were rule outputs; labels it contradicts reflect discretion, exceptions or errors, and they should be reviewed rather than overwritten by the new rule.

Why keep the old labels after relabeling?

Because rules can change back. The U.S. Department of Labor's 2024 overtime salary level took effect on July 1, 2024 and was vacated by a federal court on November 15, 2024, so a dataset that overwrote its labels in between lost the version that returned.

What should a dataset request include so labels can be replayed later?

The rule versions and effective dates that govern the task, the input fields each version reads, step-level timestamps, any record of which rule clause or override applied, and an acceptance test that caps the undecidable share per segment.

Sources

  • The Department of Labor says the Eastern District of Texas vacated the 2024 overtime rule on November 15, 2024 and that it enforces the 2019 rule's $684 weekly salary level and $107,432 highly compensated threshold; the government appealed. Source
  • The Department of Labor's FAQ on the 2024 final rule describes the $844 weekly standard salary level (about $43,888 a year) and $132,964 highly compensated threshold from July 1, 2024, with calculation changes set for January 1, 2025. Source
  • The review Learning under Concept Drift defines concept drift as a change in the joint distribution of inputs and labels between time points. Source
  • OMG's Decision Model and Notation specification defines decision tables with hit policies, such as Unique, Any, First, Priority and Collect, that determine how matching rules produce a result. Source
  • OpenFisca's documentation separates parameter evolutions, new values with start dates used by unchanged formulas, from formula evolutions that need a new dated formula. Source
  • OpenFisca applies a reform to a copy of the reference tax and benefit system and distinguishes structural reforms (formulas) from parametric reforms (parameters). Source
  • Snorkel creates training labels from user-written labeling functions, heuristics applied in code, instead of hand labeling. Source

Tell us what data you need

Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.

See if your company qualifies