Code-Set Revisions and Process Changes in Multi-Year Operational Datasets

Why label meaning changes inside operational records

Label meaning changes because the systems that generate operational labels are governed by taxonomies, policies and people, and all three get revised. A support desk that had one "Billing" disposition in 2021 may have split it into "Billing – refund" and "Billing – dispute" in 2023. A claims team may tighten the threshold for "approved with adjustment."

Neither change shows up in the text of the records; it shows up only in what the label column now means.

This is different from input drift, which is covered in detecting drift between licensed historical data and your production traffic. Input drift is a shift in P(x). Label-definition drift is a shift in P(y | x): the same ticket written in 2020 and in 2024 would receive a different code.

Research on temporal degradation reports that model performance drops as training data ages relative to test data [1]. Both the facts and the language used to describe them change over time [2]. Representativeness itself is time- and place-dependent [3], so a five-year export is effectively several related datasets stacked on one another.

Fraud is the clearest example. Labels arrive late, only some cases are investigated, and the patterns being labeled shift as attackers adapt. Dal Pozzolo and colleagues treat concept drift, class imbalance and delayed supervision as a single linked problem [4]. Operational buyers inherit the same structure in chargebacks, claims denials and escalation outcomes.

Where code-set revisions come from

Code-set revisions come from two sources: external standards that publish versioned releases, and internal taxonomies that change without notice. External changes are easier to handle because they have release dates and published crosswalks or addenda.

• Healthcare remittance. Claim Adjustment Reason Codes (CARCs) and Remittance Advice Remark Codes (RARCs) on X12 835 transactions are maintained by a national committee that adds, changes and retires codes, with list updates published several times a year [5]. A denial classifier trained on 835 data spanning many releases sees codes whose descriptions were reworded or whose use was narrowed.
• Diagnosis coding. NCHS publishes ICD-10-CM code files and addenda by fiscal-year release [7]. Codes are added, expanded to more specific children and occasionally deleted, so a diagnosis label at one specificity in one year may appear only at a finer specificity the next.
• Merchant categories. ISO 18245 defines merchant category code values, and as of October 2026 the current edition is ISO 18245:2023 [6]. Card networks and acquirers also maintain their own assignments, so a merchant's MCC in transaction history can change even when the business does not.
• Internal taxonomies. These include CRM disposition codes, Zendesk or Salesforce Service Cloud case reasons, ERP return reason codes and loss-reason picklists. They change when an admin edits a picklist, often with no record beyond the field's audit history. Return reasons and RMA codes are a common case; see product returns and RMA reason data.

Process and policy changes are the hardest to detect, because the code set stays the same while the decision rule behind it moves. Examples include a new underwriting cutoff, a revised refund policy, a reorganized tier-2 escalation path, or a new QA rubric for agents. In lending, adverse action reasons interact with this directly; see credit decision records with adverse action reasons. The bias implications are covered separately in historical decision bias in operational labels.

What to ask the supplier before you profile anything

Ask the supplier for the change history before you run any statistics, because a change point you find without context is only a hypothesis. Useful requests include:

1. Code-set version history for every label field, with effective dates and the system of record (for example, picklist audit history in Salesforce, or a reason-code master table in SAP or Oracle).
2. Dates of policy, SOP or rubric changes that affect how the label is assigned. Include who assigns it: an agent, a rules engine, an auditor, or a downstream system.
3. Platform migrations, such as a move from one ticketing system to another, and how legacy codes were converted during the migration. Data acquired through mergers often carries two taxonomies; see checking title on data acquired through a merger or asset sale.
4. Any crosswalk tables already used internally for reporting continuity.
5. Whether historical records were re-coded in place when a code was retired. In-place re-coding silently erases the change point.

Data Cards recommend documenting collection and annotation methods and the decisions that affect model performance [8]. Code-set history belongs in that documentation, attached to the delivery rather than buried in an email thread. When SourceX sources a dataset, diligence materials covering source, rights, preparation and allowed use are prepared per dataset. Ask for label-history documentation alongside those materials when you describe your request through the SourceX buyer request.

Detecting change points in label distributions

The fastest detection method is to plot each label's share of records per month and look for step changes that do not match seasonality. A gradual trend usually reflects the business. A vertical step on the first day of a month or quarter usually reflects a taxonomy edit, a migration or a policy change.

Practical checks, in rough order of cost:

• Code appearance and disappearance. List each code's first-seen and last-seen month. Codes that appear the month after another disappears are candidate splits or renames.
• Share plots with a stacked view. If "Billing" drops from 18% to 4% while "Refund" and "Dispute" appear at a combined 14%, you have found a split.
• Change-point detection. Run a standard detector (for example, PELT or binary segmentation from the ruptures library) on the monthly share series for each label. Treat the detected dates as candidates to reconcile against supplier history, not as findings.
• Conditional checks. Fix a slice of inputs, such as tickets that mention "chargeback," and plot their label distribution over time. A shift here is evidence of P(y | x) drift rather than a change in input mix.
• Classifier probe. Train a model on year one and score year three. A cluster of confident disagreements concentrated after one date points to a definition change, not random noise. Confident-learning methods were built to surface likely label errors [9], and the same tooling can help rank period-specific disagreements for review.
• Field-level metadata. Check for nulls that begin on a date, new free-text "other" values, and changes in average text length of the disposition notes.

For thresholds on how many records to hand-review once a change point is suspected, see sample sizes for estimating a dataset's error rate.

Choosing a strategy: remap, truncate or condition on period

The right strategy depends on whether the old and new labels can be related one to one, and on how much post-change data you have. Most multi-year projects end up combining the strategies per field.

Illustrative example: invented to show structure; it does not describe an available dataset.

When conditioning on period, set the regime feature to the current value at inference time. Then verify that predictions on recent holdout data match the current definition. Keep remapped and raw labels side by side so you can audit the crosswalk later, and log the crosswalk version in your training metadata.

A label-regime register for each dataset

A label-regime register is a small table that records each period during which a label field had stable meaning. Training and evaluation code can then filter or condition on it. It belongs next to the data, versioned with it.

Illustrative example: invented to show structure; it does not describe an available dataset.

field: case_disposition
system_of_record: "ticketing system picklist (audit history export)"
regimes:
  - id: R1
    effective_from: 2020-01-01
    effective_to: 2022-06-30
    code_set_version: "picklist v3"
    assigned_by: agent_at_close
    notes: "Billing is a single code"
  - id: R2
    effective_from: 2022-07-01
    effective_to: 2024-02-29
    code_set_version: "picklist v4"
    assigned_by: agent_at_close
    notes: "Billing split into BILL_REFUND and BILL_DISPUTE; no back-fill"
  - id: R3
    effective_from: 2024-03-01
    effective_to: null
    code_set_version: "picklist v4"
    assigned_by: qa_auditor_override_allowed
    notes: "Same codes; new QA rubric changes dispute criteria"
crosswalk:
  - { from: "BILLING", to: ["BILL_REFUND", "BILL_DISPUTE"], type: split, from_regime: R1 }
detected_change_points: [2022-07-01, 2024-03-04]
reconciled: true

The R3 regime is the dangerous case. The code set is identical, so a version check passes, but the assignment rule changed. Only the supplier's process history or a conditional label-share check would reveal it.

How label drift affects evaluation and SFT

Label drift distorts evaluation most, because a mixed-regime test set rewards a model for matching outdated definitions. Build evaluation sets from the most recent regime. If you need longitudinal evaluation, report metrics per regime rather than pooled. Our guidance on verifying outcome fields as ground truth covers the reliability checks that should run inside each regime.

For supervised fine-tuning on decisions, such as "given this case, choose a resolution and explain it," old-policy examples teach the model to reason under rules that no longer apply. Filter them out, or put the policy version in the prompt so the model can condition on it. Track label-regime coverage alongside the other metrics in training data quality metrics.

For how often buyers refresh operational data to limit this problem, see how often AI buyers want fresh data. More quality checks are collected in the training data quality assessment hub.

Licensing multi-year operational records with label history

SourceX sources operational datasets from US companies on request, including support and sales histories, engineering records, and finance and legal workflows. It then manages the licensing process. Each dataset is rights-reviewed and delivered under a license that defines records, uses, term and delivery, with diligence materials prepared per dataset. Describe the fields, years and label history you need at SourceX for AI data buyers.