Skip to content

AI uses for records

Post-training dataset refresh triggers after a business change

By SourceX Editorial · Updated

10 sections

Short answer

Refresh training data when a business change makes specific examples teach the wrong action, not on a calendar. Log each process change (a new policy, status code, form field, system or owner) in a refresh trigger register, select the examples its rule, label or inputs touch, then relabel, retire or replace them. Accept the refresh only if the failure cohort improves on untouched held-out cases without regressions elsewhere.

Call to transcript

Post training dataset

This page covers post-training dataset refresh triggers after a business change.Illustrative

Key takeaways

  • A model-level retrain trigger says something must change; a refresh trigger register says which examples are now wrong.
  • Only changes to the correct output make examples stale; a shift in case mix calls for reweighting, not relabeling.
  • Read changes from audit trails and policy versions early, because configuration history may only be kept for a limited time.
  • Give each affected example one action: relabel, retire, replace, keep as tagged history or escalate.
  • Accept a refresh only on a matched-budget test where the failure cohort improves and regression suites hold.
  • Relabeling or retiring examples is a major dataset version change and belongs in the maintenance record.

How do you decide which training data needs refreshing after a business change?

Decide example by example, starting from the business change rather than from a calendar. List the process changes since the last training cut (a new policy, a renamed status, a new form field, a migrated system, a different approver), find the training examples whose rule, label or inputs each change touches, and act only on those. Then prove the refresh fixed the failure it targeted on held-out cases nobody used for selection.

This is a narrower question than when to re-tune a model. A model-level trigger, such as a base model retirement or falling scores on recent traffic, tells you that something must change. It does not tell you which of thousands of supervised examples now teach the wrong action. A refresh trigger register answers that second question, one process change at a time.

The register matters most for post-training data built from business records: support resolutions, claims decisions, order exceptions, approvals and similar workflows. Those examples encode a company's rules at the moment they were written. When the rule moves, the example keeps teaching the old one with full confidence.

Which business changes make post-training examples stale?

Changes that alter what the correct output is make examples stale; changes that only alter how often a case appears usually do not. A refund window that shrinks invalidates every example that applied the old window. A seasonal spike in refund requests changes the mix but leaves each example correct.

The table below sorts common business changes by where they show up first and what they break inside a training example. Use it to decide which changes deserve a register entry at all.

Which business changes make post-training examples stale?
Business changeWhere it shows up firstWhat goes stale in the exampleUsual action
Policy or rule change (refund window, approval limit, eligibility rule)Policy document, knowledge base article, configuration valueThe target output: the decision or the explanationRelabel or retire examples decided under the old rule
New, merged or renamed status or reason codePicklist or code table in the CRM, ticketing or ERP systemLabels and any output that names the codeRemap labels; retire codes that no longer exist
Form or field change (new required field, removed field)Intake form, object schema, export layoutInputs: examples lack a fact the new process depends onReplace with examples that carry the new field
System migration or new tool in the workflowCutover plan, new object names, different IDsInputs and actions: tool names, steps and identifiersReplace action traces; keep text-only examples if still valid
Ownership or routing change (new team, new queue)Assignment rules, queue configurationRouting targets and escalation stepsRelabel routing targets; keep the case content
Product or catalog change (new tier, retired item)Product master, price book, catalogEntities and entitlements named in outputsRetire examples about retired items; add new ones
Volume or seasonality shift onlyDashboards and queue countsNothing in the example; only the mixReweight or resample; no relabeling

Catching the change in the source system before the model repeats the old rule

Read the change from the system of record, not from memory, because configuration history is often kept only for a limited time. In Salesforce, for example, Setup Audit Trail shows the 20 most recent setup changes in the org with who made them and when, and Salesforce says the full setup history can be downloaded for the past 180 days. A team that checks only once a year will miss changes that have already aged out of that window.

At a glance

Catching the change in the source system before the model repeats the old rule

Configuration logs catch deliberate changes. They miss changes that happen in practice before anyone edits a setting, such as agents starting to use a new reason in free text. A distribution check on the exported records catches some of those. Evidently, an open-source evaluation library, lists checks for missing values, duplicates, min-max ranges and new categorical values alongside drift tests that compare a current batch with a reference batch. A new categorical value in a status or reason field is a cheap, concrete signal that a register entry may be needed.

Neither signal proves an example is wrong. Each one is a lead that a person who knows the process confirms or dismisses, and the confirmation itself goes in the register so the decision can be audited later.

  • Configuration history: audit trails, change requests and release notes for the systems the examples came from.
  • Document history: dated versions of policies, playbooks and macros the examples quote or apply.
  • Data signals: new category values, fields that suddenly fill or empty, and shifts in outcome rates by period.
  • Model signals: a failure cohort found in evaluation or production review that clusters after a known date.

The refresh trigger register, field by field

The register is one row per business change, with enough detail to rebuild the selection and repeat the test. Keep it next to the dataset, version it with the dataset, and never edit a closed row; add a new one instead. The entries in the third column describe an illustrative returns-policy change and are not drawn from a real company.

The refresh trigger register, field by field
FieldWhat to recordIllustrative entryWhy it matters
Trigger ID and effective dateA stable ID and the date the change took effect in the business, not the date it was noticedRT-07, effective on the first day of Q3Separates examples decided before and after the change
Change and evidenceWhat changed and where it is recordedReturns window shortened from 60 to 30 days; policy version 4 and the audit trail entry for the window settingLets a reviewer confirm the change happened as stated
Affected rule or labelThe decision rule, label or field the change touchesLabel return_approved; rule return windowDefines what can be wrong in an example
Selection ruleA query a second person can rerun to find affected examplesReturn requests decided after day 30 and on or before day 60 of the purchase dateKeeps the selection reproducible and auditable
Affected count and action mixHow many examples matched and how many get each actionIllustrative: 410 matched; 290 relabel, 80 retire, 40 keep as historyShows the size of the change before training spend
Failure cohort and held-out setThe model failure this refresh should fix and the untouched cases that measure itApprovals of day 31 to 60 returns; 150 held-out cases from after the effective dateTies the refresh to a measurable outcome
Regression suitesExisting capabilities that must not get worseReturn requests inside 30 days; warranty claims; tone checksCatches damage to behaviors the refresh did not target
Decision, owner and versionAccept, revise or reject; who decided; resulting dataset versionAccepted by the returns process owner and the evaluation lead; dataset version 3.0.0Makes the refresh traceable in later audits

Retire, relabel, replace or keep as history: one call per affected example

Each selected example gets exactly one action, and the default is the least destructive one that removes the wrong lesson. Deleting everything from before the change is tempting and usually wrong, because many older examples still teach skills the change did not touch, such as tone, evidence gathering or how to ask for a missing order number.

At a glance

Retire, relabel, replace or keep as history: one call per affected example

  • Relabel when the input is still realistic and only the correct decision changed. The case content stays; the target output is rewritten under the new rule by someone who knows it.
  • Retire when the input itself no longer occurs, such as a request about a product that was discontinued or a step in a system that no longer exists.
  • Replace when the new process depends on a fact the old example lacks, such as a newly required field. Rewriting the old example would invent that fact, so source real cases recorded under the new process instead.
  • Keep as history when the task explicitly involves past rules, such as explaining why an older order was handled differently. Tag these examples with the rule version so they are never mixed into the general training pool by accident.
  • Escalate when the selected examples disagree with each other about what the new rule means. That disagreement is a policy question for the process owner, not a labeling task.

Did the failure cohort improve, and did anything else regress?

Accept a refresh only when the targeted failure cohort improves on held-out cases and the regression suites hold. Both halves matter. A refresh can fix the new rule while quietly degrading something older, and continual fine-tuning is a known source of that kind of loss: Luo and co-authors report that catastrophic forgetting was generally observed in the language models they tested, from 1 billion to 7 billion parameters, during continual instruction tuning.

Make the comparison fair by holding the training budget fixed. Train one candidate on the current dataset and one on the refreshed dataset with the same number of examples, steps and settings, so any difference comes from the refreshed examples rather than from more data. Draw the held-out cohort from cases recorded after the effective date, and keep it out of the selection query, the relabeling work and any prompt tuning.

Report the result per cohort, with the number of cases behind each figure and an interval that reflects how few cases a narrow cohort usually has. A small held-out cohort can swing on a handful of cases, so decide in advance what improvement counts and what regression is tolerable. Write those thresholds in the register row before the run, not after seeing the numbers.

  • Failure cohort: the cases the change made wrong, measured before and after.
  • Neighbor cohort: cases just inside the old and new boundaries, where relabeling mistakes show up first.
  • Regression suites: earlier capabilities and policies that the change did not touch.
  • Lineage check: every training example in the refreshed set maps back to its register action or to no action.

Worked example: a returns policy that changed mid-year (illustrative)

Suppose a retailer's support model was tuned on two years of return conversations. Midway through the year the returns window drops from 60 to 30 days, and reviewers notice the model still approves returns on day 45. Every figure below is illustrative.

The audit trail and policy versions confirm the effective date, so the team opens a register row. The selection rule finds 410 training examples where a return between day 31 and day 60 was approved under the old policy. A returns specialist relabels 290 of them as declines with the new explanation, retires 80 that concern a membership tier that no longer exists, and keeps 40 as history examples about orders placed before the change, tagged with the old rule version.

With the same training budget, the refreshed candidate is compared with the current model on 150 held-out cases recorded after the effective date. Suppose the share of day 31 to 60 returns handled correctly rises from roughly 20% to roughly 85%, while the within-30-day suite and the warranty suite stay level. The row is accepted and the dataset becomes a new major version.

The exception is just as instructive. Suppose the warranty suite had dropped instead, because several relabeled examples also mentioned warranty repairs and the specialist declined those too. The fix is not more data; it is splitting the selection rule so warranty requests are excluded, relabeling again and rerunning the same comparison.

When refreshing training examples is the wrong fix

Stop and rethink the refresh when the changing fact moves faster than a training cycle, when the failure is not caused by the examples, or when the meaning of the new rule is still disputed. Retraining around a fact that changes weekly, such as a price or a stock level, produces a model that is stale on arrival; that fact belongs in retrieval or a tool call that reads the current value.

Narrow the scope when only one segment changed. If the new window applies to one region or one product line, the selection rule and held-out cohort should be limited to that segment, and the register row should say so. A global relabel would teach the wrong rule everywhere else.

Bring in specialist review before relabeling when the change follows a legal or regulatory requirement, when examples contain personal or sensitive records that the refresh would copy into new versions, or when the business and the data team read the new rule differently. Those calls belong to counsel, the privacy owner or the process owner, and the register records who made them.

Recording the refresh so the next team can trust the dataset version

Version the dataset so a reader can tell whether existing examples changed or new ones were only added. The Croissant metadata format for machine learning datasets recommends semantic versioning and spells out the difference: a minor version may add data while existing data stays retrievable as is, while a major version means existing data was edited, removed or reshuffled across splits. Relabeling or retiring examples is therefore a major change under that scheme.

Croissant also recommends recording a SHA-256 checksum for each file in versioned datasets, so a training run can confirm it used the files the version declares. For datasets that keep growing, it describes a live dataset flag and advises updating the version when structure or meaning changes rather than every time new data arrives.

Documentation should say how refreshes happen, not only what the current version contains. Datasheets for Datasets asks creators whether the dataset will be updated to correct labeling errors or add or delete instances, how often, by whom and how users will be told. Croissant's responsible AI vocabulary has a similar field for a release maintenance plan covering update timeframe, maintainers and deprecation policy. The register is the evidence behind those answers.

Where replacement examples from real businesses come in

The replace action is where many refreshes stall, because the new process has not yet produced enough real cases to train on. Waiting for volume, generating synthetic cases or licensing recorded cases from businesses that already run a similar process are the usual options, and each needs the same held-out test before acceptance.

SourceX licenses selected operational records directly from businesses and handles rights review, contracting, preparation and delivery. Because it sources datasets to order from U.S. businesses rather than holding stock, a post-training team can request cases recorded under the new process and describe the workflow, volume, format, timing and licensing terms its refresh row requires.

Frequently asked questions

How often should post-training data be refreshed?

There is no safe fixed interval. Review the register on a schedule, such as monthly, but refresh only when a confirmed business change makes some examples teach the wrong output or when a failure cohort points to stale examples. A calendar review catches changes; it should not be the reason to retrain.

Is a refresh trigger the same as data drift?

No. Drift describes a change in the distribution of inputs or outcomes, which may or may not make individual examples wrong. A refresh trigger is a confirmed business change that alters the correct output for identifiable examples. Drift signals such as a new category value are useful leads for finding triggers.

Should old examples be deleted when a policy changes?

Usually not all of them. Relabel examples whose inputs are still realistic, retire those whose inputs no longer occur, and keep some as tagged history when the model must explain past decisions. Deleting everything before the change also deletes skills the change did not touch.

How many new examples does a refresh need?

Enough to move the failure cohort on held-out cases without hurting the regression suites, which depends on the task and the model. Start from the affected count in the register, run the matched-budget comparison, and add examples only if the cohort result is still short of the threshold set before the run.

Who should own the refresh trigger register?

Ownership works best when split. The data or evaluation lead owns the register, the selection rules and the tests, while the business process owner confirms each change and the meaning of the new rule. Counsel or the privacy owner signs off when a change is legal in origin or the examples hold sensitive records.

Sources

  • Salesforce Setup Audit Trail lists the 20 most recent setup changes with date and user, and Salesforce says the org's full setup history can be downloaded for the past 180 days. Source
  • Evidently's README lists data quality checks such as missing values, duplicates, min-max ranges and new categorical values, plus drift tests that compare current data with a reference. Source
  • Luo and co-authors report that catastrophic forgetting is generally observed in language models from 1 billion to 7 billion parameters during continual fine-tuning. Source
  • The Croissant 1.1 specification recommends semantic versioning for datasets, defines minor and major changes, recommends SHA-256 file checksums and describes live datasets. Source
  • The Croissant RAI specification defines a release maintenance plan property covering the updating timeframe, maintainers and deprecation policies. Source
  • Datasheets for Datasets asks dataset creators whether the dataset will be updated to correct labeling errors or add or delete instances, how often, by whom and how consumers are told. Source

Tell us what data you need

Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.

See if your company qualifies