Refreshing fine-tuning data: when to re-tune and what new data to buy

What should trigger a re-tune of a fine-tuned model?

There are four triggers, and each needs a different kind of new data. Treat the calendar as a review schedule. It should never be the reason you retrain.

• Base-model retirement. Hosted fine-tuning providers tie a tuned model's lifetime to its base model and publish deprecation schedules with notice periods, and some also restrict who can start new fine-tuning jobs. Check your provider's current deprecation terms as of October 2026 before you plan a migration. A retirement forces a full re-run on the new base, whether or not your data has changed.
• Input drift. New products, new ticket categories, new document templates or changes in customer vocabulary move the input distribution away from your training set. The signal is a falling score on a recent-traffic eval slice while the frozen eval set holds steady.
• Target change. A new refund policy, a revised style guide, a regulatory update or a renamed product makes previously correct outputs wrong. These changes are urgent because the model will confidently repeat the old answer.
• Label or taxonomy change. If routing classes, intent codes or extraction fields are added, merged or renamed, the training labels have to be remapped. Adding new examples on top of the old labels does not fix this.

If a fact changes every week, such as prices, inventory or policy text, consider moving it into retrieval instead of the weights. The fine-tuning vs RAG vs continued pre-training guide covers that split.

Why incremental fine-tuning on new data is riskier than it looks

Stacking a small new batch on top of the current fine-tuned checkpoint is cheap, but it can quietly damage the model. Studies of continual fine-tuning in LLMs report catastrophic forgetting: sequential tuning on new data erodes general and domain skills the model had before [1]. Gekhman et al. found that fine-tuning examples which introduce new factual knowledge are learned more slowly than examples consistent with what the model already knows. As the model eventually fits them, its tendency to hallucinate rises [2].

In practice, decide between two approaches. Replay means retraining from the base model on the merged corpus. Mixing means adding the new batch together with a sampled share of earlier versions. Avoid the third option, tuning only on the delta, unless your evals show it is safe.

Quality also counts for more than volume here. LIMA showed that 1,000 carefully curated pairs could produce strong alignment on a 65B model [3], so a small, clean refresh batch can be better than a large, noisy one. For preference refreshes, DPO trains directly on chosen and rejected pairs without a separate reward model [4]. That makes it practical to re-run when you buy new preference data.

A refresh cadence by data type and model role

How often you refresh depends on how fast the underlying domain changes. This table is a starting point to calibrate against your own drift metrics.

Illustrative example: invented to show structure; it does not describe an available dataset.

Whatever the cadence, keep a frozen regression eval and a rolling "last 30 days" eval that the refresh delivery never touches. The pre-purchase evaluation guide explains how to build both. If you need to size the batch, see how much data you need to fine-tune an LLM.

What to specify in a refresh data purchase

A refresh purchase only works if each delivery can be merged with the previous ones without manual cleanup. Write the following terms into the delivery specification:

• Period and cutoff. State the record date window, such as tickets closed between two dates, and say how late edits to older records are handled.
• Incremental vs full re-delivery. An incremental delivery sends only new or changed records. A full refresh replaces the whole set [6]. Incremental deliveries need stable record IDs and a change flag (insert, update, tombstone), or you cannot merge or delete correctly.
• Schema contract. Fix field names, types, the chat format (messages with role and content) and label vocabularies. Require advance notice and a mapping file for any change. On your side, choose whether ingestion fails fast or evolves the schema when unexpected columns appear [5].
• Version identifiers. Each delivery gets a version (for example v2026.10), a manifest with row counts and SHA-256 hashes, and a changelog.
• De-identification consistency. If names and account numbers are replaced with surrogates, the same method should be used in every delivery so that cross-version deduplication and conversation threading still work.
• Eval slice. Ask for a held-out sample from the new period that is never mixed into training.
• Rights per delivery. The license should cover each future delivery under the same allowed uses. Do not assume that a renewal carries over the earlier scope. Read fine-tuning-only data licenses for scope pitfalls.

The ongoing data supply agreements guide covers contract structure in more depth. For the supplier's side of the cadence question, see how often AI buyers want fresh data and whether data can be licensed every year.

Illustrative example: invented to show structure; it does not describe an available dataset.

refresh_delivery:
  dataset_id: support-sft
  version: v2026.10
  previous_version: v2026.07
  record_window: { closed_from: 2026-07-01, closed_to: 2026-09-30 }
  mode: incremental            # incremental | full
  change_types: [insert, update, tombstone]
  schema_version: 3            # unchanged since v2026.04
  format: jsonl                # one conversation per line, messages[role, content]
  rows: { insert: 18240, update: 312, tombstone: 41 }
  deidentification: { method: surrogate-replace, consistent_with: v2026.07 }
  eval_holdout: { rows: 500, never_train: true }
  manifest: manifest-v2026.10.sha256

Versioning, retention and deletion across refreshes

Keep every delivered version and every training mixture as an immutable snapshot, so that any checkpoint can be traced to the exact data used to produce it. Snapshots let you diff evals across versions and roll back a bad refresh. They also let you show which records were in which model when a supplier asks for records to be removed. Tombstones in an incremental feed only help if your merge job applies them before the next training run and you can identify which checkpoints still contain those records.

Provenance gaps add up across refreshes. The Data Provenance Initiative found license omissions and errors to be common on popular dataset hosting sites [7], so record the source, license version and delivery version for each row, not just for the whole dataset. As of October 2026, Article 53 of the AI Act has required providers of general-purpose AI models placed on the EU market to keep technical documentation, a copyright compliance policy and a public summary of training content since 2 August 2025 [8]. Versioned refresh records make these much easier to maintain.

To check what an existing corpus already contains, run a provenance audit before you add the next delivery.

Budgeting the re-tune, not just the data

A refresh costs more than the data itself. It includes eval runs, labeling or review of the new slice, compute for a full replay, and regression triage. If the budget covers only data, the tempting shortcut is to skip the replay and tune on the delta, the path most exposed to forgetting [1] and new-knowledge hallucination [2]. Plan each cycle as data intake, merge and dedupe, held-out eval, re-tune, regression gate and staged rollout, and keep the previous checkpoint deployable until the new one passes.

You can find more fine-tuning data topics in the fine-tuning and post-training data hub and the main AI data guide. Teams sourcing operational records for a recurring refresh can describe the data they need to SourceX.

Plan your fine-tuning refresh data purchase

SourceX sources operational datasets from US companies on request, such as support and sales histories, engineering records, documents, and finance and legal workflows, and manages the commercial process, including licensing agreements and ongoing purchases. Each dataset is rights-reviewed and delivered under a license that defines the records, allowed uses, term and delivery; nothing is contracted until a supplier agrees, and a request does not guarantee a match. Describe the refresh data your model needs.