Skip to content

AI uses for records

Post-training negative transfer: can new data make an agent worse?

By SourceX Editorial · Updated

10 sections

Short answer

Yes, new training data can make an agent worse at its old job. To test for it, freeze a suite covering the agent's existing capabilities before training, then compare a candidate arm trained with the new data against a budget-matched baseline on the target failure and every protected capability, using repeated trials and regression margins set in advance. Trace any regression to the data slice that caused it.

Agent run

Post training negative

This page covers post-training negative transfer: can new data make an agent worse.Illustrative

Key takeaways

  • Narrow post-training data can lower an agent's performance on tasks it already handled, not only improve the target.
  • Freeze a capability inventory as a test suite before anyone sees the candidate data.
  • Compare a candidate arm with a budget-matched baseline, with repeated trials and margins fixed in advance.
  • Score consistency across trials, because agent regressions can appear there before single-trial scores move.
  • Trace a regression to a data slice by retraining without it before keeping or dropping the batch.

How do you test whether new training data harms existing agent capabilities?

Test it the way you would test a code change: freeze a suite that measures the agent's existing job before the new data is used, train a candidate arm with the new data and a baseline arm with the same budget and none of it, and compare both arms on the target failure and on every existing capability, with repeated trials and regression margins fixed in advance. If a capability falls past its margin, trace it to the slice of new data responsible before deciding anything.

The question matters because post-training data is narrow by design. A batch built to fix one failure, say an agent mishandling warranty claims, is concentrated on that task, and concentrated data can pull a model away from behavior it already had. That effect is called negative transfer when the new data hurts performance elsewhere, and it is easy to miss when evaluation looks only at the failure the data was bought to fix.

This guide gives a regression and negative transfer test plan for teams that post-train agents on new data, whether licensed, commissioned or generated. It covers what to freeze, how to set up the comparison, why agents need repeated trials, how to attribute a regression to a data slice, and how to read the outcome.

Negative transfer, forgetting and the alignment tax

These terms describe overlapping costs of the same move: training on new data can lower performance on what the model already did. In their survey, Zhang and colleagues define negative transfer as source data or knowledge undesirably reducing learning performance in the target domain. Catastrophic forgetting is the related loss of earlier skills during sequential training; Kirkpatrick and colleagues' elastic weight consolidation paper in PNAS addressed it by slowing learning on the weights most important to earlier tasks.

Post-training literature has its own version. OpenAI's InstructGPT paper reports that its fine-tuned models showed improvements in truthfulness and reductions in toxic output while having minimal performance regressions on public NLP datasets, which is a reminder that regressions were measured and not assumed away. Qi and colleagues showed a sharper case: fine-tuning aligned models even on benign, utility-oriented datasets can steer them away from harmlessness, and they note that catastrophic forgetting of initial safety alignment may happen during fine-tuning.

For an agent, the old job is wider than a benchmark score. It includes calling tools with valid arguments, following written business policy, finishing multi-turn tasks, refusing out-of-scope requests and keeping a stable output format. Any of these can regress while the target metric improves.

Write down the old job before the new data arrives

Build a capability inventory and freeze it as a test suite before anyone looks at the candidate data. The inventory lists every behavior the deployed agent is expected to keep, how to check it, and where the test cases come from. Cases must come from real tasks or carefully written scenarios that the candidate data does not contain.

General capability can be checked with open tooling. EleutherAI's lm-evaluation-harness provides a unified framework for testing generative language models on a large number of evaluation tasks, and OpenAI's Evals framework supports private evals built from a team's own data without exposing it publicly. Neither replaces business-task checks; they cover the broad floor beneath them.

Write down the old job before the new data arrives
Capability familyWhat to checkWhere cases come fromRegression signal
Tool useCorrect tool chosen, arguments valid against schema, no invented toolsLogged production tasks with known correct callsSchema errors or wrong tool rate rises
Policy adherenceDecisions follow the written business rules in forceReal cases with a policy reference and the correct outcomeMore rule violations on unchanged policies
Task completionMulti-turn tasks reach the correct end stateReplayable scenarios with a checkable final stateLower completion or more turns per success
Refusal and scopeDeclines out-of-scope or unsafe requests appropriatelyCurated refusal set plus real out-of-scope requestsFewer appropriate refusals or more over-refusals
Output contractFormat, language and required fields are preservedDownstream parser requirementsParser failures or missing fields
General floorBroad knowledge and reasoning stay near baselineOpen benchmark tasks run with a fixed harnessDrop beyond the pre-set margin

The regression and negative transfer test plan

Fill in every row before the first training run, and treat a row left blank as a reason not to start. The example entries are illustrative and show the level of specificity needed.

The regression and negative transfer test plan
Plan elementFix in advanceWhy it mattersExample entry (illustrative)
Target failureOne sentence naming the task, the wrong behavior and its business effectWithout it, any improvement can be claimedAgent denies valid warranty claims when the receipt is a photo
Target cohortsTwo to six slices where the failure concentratesShows where the data should helpPhoto receipts, expired-looking dates, multi-item orders
Frozen capability suiteInventory cases, version and hashPrevents tuning the test to the resultSuite v3, cases drawn from the prior quarter's tasks
ArmsBaseline continuation and candidate, same base checkpointIsolates the data from everything elseBaseline mix versus baseline mix plus warranty batch
Budget matchTraining examples or tokens, steps, learning rate and seeds held equalMore compute can masquerade as better dataSame token count; candidate replaces part of the baseline mix
Trials and seedsTrials per task and training seeds per armAgent outcomes vary run to runRepeated trials per task, more than one seed per arm
MetricsPer-cohort success, per-capability pass rates, consistency across trialsAverages hide trade-offsSuccess on all trials of a task, not only one
Regression marginsLargest drop accepted per capability, set before resultsStops post hoc rationalizingPolicy adherence may not fall at all; general floor small tolerance
Attribution methodHow a regression is traced to data slicesDecides whether to fix or drop dataLeave-one-slice-out reruns on the flagged capability
LineageData batch ids, filters, generator or source, and license referenceLets a later team reproduce or remove the dataBatch manifest with record ids and source agreement

Why one run per task hides an agent regression

Agents are stochastic over a whole conversation, so a single run per task can show an improvement or a regression that is only luck. Measure each task several times and score consistency, not just best-case success.

The tau-bench benchmark, which emulates conversations between a simulated user and an agent with domain API tools and policy guidelines, introduced pass^k for this reason: the probability that all k independent trials of a task succeed, averaged across tasks, as opposed to pass@k, where one success in k counts. Its authors propose pass^k for tasks that need reliability and consistency, such as customer service, and its leaderboard shows success falling as k rises for every model with reported scores.

Regressions can show up in consistency first. An agent may still solve an exchange task most of the time after new training but fail it on a larger share of repeated attempts. Report pass rates at more than one k for each capability in both arms, and set the regression margin on the consistency measure for any capability where a single failure is costly.

Which slice of the new data did the damage?

When a capability falls past its margin, find out which part of the new data caused it before deciding to keep or drop the batch. A batch is seldom uniform, and one source, template or label policy can explain most of the damage.

At a glance

Which slice of the new data did the damage

Use the plan's attribution method. Split the batch into slices along the lines most likely to matter: source system, task type, answer style, label policy, generator version. Retrain with each slice removed, at the same budget, and rerun only the regressed capability and the target cohort. If removing one slice restores the capability while keeping most of the target gain, that slice is the problem. Check a dose response as well, by training with a smaller share of the batch; a regression that scales with the share points to the data, while one that does not points elsewhere.

Training method interacts with the result. Biderman and colleagues report that LoRA learns less and forgets less than full fine-tuning in their experiments, so the same batch can regress a full fine-tune and not an adapter. Record the method in the plan and do not compare arms trained differently.

  • Content conflict: the new data teaches a rule that contradicts an existing policy, often from an outdated source document.
  • Style drift: answers in the new data are shorter, more terse or formatted differently, and the agent copies that everywhere.
  • Tool misuse in examples: trajectories call tools in ways that work for the target task but break others.
  • Refusal erosion: the batch contains no refusals, and compliance on out-of-scope requests rises.
  • Contamination: suite cases or close copies leaked into training, inflating the target and hiding a regression.

Worked example (illustrative): a returns agent learns warranty claims

This example is illustrative and uses counts and percentages only. A retailer's support agent handles returns, exchanges and order status, and fails warranty claims where the receipt is a photo. A batch of warranty conversations is prepared to fix it.

The candidate arm improves the photo-receipt cohort clearly. The frozen suite flags exchanges: completion on a single trial looks unchanged, but success across all repeated trials of each exchange task drops past its margin. Leave-one-slice-out shows the cause is a slice of warranty conversations from one store system that always ends with a refund, so the agent began offering refunds where policy calls for an exchange.

Worked example (illustrative): a returns agent learns warranty claims
Measure (illustrative)Baseline armCandidate armCandidate without suspect slice
Photo-receipt warranty cohort, all trials succeed41% of tasks73% of tasks70% of tasks
Exchange tasks, all trials succeed82% of tasks64% of tasks81% of tasks
Policy violations per 100 tasks393
Tool schema errors per 100 calls111
General floor suiteWithin marginWithin marginWithin margin

Real regression, noise, or a changed test harness?

Rule out the test harness before blaming the data. Agent evaluations depend on simulators, tool mocks, graders and policy documents that can change between runs, and any of them can create a regression the model did not cause.

At a glance

Real regression, noise, or a changed test harness

  • Grader version: the same grader, rubric and judge model scored both arms.
  • Tool mocks and environment: API stubs, databases and fixtures are identical and reset between trials.
  • Policy documents: the agent and grader saw the same policy version in both arms.
  • User simulator: if a model simulates the customer, its version and settings are pinned.
  • Seeds and sampling: temperature and decoding settings match, and more than one training seed agrees.
  • Suite integrity: the suite hash matches the frozen version, and no case changed after results were seen.

Reading the result: ship, reweight or drop the batch

Ship the batch only if the target cohort improves beyond noise and every capability stays within its pre-set margin. Reweight or clean when a regression traces to an identifiable slice that can be removed or fixed without losing the gain. Drop the batch when the gain disappears once the harmful slice is removed, or when no slice explains the regression.

One exception matters. Sometimes a regression is intended: a business retires an old policy, and the agent should stop doing what the suite rewarded. Update the suite first, with a dated record of why, and then rerun; never accept an unexplained regression on the grounds that it might be intended.

The plan has limits. A frozen suite covers known behavior, not every future one, and repeated trials cost compute. Narrow the plan to the capabilities most exposed to the new data when budget is tight, and bring in an evaluation lead when the target and a protected capability conflict by design.

Held-out business tasks for the next round

A regression plan is only as good as its held-out tasks, and the strongest ones come from real operating records: resolved tickets with policy references, order histories with known outcomes, and logged tool calls that can be replayed. Those records also make the frozen suite credible to anyone who later reviews the decision.

Held-out business tasks of this sort are located per request: SourceX looks for them among U.S. businesses instead of keeping a catalog, then handles rights review, contracting and preparation with the record owner. A request can name the capabilities to protect, the target failure, the volume, format and timing needed, and whether records are for training, held-out evaluation or both.

Frequently asked questions

What is negative transfer in post-training?

It is when training on new data lowers performance on something the model already did, outside the task the data was meant to improve. A survey by Zhang and colleagues defines it as source data or knowledge undesirably reducing performance in the target domain. In agents it can show up as policy or tool-use regressions.

Is negative transfer the same as catastrophic forgetting?

They overlap. Catastrophic forgetting describes losing earlier skills during sequential training, while negative transfer describes new data hurting performance elsewhere. In post-training both appear as regressions, and the same frozen suite detects them.

How large should the capability suite be?

Large enough that a regression past your margin is distinguishable from noise for each capability you protect. Start from the capabilities most exposed to the new data and grow the suite from real production failures.

Does mixing general data into the batch prevent regressions?

Blending the new batch with a share of the existing training mix can reduce them. It does not remove the need to test, because a single slice with a conflicting rule can still cause a targeted regression.

Should regression tests use real or synthetic tasks?

Use real tasks wherever possible for the frozen suite, because they reflect the policies and edge cases the agent actually meets. Synthetic scenarios can fill gaps but should be flagged and kept separate when reporting results.

Sources

  • Zhang and colleagues' survey defines negative transfer as source domain data or knowledge undesirably reducing learning performance in the target domain. Source
  • Kirkpatrick and colleagues (PNAS 2017) address catastrophic forgetting with elastic weight consolidation, which selectively slows learning on weights important to earlier tasks. Source
  • The InstructGPT paper reports improvements in truthfulness and reductions in toxic output with minimal performance regressions on public NLP datasets. Source
  • Qi and colleagues report that fine-tuning aligned models on benign utility-oriented data can steer them away from harmlessness, and that catastrophic forgetting of initial safety alignment may happen during fine-tuning. Source
  • EleutherAI's lm-evaluation-harness provides a unified framework to test generative language models on a large number of evaluation tasks. Source
  • OpenAI Evals is a framework for evaluating LLMs and LLM systems that supports private evals built from a team's own data without exposing it publicly. Source
  • tau-bench emulates dynamic conversations between a simulated user and a language agent with domain-specific API tools and policy guidelines, and reports Pass^1 to Pass^4 results. Source
  • The tau-bench paper defines pass^k as the probability that all k i.i.d. trials of a task succeed, averaged across tasks, for tasks needing reliability and consistency. Source
  • Biderman and colleagues report that LoRA learns less and forgets less than full fine-tuning in their experiments. Source

Tell us what data you need

Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.

See if your company qualifies