Skip to content

AI data market

Post-training task contamination controls: keeping evaluation cases out of the training loop

By SourceX Editorial · Updated

9 sections

Short answer

Prevent evaluation cases entering a post-training loop by separating them by access, not convention. Keep items in a store data producers cannot read, pass only cohort failure descriptions and scores back, run an overlap gate on every training, preference, judge or seed batch before each job, iterate on a development split, and keep lineage that proves each control ran. Retire and replace any item that leaks.

Evaluation split

Post training task

This page covers post-training task contamination controls: keeping evaluation cases out of the training loop.Illustrative

Key takeaways

  • In post-training, contamination usually comes from the team's own loop: error analysis, authoring, synthetic seeds, preference data, RL environments and checkpoint selection.
  • Data producers should receive written cohort failure descriptions, never evaluation items or near-quotes.
  • An overlap gate with n-gram and embedding matching should run on the exact batch each job reads, and a job with unresolved matches should not start.
  • Every use of the final test to make a choice spends some of its independence, so iterate on a development split and count final-test runs.
  • Items found in training are retired and replaced, and scores are re-baselined, with the evidence kept.

How do you prevent evaluation cases entering a post-training feedback loop?

Separate the evaluation set by access, not by convention. Keep evaluation items in their own store that the people and pipelines producing training data cannot read, let only cohort-level failure descriptions and scores cross back, run an overlap check between every candidate training batch and the evaluation items before each job, and record the lineage that proves it. Use a development split for iteration so the final test is consulted rarely.

Most contamination advice is about the outside world: benchmarks scraped from the web, eval sets sent to hosted APIs, vendors who see the items. Post-training adds a closer threat. The team that measures a failure is often the same team that writes or buys the data to fix it, and the fastest way to fix a failing case is to train on something that looks just like it. Each step is reasonable. The result is a test that measures memory of the test.

The protocol in this guide treats the post-training loop as the risk surface: error analysis, data authoring, synthetic generation, preference collection, reward models, environment building and checkpoint selection. It assumes you already have a held-out set worth protecting.

Seven internal routes from the test set into training

Contamination inside a post-training loop rarely looks like copying a file. It arrives through ordinary work, which is why each route needs its own control and its own evidence that the control ran.

Seven internal routes from the test set into training
RouteHow it happensControlEvidence to keep
Error analysis to authoringAnnotators read failing eval items and write training examples that mirror themAuthors see failure descriptions, never itemsAccess log for the eval store; brief templates used
Synthetic generation seedsFailing eval prompts are used as seeds or few-shot examples for a generatorGenerator seed pools exclude eval items by ID and by overlap checkSeed pool manifest with overlap report
Preference and judge dataEval prompts appear in preference pairs, or a judge is tuned on eval-like ratingsSame overlap gate for preference and judge training dataGate report per preference batch
Environment and task reuseRL tasks or tools built from eval scenarios become training environmentsEnvironment specs are reviewed against the eval task list before useEnvironment register with source tasks
Checkpoint and setting selectionThe final test is used to choose checkpoints, mixtures or prompts repeatedlySelection on a development split; final test opened at release gates onlyCount of final-test runs per release
Production log feedbackEval traffic sent through a production endpoint lands in logs later mined for trainingTag eval traffic and filter it from log exportsFilter rule and excluded-row count
Shared preprocessingNormalizers, vocabularies or filters fitted on all data including the testFit transforms on training data onlyPipeline config showing fit scope

Who may see evaluation items, and what crosses the wall instead?

Only the evaluation owners and the automated scorer see items; everyone who produces training data sees descriptions. That one asymmetry closes most of the routes above, because an author cannot mirror an item they have never read.

Define four roles. The evaluation owner curates, versions and retires items and approves any access. The scorer is a service that runs models against items and returns per-item results only to the owner and aggregate results to everyone else. Data producers, whether staff annotators, synthetic pipelines or outside suppliers, receive a written failure description per cohort. Reviewers audit logs and gate reports but do not author data.

The failure description is the hand-off that replaces the item. It names the cohort, the input pattern in general terms, what the model did, what a correct response requires and the policies or fields involved, without quoting the item. A useful test: if two different authors could write a matching training example from the description alone, it is specific enough; if they could reproduce an eval item from it, it is too specific.

The same principle appears in ordinary machine learning practice. The scikit-learn documentation on common pitfalls says test data should never be used to make choices about the model, and that preprocessing should be learned from the training data only. A post-training team applies the same rule to people and pipelines, not just to fit calls.

Outside suppliers belong on the data-producer side of the wall. When you commission fine-tuning or preference data from a vendor, send the failure descriptions, not the failing items, and say in the order that the vendor must not request, receive or reuse evaluation material. If a vendor also builds your evaluation set, keep the two engagements under different people and different delivery locations, and run the overlap gate on everything the vendor sends for training, whatever the vendor's own checks say.

The training access and evaluation separation protocol

Adopt the ten controls below as a written protocol, assign each an owner, and keep the evidence for every training run. The protocol is versioned with the evaluation set, so a reviewer can see which controls applied to which result.

The training access and evaluation separation protocol
ControlRuleOwnerEvidence retained
1. Separate storeEvaluation items live in a store with its own access list; training buckets cannot read itEvaluation ownerAccess list and quarterly access review
2. Item identityEvery item has a stable ID, a content hash and a versionEvaluation ownerItem register
3. Description hand-offData producers receive cohort failure descriptions, never items or near-quotesEvaluation ownerDescriptions with reviewer sign-off
4. Overlap gateEvery candidate training, preference, judge or seed batch is checked against items before a job startsTraining leadGate report with match scores and removed rows
5. Split disciplineIteration uses a development split; the final test opens only at release gatesEvaluation ownerLog of final-test runs
6. Traffic taggingEvaluation calls are tagged and excluded from any log export used for trainingPlatform ownerFilter configuration and excluded counts
7. Environment reviewNew RL environments and tool tasks are compared with eval scenarios before useTraining leadEnvironment register entries
8. Canary markingItems and their documentation carry a canary string so leaks into scraped or shared corpora can be detectedEvaluation ownerCanary value and search results
9. LineageEach training example records its source, producer, brief or seed, and gate resultData operationsLineage fields in the dataset manifest
10. Retire and replaceAny item found in training is retired, replaced and scores are re-baselinedEvaluation ownerRetirement log and before-and-after scores

Running the overlap gate before every training job

Run it on the exact batch the job will read, after all transformations, and fail the job if unresolved matches remain. Checking the source dataset and then applying a paraphrasing or templating step afterwards defeats the gate.

N-gram overlap is the established starting point. EleutherAI's evaluation harness documents a decontamination procedure based on Appendix C of the GPT-3 paper, which treated a test document as contaminated if any n-gram overlapped a training document, with n between 8 and 13 depending on the dataset; the harness uses 13. AllenAI's open-instruct repository provides scripts that index instruction-tuning data and query it with test sets using n-gram, full-text or embedding matches, and can remove matching training instances.

Two cautions from those same tools. First, exact and n-gram matches miss paraphrases, which is exactly what an author working from an item would produce, so add an embedding-based match for post-training data and review the near misses by hand. Second, open-instruct's documentation notes that its search backend returns a limited number of hits, so a decontamination pass can miss matches; rerun the check after removal.

Canary strings cover a different route. BIG-bench places a canary string in every task file to help prevent tasks from leaking into web-scraped training data. Inside a company, the same idea helps detect items that escaped into shared drives, tickets or vendor deliveries that later feed training.

Decide in advance what happens to a match. A practical rule is three bands: exact or long n-gram matches are removed automatically, embedding matches above an agreed similarity go to a reviewer who removes or keeps each one with a reason, and everything below is kept. Write the thresholds into the protocol version, keep the reviewer's decisions in the gate report, and treat a sudden rise in matches as a signal that an item has leaked somewhere upstream, not just as rows to delete.

Did the target cohort really improve without breaking what already worked?

Answer it on the final test only after the comparison is fixed in advance: the target cohort, the guard cohorts that must not regress, the baseline checkpoint and the margin that counts as a change. Deciding which cohorts to report after seeing the results is a quiet form of contamination.

Repeated use of a holdout is the subtle risk. Dwork and colleagues showed in Science that adaptive reuse of a holdout set can lead to overfitting the holdout itself, and proposed a mechanism for reusing it safely. The practical lesson for a post-training team is simpler: every time the final test chooses a checkpoint, a mixture or a prompt, some of its independence is spent. Count those uses and keep most of them on the development split.

Time can also separate the two sides. LiveCodeBench collects new problems over time and lets users score models on problems released within a chosen window, so results can be restricted to problems published after a model's training data was gathered. For business tasks, a holdout drawn from records created after the training data was frozen gives a similar check that cannot be contaminated by anything that existed earlier.

  • Target cohort: did the failing cohort improve by more than the agreed margin on final-test items no data producer saw?
  • Guard cohorts: did any cohort the model already handled fall by more than the margin?
  • Format and safety guards: do refusal behavior, output format and tool-call validity hold steady?
  • Time-split check: does the gain hold on records created after the training data was frozen?
  • Gate audit: did every training batch in the run pass the overlap gate, with the report attached?

Worked example (illustrative): a claims-intake assistant that kept failing on multi-policy submissions

Consider a team fine-tuning an assistant that reads insurance submission emails and fills an intake form. All counts here are illustrative.

The evaluation owner reports that the model mishandles submissions covering several policies at once: of 120 final-test items in that cohort, the assistant fills the form correctly on 54. Under the old way of working, an annotator would open the failing items and write 500 similar emails. Under the protocol, the owner writes a two-paragraph failure description instead: multi-policy submissions where policy numbers appear in attachments rather than the body, and the model merges coverages that must stay separate.

Authors and a synthetic pipeline produce 800 candidate examples from that description. The overlap gate flags 6 by n-gram match and 19 by embedding similarity; on review, 4 of the embedding matches turn out to be close paraphrases of real final-test items, because one author had seen a forwarded email in a shared channel. All 25 flagged examples are removed, the forwarded email is traced, and the 4 items it touched are retired and replaced with new ones from the evaluation owner's reserve.

Iteration runs against the development split. When the release gate opens, the final test shows the cohort at 83 of 120, the guard cohorts within margin and the gain holding on a time-split set of submissions received after the data freeze. Without the gate, the 4 leaked items would have been scored as wins, and nobody would have known which part of the gain was real.

Legitimate crossings, and the limits of separation

Some crossings are legitimate when they are deliberate and recorded. A retired evaluation item may move into training after it is replaced and the evaluation set is re-baselined. Public benchmark items you never planned to use for decisions can be trained on if you stop reporting that benchmark. A development split exists to be looked at, and authors may read it, as long as the final test stays closed.

Agent and tool-use training needs one more check. When an evaluation task is defined by an environment state, a tool sequence and a goal rather than by text, an n-gram gate sees nothing to match. Compare new training environments with evaluation scenarios on their structure instead: the starting state, the tools available, the goal and the success check. Two tasks that share all four are the same task with different wording, and only one of them can stay on the evaluation side.

What the protocol cannot do is also worth stating plainly. It controls what your team and pipelines touch; it does not tell you what a base model saw in pretraining, which is where time-split holdouts and externally sourced, never-published records help. Overlap gates reduce but do not eliminate paraphrase leakage. And a small team where one person both evaluates and authors will need compensating controls, such as a second reviewer for failure descriptions and a periodic audit of gate reports, rather than pure role separation.

If an audit finds that training data was derived from final-test items and the results have already been published or shared with a customer, treat it as a reporting issue as well as a technical one: retire the items, re-run the comparison and correct the record.

Licensing held-out records for the evaluation side

The cleanest evaluation items are ones your training pipeline could never have seen: real business records that were not published, licensed for evaluation and held by the evaluation owner from the day they arrive. Licensing them for evaluation only, and saying so in the agreement, puts a contractual wall behind the technical one.

SourceX starts that search at U.S. businesses only once a team has asked, so the request can state an evaluation-only use and delivery straight to the evaluation owner, next to scope, volume, language, format, timing and licensing needs. Those terms then sit in the agreement the supplying business signs before any record reaches the team.

Frequently asked questions

What is post-training task contamination?

It is the presence in fine-tuning, preference, reward or RL training data of evaluation items, or examples derived from them, so that later scores reflect exposure to the test rather than learned ability. In post-training it often comes from the team's own error analysis and data authoring rather than from the web.

Is it acceptable for annotators to look at failing evaluation items?

Not for the final test. Give annotators a written failure description of the cohort instead, and let them read a development split if they need concrete examples.

Which n-gram size should the overlap gate use?

EleutherAI's harness uses 13-grams, following the GPT-3 paper, which used values between 8 and 13 depending on the dataset. For post-training data, pair an n-gram check with an embedding-based match and review near misses, because paraphrased items slip past exact overlap.

Do canary strings stop contamination?

No. They help detect it. A canary marks evaluation material so that its appearance in a scraped corpus, shared drive or vendor delivery can be found, but they do nothing about paraphrases written from the items.

How often should the final test be used?

As rarely as practical, typically at release gates. Each use to choose a checkpoint or setting reduces its independence, so record the number of final-test runs per release and do day-to-day selection on a development split.

Sources

  • scikit-learn's common pitfalls guide says test data should never be used to make choices about the model and transformations should be learned from training data only. Source
  • EleutherAI's evaluation harness bases decontamination on GPT-3 Appendix C n-gram overlap (N from 8 to 13 there) and uses 13-grams. Source
  • AllenAI open-instruct scripts index instruction-tuning data and query it with test sets by n-gram, full-text or embedding match, can remove matches, and may miss some matches. Source
  • BIG-bench task files carry a canary string intended to prevent tasks leaking into web-scraped training data. Source
  • LiveCodeBench continuously collects new problems and scores models over chosen release-date windows to avoid contamination. Source
  • Dwork et al. (Science, 2015) address overfitting from adaptive reuse of a holdout set and show how to reuse it safely. Source

Tell us what data you need

Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.

See if your company qualifies