AI uses for records
Benchmark contamination canary tests: a consented design with known limits
By SourceX Editorial · Reviewed by Noah Loul ·
10 sections
Short answer
Canary records help investigate unauthorized benchmark exposure by planting unique, consented, fictional items with random secret answers in each copy or split of a private benchmark, then testing whether a model prefers those secrets over matched unplanted twins. A hit is evidence that one copy or split reached training; a clean probe proves little, because deduplication, rewording and low repetition defeat canaries.
Evaluation split
Benchmark contamination
Key takeaways
- A canary string is a filtering request; canary records with random secrets are a test.
- Plant several canaries per copy, entity group, time window or task family so a hit points to one partition.
- Random secrets make the false positive rate computable; fix alpha and corrections before the first probe.
- Negative probes are weak evidence: deduplication, filtering, rewording and low repetition all hide exposure.
- Consent means the records owner agrees, recipients are told copies may be marked, and no canary uses a real person.
How can canary records help investigate unauthorized benchmark exposure?
Canary records help by turning a suspicion into a test with a known chance of a false alarm. You plant a small number of unique, fictional items, each with a secret answer that could not be guessed, in a private benchmark before anyone outside the evaluation team receives it. Later you ask a model to complete those items and compare its preference for the secret answers with its preference for matched alternatives that were never planted.
If a model reliably prefers the planted secrets, the copy or split that carried them probably reached its training data. If it does not, you have learned much less than it seems: canaries are easy to lose to deduplication, filtering and paraphrase, and some models barely memorize rare text. The design below is built around that asymmetry, with the false positive budget fixed before the first probe.
Canary string, canary record, watermark or backdoor: which design answers your question?
Pick the design by the question you need answered, not by habit. A set-level canary string asks corpus builders to filter the benchmark out; it is a request, not a detector. BIG-bench put a canary string in every task file to keep tasks out of web-scraped training data and asks contributors never to edit it. The other designs plant something a model can only know by training on it.
| Design | What is planted | How exposure is tested | Main weakness |
|---|---|---|---|
| Set-level canary string | One GUID in every file of the benchmark | Filtering by corpus builders; asking a model to complete the GUID | Only as good as the filter; travels with the file, not the items |
| Per-copy canary records | Fictional items with random secret answers, unique to each copy or partition | Rank of the secret among candidates of the same format | Lost to deduplication, filtering or rewording; needs repetition to be memorized |
| Random-sequence watermarks | Random character sequences or lookalike substitutions in documents | Hypothesis test on model likelihood with a bounded false detection rate | Power falls as the training set grows; needs several marked documents |
| Backdoor items with random targets | Items whose trigger maps to a randomly chosen answer | Count of backdoors a model activates; false positive rate computed exactly | Few backdoors give a weak bound; the items sit inside the test set |
| Ordering test, nothing planted | Nothing; uses the benchmark's published order | Likelihood of the canonical order vs shuffled orders | Only detects exposure to that ordered file, not to reordered copies |
The consented canary design, field by field
Consent here means three things. The owner of the records the benchmark was built from agrees in writing that fictional items may be added. Every recipient's agreement says copies may carry recipient-specific markers, without saying which items they are. And no canary uses a real person, a real account or a real contact point, so a canary that escapes harms nobody. Have counsel check that the marker clause does not contradict any promise that every record is genuine.
The table is the design record. Fill every row before planting, and keep the filled copy in the canary registry, outside the benchmark files.
| Field | Specification | Why it matters |
|---|---|---|
| Partition key | One canary set per recipient copy, and per entity group, time window or task family where those are split | A hit then points to a copy or split, not just to the benchmark |
| Canaries per partition | Several independent canaries, never one; count set by the false positive budget | One canary gives a weak test; several give a usable bound |
| Secret format | Random value from a large, stated candidate space, for example a 10-character code in a ticket reference field | Known candidate space gives a known chance hit rate |
| Carrier | A realistic item of the same type as its neighbors, with the secret in a field the task asks for | Odd-looking items are filtered out or spotted by annotators |
| Twin set | Unplanted canaries from the same template, kept in the registry only | Twins measure how a clean model scores the template |
| Repetition | Planned number of occurrences per canary within the copy, recorded | Single occurrences are often not memorized |
| Scoring rule | Canaries excluded from every reported benchmark score | Fictional items must not move the leaderboard |
| Decision rule | Alpha per probe, correction for the number of models and cycles, written before probing | Prevents finding a hit by probing until one appears |
| Registry | Canary id, partition, secret, twins, recipient, delivery date, file hash | Lets a hit be traced and audited later |
Placing canaries by company, task family and time so a hit points somewhere
Canaries can only localize a leak along the lines you split on. If the benchmark is split by company, give each company group its own canaries; if it has a temporal holdout, give each window its own; if vendors each received a different subset, give each copy its own. A hit on one partition's canaries then says which split or copy reached training.
Keep the canary density low enough that it does not change what the partition measures. A handful of canaries in a partition of a few hundred tasks is invisible in the task-family mix; dozens are not, and annotators start to notice the pattern. If a partition is too small to hide several canaries, merge it with a neighbor for canary purposes and accept that a hit will localize less precisely.
Canaries do not repair a bad split. If the same company, customer or document thread appears in both the training pool and the test set, a model can answer test items from legitimate training data, and no canary will show it. Split first by company, entity group, time and task family, check for near-duplicates across the boundary, and only then plant canaries to watch the boundary you built.
Running the probe and fixing the false positive budget
Probe each model with the canary prompts and with the twin prompts, and score the rank of the true secret among a sample of candidates from the same format. This is the exposure idea from the Secret Sharer work: compare the model's likelihood for an inserted secret with its likelihood for random sequences of the same format that were never inserted. A memorized canary ranks near the top; an unseen one ranks anywhere.
The random secret is what makes the false positive rate computable. Under the null hypothesis that a model never saw the canary, its chance of ranking the true secret first among N equally formatted candidates is about 1 in N, and the twins let you check that the template behaves that way. With several independent canaries per partition, the chance that a clean model ranks several of them first is small and can be computed with a binomial calculation.
Work in the literature shows why the count matters. DyePack hides several backdoors with random targets in a test set so that the false positive rate of flagging a model can be computed exactly; its authors report that one or two backdoors leave the rate high, while around eight give very low rates. The data watermark study by Wei and colleagues frames detection as a hypothesis test with a bounded false detection rate and finds that watermark length, duplication and interference all affect its power.
- Probe on a schedule tied to model releases, not continuously, so each probe maps to one model version and the multiple-testing count stays small.
- Fix alpha, the number of models you will probe and the number of cycles before the first probe, and correct for all of them.
- Probe the twins with every model; if clean twins rank high, the template is guessable and the canary set is void.
- Log the model snapshot, prompt, decoding settings and date of every probe.
- Report a hit as evidence about one partition and one model version, with its p-value, never as a verdict about the vendor.
Why a clean probe proves little: the false negative problem
A negative result usually means the test lacked power, not that the model is clean. Copyright trap experiments by Meeus and colleagues found that medium-length trap sentences repeated 100 times were not detectable with existing methods, while longer sequences repeated many times could be detected, and that traps work less well on smaller models that memorize less.
Several ordinary data practices also erase canaries before training. Exact deduplication drops repeated items, quality filters drop odd-looking ones, and a team that rewrites items into a new format may change the secret field. Exposure can also travel without the canaries: a vendor that trained only on the answer column, or on summaries of the tasks, may never have seen a canary item at all.
When planting is not possible, for example because the benchmark is already public, the ordering test by Oren and colleagues offers a black-box alternative: a model that prefers the canonical order of a dataset over shuffled orders must have seen that ordering. It answers a narrower question, exposure to that ordered file, and needs no canaries.
Answer keys, attachments and vendor exports need canaries too
Plant canaries in the surrounding files as well as in the items, because answers can leak through keys, attachments and exports just as easily as through the items themselves. Treat every artifact that contains or implies an answer as a carrier.
- Answer keys and grader rubrics: plant a canary answer in the key file as well as the item, since keys are often shared separately.
- Attachments: invoices, PDFs or screenshots that the task refers to can carry a canary value in a field the answer depends on.
- Annotation exports: a vendor's labeled copy of the benchmark is a derivative; give it its own canary set if it returns to you.
- Summaries and paraphrases: a canary secret phrased as an exact code survives summarization better than one embedded in prose.
- Bug reports and tickets about the benchmark: your own team can leak a canary into a public tracker, which later looks like a vendor leak.
Worked example (illustrative): a support-ticket benchmark shared with three vendors
Everything in this example is illustrative, invented to show how the design reads; it reports no results. An evaluation team built a private benchmark of 600 support-ticket tasks from one company's records, split by customer group and by quarter, and sent copies to three annotation vendors. Each copy carried eight canary tickets unique to that vendor, each with a random 10-character order reference as the secret answer and a matched set of unplanted twins in the registry.
The steps below show the order of work and what each step lets the team say. The point of the example is the reading in the last column: each conclusion is narrower than the instinctive one.
| Step | What the team did | What it can conclude |
|---|---|---|
| Before release | Wrote alpha, three vendors, four probe cycles and the correction into the registry | Any later hit is judged against a rule set in advance |
| Twin check | Probed twins on two public models; true secrets ranked no better than chance | The template is not guessable |
| Cycle 2 probe | A new model ranked six of vendor B's eight secrets first; none of A's or C's | Evidence that vendor B's copy reached training for that model version |
| Cross-check | Searched internal trackers and shared drives for vendor B's secrets; none found | Rules out the team's own leak as the source |
| Response | Retired vendor B's partition from scoring, issued a fresh copy and referred the contract question to counsel | Scores stay comparable; enforcement is a separate decision |
When a hit is not enough, and who should review it
Stop and narrow the claim when the twins rank high, when the canaries were ever pasted into a public channel, or when probing used a prompt that already contained the secret. Each of these produces hits from clean models. Also be careful with repeated probing through a hosted API: probe prompts are themselves data you have sent to that provider, so keep them free of secrets and check the provider's retention terms.
Bring in a dataset engineer to confirm the split and registry, and the evaluation lead to confirm the decision rule was followed. Anything that turns into a contract dispute belongs with counsel, because a canary hit is technical evidence about one model version, not a finding about who breached what.
Never-published business records for benchmarks you can watch
Canaries work best in a benchmark nobody has seen, built from records that were never public. SourceX assembles datasets from U.S. businesses only once a team asks and resells nothing from stock, so an evaluation team can specify the task families, time windows, volume and licensing terms a private benchmark needs. Before delivery, whichever business provides the records approves scope and terms and signs, and that is the moment to settle the marker clause.
Frequently asked questions
Is a canary string enough to keep a benchmark out of training data?
No. A set-level canary string asks corpus builders to filter the benchmark out, and it only works if they do. It also travels with the file, so items copied into prompts, spreadsheets or papers lose it.
How many canary records does each copy need?
Enough that a clean model is very unlikely to match several of them by chance, given the candidate space and the number of models and cycles you will probe. Backdoor studies show that one or two planted items leave a high false positive rate while around eight give a very low one, so plan for several per partition.
Does a negative canary probe prove a model never saw the benchmark?
No. Deduplication, filtering, rewording, low repetition and models that memorize little all hide real exposure. Report a negative result as no evidence of exposure from this test.
Can canary records distort benchmark scores?
Only if they are scored. Exclude every canary from reported results, keep the registry outside the benchmark files, and check that canary items do not change the task-family mix.
Who should approve planting canaries in a benchmark built from business records?
The owner of the underlying records, the evaluation lead and counsel for the recipient agreements. The records owner must accept fictional items in the delivered copies, and recipients should be told copies may be marked.
Sources
- The BIG-bench README says all task files contain a canary string, which should not be edited, to prevent benchmark tasks from leaking into web-scraped training data. Source
- The Secret Sharer paper inserts random canary sequences into training data and measures exposure by comparing their perplexity with equivalent random sequences that were never inserted. Source
- Oren and colleagues test for test set contamination in black-box models by checking whether a model assigns higher likelihood to a dataset's canonical ordering than to shuffled orderings. Source
- Copyright trap experiments found that medium-length trap sequences repeated 100 times were not detectable, longer sequences repeated many times could be detected, and traps work less well on models that memorize less. Source
- Wei, Wang and Jia frame detection of watermarked documents in pretraining data as a hypothesis test with a bounded false detection rate and find that watermark length, duplication and interference affect its power. Source
- DyePack plants multiple backdoors with random targets in a test set so the false positive rate of flagging a model can be computed exactly, and reports that more backdoors give much lower false positive rates. Source
Tell us what data you need
Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.