Training data poisoning: acceptance tests for licensed datasets

What is training data poisoning?#

Training data poisoning is an integrity attack in which someone alters the data a model learns from so the trained model behaves the way the attacker wants. The OWASP Top 10 for LLM Applications (LLM04:2025, Data and Model Poisoning) defines it as manipulation of pre-training, fine-tuning or embedding data to introduce vulnerabilities, backdoors or biases, and notes the risk is highest with external data sources that may contain unverified content.

NIST's adversarial machine learning taxonomy (NIST AI 100-2e2025) distinguishes poisoning by goal and violation type. Availability poisoning, an availability violation, degrades a model broadly. Targeted poisoning and backdoor poisoning are both listed as integrity attacks: targeted poisoning changes the model's output on specific inputs, while backdoor poisoning ties a behavior to a trigger pattern, so the model acts normally until an input contains the trigger.

For a buyer, the practical distinction is between accidental contamination (noise, duplicates, mislabeled records, model-generated text) and deliberate manipulation. Both show up in the same acceptance tests, but deliberate poisoning is designed to pass casual inspection, which is why the tests below are aimed at small, concentrated anomalies rather than average quality.

Is a licensed dataset safer than scraped web data?#

A licensed dataset removes some poisoning paths and adds others, so it is safer only if the supplier's pipeline is controlled. The best-documented web-scale attacks depend on the open internet: Carlini et al. describe split-view poisoning, where content behind a URL changes between the time a curator indexes it and the time a trainer downloads it, and frontrunning poisoning, where malicious edits are timed just before a snapshot of a crowd-sourced site such as Wikipedia.

Data exported once from a company's own systems (a Zendesk ticket history, a Salesforce case archive, a NetSuite transaction ledger) is not exposed to those two attacks, because nobody outside the supplier can change it after export. The remaining paths are inside the pipeline: someone with write access to the source system or the export, a labeling vendor, a transformation script, or an unauthenticated delivery channel. Commissioned and crowd-labeled data add the labelers themselves as a path.

Use the checklist below to decide which evidence to ask for, based on where the data came from.

How many poisoned records does it take to matter?#

Research shows that a small absolute number of crafted examples can install a behavior in controlled experiments, so a low contamination percentage is not evidence of safety. Wan et al. reported that, in their instruction-tuning experiments, as few as 100 poison examples could make arbitrary trigger phrases produce consistently negative or degenerate outputs across many held-out tasks.

Souly et al. (2025) pretrained models from 600M to 13B parameters and found that a near-constant number of poisoned documents, around 250 in their setup, installed a trigger that made the model output gibberish, regardless of how much clean data the model saw. The tested behavior was a denial-of-service backdoor in models up to 13B parameters, so the result should not be read as showing that the same count works for larger models or more harmful behaviors.

The buyer takeaway is about test design, not alarm. Random sampling sized to estimate an average error rate will usually miss a few hundred crafted records in a million-record delivery. Tests should look for concentration: repeated rare strings, clusters of near-duplicates, one annotator or one batch that behaves differently.

Why can't you just test the model after training?#

Post-training tests are necessary but not sufficient, because a backdoored model can behave normally on every input that lacks the trigger. OWASP notes that backdoors may leave model behavior untouched until a trigger causes it to change, which makes them hard to test for and detect.

Hubinger et al. (Sleeper Agents) built proof-of-concept models with deliberately planted backdoors and found the behavior persisted through supervised fine-tuning, reinforcement learning and adversarial training, and that adversarial training could teach the model to recognize its trigger more reliably and hide the behavior. That study planted backdoors on purpose, but the lesson for buyers holds: safety tuning after the fact does not reliably clean a poisoned dataset out of a model.

One further limit matters for licensed data. Clean-label attacks (Shafahi et al., Poison Frogs) poison a model without changing any labels, so a label audit alone will not catch everything. And retraining after a bad dataset is found is expensive, so the cheapest control point is acceptance, before the data enters a training run.

Which acceptance tests should buyers run on a training data delivery?#

Run tests that verify the delivery is the data the supplier described, then tests that look for small, concentrated anomalies. The table is written so each row can go into an acceptance schedule with a pass criterion agreed before delivery; the thresholds are yours to set from your own baseline, not universal values.

What integrity evidence should a data supplier provide?#

Ask for evidence that connects each delivered record to its origin and shows who could change it along the way. OWASP's LLM04 mitigations include tracking data origins and transformations (it names OWASP CycloneDX and ML-BOM), vetting data vendors rigorously, using anomaly detection to filter adversarial data, and using data version control to track changes and detect manipulation. CycloneDX's ML-BOM format can describe datasets and their provenance alongside models, which gives buyer and supplier a shared structure for the record.

A supplier of operational records will rarely have a full ML-BOM, and that is fine. What matters is that the following exist in writing at delivery:

• 1Source statement: which system the records came from (for example ServiceTitan jobs or Jira issues), the date range, and the export method.
• 2Hash manifest generated at export time, before any transfer, with file names, sizes, record counts and hashes.
• 3Transformation log: every step between export and delivery (de-identification, filtering, format conversion), with the script or tool version.
• 4Write-access statement: who could create or edit records in the source system and in the export pipeline during the covered period.
• 5Labeling record, where labels exist: annotator or labeler ID per label, guidelines version, agreement and gold-question results.
• 6Versioned change log for any refresh, keyed by stable record IDs.
• 7Delivery channel description: how the files are transferred and how the recipient authenticates them.

What contract wording covers training data poisoning?#

Put the acceptance tests and integrity evidence into the agreement so a failed test has a defined remedy. The clauses below are starting wording for counsel to adapt; they complement, not replace, the title, rights and accuracy warranties a training data license already needs.

• 1Integrity warranty: Supplier warrants that, to its knowledge after reasonable inquiry, it has not inserted, and has not permitted any person to insert, records or labels into the Dataset for the purpose of altering the behavior of a model trained on it.
• 2Manifest: Supplier will deliver a manifest listing each file, its size, record count and SHA-256 hash, generated at export and before transfer. Files that do not match the manifest are not delivered for purposes of acceptance.
• 3Acceptance period: Buyer may run the acceptance tests in Schedule X within N business days of delivery. Buyer will notify Supplier of any failed test with the affected record IDs.
• 4Cure: On a failed test, Supplier will investigate, explain or remove the affected records, and re-deliver a corrected release with an updated manifest and change log within the agreed period.
• 5No unlogged changes: Supplier will not modify, add or remove records in a delivered release without a versioned change log keyed to stable record IDs.
• 6Provenance cooperation: Supplier will, on request and for a sample of records, show source system identifiers and export timestamps sufficient to trace each sampled record to its origin.
• 7Incident notice: Supplier will notify Buyer promptly after becoming aware of unauthorized access to the source data, export pipeline or labeling workflow during the covered period.

What should you do if an acceptance test fails?#

Quarantine the delivery and keep it out of every training, evaluation and retrieval pipeline until the failure is explained. A failed test is often benign (a support macro repeated thousands of times, a bulk import, a labeler who misread the guidelines), and the supplier can usually explain it from the source system.

Share the flagged record IDs and the test output with the supplier, ask for the matching rows from the export log, and decide per cluster: explained and kept, explained and removed, or unexplained and removed. Record the decision against the dataset version so your lineage records show which release was accepted and why. If a model has already trained on the data, use those lineage records to find every checkpoint that used it.

How this applies when sourcing operational data through SourceX#

SourceX sources datasets for AI buyers on request from U.S. businesses rather than holding them in stock, and buyers share scope, volume, format, timing and licensing needs through the request form. That makes it practical to name the acceptance tests and integrity evidence above in the request, before a supplier exports anything.

On the supply side, the company approves exactly what is released, once per deal, and de-identification is performed by SourceX directly or through its processing partner. Large deliveries stay in the seller's own storage or ship on encrypted drives, so agree manifest and hash checks for that transfer route as part of the delivery terms.