Unlawfully processed personal data in training sets: downstream consequences for model buyers and deployers

This page is general information, not legal advice. Confirm requirements with counsel for your jurisdiction and use case.

What Opinion 28/2024 actually decided about unlawful training

The EDPB treats unlawful development as a fact that supervisory authorities assess case by case, not as an automatic ban on every later use of the model [1]. The opinion was requested by the Irish supervisory authority and adopted in December 2024; it answers four questions, the last of which concerns how unlawful processing during development affects later processing or operation of the model [1][3]. Law-firm commentary on the opinion discusses what this question means for models trained on unlawfully processed personal data, which is where licensees and deployers are exposed [4][5].

The analysis rests on ordinary GDPR mechanics rather than a new AI rule. Lawfulness under Article 5(1)(a) and a legal basis under Article 6 attach to each processing operation, and Article 24 makes each controller responsible for demonstrating its own compliance [6]. The opinion's core move is to ask whether personal data is still "in" the model: if the model memorizes or can regurgitate training records, deployment is itself processing of that personal data, and the upstream defect is in play [1].

Two framing points matter for buyers. First, the AI Act does not displace this: Article 2(7) keeps Union data protection law fully applicable to personal data processed in connection with AI systems [7]. Second, as of October 2026 the Digital Omnibus proposals that would narrow the definition of personal data were still proposals, not law, so the Opinion 28/2024 analysis remains the operative reference [11].

The three scenarios and where a buyer sits

Your exposure depends on which scenario describes your relationship to the model and whether personal data survives in the weights. The opinion distinguishes three situations [1][2].

Scenario 3 is the escape hatch, and it is narrow. The anonymity bar in the opinion requires that the likelihood of extracting personal data from the model, directly or through queries, be insignificant, which is a high threshold under Recital 26 [1][6]. Our companion guide on when a model trained on personal data counts as anonymous covers the tests; this page assumes you cannot yet rely on them.

What "appropriate assessment" means for a deploying controller

The deployer's duty is evidentiary: you must be able to show you asked the right questions about the model's training data and acted on the answers. The opinion points supervisory authorities toward factors such as the source of the data and whether the developer's processing has already been found to infringe the GDPR by an authority or a court [1]. A model built from a personal-data breach, or from a scrape an authority has already condemned, is a red flag a deployer cannot claim not to have seen.

The opinion also notes that the depth of the assessment can scale with risk [1]. A customer-facing assistant handling health or financial queries, or a model that will be fine-tuned further on employee records, warrants more scrutiny than a narrow classifier with no generative output. Where the provider is subject to AI Act documentation duties, those artifacts are useful inputs but not a substitute for your own GDPR accountability [1][7].

Useful upstream evidence usually exists if you ask for it. General-purpose model providers must publish a training-content summary under Article 53(1)(d) using the Commission's July 2025 template [9]. For high-risk systems, Article 10 requires data governance covering data collection processes and the origin of data, which gives buyers a concrete document set to request [8]. As of October 2026, Article 10's high-risk dates have reportedly moved to December 2027 (Annex III) and August 2028 (Annex I) under Regulation (EU) 2026/1744, but the documentation it describes is already a sound diligence baseline [8].

Why provenance claims fail under scrutiny

Most training-data defects surface as broken provenance chains rather than obvious misconduct. The Data Provenance Initiative's audit of more than 1,800 text datasets found license omission above 70% and license error rates above 50% on popular hosting sites [10]. If licenses are this unreliable, the lawful-basis and consent metadata behind personal data is unlikely to be better.

Recurring failure modes buyers should test for:

• Purpose drift. CRM, ticketing or call-recording data collected for service delivery was repurposed for model training without a compatibility assessment or a new legal basis.
• Broken processor chains. A vendor acting as a processor under Article 28 used client data to train its own model, which can make it a controller for that processing without a legal basis [6].
• Laundering through intermediaries. A reseller or aggregator cannot hand over rights it never received; see our guide to buying training data through a broker or reseller.
• Ignored objections and erasure. Article 21 objections or Article 17 erasure requests honored in the source system but not propagated to training snapshots.
• Special-category leakage. Health, biometric or union-membership data inside free-text fields, where Article 9 conditions were never met [6].
• "Public" equals "lawful." Scraped forum posts or profile pages treated as free to use; the opinion's legitimate-interest analysis does not support that shortcut [1].

These are also the reasons many buyers prefer licensed operational data with documented rights over scraped corpora; our comparison of licensed, synthetic and scraped training data sets out the trade-offs.

Deployer diligence request for a third-party model or dataset

A written request, answered in writing and kept on file, is the core of a defensible assessment. The template below adapts the opinion's factors into questions a governance lead can send to a model developer or dataset supplier [1][8][9]. Pair it with our broader data provider due diligence questionnaire.

Illustrative example: invented to show structure; it does not describe an available dataset.

For memorization testing in item 6, see training-data extraction and memorization risk; for item 5 evidence, see the de-identification evidence package checklist.

Corrective measures and what remediation looks like

Supervisory authorities have the full Article 58(2) toolkit, and the opinion lists outcomes up to erasure of the model itself [1][6]. Measures named include fines, temporary limitations on processing, erasure of the unlawfully processed part of the dataset and, where that is not possible, erasure of the whole dataset or the model, depending on the facts [1]. For a deployer, the practical consequence is operational: a model you depend on may have to be withdrawn or retrained on a timetable you do not control.

Remediation options, roughly in order of cost:

1. Suspend the affected feature while the developer's position is clarified, and document the decision.
2. Block regurgitation with output filtering and retrieval controls; this reduces risk but does not cure unlawful development.
3. Demand a cleaned retrain excluding the tainted sources, with a written description of what was removed and how.
4. Demonstrate anonymity of the retrained or existing model under the opinion's tests, which moves you toward Scenario 3 [1].
5. Replace the model with one whose provenance you can evidence end to end.

If the defect is in a dataset you licensed directly, follow your response playbook for personal data found in a licensed dataset and check the record-level takedown obligations in your license. Weights already trained on the data are the hard part: deleting the source records does not, by itself, remove what the model learned.

Allocating the risk in contracts

Contracts cannot make unlawful processing lawful, but they decide who pays and who must cooperate. Because each controller carries its own accountability, an indemnity from the developer does not discharge your duty to assess; it only funds the response [1][6]. Draft for the response you will need, not just for damages. Teams that request licensed datasets through SourceX's buyer intake still need these clauses in every license they sign.

Clauses buyers commonly negotiate with model and data suppliers:

• Provenance warranty that personal data in training sets was processed with a valid legal basis, tied to the source register in the diligence request.
• Prompt notification of any supervisory inquiry, complaint or finding touching the training processing.
• Cooperation and retraining duty, including delivery of a cleaned checkpoint or dataset version on request.
• Suspension right for the buyer without breach if an authority's action makes continued use doubtful.
• Audit or attestation rights over de-identification and memorization testing.

Where your own lawful basis for further fine-tuning rests on legitimate interest, document it separately; see the legitimate-interest assessment buyers must document. The privacy and de-identification hub collects the related guides.

Sourcing training data with documented rights

The cheapest way to avoid an unlawful-training problem is to start from data whose ownership, consents and preparation are documented before you train. SourceX sources operational datasets from US companies on request, rights-reviews each one for ownership and consents, removes or replaces personal details before delivery with the method recorded, and prepares diligence materials per dataset, all under a license that defines records, uses, term and delivery. Describe the data you need at sourcex.si/buyers.