Training and evaluation data for legal AI
Legal AI needs private legal work product on top of public case law and statutes: contracts traced from first draft through each round of redlines to the signed version, the playbooks and fallback positions behind each change, and the edits and approvals of supervising lawyers. SourceX sources negotiation histories, matter files, playbooks and review records from law firms and in-house legal teams, with privilege, client confidentiality and counterparty content reviewed before anything is licensed.
Dataset types to start with
- Contract negotiation and redline histories
Version chains with tracked changes, negotiation emails and the executed agreement show which positions were proposed, conceded or held, so a redlining model learns the markup and an evaluator can check its proposals against what was finally signed.
- Enterprise document archives
Matter folders mix agreements, amendments, side letters, memos, scans and correspondence, the material that due diligence, clause extraction and legal search tools have to read together.
- SOPs, playbooks and internal knowledge bases
Playbooks, clause libraries, approval matrices and fallback positions are the rules a review agent is meant to apply. Set against redlines, they show where negotiators held the standard position and where an approved exception was made.
- Human feedback and QA-scored work
Senior lawyers' edits to junior drafts, approval and escalation decisions and second-level review overturns grade legal work the way a supervisor does, supplying preference signals and rubrics.
Why this data is hard to get
Public legal text is the finished half
Case law, statutes and contracts filed with securities regulators are public, but they are final texts. The drafts, markups and reasons for each change that produced a signed agreement stay inside law firms and legal departments.
Client confidentiality and privilege come first
Lawyers generally may not reveal information about a client's matter without the client's informed consent, and disclosing privileged advice to a third party can waive privilege. Law firm data therefore needs client authorization as well as the firm's agreement.
Half of every negotiation belongs to the other side
A negotiated contract is partly drafted by the counterparty and its lawyers. Confidentiality clauses in the agreement itself can restrict sharing its terms, so counterparty drafts and comments need their own review.
The right answer depends on the matter
Whether a clause is acceptable depends on governing law, party role, deal size, bargaining power and the client's risk appetite. Without matter context, the same clause looks accepted in one deal and rejected in the next, with nothing to explain why.
Exam-style benchmarks miss the work
Bar-exam and doctrinal question sets test legal knowledge, not issue spotting in a counterparty draft or proposing a fallback a client will accept, and items drawn from published sources can be memorized during pretraining.
What a negotiation history teaches
One contract's version chain yields several kinds of training and evaluation examples. Given the counterparty's opening draft and the client's playbook, which clauses did the reviewing lawyer flag? What markup did they propose, and which comment explained it? Which proposals survived into the executed agreement, and which were traded away? The signed text is the outcome label: it shows which positions held, and the approval records show which deviations a senior lawyer accepted and why.
Ordering matters. Training on the final agreement alone teaches a model what a good contract looks like, not how to get there from the other side's draft. Keep every round, attribute each version to the side that produced it, and link the emails that carried each draft, because explanations such as "we accept the cap only if data breaches are carved out" are where the reasoning lives.
Splitting by matter, client and template
Legal corpora are full of near-duplicates. A company signs agreement after agreement on its own template, and a law firm reuses precedents across clients. Split by matter, client or template family rather than by document or clause, and hold out the most recent period for evaluation, so the model cannot score well by recognizing a precedent it trained on. If the playbook is part of the model's input, keep the evaluation playbook versions out of training too.
Scoping a legal data request
State the contract types, your party side, governing laws, languages and period, and whether you need counterparty drafts, internal commentary or only the exchanged versions. Say how party names, deal values and dates should be de-identified, and whether you need the playbooks that applied. In-house legal departments and law firms clear differently, so say if either is excluded. On the sample, check that version chains run unbroken from first draft to signature and that removing privileged emails did not break the thread. The due diligence checklist covers the rights questions in more detail.
What good data looks like
- Each contract is a complete version chain from first draft to executed agreement, with every version dated and attributed to the party that produced it.
- Tracked changes and comments are preserved in native files or structured diffs, with negotiation emails linked to the round they belong to.
- Matter metadata records contract type, governing law, party role, deal size band and the final outcome, such as signed, abandoned or later amended.
- The playbook and clause library versions in force are included, and approved deviations are linked to their approval records.
- Privileged advice, client identities and counterparty-confidential terms are removed or de-identified under a documented review, with client authorization recorded.
- Templates are grouped into families, so one client's standard agreement cannot appear on both sides of a train/test split.
Questions buyers ask
Can law firms license client documents for AI training?
Only with the client's authorization. In the US, ABA Model Rule 1.6 and its state equivalents generally bar lawyers from revealing information about a representation without the client's informed consent, and other jurisdictions have similar duties. SourceX checks that authorization before law firm data is offered. In-house legal teams can be simpler sources, because the company owns its contracts, though counterparty confidentiality still applies.
How is privileged material handled?
Privileged advice is usually excluded, because disclosing it to a third party can waive privilege. Internal emails about negotiation strategy and advice memos are typically removed, while redlines and emails exchanged with the counterparty, which were never privileged, can remain if confidentiality terms allow. The privilege review approach is agreed in scope, so state whether your use case needs internal commentary.
How do I evaluate a contract review model on real negotiations?
Hold out whole matters, preferably the most recent. Give the model the draft the other side sent and the playbook in force, then compare its flags and markup with the reviewing lawyer's markup and with the executed text. Grade issue spotting, positions and drafting separately, using rubrics written by lawyers in the relevant practice area, and accept alternative wording that achieves the same position.
Can data for litigation or e-discovery models be sourced?
Sometimes, but it is harder to clear than transactional data. Documents produced in discovery are often bound by protective orders that limit their use to the case, and litigation files are dense with privileged communications and attorney work product. Records of the review process, such as relevance and privilege coding and quality-control overturns, are more promising where clients and review providers allow it.
Which contract types and jurisdictions can be requested?
Name the contract types, such as NDAs, master services agreements, SaaS subscriptions, data processing agreements, leases or credit agreements, plus your party side, governing laws and languages. Supply depends on which firms and legal departments hold matching negotiation histories and agree to license them, so it is not guaranteed, and specialized deal types are scarcer than high-volume commercial contracts.
Tell us what you are building
Describe the model or agent, the tasks it must handle, and the volume, format and permitted use you need. SourceX will match it to partner data.
Updated 3 October 2026.