AI uses for records
Document comparison change benchmark: what changed in a contract exhibit?
By SourceX Editorial · Reviewed by Noah Loul ·
10 sections
Short answer
Pair two versions of the exhibit whose status you can prove, align them by section, table cell or page region, and label every difference with its location in both files, its type and a materiality grade tied to the clause of the governing agreement that gives it effect. Add unchanged and redacted pairs as negatives, then score systems on detection, localization, typing and materiality-weighted recall.
Evaluation split
Document comparison change
Key takeaways
- A comparison item needs proven provenance: which file was executed and which agreement the exhibit belongs to.
- Materiality grades must cite the governing clause and one stated reader, so reviewers agree for the same reason.
- Use two diff tools, tokenize tables by cell and adjudicate the union; tool defaults can hide one-cell changes.
- Redactions and omitted schedules in filed copies are labeled separately and excluded from change scoring.
- Report detection, localization, type accuracy, materiality-weighted recall and false alarms on unchanged pairs.
How do you build a comparison task from two versions of a contract exhibit?
Start from two versions whose status you can prove, such as the approved draft and the executed copy, align them unit by unit, and label every difference with its location in both files, its type and a materiality grade tied to the agreement the exhibit belongs to. Then add pairs with no real change, so a system is also tested on staying quiet.
At a glance
How do you build a comparison task from two versions of a contract exhibit
The order matters because each step depends on the one before. A materiality grade means little if the change was located on the wrong page, and a location means little if nobody can show which file was signed.
- Collect the pair and its provenance: file hashes, version labels, which copy was executed and the governing agreement that incorporates the exhibit.
- Choose the alignment unit: numbered section, table row, table cell or page region for scanned pages.
- Generate candidate differences with two independent tools and keep both outputs.
- Have a reviewer accept, merge, split or reject each candidate and add any the tools missed.
- Assign a difference type and a materiality grade, citing the clause of the governing agreement that makes the change matter.
- Add negative pairs: the same version exported twice, rescanned, or redacted for filing.
- Freeze the item, record who adjudicated it, and version the benchmark.
Why are exhibits harder to compare than the body of the agreement?
Exhibits carry the operational and economic detail, often in tables, schedules and scanned attachments, and their meaning depends on clauses that live in a different file. A one-cell change in a pricing schedule can matter more than a rewritten paragraph in the main agreement.
Three properties make exhibits hard. They are table-heavy, so a change can be a single number in a row that a text diff renders as noise. They are often re-created rather than edited, so the second version may come back as a scan with different pagination. And their meaning is borrowed: a service level table only matters through the remedy clause in the master agreement that refers to it.
Public filings show both materiality and redaction at work. Under SEC Regulation S-K Item 601, schedules and similar attachments to filed exhibits need not be filed if they contain no information material to an investment or voting decision, provided the exhibit lists what was omitted. A registrant may also redact specific terms of a material contract that it customarily and actually treats as private or confidential, if the omitted information is not material. A filed exhibit can therefore differ from the executed one by design, and a benchmark built from filings has to label those gaps as omissions or redactions, not as changes in terms.
What makes a difference in a contract exhibit material?
A difference is material when it changes what a party must do, pay, receive or risk under the governing agreement; everything else is clerical or cosmetic. The grade must name the clause that makes it matter, so two reviewers reach the same label for the same reason.
Grade from the reader's position, stated once for the whole benchmark, such as counsel for the customer. Use the clause categories from expert-annotated contract datasets as a checklist of what usually carries risk: the Contract Understanding Atticus Dataset defines categories such as Expiration Date, Renewal Term, Price Restrictions, Minimum Commitment and Cap on Liability. A change touching one of those categories starts at the operational grade and is raised if the governing agreement ties money, term or remedies to it.
| Grade | Definition | Exhibit examples | Evidence the label must cite |
|---|---|---|---|
| M0 cosmetic | Changes appearance only | Font, pagination, moved page break, rescan artifacts | Rendered pages showing identical content |
| M1 clerical | Changes wording without changing obligations | Typo fix, renumbered rows, updated cross-reference that points to the same clause | Both versions of the passage and the target of the reference |
| M2 operational | Changes how the work is done but not price, term or risk | New notice contact, revised delivery window, added reporting step | The exhibit row and the clause that makes it binding |
| M3 economic or risk | Changes price, quantity, term, remedies or liability | Volume tier threshold, unit price, renewal period, service credit percentage, liability cap | The changed cell or passage, the governing clause and the party affected |
| R redaction or omission | Content withheld from one version on purpose | Bracketed redaction in a filed copy, omitted schedule listed in the exhibit | The redaction marker or omission list; excluded from change scoring |
Which fields does each document difference benchmark item carry?
One item is one difference, or one confirmed absence of difference, in one aligned unit of one version pair. The schema below can be used as is; every field is required unless marked optional.
At a glance
Which fields does each document difference benchmark item carry
| Field | Content | Illustrative value | Rule |
|---|---|---|---|
| pair_id and provenance | Hashes of both files, version labels, which is executed, governing agreement id | PAIR-031; v3 approved draft, v4 executed scan; MSA-12 | No item without a proven executed copy or an explicit draft-to-draft label |
| alignment_unit | Section number, table row key, cell or page region | Exhibit B, table 2, row Tier 2 | Pick the smallest unit both versions share |
| location_a and location_b | Page and bounding box in each file, or none if absent | p3 (72, 410, 540, 432); p4 (70, 118, 538, 140) | Coordinates in PDF points from the top of the page |
| diff_type | Insert, delete, replace, move, cell change, row added, row removed, format only | cell change | Moves are one item, not a delete plus an insert |
| before and after | Exact text or cell value from each version | 10,000 units to 12,000 units | Copied from the source, never retyped |
| materiality and rationale | Grade M0 to M3 or R, with the governing clause | M3: MSA section 4.2 ties the volume discount to this tier | Rationale must cite a clause, not an opinion |
| cross_file_links | Other records needed to interpret the change (optional) | MSA-12 section 4.2; amendment 2 recital | Link by id and section, not by quoting |
| capture_quality | Native or scanned, OCR confidence where scanned | scan, low confidence on row 3 | Low-confidence items are flagged, not dropped |
| adjudication | Reviewer ids, agreement status, date | two reviewers, agreed, 2026-10-18 | Disagreements go to a third reviewer and stay recorded |
How is each change located in both files?
Every label points to a place in both files that a person can open and see. For native PDFs, character-level coordinates give that place; for Word files, the revision markup does; for scans, a region drawn on the page image does.
Tools such as pdfplumber expose, for every character, its distance from the left edge (x0) and from the top of the page (top and bottom) and of the document (doctop), and can crop a page to a bounding box. Storing the box in both versions lets a grader check localization without re-running extraction, and lets a reader jump straight to the changed cell.
Native Word files already hold part of the label. In Office Open XML, tracked insertions and deletions are wrapped in w:ins and w:del elements, each of which can carry the revision author and date. When a pair comes from tracked changes, use that markup to seed candidates, but adjudicate anyway: authors accept and reject changes before sending, so the markup describes the editing session, not necessarily the difference between the two files that were exchanged.
Word's legal blackline compare is a useful baseline and a useful trap. It writes the result to a third document by default and, if either file contains tracked changes, it offers to accept them first, treating the original's changes as accepted. Record that setting in the item provenance, because a baseline that silently accepted revisions is comparing different texts from the ones a human reviewer saw.
Why should candidate diffs come from two tools and never be trusted alone?
Different diff algorithms produce different, equally correct edit scripts, and their defaults can hide exactly the repeated numbers that exhibits are full of. Using two tools and adjudicating the union catches more real changes than tuning one.
The diff-match-patch library implements Myers' diff algorithm with pre-diff speedups and post-diff cleanups. Python's difflib uses an algorithm related to Ratcliff and Obershelp's gestalt pattern matching and applies a junk heuristic: by default, when the second sequence is at least 200 items long, items that account for more than 1% of it are treated as junk, and because only the second sequence is inspected, comparing A to B can give different results than comparing B to A. In a pricing table where the same unit labels and digits recur in every row, that default can turn a one-cell change into a confusing block replacement. Turn the heuristic off or set the junk function explicitly for exhibit work, and record the setting.
Tokenize tables by cell rather than by line, so a changed value is one candidate and a reflowed row is not. Keep both tools' raw outputs with the item, because a later reviewer disputing a label will want to see what the machine proposed.
How would a revised pricing exhibit be labeled? (illustrative)
This example is illustrative; the agreement, numbers and dates are invented. The pair is Exhibit B to a supply agreement: version 3 is the approved draft in Word, version 4 is the countersigned PDF scan. The master agreement says volume discounts apply at the tiers in Exhibit B and that service credits are calculated as set out in its table 3.
The reviewers confirmed six differences. Two are M3: the Tier 2 threshold rose from 10,000 to 12,000 units, and the service credit for a missed delivery window fell from 5 percent to 3 percent. One is M2: the notice contact for delivery disputes changed. Two are M1: rows in table 2 were renumbered and a cross-reference was corrected to point to the same section. One is M0: the signature block moved to a new page. Both diff tools reported the renumbering as a block of changes; only the cell-tokenized pass isolated the threshold change as one item.
The exception: the same exhibit also appears in a public filing, where the unit price column is replaced by bracketed redaction marks. Compared against the executed scan, the filed copy shows a deleted column. That is labeled R and excluded from change scoring, because the filer withheld it as confidential and not material; a system that reports it as a price deletion fails the item.
How is a document comparison change benchmark scored?
Score detection, localization, typing and materiality separately, then weight by materiality so a missed M3 change costs far more than a missed page break. A single overall accuracy number hides the failure that matters most, which is a quiet miss on a material term.
Report these metrics for every benchmark release, broken down by capture type (native to native, native to scan, scan to scan):
- Detection precision and recall at the change level, counting a move as one change.
- Localization: the share of detected changes whose region overlaps the gold box in both versions.
- Type accuracy among correctly detected changes.
- Materiality-weighted recall, with weights fixed in the benchmark card before any system is run.
- Ranking quality: whether M3 changes appear before M0 and M1 changes in the system's output.
- False alarm rate on negative pairs, including rescans and redacted filed copies.
Where does the benchmark stop and a contracts reviewer take over?
The benchmark measures whether a system finds and ranks differences the way trained reviewers labeled them; it does not decide what a change means legally for a specific deal. Materiality grades here are a labeling convention tied to one stated reader, not legal advice.
Hand an item to a contracts specialist when the governing agreement is missing or ambiguous, when an amendment or side letter may override the exhibit, when the change depends on a defined term whose definition also changed, or when reviewers disagree on grade after adjudication. Narrow the scope instead of guessing: an item without its governing clause can still be scored for detection and localization and left ungraded for materiality.
Where do exhibit pairs with a provable signed copy come from?
Real exhibit pairs with a provable executed copy, the governing agreement and any amendments are hard to assemble from public material, because filings omit schedules and redact terms. AI teams building comparison products therefore look for operating records from businesses that kept drafts, executed scans and amendment chains together.
A team can request pairs like these through SourceX, which looks for them at U.S. businesses instead of keeping stock: the request names pair types, volume, format, timing and licensing needs, and the supplying company agrees scope and terms and signs before anything is delivered. Counterparty contracts often carry confidentiality clauses, so having the files, being permitted to share them and holding a license to use them for AI are checked separately, and identifying details are removed before delivery.
Frequently asked questions
What is a document comparison change benchmark?
It is a fixed set of document version pairs in which every real difference is located in both files, typed and graded for materiality, plus pairs with no real change. Systems are scored on finding, locating, typing and ranking those differences. It differs from training data because its labels are adjudicated and frozen for measurement.
How do you decide whether a change in a contract exhibit is material?
Tie the grade to the clause in the governing agreement that gives the exhibit its effect, and grade from one stated reader's position. Changes to price, quantity, term, remedies or liability are usually material; formatting and renumbering are not. Disputed grades go to a contracts specialist.
Can tracked changes in Word be used as ground truth?
They are a good source of candidates, since Office Open XML marks insertions and deletions with w:ins and w:del and can record author and date. They are not ground truth on their own, because accepted, rejected or untracked edits mean the markup may not match the difference between the files actually exchanged.
Why include pairs with no real change?
Production comparison tools see many rescans, re-exports and redacted copies that differ in pixels or layout but not in terms. Without negative pairs, a benchmark rewards systems that flag everything. A false alarm rate on these pairs shows whether reviewers can trust a quiet result.
Are SEC filings a good source of exhibit version pairs?
They are useful for structure but limited as ground truth. Item 601 lets filers omit non-material schedules and redact confidential, non-material terms, so a filed exhibit may differ from the executed one on purpose. Those gaps must be labeled as omissions or redactions rather than changes.
Sources
- Regulation S-K Item 601 lets registrants omit non-material schedules if the exhibit lists them, and redact specific terms of material contracts that are customarily treated as confidential and are not material. Source
- The Contract Understanding Atticus Dataset is an expert-annotated contract review benchmark whose categories include Expiration Date, Renewal Term, Price Restrictions, Minimum Commitment and Cap on Liability. Source
- diff-match-patch implements Myers' diff algorithm surrounded by pre-diff speedups and post-diff cleanups. Source
- Python's difflib uses a junk heuristic that by default treats items over 1% of a second sequence of 200 or more items as junk, and is asymmetric between A-to-B and B-to-A comparisons. Source
- pdfplumber reports character positions such as x0, top, bottom and doctop and can crop a page to a bounding box. Source
- Word's legal blackline compare writes results to a new document by default and offers to accept existing tracked changes before comparing. Source
- Office Open XML marks tracked inserted runs with w:ins, with author and date attributes for the revision. Source
- Office Open XML marks tracked deleted runs with w:del, which inherits author and date attributes from the run track change type. Source
Tell us what data you need
Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.