Skip to content

Systems and records

Contract exhibit inconsistency dataset: can an agent spot the conflict?

By SourceX Editorial · Reviewed by Noah Loul ·

9 sections

Short answer

To test whether an agent notices a conflict between an agreement and its exhibit, pair executed, nonprivileged documents with gold items that counsel prepared: both conflicting passages, the conflict type, the precedence language that decides it, and a label of body, exhibit or amendment controls, or not settled by the documents. Score detection, location, precedence and deferral separately, and include consistent pairs so false alarms count.

License terms

Contract exhibit

This page covers contract exhibit inconsistency dataset: can an agent spot the conflict.Illustrative

Key takeaways

  • A cross document task scores whether the agent finds, quotes and correctly defers on disagreements, not whether it picks a winner alone.
  • The contract's own order of precedence decides the gold label, and counsel records the exact sentence relied on.
  • A not settled label with a reason code is a legitimate gold answer and teaches the agent where its authority ends.
  • Consistent pairs where an exhibit only adds detail are needed so that flagging everything scores badly.
  • Only executed documents screened as nonprivileged, with counterparty confidentiality reviewed and personal data masked consistently, belong in the set.

How do you test whether an agent notices a conflict between an agreement and its exhibit?

Give the agent the governing agreement and one exhibit, ask it to report every point where the two disagree, and compare its answer with a gold record that counsel prepared from the same documents. Each gold item names the two passages, the type of disagreement, the provision that decides which text controls, and whether the documents alone settle it.

The test is not whether the agent can pick a winner on its own. A useful agent finds the disagreement, quotes both sides, points to the precedence language, and says plainly when the answer depends on facts or law outside the packet. Scoring those four behaviors separately tells you far more than one pass or fail grade.

Most public contract benchmarks ask a narrower question. They label clauses inside one document or test whether a single contract supports a stated hypothesis. A cross document task adds the step practitioners actually perform: reading the body and its attachments together and noticing that they no longer agree.

Which disagreements between a contract body and an exhibit belong in the task?

Include the disagreements that change what a party must do, by when, or for whom, and leave out differences in wording that mean the same thing. An exhibit that restates an obligation in more detail is consistent; an exhibit that shortens a deadline, adds a deliverable or names a different entity is not.

The categories below cover most of what legal operations teams see in services agreements, statements of work and order forms. Every example is illustrative, written for this guide rather than taken from a real contract.

Which disagreements between a contract body and an exhibit belong in the task?
Conflict typeWhere it usually hidesIllustrative exampleWhat counts as detection
Deadline or periodAcceptance, cure, notice and renewal clauses vs a scheduleBody allows 10 business days to accept; exhibit says 5 calendar daysBoth passages cited and the unit difference named
Quantity or service levelService level exhibit vs performance warrantyBody promises 99.5 percent uptime; exhibit lists 99.0 percent for the same serviceThe metric, both values and the affected service named
Scope of workStatement of work deliverables vs body definition of ServicesExhibit adds data migration that the body excludesThe added or removed item and the exclusion cited
Party or entitySignature blocks, order forms and notices vs the preambleExhibit names a subsidiary while the body names the parentBoth legal names quoted and the mismatch flagged as possibly material
Defined termExhibit uses a capitalized term with its own definitionExhibit defines Business Day to include SaturdaysBoth definitions cited and the clauses they affect listed
Precedence or override languageExhibit preamble or footerExhibit says it controls notwithstanding anything in the AgreementThe override sentence surfaced together with the body's precedence clause
Version or dateExhibit header, amendment referencesExhibit dated after Amendment 1 but cites the original fee scheduleThe dates and the referenced version mismatch reported

Which document wins, and who decides that for the gold label?

The contract's own order of precedence decides first, and counsel records how it applies; the agent is never the authority for the gold label. Many agreements rank their parts explicitly, and the ranking is not always the one a reader expects.

Federal procurement shows how specific these rankings get. The FAR clause 52.215-8, Order of Precedence, resolves any inconsistency by ranking the Schedule (excluding the specifications) first, then representations and other instructions, then contract clauses, then other documents, exhibits and attachments, and the specifications last. The commercial terms clause 52.212-4, paragraph (s), uses a nine-step order in which addenda, including software license agreements, sit above other documents, exhibits and attachments, and the specification again comes last.

Private agreements vary more. Some say the body always controls, some let a statement of work override the body only for that project, and some let an exhibit win only where it expressly says so. Later amendments may change the ranking itself. The gold record therefore stores the exact precedence sentence, the amendment that last touched it, and counsel's reading, so that the reasoning is checkable and the label can be revisited when a reviewer disagrees.

When there is no precedence clause, or two clauses each claim to control, the right gold answer is often that the documents do not settle the question. Recording that outcome is as valuable as recording a winner, because it teaches the agent when to stop.

The counsel reviewed cross document inconsistency task, field by field

Each task item is one disagreement, or one confirmed absence of disagreement, with enough context that a second reviewer can rebuild the label without asking anyone. The schema below is the asset; a team can copy it into a spreadsheet or a JSON schema as is.

At a glance

The counsel reviewed cross document inconsistency task, field by field

The counsel reviewed cross document inconsistency task, field by field
FieldWhat it holdsFilled byValidation rule
packet_id and document_idsStable IDs for the agreement, each exhibit and each amendment in the packetLegal operationsEvery ID resolves to a stored, signed or executed copy
version_and_datesExecution date, effective date and amendment dates for each documentLegal operationsExhibit date is consistent with the amendment chain or flagged
passage_a and passage_bCharacter spans in the body and the exhibit that disagreeAnnotatorSpans quote the text exactly; no paraphrase
conflict_typeOne value from the conflict table, or none for a negative itemAnnotatorSingle value; negatives are at least a fifth of items
precedence_basisThe precedence or override sentence relied on, with its spanCounselMust cite text in the packet or say none found
resolution_labelbody controls, exhibit controls, amendment controls, or not settled by the documentsCounselNot settled requires a reason code
reason_codeWhy the documents do not settle it: missing document, extrinsic facts, law, ambiguityCounselRequired when resolution_label is not settled
materiality_noteWhether the difference changes an obligation, date, amount of work or partyCounselFree text, one or two sentences
privilege_screenScreened by, date, and the basis for treating the packet as nonprivilegedCounselBlank screen blocks the item
exclusion_recordPackets removed and why, kept in a separate logLegal operationsEvery excluded packet has a reason and date

Scoring detection, location and deferral instead of a single right answer

Score four things separately: whether the agent found the disagreement, whether it located both passages, whether it cited the right precedence basis, and whether it deferred when counsel marked the item as not settled. An agent that finds every conflict but confidently resolves the unsettled ones is more dangerous than one that finds fewer and defers correctly.

Published contract datasets offer useful building blocks. ContractNLI, from Stanford NLP, gives a system a contract and a hypothesis and asks whether the contract entails it, contradicts it or does not mention it, and it requires evidence spans for the decision. CUAD, curated by the Atticus Project, labels 41 categories of clauses, including agreement date, renewal term, notice period to terminate renewal and anti-assignment, which makes it a reasonable vocabulary for naming the provisions involved. LegalBench collects 162 legal reasoning tasks from 40 contributors, several of them built on ContractNLI and CUAD. None of these pairs a body with its own exhibit, so treat them as references for label design, not as substitutes for the task.

Report negatives too. A task where every item is a real conflict rewards an agent that flags everything. Include consistent pairs where the exhibit adds detail without contradicting the body, and count each false alarm against the agent.

  • Detection: share of counsel-labeled conflicts the agent reported.
  • False alarms: share of negative items the agent reported as conflicts.
  • Location: share of detected conflicts where both cited spans overlap counsel's spans.
  • Precedence: share of detected conflicts where the agent cited the same controlling text as counsel.
  • Deferral: share of not settled items where the agent declined to resolve and gave a usable reason.

Worked example (illustrative): an acceptance window that moved in Exhibit B

This case is illustrative; no real contract is described. A software services agreement says in section 6 that the customer has 10 business days to accept each deliverable, and in section 18 that the body controls over any exhibit unless the exhibit expressly states that it overrides a named section. Exhibit B, the statement of work, says deliverables are deemed accepted 5 calendar days after delivery. Amendment 1, signed later, reprints Exhibit B in full with a new milestone list and leaves its acceptance sentence unchanged.

Counsel produces three gold items from this packet. The first is a conflict of type deadline: passage A is section 6, passage B is the Exhibit B acceptance sentence, the precedence basis is section 18, and the label is body controls, because Exhibit B does not expressly override section 6. The second is a negative item: Exhibit B's description of the acceptance test environment adds detail that section 6 does not contradict. The third concerns Amendment 1, which reprints Exhibit B in full with the 5-day sentence intact; counsel marks it not settled, reason code ambiguity, because a reader could argue that re-executing the exhibit after the body shows a later intent.

An agent that reports the first item, ignores the second, and flags the third for review scores well on every measure. An agent that announces that acceptance takes 5 days has failed on precedence; one that calls the test environment description a conflict has produced a false alarm.

The exception worth keeping in the set is the third item. Most teams would be tempted to force a label onto it. Leaving it unresolved, with a reason code, is what teaches the agent where its authority ends.

Only executed or final documents that counsel has screened as nonprivileged, and that the contract's confidentiality terms allow to be used, belong in the set. Drafts carrying counsel's comments, redlines exchanged with outside lawyers and internal analyses of which clause controls are usually privileged or attorney work product, and they stay out.

Professional duties reach further than privilege. ABA Model Rule 1.6 bars a lawyer from revealing information relating to the representation of a client without informed consent or another exception, so in-house counsel should confirm that their own analysis notes are excluded before anything leaves the department. Federal Rule of Evidence 502 limits when a disclosure in a federal proceeding or to a federal agency waives privilege, and it covers inadvertent disclosure when reasonable steps were taken; it is a litigation rule, not permission to reuse privileged material elsewhere.

Two further screens matter in practice. Many commercial contracts restrict disclosure of their own terms, so the counterparty's confidentiality clause is part of the rights review. Signature blocks, notice addresses and named contacts contain personal data, which is masked or replaced consistently across the body and every exhibit so the conflict survives the masking.

  • Executed versions only; drafts and redlines with counsel comments excluded.
  • Counterparty confidentiality and use restrictions reviewed for each agreement.
  • Personal data in signature, notice and contact fields masked the same way in every document of the packet.
  • Counsel's resolution notes kept separate from any material that leaves legal, unless counsel approves their release.
  • Exclusion log kept with the packet ID, reason and date for every removed agreement.

Which conflicts can the documents alone not settle?

Stop and route the item to counsel whenever the answer depends on something the packet does not contain. That includes documents incorporated by reference but not supplied, such as online terms or a master agreement held by the other party, and facts about how the parties actually performed.

Course of dealing and course of performance can matter. If the parties have accepted deliverables on the exhibit's 5-day schedule for two years, a court in some jurisdictions may weigh that conduct. The documents alone cannot tell the agent or the annotator about it, so the item gets the reason code extrinsic facts rather than a forced label.

Jurisdiction changes outcomes as well. Rules on interpreting conflicting terms, on whether a later document signals a new intent, and on the effect of unsigned exhibits differ by governing law. A task built from agreements governed by one state's law should say so, and results should not be generalized beyond it.

Version uncertainty is the most common practical failure. If nobody can confirm which Exhibit B was attached at signing, the packet is excluded rather than labeled, because a conflict built on the wrong exhibit teaches the agent to find disagreements that never existed.

Executed agreement packets with counsel-reviewed conflict labels are the kind of legal operations record that AI developers find hard to create on their own. When a legal department licenses contract packets through SourceX, the rights review, contracting, preparation and delivery are SourceX's to run. The department approves scope and terms and signs before anything is delivered, after a first step that is a short company assessment with no data upload.

Frequently asked questions

What is a contract exhibit inconsistency dataset?

It is a set of task items built from real or illustrative contract packets, where each item records a point at which the main agreement and an exhibit disagree, or a confirmed point where they agree. Counsel labels which provision controls, or that the documents do not settle it, so an agent can be scored against that judgment.

Does the main agreement always override an exhibit?

No. The contract's order of precedence decides, and it varies. The FAR clause 52.215-8, for example, ranks contract clauses above other documents, exhibits and attachments, while some private agreements let a statement of work override the body for its own project.

Can an AI agent decide which clause controls?

An agent can find and quote the conflicting passages and the precedence language, which is the useful part of the work. Deciding which term governs in a dispute depends on the full facts and governing law, so it stays with counsel, and a good test rewards the agent for deferring.

How many negative examples should the task include?

Enough that an agent which flags everything scores badly. A practical floor is that at least a fifth of items are consistent pairs where the exhibit adds detail without contradicting the body, though teams should set the share from what their own contracts look like.

Are redlines and drafts useful for this task?

They can show how conflicts arise, but drafts with counsel comments are usually privileged or work product and should be excluded. The task works best with executed versions, where the conflict is the one the parties actually signed.

Sources

  • FAR 52.215-8 resolves inconsistencies by ranking the Schedule, representations and instructions, contract clauses, other documents, exhibits and attachments, and the specifications last. Source
  • FAR 52.212-4(s) sets a nine-step order of precedence in which addenda, including software license agreements, rank above other documents, exhibits and attachments, with the specification last. Source
  • ContractNLI asks whether a contract entails, contradicts or does not mention a hypothesis and requires evidence spans for the decision. Source
  • CUAD is a dataset for legal contract review curated by the Atticus Project; its 41 categories include agreement date, renewal term, notice period to terminate renewal and anti-assignment. Source
  • LegalBench is a collaboratively built benchmark of 162 legal reasoning tasks from 40 contributors. Source
  • ABA Model Rule 1.6 bars a lawyer from revealing information relating to the representation of a client without informed consent or another listed exception. Source
  • Federal Rule of Evidence 502 limits waiver of privilege and work product for disclosures in federal proceedings or to federal agencies, including inadvertent disclosures where reasonable steps were taken. Source

See if your company qualifies

A short company assessment. No data uploads are needed.

See if your company qualifies