Skip to content

AI uses for records

Agent wrong document version failure: it read the contract but missed the amendment

By SourceX Editorial · Reviewed by Noah Loul ·

10 sections

Short answer

To test an agent that reads a contract but misses its amendment, build a case where a later executed amendment replaces the clause the question depends on, then grade three things separately: whether the evidence set included every executed amendment touching that clause, whether the agent applied the controlling document for the decision date, and whether the answer is right. An accurate quote from a superseded clause fails.

Agent run

Agent wrong document version

This page covers agent wrong document version failure: it read the contract but missed the amendment.Illustrative

Key takeaways

  • A missed amendment produces an accurate quote from a superseded clause, so citation and faithfulness checks pass it.
  • Grade the evidence set: every executed document that modifies the clause must be in it.
  • Write the authority rule into the case: executed amendments control the clauses they name from their effective date; drafts never control.
  • Make the reference answer name the authority chain so retrieval metrics can see the missing amendment.
  • Plant variants and a not-yet-effective counterexample so a newest-document shortcut fails.
  • Score evidence, authority, answer and abstention separately, over several runs per case.

How do you test an agent that reads a contract but misses its amendment?

Build a case where the answer depends on a clause that a later executed amendment replaced, give the agent the tools it would use in production, and grade three things separately: whether its evidence set included every executed document that touches the clause, whether it applied the right one for the decision date, and whether the final answer matches. An agent that quotes the base contract accurately and misses the amendment fails, however confident the reply.

This failure is worth its own test because nothing in the transcript looks wrong. The quoted text exists. The citation points at a real section of a signed agreement. A reviewer who checks that the citation supports the answer will pass it, because it does. The defect is an absence: a document the agent never retrieved, or retrieved and did not treat as controlling.

The case below is illustrative. The companies, documents, dates and percentages are constructed for this article and do not describe any real contract, customer or model run. It shows the starting state, the step where the run went wrong, the case record a grader needs, and the variants and counterexample that stop the test from rewarding blanket caution.

What does the illustrative contract folder contain?

The illustrative environment is a contract repository behind two tools: search_documents, which returns ranked passages, and get_document, which returns a full file with its metadata. The account team asks the agent a renewal question on a decision date of October 18, 2026: what is the most the supplier can raise unit prices at the January 1, 2027 renewal for this customer?

Four files sit in the customer's folder. Only one combination of them gives the right answer.

What does the illustrative contract folder contain?
Document (illustrative)StatusEffective dateWhat it says about pricing
Master supply agreementExecutedMarch 1, 2023Section 6.2 caps annual unit price increases at 5%
First AmendmentExecutedJune 15, 2024Changes the notice address only
Second AmendmentExecutedJanuary 1, 2026Section 6.2 is deleted and replaced: increases capped at 3% per contract year
Third Amendment (redline draft)Unsigned draftNoneProposes a 2% cap; never executed
Ground truth on the decision dateNot a fileOctober 18, 2026The cap is 3% under Section 6.2 as replaced by the Second Amendment

How did the run end with a correct quote from a superseded clause?

The run below is a constructed trace. It is typical of how retrieval-first agents miss amendments: the amendment changes meaning by section number, so it shares few words with the question.

  • Step 1: the agent calls search_documents with the query price increase cap for this customer. The top passages are Section 6.2 of the master agreement, a paragraph from the unsigned Third Amendment, and the notice clause of the First Amendment.
  • Step 2: the Second Amendment does not rank. Its operative sentence reads Section 6.2 is hereby deleted in its entirety and replaced with the following, followed by text that uses the phrase adjustment to unit pricing rather than price increase cap.
  • Step 3: the agent correctly sets aside the Third Amendment because its metadata shows no signatures. It reads the First Amendment, sees nothing about pricing, and concludes that Section 6.2 is unchanged.
  • Step 4: the agent answers that increases are capped at 5% per year under Section 6.2 of the master supply agreement, quotes the clause, and adds that it checked the amendments on file.
  • Step 5: the account team drafts a renewal quote with a 5% increase. Every sentence the agent wrote was traceable to a signed document, and the decision it produced was wrong.

Why does a faithful, cited answer still fail this case?

The completeness condition failed: the evidence set did not include every executed document that modifies the clause in question. The answer was faithful to what the agent retrieved, so faithfulness and citation checks pass. The error sits one step earlier, in what was never retrieved.

That is why the reference answer has to name the authority chain, not just the number. Ragas defines context recall as the share of claims in a reference answer that can be attributed to the retrieved context. If the reference says only that the cap is 3%, the metric has little to check. If it says the cap is 3% because the Second Amendment replaced Section 6.2 effective January 1, 2026, the missing amendment shows up as an unsupported claim.

State-based grading has a similar gap. The tau-bench paper evaluates agents by comparing the database state at the end of a conversation with an annotated goal state. A question-answering run like this one changes nothing, so it needs an answer-level check. If the task instead ends with the agent writing the renewal quote, the goal state must encode the 3% cap, or the grader will accept the 5% quote as a completed task.

What must the case record hold so every grader reaches the same verdict?

This is the asset: the fields a test author fills in so that any reviewer, human or automated, reaches the same verdict. Values are for the illustrative case above.

At a glance

What must the case record hold so every grader reaches the same verdict

What must the case record hold so every grader reaches the same verdict?
FieldValue in this caseEvidence the grader readsPass condition
Case labelIllustrative, constructed for testingCase headerReal cases carry an approval reference instead
Question and decision dateMaximum renewal price increase, as of October 18, 2026Task promptAnswer is stated for that date
Document familyMaster agreement plus three amendment filesFolder manifest with file hashesEvery family member is listed
Execution statusTwo amendments executed, one draftSignature metadata per fileDrafts never control the answer
Clause mapSection 6.2 replaced by the Second AmendmentOperative amendment sentenceAgent identifies the replacing document
Applicable ruleExecuted amendment controls the clauses it names from its effective dateRule text stored with the caseRule cited by version
Failure stepStep 2: Second Amendment never retrievedTool call log with resultsEvidence set includes the Second Amendment
Expected answer3% per contract year, citing the Second AmendmentReference answer with authority chainNumber and controlling document both correct
SeverityHigh: a pricing decision taken on superseded termsSeverity rubricScored separately from fluency
Recovery checkDoes the agent revise when told a Second Amendment exists?Follow-up turnRetracts, retrieves and corrects without inventing terms

Which authority rule should the case write down?

Write the rule into the case in plain words before anyone labels it, because the right answer depends on it. A workable default: the executed base agreement controls; an executed amendment replaces or adds only the clauses it names, from its stated effective date; unsigned drafts never control; and an amended and restated agreement replaces the base and every earlier amendment.

Many agreements also carry their own order-of-precedence clause, which ranks the parts of a contract against each other. Federal procurement shows the pattern in public: FAR clause 52.215-8 resolves any inconsistency by giving precedence first to the Schedule, then to representations and other instructions, then to contract clauses, then to other documents, exhibits and attachments, and last to the specifications. A case built on a contract with such a clause should store it and test whether the agent applied it.

Pick clause types where amendments matter. The Contract Understanding Atticus Dataset (CUAD) defines clause categories such as Effective Date, Price Restrictions and Cap on Liability, which make a useful checklist of terms that business amendments commonly change and that downstream decisions depend on.

Which records show the amendment controlled and the quote went out wrong?

Three linked records settle the verdict, and the case should ship with all of them rather than a summary.

  • The executed Second Amendment with its signature evidence and effective date. This proves the 3% cap was in force on the decision date.
  • The clause map that links the amendment's operative sentence to Section 6.2 of the base agreement. This proves the amendment replaced the clause the question asked about, rather than adding a separate term.
  • The downstream record: the renewal quote or order form the account team produced from the agent's answer. A 5% increase on that record is the eventual task outcome, and it is what makes the failure costly rather than academic.
  • For public examples of how amendments are filed as separate documents, Item 601 of Regulation S-K requires public companies to file amendments or modifications to previously filed material contract exhibits as exhibits to later reports. Those filings are a realistic source of base-and-amendment pairs for constructed cases, since each amendment arrives as its own document.

Which variants and counterexample keep the case honest?

One case teaches one route to the miss. A useful set plants several, and includes at least one case where the base agreement is correct, so the agent is not rewarded for assuming that something always changed.

  • Vary the amendment's position in the context. Lost in the Middle reports that model performance is often highest when relevant information is at the beginning or end of the input and degrades when it sits in the middle of long contexts, so a set that always puts the amendment last will overstate reliability.
  • In the counterexample, a correct agent answers 5% for a decision date before January 1, 2026, and notes that a 3% cap takes effect then. Grade it as a pass; an agent that always picks the newest document fails it.
Which variants and counterexample keep the case honest?
VariantHow the amendment gets missedCorrect behaviorGrader signal
Replacement by section numberAmendment shares few words with the questionSearch the family for amendments, then read each operative clauseAmendment absent from evidence set
Amendment buried in a long fileAgent loads the whole folder and the amendment sits mid-contextSame answer wherever the amendment appearsAccuracy changes with amendment position
Unsigned redlineDraft text ranks highly and looks newerIgnore drafts; say a draft exists if relevantDraft terms cited as controlling
Amended and restated agreementAgent stacks old amendments on the restated textTreat the restatement as the new baseSuperseded amendment cited
Amendment in another systemSigned file lives in the e-signature archive, not the repositoryReport that the family may be incompleteConfident answer with no completeness check
Not yet effective (counterexample)Second Amendment executed but effective after the decision dateBase clause controls for that dateAgent applies the amendment early

Why score evidence, authority and answer as separate fields?

Score evidence completeness, authority reasoning, the final answer and abstention as separate fields. A single pass or fail hides which control broke, and an agent that is usually right on easy cases can average away a habit of missing amendments.

At a glance

Why score evidence, authority and answer as separate fields

  • Evidence completeness: did the evidence set include every executed family member that touches the clause? Pass or fail per case.
  • Authority: did the agent name the controlling document and its effective date, and set aside drafts and superseded text? Pass or fail.
  • Answer: is the number or term correct for the decision date? Pass or fail, scored only after the first two so a lucky guess is visible.
  • Abstention: when the family is incomplete by design, did the agent say so instead of answering? A correct refusal passes; a silent answer fails.
  • Repeat each case several times. The tau-bench paper introduces pass^k, the chance that all k trials of a task succeed, because one passing run says little about an agent that misses amendments some of the time.
  • No results are reported on this page. The case defines what to measure, not how any agent performs.

Where do real amended contract families come from?

Constructed cases show the mechanism; real contract families supply the variety that matters, such as amendments that renumber sections, restatements that fold in side letters, and signed files stored outside the repository. A test set built from de-identified executed agreements with their amendment chains intact exercises an agent against the structures it will meet in deployment.

SourceX has no shelf of contract families; it sources them from U.S. businesses when a team asks, and its request-data form is where the team sets the scope, volume, language, format, timing and licensing needs, including that amendment chains and effective dates must stay linked. Before any agreement is delivered, the business that owns the contracts approves the scope and terms and signs, and SourceX performs the de-identification through approved processing routes, so the family links are best named in the request rather than rebuilt afterward.

Frequently asked questions

Why do AI agents miss contract amendments?

Amendments often change meaning by section number, as in Section 6.2 is deleted and replaced, so they share few words with the user's question and rank poorly in search. Agents also stop searching once they find a plausible clause in the base agreement. A test should check the evidence set, not just the final answer.

Is an accurate citation enough to pass a contract question?

No. A citation can accurately quote a clause that a later amendment replaced. The grader has to check that the cited document was controlling on the decision date and that every executed amendment touching the clause was considered.

Should an agent treat the newest document as controlling?

Not automatically. Unsigned drafts never control, and an executed amendment controls only the clauses it names from its effective date. Include a counterexample where the amendment is not yet effective so that a newest-wins shortcut fails.

How many runs should each case get?

Several. A single passing run can hide an agent that misses the amendment some of the time. The pass^k measure from the tau-bench paper, the chance that all k trials succeed, is one way to report consistency.

Can public filings be used to build amendment test cases?

Public company filings include material contracts and their later amendments as separate exhibits, which makes them a reasonable source of structure for constructed cases. Check the licensing and confidentiality position before using any document in a commercial test set.

Sources

  • Ragas defines context recall as the number of claims in the reference supported by the retrieved context divided by the total number of claims in the reference. Source
  • The tau-bench paper grades agents by comparing the database state at the end of a conversation with an annotated goal state, and introduces pass^k, the chance that all k trials succeed. Source
  • FAR 52.215-8 resolves inconsistencies by precedence: the Schedule, representations and other instructions, contract clauses, other documents, exhibits and attachments, then the specifications. Source
  • CUAD's category descriptions include Effective Date, Price Restrictions and Cap on Liability clause categories. Source
  • Item 601 of Regulation S-K requires amendments or modifications to previously filed exhibits to be filed as exhibits to subsequent reports. Source
  • Lost in the Middle reports that model performance is often highest when relevant information is at the beginning or end of the input context and degrades when it is in the middle of long contexts. Source

Tell us what data you need

Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.

See if your company qualifies