AI uses for records
Agent wrong document version failure: it read the contract but missed the amendment
By SourceX Editorial · Reviewed by Noah Loul ·
10 sections
Short answer
To test an agent that reads a contract but misses its amendment, build a case where a later executed amendment replaces the clause the question depends on, then grade three things separately: whether the evidence set included every executed amendment touching that clause, whether the agent applied the controlling document for the decision date, and whether the answer is right. An accurate quote from a superseded clause fails.
Agent run
Agent wrong document version
Key takeaways
- A missed amendment produces an accurate quote from a superseded clause, so citation and faithfulness checks pass it.
- Grade the evidence set: every executed document that modifies the clause must be in it.
- Write the authority rule into the case: executed amendments control the clauses they name from their effective date; drafts never control.
- Make the reference answer name the authority chain so retrieval metrics can see the missing amendment.
- Plant variants and a not-yet-effective counterexample so a newest-document shortcut fails.
- Score evidence, authority, answer and abstention separately, over several runs per case.
How do you test an agent that reads a contract but misses its amendment?
Build a case where the answer depends on a clause that a later executed amendment replaced, give the agent the tools it would use in production, and grade three things separately: whether its evidence set included every executed document that touches the clause, whether it applied the right one for the decision date, and whether the final answer matches. An agent that quotes the base contract accurately and misses the amendment fails, however confident the reply.
This failure is worth its own test because nothing in the transcript looks wrong. The quoted text exists. The citation points at a real section of a signed agreement. A reviewer who checks that the citation supports the answer will pass it, because it does. The defect is an absence: a document the agent never retrieved, or retrieved and did not treat as controlling.
The case below is illustrative. The companies, documents, dates and percentages are constructed for this article and do not describe any real contract, customer or model run. It shows the starting state, the step where the run went wrong, the case record a grader needs, and the variants and counterexample that stop the test from rewarding blanket caution.
What does the illustrative contract folder contain?
The illustrative environment is a contract repository behind two tools: search_documents, which returns ranked passages, and get_document, which returns a full file with its metadata. The account team asks the agent a renewal question on a decision date of October 18, 2026: what is the most the supplier can raise unit prices at the January 1, 2027 renewal for this customer?
Four files sit in the customer's folder. Only one combination of them gives the right answer.
| Document (illustrative) | Status | Effective date | What it says about pricing |
|---|---|---|---|
| Master supply agreement | Executed | March 1, 2023 | Section 6.2 caps annual unit price increases at 5% |
| First Amendment | Executed | June 15, 2024 | Changes the notice address only |
| Second Amendment | Executed | January 1, 2026 | Section 6.2 is deleted and replaced: increases capped at 3% per contract year |
| Third Amendment (redline draft) | Unsigned draft | None | Proposes a 2% cap; never executed |
| Ground truth on the decision date | Not a file | October 18, 2026 | The cap is 3% under Section 6.2 as replaced by the Second Amendment |
How did the run end with a correct quote from a superseded clause?
The run below is a constructed trace. It is typical of how retrieval-first agents miss amendments: the amendment changes meaning by section number, so it shares few words with the question.
- Step 1: the agent calls search_documents with the query price increase cap for this customer. The top passages are Section 6.2 of the master agreement, a paragraph from the unsigned Third Amendment, and the notice clause of the First Amendment.
- Step 2: the Second Amendment does not rank. Its operative sentence reads Section 6.2 is hereby deleted in its entirety and replaced with the following, followed by text that uses the phrase adjustment to unit pricing rather than price increase cap.
- Step 3: the agent correctly sets aside the Third Amendment because its metadata shows no signatures. It reads the First Amendment, sees nothing about pricing, and concludes that Section 6.2 is unchanged.
- Step 4: the agent answers that increases are capped at 5% per year under Section 6.2 of the master supply agreement, quotes the clause, and adds that it checked the amendments on file.
- Step 5: the account team drafts a renewal quote with a 5% increase. Every sentence the agent wrote was traceable to a signed document, and the decision it produced was wrong.
Why does a faithful, cited answer still fail this case?
The completeness condition failed: the evidence set did not include every executed document that modifies the clause in question. The answer was faithful to what the agent retrieved, so faithfulness and citation checks pass. The error sits one step earlier, in what was never retrieved.
That is why the reference answer has to name the authority chain, not just the number. Ragas defines context recall as the share of claims in a reference answer that can be attributed to the retrieved context. If the reference says only that the cap is 3%, the metric has little to check. If it says the cap is 3% because the Second Amendment replaced Section 6.2 effective January 1, 2026, the missing amendment shows up as an unsupported claim.
State-based grading has a similar gap. The tau-bench paper evaluates agents by comparing the database state at the end of a conversation with an annotated goal state. A question-answering run like this one changes nothing, so it needs an answer-level check. If the task instead ends with the agent writing the renewal quote, the goal state must encode the 3% cap, or the grader will accept the 5% quote as a completed task.
What must the case record hold so every grader reaches the same verdict?
This is the asset: the fields a test author fills in so that any reviewer, human or automated, reaches the same verdict. Values are for the illustrative case above.
At a glance
What must the case record hold so every grader reaches the same verdict
| Field | Value in this case | Evidence the grader reads | Pass condition |
|---|---|---|---|
| Case label | Illustrative, constructed for testing | Case header | Real cases carry an approval reference instead |
| Question and decision date | Maximum renewal price increase, as of October 18, 2026 | Task prompt | Answer is stated for that date |
| Document family | Master agreement plus three amendment files | Folder manifest with file hashes | Every family member is listed |
| Execution status | Two amendments executed, one draft | Signature metadata per file | Drafts never control the answer |
| Clause map | Section 6.2 replaced by the Second Amendment | Operative amendment sentence | Agent identifies the replacing document |
| Applicable rule | Executed amendment controls the clauses it names from its effective date | Rule text stored with the case | Rule cited by version |
| Failure step | Step 2: Second Amendment never retrieved | Tool call log with results | Evidence set includes the Second Amendment |
| Expected answer | 3% per contract year, citing the Second Amendment | Reference answer with authority chain | Number and controlling document both correct |
| Severity | High: a pricing decision taken on superseded terms | Severity rubric | Scored separately from fluency |
| Recovery check | Does the agent revise when told a Second Amendment exists? | Follow-up turn | Retracts, retrieves and corrects without inventing terms |
Which authority rule should the case write down?
Write the rule into the case in plain words before anyone labels it, because the right answer depends on it. A workable default: the executed base agreement controls; an executed amendment replaces or adds only the clauses it names, from its stated effective date; unsigned drafts never control; and an amended and restated agreement replaces the base and every earlier amendment.
Many agreements also carry their own order-of-precedence clause, which ranks the parts of a contract against each other. Federal procurement shows the pattern in public: FAR clause 52.215-8 resolves any inconsistency by giving precedence first to the Schedule, then to representations and other instructions, then to contract clauses, then to other documents, exhibits and attachments, and last to the specifications. A case built on a contract with such a clause should store it and test whether the agent applied it.
Pick clause types where amendments matter. The Contract Understanding Atticus Dataset (CUAD) defines clause categories such as Effective Date, Price Restrictions and Cap on Liability, which make a useful checklist of terms that business amendments commonly change and that downstream decisions depend on.
Which records show the amendment controlled and the quote went out wrong?
Three linked records settle the verdict, and the case should ship with all of them rather than a summary.
- The executed Second Amendment with its signature evidence and effective date. This proves the 3% cap was in force on the decision date.
- The clause map that links the amendment's operative sentence to Section 6.2 of the base agreement. This proves the amendment replaced the clause the question asked about, rather than adding a separate term.
- The downstream record: the renewal quote or order form the account team produced from the agent's answer. A 5% increase on that record is the eventual task outcome, and it is what makes the failure costly rather than academic.
- For public examples of how amendments are filed as separate documents, Item 601 of Regulation S-K requires public companies to file amendments or modifications to previously filed material contract exhibits as exhibits to later reports. Those filings are a realistic source of base-and-amendment pairs for constructed cases, since each amendment arrives as its own document.
Which variants and counterexample keep the case honest?
One case teaches one route to the miss. A useful set plants several, and includes at least one case where the base agreement is correct, so the agent is not rewarded for assuming that something always changed.
- Vary the amendment's position in the context. Lost in the Middle reports that model performance is often highest when relevant information is at the beginning or end of the input and degrades when it sits in the middle of long contexts, so a set that always puts the amendment last will overstate reliability.
- In the counterexample, a correct agent answers 5% for a decision date before January 1, 2026, and notes that a 3% cap takes effect then. Grade it as a pass; an agent that always picks the newest document fails it.
| Variant | How the amendment gets missed | Correct behavior | Grader signal |
|---|---|---|---|
| Replacement by section number | Amendment shares few words with the question | Search the family for amendments, then read each operative clause | Amendment absent from evidence set |
| Amendment buried in a long file | Agent loads the whole folder and the amendment sits mid-context | Same answer wherever the amendment appears | Accuracy changes with amendment position |
| Unsigned redline | Draft text ranks highly and looks newer | Ignore drafts; say a draft exists if relevant | Draft terms cited as controlling |
| Amended and restated agreement | Agent stacks old amendments on the restated text | Treat the restatement as the new base | Superseded amendment cited |
| Amendment in another system | Signed file lives in the e-signature archive, not the repository | Report that the family may be incomplete | Confident answer with no completeness check |
| Not yet effective (counterexample) | Second Amendment executed but effective after the decision date | Base clause controls for that date | Agent applies the amendment early |
Why score evidence, authority and answer as separate fields?
Score evidence completeness, authority reasoning, the final answer and abstention as separate fields. A single pass or fail hides which control broke, and an agent that is usually right on easy cases can average away a habit of missing amendments.
At a glance
Why score evidence, authority and answer as separate fields
- Evidence completeness: did the evidence set include every executed family member that touches the clause? Pass or fail per case.
- Authority: did the agent name the controlling document and its effective date, and set aside drafts and superseded text? Pass or fail.
- Answer: is the number or term correct for the decision date? Pass or fail, scored only after the first two so a lucky guess is visible.
- Abstention: when the family is incomplete by design, did the agent say so instead of answering? A correct refusal passes; a silent answer fails.
- Repeat each case several times. The tau-bench paper introduces pass^k, the chance that all k trials of a task succeed, because one passing run says little about an agent that misses amendments some of the time.
- No results are reported on this page. The case defines what to measure, not how any agent performs.
Where do real amended contract families come from?
Constructed cases show the mechanism; real contract families supply the variety that matters, such as amendments that renumber sections, restatements that fold in side letters, and signed files stored outside the repository. A test set built from de-identified executed agreements with their amendment chains intact exercises an agent against the structures it will meet in deployment.
SourceX has no shelf of contract families; it sources them from U.S. businesses when a team asks, and its request-data form is where the team sets the scope, volume, language, format, timing and licensing needs, including that amendment chains and effective dates must stay linked. Before any agreement is delivered, the business that owns the contracts approves the scope and terms and signs, and SourceX performs the de-identification through approved processing routes, so the family links are best named in the request rather than rebuilt afterward.
Frequently asked questions
Why do AI agents miss contract amendments?
Amendments often change meaning by section number, as in Section 6.2 is deleted and replaced, so they share few words with the user's question and rank poorly in search. Agents also stop searching once they find a plausible clause in the base agreement. A test should check the evidence set, not just the final answer.
Is an accurate citation enough to pass a contract question?
No. A citation can accurately quote a clause that a later amendment replaced. The grader has to check that the cited document was controlling on the decision date and that every executed amendment touching the clause was considered.
Should an agent treat the newest document as controlling?
Not automatically. Unsigned drafts never control, and an executed amendment controls only the clauses it names from its effective date. Include a counterexample where the amendment is not yet effective so that a newest-wins shortcut fails.
How many runs should each case get?
Several. A single passing run can hide an agent that misses the amendment some of the time. The pass^k measure from the tau-bench paper, the chance that all k trials succeed, is one way to report consistency.
Can public filings be used to build amendment test cases?
Public company filings include material contracts and their later amendments as separate exhibits, which makes them a reasonable source of structure for constructed cases. Check the licensing and confidentiality position before using any document in a commercial test set.
Sources
- Ragas defines context recall as the number of claims in the reference supported by the retrieved context divided by the total number of claims in the reference. Source
- The tau-bench paper grades agents by comparing the database state at the end of a conversation with an annotated goal state, and introduces pass^k, the chance that all k trials succeed. Source
- FAR 52.215-8 resolves inconsistencies by precedence: the Schedule, representations and other instructions, contract clauses, other documents, exhibits and attachments, then the specifications. Source
- CUAD's category descriptions include Effective Date, Price Restrictions and Cap on Liability clause categories. Source
- Item 601 of Regulation S-K requires amendments or modifications to previously filed exhibits to be filed as exhibits to subsequent reports. Source
- Lost in the Middle reports that model performance is often highest when relevant information is at the beginning or end of the input context and degrades when it is in the middle of long contexts. Source
Tell us what data you need
Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.