AI uses for records
Retrieval document version benchmark: finding the controlling version
By SourceX Editorial · Updated
10 sections
Short answer
Label, for every test question, which document version controls the answer for that user at that date, then score whether retrieval ranks it above superseded, draft and out-of-scope look-alikes. Use graded judgments with versions as hard negatives, measure the controlling hit, wrong-version-first rate and forbidden retrievals, and check that the cited passage supports the exact answer.
Evaluation split
Retrieval document version
Key takeaways
- The controlling version depends on the as-of date, approval status, precedence, scope and the user's permissions, not on recency alone.
- Record an authority record per document family before writing questions; families you cannot label stay out.
- Use six query types so that a retriever sorting by date or similarity cannot pass by accident.
- Grade other versions as labeled hard negatives and score wrong-version-first and forbidden retrievals separately.
- Check that the cited passage supports the exact answer, and tag history questions where an older version is correct.
How do you evaluate retrieval of the controlling version rather than similar text?
Build the benchmark around authority, not similarity. Each test question carries an as-of date, the asking user's role and the scope it applies to, and a reviewer records which document version controls the answer under those conditions. Every other version of the same document goes into the judgments as a labeled hard negative, because those are the passages a similarity search will find first.
Then score four things: whether the controlling version appears in the top results, whether any non-controlling version outranks it, whether the full evidence set the answer needs was retrieved, and whether anything the user may not see was returned. A final answer-level check confirms that the cited passage supports this exact answer for this user and date, or that the system abstained when nothing controls.
This guide sets out a version authority retrieval evaluation design: the authority record each document family needs, six query types, a grading scale, metrics with pass conditions, a worked example with illustrative counts, and the cases where an older version is the right result.
Why the controlling version is often not the newest one
Most stale-document tests assume the latest version is correct. In business records that assumption breaks in at least four ways, and a benchmark that only rewards recency will pass a retriever that fails all four.
Effective dates differ from creation dates: a pricing policy approved in May may not apply until July, and a question about a June order is governed by the old one. Approval status differs from existence: in SharePoint libraries that track minor versions, drafts can be limited to people who can edit, and other readers see the last published major version, so a draft newer than the published file is not the answer for most users. Several documents can be valid at once with a precedence rule between them: the federal clause at FAR 52.215-8 resolves inconsistencies in a contract by giving precedence to the Schedule over representations, then contract clauses, then other documents and attachments, then the specifications. And scope limits authority: the policy for one region, product line or customer does not control a question about another.
The broader research literature points the same way. The original retrieval-augmented generation paper named providing provenance for outputs and updating world knowledge as open problems that a retrieval memory can help with, and the FreshQA benchmark separates never-changing, slow-changing, fast-changing and false-premise questions because models struggle with knowledge that changes. Version authority is the enterprise form of that problem.
The authority record each document family needs
Before writing a single question, group documents into families, meaning every version and amendment of one policy, contract or procedure, and record their authority metadata. The Dublin Core terms give a neutral vocabulary for most of it: replaces and isReplacedBy link versions in a chain, valid records the date or range when a resource is valid, and dateAccepted records when it was accepted.
If a family cannot be filled in, it cannot be graded. Leave it out of the benchmark and note the gap rather than guessing which version controls.
| Field | What it captures | Example (illustrative) |
|---|---|---|
| Family id | Which documents are versions or parts of one thing | Returns policy, all versions |
| Version and links | Order in the chain, using replaces and isReplacedBy | v4 replaces v3; v5 draft replaces v4 |
| Valid range | Dates the version governs, not the date it was written | v3 valid Jan 1 to Jun 30; v4 from Jul 1 |
| Approval status | Draft, approved, published, withdrawn | v5 draft, visible to editors only |
| Precedence rank | Order among documents valid at the same time | Amendment over master agreement over order form |
| Scope | Region, product, customer or entity it applies to | U.S. retail customers only |
| Audience | Roles allowed to see it | All staff; v5 editors only |
Six query types that separate authority from similarity
Write questions so that the wording matches several versions about equally well and only the authority record decides the answer. Phrase each question the way a real user would, then attach the as-of date, role and scope as structured fields the retriever receives, just as a production system would pass them.
Aim for a balanced mix. A benchmark of only current-date questions will reward a retriever that sorts by date and nothing else.
| Query type | Condition | Controlling result | Hard negatives |
|---|---|---|---|
| Current, single chain | As-of today, one valid version | Latest approved version | All earlier versions |
| Past as-of date | As-of a date inside an older valid range | The version valid on that date | Newer versions with similar wording |
| Approved after, effective later | As-of between approval and effective date | The still-effective older version | The approved but not yet effective version |
| Draft present | A newer draft exists; user cannot see drafts | Last published version | The draft |
| Precedence conflict | Two valid documents disagree | The higher-precedence document plus the precedence clause | The lower-precedence document |
| Nothing controls | No version valid for that date or scope | None; the system should abstain | Every version of the family |
Grading versions as labeled hard negatives
Use graded relevance rather than relevant or not. The standard qrels format read by NIST's trec_eval tool is one line per judgment with a query id, an ignored iteration field, a document id and an integer relevance value, and trec_eval's level_for_rel option sets the minimum value counted as relevant on a multi-level scale. That lets one judgment file serve both a strict authority score and a looser topical score.
Store the reason with each judgment in a side file, because a zero for a superseded version and a zero for an unrelated page mean different things when you analyze failures.
| Grade | Meaning | Counted when level_for_rel is 3 | Counted when level_for_rel is 1 |
|---|---|---|---|
| 3 | Controlling version, passage that answers | Yes | Yes |
| 2 | Required supporting document, such as a precedence clause or base agreement | No; scored in evidence-set recall | Yes |
| 1 | Same family, topically right, but not controlling for this date, status or scope | No | Yes |
| 0 | Unrelated or wrong family | No | No |
| Forbidden list | Document the user may not see, kept outside qrels | Scored separately as a violation | Scored separately as a violation |
Metrics and pass conditions for a version authority benchmark
Report authority metrics alongside ordinary ranking metrics, never instead of them. A retriever can have strong topical recall and still put the wrong version first on a third of questions.
Set pass conditions with the person who owns the source documents, since they decide how costly a wrong version is in their process.
| Metric | Definition | Typical pass condition (set per corpus) |
|---|---|---|
| Controlling hit at k | Share of answerable queries with a grade 3 result in the top k | High floor at the k the generator actually reads |
| Wrong version first | Share of queries where a grade 1 version from the same family ranks above the grade 3 one | Near zero on precedence and past-date types |
| Evidence-set recall | Share of queries where every grade 3 and grade 2 item needed is in the top k | Floor for precedence-conflict queries |
| Forbidden retrieval rate | Share of queries returning any document the user may not see | Zero; any hit fails the run |
| Abstention accuracy | Nothing-controls queries where the system declined, plus answerable ones where it did not | Both directions reported |
| Citation support | Answers whose cited passage states the fact, for this date and scope | Checked by a reviewer on a sample |
Worked example: one policy family, four dates
The family and counts are illustrative. A returns policy has v3 valid January 1 to June 30, v4 approved in May and valid from July 1, and a v5 draft visible only to editors. The benchmark asks the same question, whether an opened item can be returned after 30 days, as of four dates and roles: March as a store associate, June 15 as an associate, August as an associate, and August as a policy editor drafting changes.
A pure similarity retriever ranks v5 first in every case, because the draft repeats the question's wording most closely. It scores a controlling hit at 1 on only the editor query, where the draft is the intended subject, and returns a forbidden draft to associates three times. A retriever that hides drafts from users who cannot see them and then sorts by date does better: it gets both August queries right but returns v4 for March and June 15, when v3 still governs. Only a retriever that filters by valid range and audience before ranking gets all four right.
| Query (illustrative) | Controlling result | Similarity only, rank 1 | Draft filter and date sort, rank 1 |
|---|---|---|---|
| March, associate | v3 | v5 draft (forbidden) | v4 (not yet valid) |
| June 15, associate | v3 | v5 draft (forbidden) | v4 (approved, not effective) |
| August, associate | v4 | v5 draft (forbidden) | v4 |
| August, editor asking about the draft | v5 | v5 | v5 |
Does the retrieved evidence authorize this exact answer for this user and time?
Retrieving the controlling version is necessary but not sufficient. The answer can still cite a passage from the right document that does not state the claimed fact, combine a clause from v4 with a number from v3, or apply a regional rule to the wrong region.
Add a reviewer check on a sample of answers with three questions: does the cited passage state the fact as given, is it from a version valid on the as-of date and in scope for the user, and is the user allowed to see it. Record the failure type when any answer is no, since a citation error and a permission error need different fixes.
For precedence questions, require both documents in the citation: the governing passage and the clause that makes it govern. An answer that quotes an amendment without showing why it overrides the master agreement is not auditable.
When an older version is the right result, and where the design stops
History questions reverse the usual rule. If a user asks what the policy said last year, or how a clause changed, the superseded version is the controlling result for that query and the current one is the hard negative. Tag these questions explicitly so the benchmark does not punish correct historical retrieval.
The design also has limits. Authority metadata is often incomplete in real repositories, and filling it in takes a records owner's time; families you cannot label should be excluded and counted. Precedence rules sometimes need legal interpretation, so contract-heavy corpora need counsel or a contracts lead to sign off the controlling labels. And a benchmark frozen on one date goes stale, so rerun it whenever the corpus, permissions, chunking or metadata change.
Sourcing version histories that make the benchmark real
The hard negatives are only hard if they come from real version chains: policies revised several times, contracts with amendments and order forms, procedures with drafts and withdrawals, each with dates and approval records. Synthetic edits tend to differ in obvious ways that a retriever can learn to spot.
When a team needs such document families, the supplier must hold the right to license them, which is separate from having access to the repository, and confidential terms or personal data need review before delivery. For version chains like these, SourceX works from a buyer's request rather than an inventory, locating U.S. businesses that hold the records; the retrieval team uses the request form to name the document types, volume, language, format, timing and licensing needs it has in mind.
Frequently asked questions
What is a controlling document version?
It is the version that governs the answer for a given user, date and scope. It may be older than the newest file, for example when a new policy is approved but not yet effective, or when a contract amendment takes precedence over a later order form.
How is this different from testing RAG on outdated documents?
Outdated-document tests usually reward the latest version. A version authority benchmark adds as-of dates, approval status, precedence and permissions, so the correct result can be an older, published or higher-precedence document.
Which retrieval metrics should a version benchmark report?
Report the controlling hit at k, the rate at which a non-controlling version ranks first, evidence-set recall for multi-document answers, forbidden retrieval rate and abstention accuracy, next to ordinary ranking metrics such as recall or nDCG.
Can trec_eval score graded version judgments?
Yes. Its qrels format carries an integer relevance value per document, and the level_for_rel option sets the minimum value counted as relevant, so the same judgments can produce a strict authority score and a looser topical score.
What should the system do when no version applies?
It should abstain or say that no document governs that date or scope. Include such questions on purpose and score abstention in both directions, so the system neither invents an answer nor declines answerable questions.
Sources
- trec_eval reads relevance judgments as qrels lines of query id, an ignored iteration field, document id and an integer relevance value. Source
- trec_eval's level_for_rel option sets the minimum judgment value needed for a document to count as relevant on a multi-level scale; the default is 1. Source
- DCMI Metadata Terms define isReplacedBy (a related resource that supplants, displaces or supersedes the described resource), its inverse replaces, valid (date or range of validity) and dateAccepted. Source
- In SharePoint libraries that track minor versions, draft visibility can be limited to people who can edit, and readers without that permission see the last major (published) version. Source
- FAR 52.215-8 resolves inconsistencies in a solicitation or contract by precedence: the Schedule (excluding specifications), representations and other instructions, contract clauses, other documents, exhibits and attachments, then the specifications. Source
- The FreshLLMs paper introduces FreshQA with never-changing, slow-changing, fast-changing and false-premise questions and reports that models struggle with fast-changing knowledge and false premises. Source
- The retrieval-augmented generation paper names providing provenance for decisions and updating world knowledge as open problems that a retrieval-based memory can help address. Source
Tell us what data you need
Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.