Skip to content

AI uses for records

Retrieval document version benchmark: finding the controlling version

By SourceX Editorial · Updated

10 sections

Short answer

Label, for every test question, which document version controls the answer for that user at that date, then score whether retrieval ranks it above superseded, draft and out-of-scope look-alikes. Use graded judgments with versions as hard negatives, measure the controlling hit, wrong-version-first rate and forbidden retrievals, and check that the cited passage supports the exact answer.

Evaluation split

Retrieval document version

This page covers retrieval document version benchmark: finding the controlling version.Illustrative

Key takeaways

  • The controlling version depends on the as-of date, approval status, precedence, scope and the user's permissions, not on recency alone.
  • Record an authority record per document family before writing questions; families you cannot label stay out.
  • Use six query types so that a retriever sorting by date or similarity cannot pass by accident.
  • Grade other versions as labeled hard negatives and score wrong-version-first and forbidden retrievals separately.
  • Check that the cited passage supports the exact answer, and tag history questions where an older version is correct.

How do you evaluate retrieval of the controlling version rather than similar text?

Build the benchmark around authority, not similarity. Each test question carries an as-of date, the asking user's role and the scope it applies to, and a reviewer records which document version controls the answer under those conditions. Every other version of the same document goes into the judgments as a labeled hard negative, because those are the passages a similarity search will find first.

Then score four things: whether the controlling version appears in the top results, whether any non-controlling version outranks it, whether the full evidence set the answer needs was retrieved, and whether anything the user may not see was returned. A final answer-level check confirms that the cited passage supports this exact answer for this user and date, or that the system abstained when nothing controls.

This guide sets out a version authority retrieval evaluation design: the authority record each document family needs, six query types, a grading scale, metrics with pass conditions, a worked example with illustrative counts, and the cases where an older version is the right result.

Why the controlling version is often not the newest one

Most stale-document tests assume the latest version is correct. In business records that assumption breaks in at least four ways, and a benchmark that only rewards recency will pass a retriever that fails all four.

Effective dates differ from creation dates: a pricing policy approved in May may not apply until July, and a question about a June order is governed by the old one. Approval status differs from existence: in SharePoint libraries that track minor versions, drafts can be limited to people who can edit, and other readers see the last published major version, so a draft newer than the published file is not the answer for most users. Several documents can be valid at once with a precedence rule between them: the federal clause at FAR 52.215-8 resolves inconsistencies in a contract by giving precedence to the Schedule over representations, then contract clauses, then other documents and attachments, then the specifications. And scope limits authority: the policy for one region, product line or customer does not control a question about another.

The broader research literature points the same way. The original retrieval-augmented generation paper named providing provenance for outputs and updating world knowledge as open problems that a retrieval memory can help with, and the FreshQA benchmark separates never-changing, slow-changing, fast-changing and false-premise questions because models struggle with knowledge that changes. Version authority is the enterprise form of that problem.

The authority record each document family needs

Before writing a single question, group documents into families, meaning every version and amendment of one policy, contract or procedure, and record their authority metadata. The Dublin Core terms give a neutral vocabulary for most of it: replaces and isReplacedBy link versions in a chain, valid records the date or range when a resource is valid, and dateAccepted records when it was accepted.

If a family cannot be filled in, it cannot be graded. Leave it out of the benchmark and note the gap rather than guessing which version controls.

The authority record each document family needs
FieldWhat it capturesExample (illustrative)
Family idWhich documents are versions or parts of one thingReturns policy, all versions
Version and linksOrder in the chain, using replaces and isReplacedByv4 replaces v3; v5 draft replaces v4
Valid rangeDates the version governs, not the date it was writtenv3 valid Jan 1 to Jun 30; v4 from Jul 1
Approval statusDraft, approved, published, withdrawnv5 draft, visible to editors only
Precedence rankOrder among documents valid at the same timeAmendment over master agreement over order form
ScopeRegion, product, customer or entity it applies toU.S. retail customers only
AudienceRoles allowed to see itAll staff; v5 editors only

Six query types that separate authority from similarity

Write questions so that the wording matches several versions about equally well and only the authority record decides the answer. Phrase each question the way a real user would, then attach the as-of date, role and scope as structured fields the retriever receives, just as a production system would pass them.

Aim for a balanced mix. A benchmark of only current-date questions will reward a retriever that sorts by date and nothing else.

Six query types that separate authority from similarity
Query typeConditionControlling resultHard negatives
Current, single chainAs-of today, one valid versionLatest approved versionAll earlier versions
Past as-of dateAs-of a date inside an older valid rangeThe version valid on that dateNewer versions with similar wording
Approved after, effective laterAs-of between approval and effective dateThe still-effective older versionThe approved but not yet effective version
Draft presentA newer draft exists; user cannot see draftsLast published versionThe draft
Precedence conflictTwo valid documents disagreeThe higher-precedence document plus the precedence clauseThe lower-precedence document
Nothing controlsNo version valid for that date or scopeNone; the system should abstainEvery version of the family

Grading versions as labeled hard negatives

Use graded relevance rather than relevant or not. The standard qrels format read by NIST's trec_eval tool is one line per judgment with a query id, an ignored iteration field, a document id and an integer relevance value, and trec_eval's level_for_rel option sets the minimum value counted as relevant on a multi-level scale. That lets one judgment file serve both a strict authority score and a looser topical score.

Store the reason with each judgment in a side file, because a zero for a superseded version and a zero for an unrelated page mean different things when you analyze failures.

Grading versions as labeled hard negatives
GradeMeaningCounted when level_for_rel is 3Counted when level_for_rel is 1
3Controlling version, passage that answersYesYes
2Required supporting document, such as a precedence clause or base agreementNo; scored in evidence-set recallYes
1Same family, topically right, but not controlling for this date, status or scopeNoYes
0Unrelated or wrong familyNoNo
Forbidden listDocument the user may not see, kept outside qrelsScored separately as a violationScored separately as a violation

Metrics and pass conditions for a version authority benchmark

Report authority metrics alongside ordinary ranking metrics, never instead of them. A retriever can have strong topical recall and still put the wrong version first on a third of questions.

Set pass conditions with the person who owns the source documents, since they decide how costly a wrong version is in their process.

Metrics and pass conditions for a version authority benchmark
MetricDefinitionTypical pass condition (set per corpus)
Controlling hit at kShare of answerable queries with a grade 3 result in the top kHigh floor at the k the generator actually reads
Wrong version firstShare of queries where a grade 1 version from the same family ranks above the grade 3 oneNear zero on precedence and past-date types
Evidence-set recallShare of queries where every grade 3 and grade 2 item needed is in the top kFloor for precedence-conflict queries
Forbidden retrieval rateShare of queries returning any document the user may not seeZero; any hit fails the run
Abstention accuracyNothing-controls queries where the system declined, plus answerable ones where it did notBoth directions reported
Citation supportAnswers whose cited passage states the fact, for this date and scopeChecked by a reviewer on a sample

Worked example: one policy family, four dates

The family and counts are illustrative. A returns policy has v3 valid January 1 to June 30, v4 approved in May and valid from July 1, and a v5 draft visible only to editors. The benchmark asks the same question, whether an opened item can be returned after 30 days, as of four dates and roles: March as a store associate, June 15 as an associate, August as an associate, and August as a policy editor drafting changes.

A pure similarity retriever ranks v5 first in every case, because the draft repeats the question's wording most closely. It scores a controlling hit at 1 on only the editor query, where the draft is the intended subject, and returns a forbidden draft to associates three times. A retriever that hides drafts from users who cannot see them and then sorts by date does better: it gets both August queries right but returns v4 for March and June 15, when v3 still governs. Only a retriever that filters by valid range and audience before ranking gets all four right.

Worked example: one policy family, four dates
Query (illustrative)Controlling resultSimilarity only, rank 1Draft filter and date sort, rank 1
March, associatev3v5 draft (forbidden)v4 (not yet valid)
June 15, associatev3v5 draft (forbidden)v4 (approved, not effective)
August, associatev4v5 draft (forbidden)v4
August, editor asking about the draftv5v5v5

Does the retrieved evidence authorize this exact answer for this user and time?

Retrieving the controlling version is necessary but not sufficient. The answer can still cite a passage from the right document that does not state the claimed fact, combine a clause from v4 with a number from v3, or apply a regional rule to the wrong region.

Add a reviewer check on a sample of answers with three questions: does the cited passage state the fact as given, is it from a version valid on the as-of date and in scope for the user, and is the user allowed to see it. Record the failure type when any answer is no, since a citation error and a permission error need different fixes.

For precedence questions, require both documents in the citation: the governing passage and the clause that makes it govern. An answer that quotes an amendment without showing why it overrides the master agreement is not auditable.

When an older version is the right result, and where the design stops

History questions reverse the usual rule. If a user asks what the policy said last year, or how a clause changed, the superseded version is the controlling result for that query and the current one is the hard negative. Tag these questions explicitly so the benchmark does not punish correct historical retrieval.

The design also has limits. Authority metadata is often incomplete in real repositories, and filling it in takes a records owner's time; families you cannot label should be excluded and counted. Precedence rules sometimes need legal interpretation, so contract-heavy corpora need counsel or a contracts lead to sign off the controlling labels. And a benchmark frozen on one date goes stale, so rerun it whenever the corpus, permissions, chunking or metadata change.

Sourcing version histories that make the benchmark real

The hard negatives are only hard if they come from real version chains: policies revised several times, contracts with amendments and order forms, procedures with drafts and withdrawals, each with dates and approval records. Synthetic edits tend to differ in obvious ways that a retriever can learn to spot.

When a team needs such document families, the supplier must hold the right to license them, which is separate from having access to the repository, and confidential terms or personal data need review before delivery. For version chains like these, SourceX works from a buyer's request rather than an inventory, locating U.S. businesses that hold the records; the retrieval team uses the request form to name the document types, volume, language, format, timing and licensing needs it has in mind.

Frequently asked questions

What is a controlling document version?

It is the version that governs the answer for a given user, date and scope. It may be older than the newest file, for example when a new policy is approved but not yet effective, or when a contract amendment takes precedence over a later order form.

How is this different from testing RAG on outdated documents?

Outdated-document tests usually reward the latest version. A version authority benchmark adds as-of dates, approval status, precedence and permissions, so the correct result can be an older, published or higher-precedence document.

Which retrieval metrics should a version benchmark report?

Report the controlling hit at k, the rate at which a non-controlling version ranks first, evidence-set recall for multi-document answers, forbidden retrieval rate and abstention accuracy, next to ordinary ranking metrics such as recall or nDCG.

Can trec_eval score graded version judgments?

Yes. Its qrels format carries an integer relevance value per document, and the level_for_rel option sets the minimum value counted as relevant, so the same judgments can produce a strict authority score and a looser topical score.

What should the system do when no version applies?

It should abstain or say that no document governs that date or scope. Include such questions on purpose and score abstention in both directions, so the system neither invents an answer nor declines answerable questions.

Sources

  • trec_eval reads relevance judgments as qrels lines of query id, an ignored iteration field, document id and an integer relevance value. Source
  • trec_eval's level_for_rel option sets the minimum judgment value needed for a document to count as relevant on a multi-level scale; the default is 1. Source
  • DCMI Metadata Terms define isReplacedBy (a related resource that supplants, displaces or supersedes the described resource), its inverse replaces, valid (date or range of validity) and dateAccepted. Source
  • In SharePoint libraries that track minor versions, draft visibility can be limited to people who can edit, and readers without that permission see the last major (published) version. Source
  • FAR 52.215-8 resolves inconsistencies in a solicitation or contract by precedence: the Schedule (excluding specifications), representations and other instructions, contract clauses, other documents, exhibits and attachments, then the specifications. Source
  • The FreshLLMs paper introduces FreshQA with never-changing, slow-changing, fast-changing and false-premise questions and reports that models struggle with fast-changing knowledge and false premises. Source
  • The retrieval-augmented generation paper names providing provenance for decisions and updating world knowledge as open problems that a retrieval-based memory can help address. Source

Tell us what data you need

Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.

See if your company qualifies