Skip to content

AI uses for records

How should a RAG evaluation handle a document that changes the answer?

By SourceX Editorial · Updated

10 sections

Short answer

Treat every corpus change as an evaluation event. Each gold item should record its controlling source, effective dates and required evidence, so that when a document is added, amended or withdrawn you can find the items it affects, re-judge their expected answers, and rerun them before and after the change. A regression is an answer that should have changed but did not, or changed when it should not have.

Retrieval

RAG corpus update regression

This page covers how should a RAG evaluation handle a document that changes the answer.Illustrative

Key takeaways

  • A gold answer is only correct relative to a corpus state, so every gold item needs its controlling source and the dates it is valid for.
  • Faithfulness and context recall can stay high after an answer goes stale, because both are measured against retrieved text and a reference that may be outdated.
  • Find affected items two ways: by document IDs in each item's evidence set, and by searching the new document against gold queries that never cited it.
  • Score the run as flips: answers correctly updated, answers left stale, answers wrongly changed, and abstentions where authority is unresolved.
  • Some questions are about the past; for them the superseded document stays controlling and a changed answer is the regression.

How should a RAG evaluation handle a new document that changes the correct answer?

By treating the corpus change as the trigger for a targeted regression run. Before the new document goes live, find every gold item whose expected answer could depend on it, have a person re-judge the expected answer against the new corpus state, and run those items against both the old and the new index. Release the update only when the answers that should change do, and the answers that should not change do not.

That requires gold items that can be re-judged. A question and an expected answer are not enough; each item needs to say which document controls the answer, from when to when that is true, who is allowed to see it, and which passages must be cited. Without that, a team cannot tell a stale expected answer from a wrong model answer.

This page covers RAG corpus update regression for enterprise knowledge bases, policy libraries, product documentation and similar corpora that change in place, where yesterday's correct answer can become unreliable overnight.

Why do standard RAG scores stay high after the answer goes stale?

Because the common metrics measure consistency with what was retrieved and with the stored reference, not with what is currently true. In the Ragas library, faithfulness checks whether every claim in a response is supported by the retrieved context. If the retriever returns last year's policy, a response that repeats it faithfully scores well.

Context recall has the mirror problem. Ragas computes it as the share of claims in a reference answer that the retrieved context supports, so it always needs a reference. When the reference itself was written against the old document, retrieving the old document scores perfectly and retrieving the new one is penalized.

Answers also change at different speeds. The FreshQA benchmark sorts questions into never-changing, slow-changing, fast-changing and false-premise groups. The same split is useful for an internal gold set: tag each item by how often its controlling source is revised, and expect most regression work to come from the fast-changing tag.

Which corpus changes should trigger a regression run?

Every change that can alter which document controls an answer, which is more than new content. Use the table to decide which gold items to re-judge and what a passing answer looks like after the change.

Which corpus changes should trigger a regression run?
Corpus changeGold items to re-judgeExpected behavior after the changePass check
New version supersedes an old documentEvery item whose evidence set cites the old documentAnswer from the new version; old version not cited for current questionsCitation points to the new document ID and version
Amendment changes one sectionItems citing that section, plus items on the same topic that cite other sectionsChanged facts updated, unchanged facts stableUnchanged items return the same answer as before
Document withdrawnItems citing itA replacement source, or abstention if none existsNo citation to the withdrawn document
New document dated in the futureItems on its topicCurrent answer until the effective date, then the new oneRun once at a date before and once after the effective date
Lower-authority document contradicts a controlling oneItems on the contested topicAnswer from the controlling source; conflict noted if relevantControlling source ranked and cited above the contradicting one
Permission change on a documentItems whose users lost or gained accessUsers without access get no content from itNo passage from the document in answers for those users
Re-chunking or metadata change onlyA stratified sample of all itemsNo answer changesAnswer and citation identical to the previous run
New document on a topic not covered beforeItems that previously required abstentionAnswer now available from the new sourceFormer abstentions become cited answers

What should each gold item record so it can be re-judged?

Enough for a reviewer who did not write it to decide, months later, whether the expected answer still holds. Public benchmarks with time-sensitive answers already work this way: the FreshQA maintainers update the dataset on a regular schedule and keep earlier dated versions available. An internal gold set should do the same, with a version per corpus state rather than per calendar date.

Two fields do most of the work. The valid-from and valid-to dates let the same question carry different expected answers for different points in time, and the hard negatives make the superseded version an explicit test rather than an accident. When a reviewer re-judges an item, close the old expected answer with an end date instead of overwriting it, so earlier runs remain interpretable.

  • Query and query intent, including whether it asks about now or about a stated past date.
  • Controlling source: document ID, version and the section or passage that decides the answer.
  • Authority rank of that source within its topic, for example policy over FAQ over forum post.
  • Valid from and valid to dates for the expected answer.
  • Permission scope: the user groups allowed to receive the answer.
  • Hard negatives: superseded versions and look-alike documents that must not be cited.
  • Required evidence set: every passage the answer must rest on, not only the first one.
  • Expected answer and the facts in it that a grader checks.
  • Abstention rule: when the correct output is to say the corpus does not settle the question.
  • Last re-judged date, by whom, and against which corpus version.

How do you find the gold items a new document affects?

Use two passes, because the obvious one misses the most dangerous cases. The first pass is mechanical. If documents are loaded with stable IDs, the indexing layer can tell you what changed: LlamaIndex, for example, supports insert, delete, update and refresh, and its refresh operation updates documents that keep an ID but have new text, inserts new ones and returns which documents were refreshed. Join that list to the evidence sets in your gold items.

The second pass looks for hidden impact. A brand-new document has no gold items that cite it, yet it may now be the controlling source for questions answered from elsewhere. Run each new or substantially changed document as a query against the gold set's questions, take the closest matches, and send them to a reviewer. Items with a shared topic tag or the same authority owner go to the reviewer too.

Keep both lists in the run record. The first proves coverage of what changed; the second shows the team looked for answers that changed without any cited document changing.

Does the retrieved evidence authorize this exact answer for this user and time?

Ask four questions of every answer in the regression run, and fail the item if any answer is no. Similarity between the question and the passage is not on the list, because a superseded document is usually the most similar passage of all.

At a glance

Does the retrieved evidence authorize this exact answer for this user and time

  • Authority: is the cited source the controlling one for this topic, or a lower-ranked document that happens to agree or disagree?
  • Version and date: was the cited version in force on the date the question is about?
  • Completeness: does the cited evidence cover every fact in the answer, including exceptions stated elsewhere in the same document?
  • Access: is this user allowed to receive content from every cited passage?

Worked example: a revised returns policy (illustrative)

All figures are illustrative. A retailer's support assistant answers from a corpus of policy documents, help-center articles and agent macros. The returns policy is revised: the return window for one product category shortens, effective on the first of next month.

The mechanical pass finds 42 gold items citing the old policy. The hidden-impact pass adds 11 more that cite help-center articles repeating the old window. Reviewers re-judge all 53: 18 expected answers change, 35 stay the same. The help-center articles have not been updated yet, so the reviewers also mark them as lower authority than the policy for this topic.

Run against the new index, 14 of the 18 answers update correctly and 4 still quote the help-center wording, because those articles rank higher on similarity. Of the 35 that should be stable, 2 change wrongly after re-chunking split a table of exceptions. The release is held until the help-center articles are updated or down-ranked and the chunking fix is in place.

The exception: 6 of the 53 items ask about orders placed before the effective date. For them the old policy remains controlling, and an answer that switches to the new window is the regression. Without a stated date in each gold item, those 6 would have been re-judged wrongly and the evaluation would have rewarded the error.

Scoring the before-and-after run

Report changes, not a single average. An overall accuracy that stays flat can hide equal numbers of fixes and new breakages. Use four counts on the affected items, plus the stable sample.

At a glance

Scoring the before-and-after run

Keep the four counts per change, not pooled across a quarter, so each count can be traced to the document that caused it. Left stale usually points to retrieval ranking or to lower-authority copies of the old wording. Wrongly changed usually points to chunking, metadata or a new document that was mistaken for a controlling one.

Set the release gate before the run. A practical gate: no left-stale or wrongly-changed answers on items tagged high impact, no access failures at all, and identical answers on the stable sample apart from items a reviewer has explained. Anything else holds the index swap, and the previous index keeps serving until the fixes are in.

  • Correctly updated: the expected answer changed and the system's answer changed to match it, citing the controlling source.
  • Left stale: the expected answer changed and the system still gives the old one.
  • Wrongly changed: the expected answer did not change and the system's answer did.
  • Correct abstention: authority is unresolved or the user lacks access, and the system says so instead of answering.
  • Stable sample: answers on unaffected items identical to the previous run.

Which authority calls need a named owner rather than a metric?

A regression run can show that the answer moved; it cannot decide which document should win. Authority rankings, effective dates and permission scopes are business decisions, and each topic needs a person who owns them. If two documents of equal rank disagree, the right fix is in the corpus, not in the retriever.

The LlamaIndex failure mode checklist describes stale data appearing after source documents were updated, and recommends document ID tracking, timestamps and version metadata, and finishing ingestion or refresh jobs before routing production traffic to the new index. Those are pipeline controls. Pair them with an owner sign-off on the re-judged gold items before the swap, and keep the previous index available until the run passes.

Stop and bring in the topic owner, and counsel where relevant, when a change affects regulated or contractual content, when no authority rank exists for a topic, or when the regression run shows answers reaching users who should not see the document.

Sourcing document histories for authority tests

Realistic authority tests need corpora with real revision histories: policies with dated versions, amendments, withdrawn documents and lower-authority material that repeats old wording. Those histories exist inside most operating companies, in document management systems, help centers and ticketing macros, and they are hard to synthesize convincingly.

SourceX keeps no datasets in stock; for each buyer request it sources records from U.S. businesses. A team that needs versioned policy documents with effective dates can set out the scope, volume, language, format, timing and licensing terms in the request form.

Frequently asked questions

What is a RAG corpus update regression?

It is a drop in answer quality caused by a change to the documents a RAG system retrieves from, rather than by a change to the model or prompt. Typical forms are answers left stale after a revision, answers that change when they should not, and citations to withdrawn or superseded documents.

How often should gold answers be re-judged?

Whenever a corpus change touches their controlling source or topic, and on a fixed schedule for fast-changing topics. Record the corpus version each item was last judged against, so stale items are easy to find.

Should the old version of a document stay in the index?

Only if users can ask about past dates and the old version carries effective dates the retriever can filter on. Otherwise remove it or mark it superseded, and keep it in the gold set as a hard negative.

Can an LLM re-judge the expected answers automatically?

It can propose updates and flag likely affected items, which saves reviewer time. The final call on which document controls an answer should stay with the person who owns that topic, because authority is a business decision.

Is re-indexing without content changes safe to skip testing?

No. Chunking, embedding and metadata changes can move which passage ranks first and split tables or exceptions across chunks. Run a stratified sample and expect identical answers.

Sources

  • Ragas defines faithfulness as whether every claim in a response is supported by the retrieved context. Source
  • Ragas context recall requires a reference and measures the share of reference claims supported by the retrieved context. Source
  • The FreshLLMs paper introduces FreshQA with never-changing, slow-changing, fast-changing and false-premise question groups. Source
  • The FreshQA maintainers update the dataset regularly and keep earlier dated versions available. Source
  • LlamaIndex documents insert, delete, update and refresh operations, where refresh updates documents with the same ID and changed text, inserts new ones and reports which were refreshed. Source
  • The LlamaIndex RAG failure mode checklist lists stale data after source updates as a symptom and recommends ID tracking, version metadata and completing refresh jobs before routing production traffic. Source

Tell us what data you need

Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.

See if your company qualifies