Skip to content

AI uses for records

RAG evidence completeness dataset: testing answers that need more than one chunk

By SourceX Editorial · Updated

9 sections

Short answer

A RAG evidence completeness dataset tests whether a system retrieved every piece of evidence an answer needs, not just one relevant chunk. For each question, label the minimal evidence set as groups of required facts (rule, condition, exception, current value), add variants where one group is missing, and score set recall, citation support and correct abstention. One matching passage counts as success only when the set has one member.

Retrieval

RAG evidence completeness

This page covers RAG evidence completeness dataset: testing answers that need more than one chunk.Illustrative

Key takeaways

  • A relevant chunk is not sufficient evidence when the answer depends on a rule, a condition, a case fact and an exception together.
  • Label each benchmark item with evidence groups and a sufficiency rule, built from the governing sources, not from one retriever's results.
  • Add an insufficient variant for each multi-group item by removing one required group, and score whether the system notices.
  • Report group-complete rate, group recall, citation support, abstention and hard negative use separately.
  • Completeness labels expire when documents change, so tie them to a refresh trigger and a source owner's sign-off.

Why one relevant chunk can still leave the answer unsupported

A chunk can be on topic and still not justify the answer, because many business questions are only answerable from several facts at once: the rule, the condition that triggers it, the exception that overrides it and the value that applies to this case. Retrieval metrics that score each chunk for relevance on its own reward the first of those facts and say nothing about the rest.

Research on sufficient context makes the gap concrete. Joren and colleagues at Google separate questions whose retrieved context is sufficient from those whose context is not, and report that larger models answer well when the context suffices but often give an incorrect answer instead of abstaining when it does not. A benchmark that never labels sufficiency cannot see that failure, because the retrieved chunk looked relevant in both cases.

Multi-hop datasets were built around the same insight. HotpotQA asks crowd workers to mark the sentence-level supporting facts behind each answer, so a system can be scored on whether it found the evidence and not only on whether the final string matched. Evidence completeness takes that idea into enterprise retrieval, where the supporting facts sit in contracts, tickets, policies and logs instead of Wikipedia.

Which questions need a multi-document evidence set?

Any question whose answer combines a general rule with case facts, or compares, counts or dates things across records, needs more than one chunk. The FRAMES benchmark from Google tags its questions with reasoning types such as numerical, tabular, multiple constraints, temporal and post-processing, and its dataset card says each question needs information from between 2 and 15 Wikipedia articles. Business corpora produce the same shapes with different documents.

The table below maps common question shapes to the smallest evidence set that can support them. The examples are illustrative and use generic business systems; replace them with your own corpus.

Which questions need a multi-document evidence set?
Question shapeIllustrative business questionMinimum evidence setWhat a single chunk gets wrong
Rule plus case factsDoes this customer qualify for an SLA service credit for last week's outage?SLA clause, the customer's order form tier, incident record with durationQuotes the credit rule but cannot say whether this customer and incident meet it
Rule plus exceptionCan this refund be issued without manager approval?Refund policy, approval matrix, any regional exception memoApplies the general limit and misses the override
Current state across versionsWhat is the current onboarding checklist for contractors?Latest approved version, change log or supersession noteReturns an older version that ranks higher because it is longer
Aggregation or countHow many open P1 tickets mention the billing migration?Every matching ticket, or a report that is itself authoritativeCounts what happened to be retrieved
ComparisonWhich of two vendor contracts has the shorter termination notice?The termination clause from each contract and any amendmentDescribes one contract and guesses the other
Temporal conditionWas this invoice paid within the agreed term?Payment terms clause, invoice date, payment recordAnswers from the terms alone, without the dates
Bridge entityWho approved the change that caused incident 4471?Incident record naming the change, change record naming the approverStops at the incident and names the wrong person or none

The multi-document evidence completeness benchmark brief

The brief below is the record each benchmark item should carry. It is written so that a retrieval engineer can build the item, a source owner can confirm it and a scorer can run it without guessing. Fill every field for every item; an item missing its sufficiency rule or abstention condition cannot be scored for completeness.

At a glance

The multi-document evidence completeness benchmark brief

The multi-document evidence completeness benchmark brief
FieldWhat to recordIllustrative valueWhy it matters
Question and intentThe question as a user would ask it, plus a one-line intent noteIs Acme owed a service credit for the 4 March outage?Intent decides which facts are required
User and as-of dateRole, permissions group and the date the answer must be true forAccount manager, sales group, as of 10 MarchThe same question can need different evidence for another user or date
Evidence unitsStable IDs for each passage, row or field that can support part of the answerMSA-7.2, ORDER-118 line 3, INC-4471 summaryLets scorers match retrieval at passage or span level
Evidence groups and rolesUnits grouped by role: rule, condition, case fact, exception, current valueRule: MSA-7.2; tier: ORDER-118; duration: INC-4471; exclusion: MSA-7.4Completeness is judged per group, not per unit
Sufficiency ruleWhich groups are all required and where alternatives existAll of rule, tier and duration; exclusion check required; either of two duration sourcesSeparates a missing fact from a redundant one
Controlling source and versionWhich document governs if two disagree, and its effective dateMSA amendment 2 replaces clause 7.2 from 1 JanuaryStops a complete but outdated set from passing
Hard negativesPlausible units that look relevant but must not be usedDraft SLA, another customer's order form, a chat restatementTests that the set is the right set, not just a full one
Expected answer and claimsReference answer broken into atomic claims, each tied to a groupCredit applies; claim 1 from rule, claim 2 from tier, claim 3 from durationLets claim-level recall and citation checks run
Insufficient variantThe same question with one group removed from the corpus, and the expected abstentionRemove INC-4471; expected answer: cannot confirm durationMeasures whether the system notices the gap
Review recordAnnotators, adjudicator, source owner sign-off and dateTwo annotators, retrieval lead adjudicated, policy owner confirmedMakes disagreement and refresh visible

How to label the required evidence set without copying one retriever's output

Build each item from the sources outward: start with documents a domain owner says govern the answer, write the question against them, then mark the minimal set. Labeling from a retriever's top results bakes that retriever's blind spots into the benchmark, so the missing group never appears as missing.

MuSiQue shows a disciplined way to do this. Its authors compose multihop questions bottom-up from single-hop questions, so each hop has its own answer and supporting paragraph, and they report that the construction makes shortcut reasoning much less effective. In an enterprise set, each evidence group plays the role of a single-hop sub-question: can this group alone answer its part of the question, and does the final answer depend on it?

Checklist · 5 items

  1. Ask two annotators to mark the minimal sufficient set independently, then adjudicate differences with the source owner.
  2. Mark alternatives explicitly: if either the incident record or the monitoring export proves duration, they form one group with an any-of rule.
  3. Test minimality by deleting each group in turn and asking whether the reference answer still follows. If it does, the group was not required.
  4. Record units at the granularity your system cites. If answers cite passages, label passages; if they cite table rows, label rows.
  5. Keep the reference answer's claims aligned to groups, so a scorer can tell which missing group caused which unsupported claim.

0 of 5 ticked. Tick each item as you go.

Scoring a run: set recall, citation support and abstention

Score completeness at the group level first, then check that the answer's claims are supported by what was cited, then check behavior on the insufficient variants. A system that retrieves every group but cites only one, or that answers confidently when a group was removed, fails in a way per-chunk relevance never shows.

Two existing tools cover parts of this. The Ragas context recall metric breaks the reference answer into claims and measures the share that can be attributed to the retrieved context. ALCE's evaluation code scores citation recall by checking with an NLI model whether each output sentence is entailed by the passages it cites, and citation precision by checking for cited passages that were not needed. The group-level and abstention metrics below are what an evidence completeness dataset adds.

Scoring a run: set recall, citation support and abstention
MetricDefinitionNeeds from the briefFailure it exposes
Group-complete rateShare of answerable items where every required group has at least one unit in the retrieved contextEvidence groups and sufficiency ruleRetriever stops after the first good chunk
Group recallRequired groups retrieved divided by required groups, averaged over itemsEvidence groupsPartial sets that per-item pass or fail scores hide
Claim-level context recallShare of reference claims attributable to retrieved contextExpected answer claimsContext that covers the topic but not the claims
Citation recall and precisionWhether each answer sentence is entailed by its citations, and whether citations are necessaryEvidence units and expected claimsAnswers that are right but unsupported by what they cite
Correct abstention rateShare of insufficient variants where the system declines or names the missing factInsufficient variantConfident answers built on incomplete evidence
Hard negative use rateShare of answers that cite a unit labeled as a hard negativeHard negativesComplete-looking sets assembled from the wrong documents

Worked example: a service credit question with five candidate passages

This example is illustrative. The documents, IDs and durations are invented to show the method, and the percentages and hours are labels for the scenario, not measurements.

The question is whether a customer is owed a service credit for an outage on 4 March. The corpus holds a master services agreement with an availability clause and an exclusions clause, the customer's order form, an incident record and a draft revision of the agreement that was never signed. In this illustration the retriever ranks the availability clause first because it shares the most words with the question.

  • Answerable item: all four required groups retrieved; the answer states the credit applies and cites each group.
  • Insufficient variant: the incident record is removed. The correct output says the credit cannot be confirmed because the outage duration and cause are not in the available records.
  • Typical failure: the system retrieves the clause and the draft, cites the draft's threshold and answers yes. Group recall is one of four, and the hard negative use flag is raised.
Worked example: a service credit question with five candidate passages
Candidate unitRole in the briefRequired?If the system relies on it alone
Availability clause: credit when monthly uptime falls below the committed level for the tierRuleYesStates the rule but not whether this customer or incident meets it
Order form: customer is on the higher availability tierConditionYesKnows the tier, not the outage length
Incident record: outage lasted 3 hours (illustrative), cause unplannedCase factYesKnows the outage, not the threshold
Exclusions clause: scheduled maintenance does not count as downtimeException checkYes, to confirm it does not applyMisses that a maintenance window could cancel the credit
Unsigned draft agreement with a different thresholdHard negativeNo, must not be citedApplies a threshold that never took effect

When is one passage genuinely enough, and when should no answer pass?

Some items really do have a one-unit evidence set, such as a definition, a single field lookup or a policy statement with no conditions, and those items should stay in the benchmark so the system is not rewarded for over-retrieving. Label them with a sufficiency rule of one group, and check that citation precision stays high on them.

At the other end, keep items where no complete set exists in the corpus at all. MuSiQue's full release pairs each answerable question with an unanswerable contrast version in which the supporting paragraph for one sub-question is absent from the context, and the authors report a further drop in model performance on that harder setting. Enterprise versions are easy to make from the brief: remove one required group and record the expected abstention.

  • Keep a mix: single-group items, multi-group items and insufficient variants, reported separately so one category cannot mask another.
  • Do not count over-retrieval as a failure on its own; count it through citation precision and hard negative use.
  • Treat an answer that names the missing fact as a pass on an insufficient variant, even if it does not use the word abstain.

Where completeness labels stop being reliable

Completeness labels are judgments about a corpus at a date, so they drift when documents are added, superseded or re-chunked, and they need a refresh trigger tied to those changes. A label that was correct before an amendment can become wrong without anyone touching the benchmark.

Annotators also disagree about minimality, especially for exception checks: is the exclusions clause required evidence or background? Record the adjudication and the reason, and have the retrieval engineer and the owner of the governing documents sign off together, since one knows the system's units and the other knows which source controls.

Automated sufficiency judges help scale, but they are not ground truth. The sufficient context paper uses an LLM autorater to classify instances and notes that simply abstaining whenever the rater says the context is insufficient can lower overall performance, because models sometimes answer correctly anyway. Use automated judges to triage and humans to label the items that decide release.

Permissions add a further limit. A set can be complete for one user and incomplete for another who cannot see the order form, so the brief's user and as-of fields are part of the label, not metadata.

Sourcing realistic multi-document corpora for completeness testing

Public multi-hop datasets are built from encyclopedia articles, so they rarely contain the contract, ticket and policy combinations that make enterprise questions hard. A completeness benchmark is most useful when its documents come from real operating systems where rules, case records and amendments actually interact.

Corpora like that are what SourceX sources from U.S. businesses on request, with nothing held in stock, so an evaluation team can set out the document types, volume, language, format, timing and licensing it needs on the request-data form. The supplying business approves the scope and terms and signs the agreement before any records are delivered.

Frequently asked questions

What is evidence completeness in RAG evaluation?

Evidence completeness asks whether the retrieved context contains every fact the answer depends on, not just one relevant passage. It is labeled as a minimal set of evidence groups per question and scored at the group level. A system passes only when all required groups are retrieved and cited.

How is evidence completeness different from context recall?

Context recall, as Ragas defines it, measures the share of reference-answer claims attributable to the retrieved context. Evidence completeness adds structure: it names which groups are required, which are alternatives and which documents are hard negatives. It also includes insufficient variants that test whether the system abstains.

How many documents should a multi-document benchmark question need?

As many as the answer genuinely depends on and no more. The business question shapes in this guide need two to four groups, while public benchmarks such as FRAMES go up to fifteen articles. Test minimality by removing each group and checking whether the reference answer still follows.

Should a RAG system refuse to answer when one piece of evidence is missing?

It should not present an unsupported answer as settled. The better behavior is to answer what the evidence supports and name the missing fact, or decline. Score that behavior on insufficient variants built by removing one required group from the corpus.

Can HotpotQA or MuSiQue replace an enterprise completeness benchmark?

They are useful for checking multi-hop retrieval and abstention in general, and their supporting-fact and contrast designs are worth copying. They do not contain contracts, tickets, policy versions or permissions, so they cannot test whether a system finds the governing clause and the case record together in your corpus.

Sources

  • HotpotQA collects sentence-level supporting facts from crowd workers for each multi-hop answer. Source
  • MuSiQue composes multihop questions from single-hop questions and pairs answerable questions with unanswerable contrast versions. Source
  • FRAMES questions carry reasoning-type labels and each needs information from between 2 and 15 Wikipedia articles. Source
  • Models often answer incorrectly instead of abstaining when retrieved context is insufficient, and blanket abstention can lower overall performance. Source
  • ALCE evaluates citation quality, scoring whether each sentence is entailed by its cited passages and whether citations were necessary. Source
  • Ragas context recall is the share of reference-answer claims attributable to the retrieved context. Source

Tell us what data you need

Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.

See if your company qualifies