AI uses for records
RAG evidence completeness dataset: testing answers that need more than one chunk
By SourceX Editorial · Updated
9 sections
Short answer
A RAG evidence completeness dataset tests whether a system retrieved every piece of evidence an answer needs, not just one relevant chunk. For each question, label the minimal evidence set as groups of required facts (rule, condition, exception, current value), add variants where one group is missing, and score set recall, citation support and correct abstention. One matching passage counts as success only when the set has one member.
Retrieval
RAG evidence completeness
Key takeaways
- A relevant chunk is not sufficient evidence when the answer depends on a rule, a condition, a case fact and an exception together.
- Label each benchmark item with evidence groups and a sufficiency rule, built from the governing sources, not from one retriever's results.
- Add an insufficient variant for each multi-group item by removing one required group, and score whether the system notices.
- Report group-complete rate, group recall, citation support, abstention and hard negative use separately.
- Completeness labels expire when documents change, so tie them to a refresh trigger and a source owner's sign-off.
Why one relevant chunk can still leave the answer unsupported
A chunk can be on topic and still not justify the answer, because many business questions are only answerable from several facts at once: the rule, the condition that triggers it, the exception that overrides it and the value that applies to this case. Retrieval metrics that score each chunk for relevance on its own reward the first of those facts and say nothing about the rest.
Research on sufficient context makes the gap concrete. Joren and colleagues at Google separate questions whose retrieved context is sufficient from those whose context is not, and report that larger models answer well when the context suffices but often give an incorrect answer instead of abstaining when it does not. A benchmark that never labels sufficiency cannot see that failure, because the retrieved chunk looked relevant in both cases.
Multi-hop datasets were built around the same insight. HotpotQA asks crowd workers to mark the sentence-level supporting facts behind each answer, so a system can be scored on whether it found the evidence and not only on whether the final string matched. Evidence completeness takes that idea into enterprise retrieval, where the supporting facts sit in contracts, tickets, policies and logs instead of Wikipedia.
Which questions need a multi-document evidence set?
Any question whose answer combines a general rule with case facts, or compares, counts or dates things across records, needs more than one chunk. The FRAMES benchmark from Google tags its questions with reasoning types such as numerical, tabular, multiple constraints, temporal and post-processing, and its dataset card says each question needs information from between 2 and 15 Wikipedia articles. Business corpora produce the same shapes with different documents.
The table below maps common question shapes to the smallest evidence set that can support them. The examples are illustrative and use generic business systems; replace them with your own corpus.
| Question shape | Illustrative business question | Minimum evidence set | What a single chunk gets wrong |
|---|---|---|---|
| Rule plus case facts | Does this customer qualify for an SLA service credit for last week's outage? | SLA clause, the customer's order form tier, incident record with duration | Quotes the credit rule but cannot say whether this customer and incident meet it |
| Rule plus exception | Can this refund be issued without manager approval? | Refund policy, approval matrix, any regional exception memo | Applies the general limit and misses the override |
| Current state across versions | What is the current onboarding checklist for contractors? | Latest approved version, change log or supersession note | Returns an older version that ranks higher because it is longer |
| Aggregation or count | How many open P1 tickets mention the billing migration? | Every matching ticket, or a report that is itself authoritative | Counts what happened to be retrieved |
| Comparison | Which of two vendor contracts has the shorter termination notice? | The termination clause from each contract and any amendment | Describes one contract and guesses the other |
| Temporal condition | Was this invoice paid within the agreed term? | Payment terms clause, invoice date, payment record | Answers from the terms alone, without the dates |
| Bridge entity | Who approved the change that caused incident 4471? | Incident record naming the change, change record naming the approver | Stops at the incident and names the wrong person or none |
The multi-document evidence completeness benchmark brief
The brief below is the record each benchmark item should carry. It is written so that a retrieval engineer can build the item, a source owner can confirm it and a scorer can run it without guessing. Fill every field for every item; an item missing its sufficiency rule or abstention condition cannot be scored for completeness.
At a glance
The multi-document evidence completeness benchmark brief
| Field | What to record | Illustrative value | Why it matters |
|---|---|---|---|
| Question and intent | The question as a user would ask it, plus a one-line intent note | Is Acme owed a service credit for the 4 March outage? | Intent decides which facts are required |
| User and as-of date | Role, permissions group and the date the answer must be true for | Account manager, sales group, as of 10 March | The same question can need different evidence for another user or date |
| Evidence units | Stable IDs for each passage, row or field that can support part of the answer | MSA-7.2, ORDER-118 line 3, INC-4471 summary | Lets scorers match retrieval at passage or span level |
| Evidence groups and roles | Units grouped by role: rule, condition, case fact, exception, current value | Rule: MSA-7.2; tier: ORDER-118; duration: INC-4471; exclusion: MSA-7.4 | Completeness is judged per group, not per unit |
| Sufficiency rule | Which groups are all required and where alternatives exist | All of rule, tier and duration; exclusion check required; either of two duration sources | Separates a missing fact from a redundant one |
| Controlling source and version | Which document governs if two disagree, and its effective date | MSA amendment 2 replaces clause 7.2 from 1 January | Stops a complete but outdated set from passing |
| Hard negatives | Plausible units that look relevant but must not be used | Draft SLA, another customer's order form, a chat restatement | Tests that the set is the right set, not just a full one |
| Expected answer and claims | Reference answer broken into atomic claims, each tied to a group | Credit applies; claim 1 from rule, claim 2 from tier, claim 3 from duration | Lets claim-level recall and citation checks run |
| Insufficient variant | The same question with one group removed from the corpus, and the expected abstention | Remove INC-4471; expected answer: cannot confirm duration | Measures whether the system notices the gap |
| Review record | Annotators, adjudicator, source owner sign-off and date | Two annotators, retrieval lead adjudicated, policy owner confirmed | Makes disagreement and refresh visible |
How to label the required evidence set without copying one retriever's output
Build each item from the sources outward: start with documents a domain owner says govern the answer, write the question against them, then mark the minimal set. Labeling from a retriever's top results bakes that retriever's blind spots into the benchmark, so the missing group never appears as missing.
MuSiQue shows a disciplined way to do this. Its authors compose multihop questions bottom-up from single-hop questions, so each hop has its own answer and supporting paragraph, and they report that the construction makes shortcut reasoning much less effective. In an enterprise set, each evidence group plays the role of a single-hop sub-question: can this group alone answer its part of the question, and does the final answer depend on it?
Checklist · 5 items
- Ask two annotators to mark the minimal sufficient set independently, then adjudicate differences with the source owner.
- Mark alternatives explicitly: if either the incident record or the monitoring export proves duration, they form one group with an any-of rule.
- Test minimality by deleting each group in turn and asking whether the reference answer still follows. If it does, the group was not required.
- Record units at the granularity your system cites. If answers cite passages, label passages; if they cite table rows, label rows.
- Keep the reference answer's claims aligned to groups, so a scorer can tell which missing group caused which unsupported claim.
0 of 5 ticked. Tick each item as you go.
Scoring a run: set recall, citation support and abstention
Score completeness at the group level first, then check that the answer's claims are supported by what was cited, then check behavior on the insufficient variants. A system that retrieves every group but cites only one, or that answers confidently when a group was removed, fails in a way per-chunk relevance never shows.
Two existing tools cover parts of this. The Ragas context recall metric breaks the reference answer into claims and measures the share that can be attributed to the retrieved context. ALCE's evaluation code scores citation recall by checking with an NLI model whether each output sentence is entailed by the passages it cites, and citation precision by checking for cited passages that were not needed. The group-level and abstention metrics below are what an evidence completeness dataset adds.
| Metric | Definition | Needs from the brief | Failure it exposes |
|---|---|---|---|
| Group-complete rate | Share of answerable items where every required group has at least one unit in the retrieved context | Evidence groups and sufficiency rule | Retriever stops after the first good chunk |
| Group recall | Required groups retrieved divided by required groups, averaged over items | Evidence groups | Partial sets that per-item pass or fail scores hide |
| Claim-level context recall | Share of reference claims attributable to retrieved context | Expected answer claims | Context that covers the topic but not the claims |
| Citation recall and precision | Whether each answer sentence is entailed by its citations, and whether citations are necessary | Evidence units and expected claims | Answers that are right but unsupported by what they cite |
| Correct abstention rate | Share of insufficient variants where the system declines or names the missing fact | Insufficient variant | Confident answers built on incomplete evidence |
| Hard negative use rate | Share of answers that cite a unit labeled as a hard negative | Hard negatives | Complete-looking sets assembled from the wrong documents |
Worked example: a service credit question with five candidate passages
This example is illustrative. The documents, IDs and durations are invented to show the method, and the percentages and hours are labels for the scenario, not measurements.
The question is whether a customer is owed a service credit for an outage on 4 March. The corpus holds a master services agreement with an availability clause and an exclusions clause, the customer's order form, an incident record and a draft revision of the agreement that was never signed. In this illustration the retriever ranks the availability clause first because it shares the most words with the question.
- Answerable item: all four required groups retrieved; the answer states the credit applies and cites each group.
- Insufficient variant: the incident record is removed. The correct output says the credit cannot be confirmed because the outage duration and cause are not in the available records.
- Typical failure: the system retrieves the clause and the draft, cites the draft's threshold and answers yes. Group recall is one of four, and the hard negative use flag is raised.
| Candidate unit | Role in the brief | Required? | If the system relies on it alone |
|---|---|---|---|
| Availability clause: credit when monthly uptime falls below the committed level for the tier | Rule | Yes | States the rule but not whether this customer or incident meets it |
| Order form: customer is on the higher availability tier | Condition | Yes | Knows the tier, not the outage length |
| Incident record: outage lasted 3 hours (illustrative), cause unplanned | Case fact | Yes | Knows the outage, not the threshold |
| Exclusions clause: scheduled maintenance does not count as downtime | Exception check | Yes, to confirm it does not apply | Misses that a maintenance window could cancel the credit |
| Unsigned draft agreement with a different threshold | Hard negative | No, must not be cited | Applies a threshold that never took effect |
When is one passage genuinely enough, and when should no answer pass?
Some items really do have a one-unit evidence set, such as a definition, a single field lookup or a policy statement with no conditions, and those items should stay in the benchmark so the system is not rewarded for over-retrieving. Label them with a sufficiency rule of one group, and check that citation precision stays high on them.
At the other end, keep items where no complete set exists in the corpus at all. MuSiQue's full release pairs each answerable question with an unanswerable contrast version in which the supporting paragraph for one sub-question is absent from the context, and the authors report a further drop in model performance on that harder setting. Enterprise versions are easy to make from the brief: remove one required group and record the expected abstention.
- Keep a mix: single-group items, multi-group items and insufficient variants, reported separately so one category cannot mask another.
- Do not count over-retrieval as a failure on its own; count it through citation precision and hard negative use.
- Treat an answer that names the missing fact as a pass on an insufficient variant, even if it does not use the word abstain.
Where completeness labels stop being reliable
Completeness labels are judgments about a corpus at a date, so they drift when documents are added, superseded or re-chunked, and they need a refresh trigger tied to those changes. A label that was correct before an amendment can become wrong without anyone touching the benchmark.
Annotators also disagree about minimality, especially for exception checks: is the exclusions clause required evidence or background? Record the adjudication and the reason, and have the retrieval engineer and the owner of the governing documents sign off together, since one knows the system's units and the other knows which source controls.
Automated sufficiency judges help scale, but they are not ground truth. The sufficient context paper uses an LLM autorater to classify instances and notes that simply abstaining whenever the rater says the context is insufficient can lower overall performance, because models sometimes answer correctly anyway. Use automated judges to triage and humans to label the items that decide release.
Permissions add a further limit. A set can be complete for one user and incomplete for another who cannot see the order form, so the brief's user and as-of fields are part of the label, not metadata.
Sourcing realistic multi-document corpora for completeness testing
Public multi-hop datasets are built from encyclopedia articles, so they rarely contain the contract, ticket and policy combinations that make enterprise questions hard. A completeness benchmark is most useful when its documents come from real operating systems where rules, case records and amendments actually interact.
Corpora like that are what SourceX sources from U.S. businesses on request, with nothing held in stock, so an evaluation team can set out the document types, volume, language, format, timing and licensing it needs on the request-data form. The supplying business approves the scope and terms and signs the agreement before any records are delivered.
Frequently asked questions
What is evidence completeness in RAG evaluation?
Evidence completeness asks whether the retrieved context contains every fact the answer depends on, not just one relevant passage. It is labeled as a minimal set of evidence groups per question and scored at the group level. A system passes only when all required groups are retrieved and cited.
How is evidence completeness different from context recall?
Context recall, as Ragas defines it, measures the share of reference-answer claims attributable to the retrieved context. Evidence completeness adds structure: it names which groups are required, which are alternatives and which documents are hard negatives. It also includes insufficient variants that test whether the system abstains.
How many documents should a multi-document benchmark question need?
As many as the answer genuinely depends on and no more. The business question shapes in this guide need two to four groups, while public benchmarks such as FRAMES go up to fifteen articles. Test minimality by removing each group and checking whether the reference answer still follows.
Should a RAG system refuse to answer when one piece of evidence is missing?
It should not present an unsupported answer as settled. The better behavior is to answer what the evidence supports and name the missing fact, or decline. Score that behavior on insufficient variants built by removing one required group from the corpus.
Can HotpotQA or MuSiQue replace an enterprise completeness benchmark?
They are useful for checking multi-hop retrieval and abstention in general, and their supporting-fact and contrast designs are worth copying. They do not contain contracts, tickets, policy versions or permissions, so they cannot test whether a system finds the governing clause and the case record together in your corpus.
Sources
- HotpotQA collects sentence-level supporting facts from crowd workers for each multi-hop answer. Source
- MuSiQue composes multihop questions from single-hop questions and pairs answerable questions with unanswerable contrast versions. Source
- FRAMES questions carry reasoning-type labels and each needs information from between 2 and 15 Wikipedia articles. Source
- Models often answer incorrectly instead of abstaining when retrieved context is insufficient, and blanket abstention can lower overall performance. Source
- ALCE evaluates citation quality, scoring whether each sentence is entailed by its cited passages and whether citations were necessary. Source
- Ragas context recall is the share of reference-answer claims attributable to the retrieved context. Source
Tell us what data you need
Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.