AI uses for records
RAG answerability dataset: when the business archive cannot answer
By SourceX Editorial · Updated
10 sections
Short answer
Label business questions per question, asker and date rather than per question alone. List the evidence a correct answer needs, check each item in a frozen archive snapshot for existence, authority, currency and the asker's access, and record which item is missing. The label then follows: answer and cite, answer part and name the gap, ask a clarifying question, or decline and route to the owner.
Retrieval
RAG answerability dataset
Key takeaways
- The same question can be answerable for one asker and not for another, so every task names its asker, date and snapshot.
- Code the missing evidence (absent, superseded, non-authoritative, out of scope, partial, restricted, ambiguous) instead of a bare unanswerable flag.
- Faithfulness to retrieved text does not prove authority; a superseded policy can support a perfectly faithful wrong answer.
- Include counterexamples that look unanswerable but are answered by combining two sources.
- Score answering, declining, gap naming and access correctness as separate numbers.
How do you label business questions that an archive cannot answer?
Label each task as a combination of question, asker and date, not as a question alone. For every task, write down the evidence a correct answer depends on, check each required item against a frozen archive snapshot for four properties (it exists, it is the controlling source, it is current for the date asked, and the asker may see it) and record which item is missing. The answerability label follows from that record, and so does the behavior you expect from the system.
Four expected behaviors cover nearly every business case: answer and cite; answer the supported part and name the gap; ask a clarifying question when the question could mean two things; or decline and point to the owner of the missing evidence. A bare answerable or unanswerable flag hides which of these the system should have chosen, and that choice is what a production user experiences.
This is a retrieval evaluation asset built for business archives, where the hard cases are not trick questions but ordinary ones: a policy that was replaced last quarter, a figure that lives in a system the index never ingested, a contract the asker is not cleared to read. The task set below is designed so each of those failures has its own code.
Why a similar passage is not the same as a supporting one
Retrieval systems rank by similarity, and the passage most similar to a question is often the one that almost answers it. SQuAD 2.0 was built around this problem: its authors added 53,775 new unanswerable questions written against the same paragraphs as answerable ones, precisely because automatically generated unanswerable questions were easy to spot. Models had to learn that a relevant-looking paragraph can still contain no supported answer.
Business archives produce these near misses naturally. The 2024 version of a policy sits next to the 2026 version; a regional price list resembles the global one; a draft contract shares nearly every sentence with the signed one. A label that only asks whether relevant text was retrieved will mark all of these answerable.
Claim-level metrics help but do not settle it. Ragas defines faithfulness as the share of claims in a response that the retrieved context supports, and context recall as the share of claims in a reference answer that the retrieved context supports. A response built entirely from a superseded policy can be perfectly faithful to its context and still wrong for the business, because faithfulness checks support, not authority. Context recall needs a reference answer, which an unanswerable task by definition does not have, so the task set has to carry the missing-evidence record itself.
The required evidence set behind each business question
Break each question into the evidence a correct answer would need, then test each piece. Most business questions need between one and four pieces: the controlling source, a date or version that makes it current, the specific entity or scope asked about, and sometimes a second source that has to be combined with the first. The RGB benchmark for retrieval-augmented generation treats this combining step, information integration, as a separate ability from rejecting questions the documents cannot answer, and business questions exercise both.
| Requirement | Question the labeler asks | Typical business failure | Missing-evidence code |
|---|---|---|---|
| Existence | Is there any record in the snapshot that states this fact? | The figure lives in a finance system that was never indexed | absent |
| Authority | Is the record that states it the controlling source for this question? | Only a sales slide restates the policy; the policy itself is missing | non_authoritative |
| Currency | Was that source in force on the date the question refers to? | Only the superseded 2024 version of the policy is present | superseded |
| Scope | Does the source cover this entity, region, product or customer? | Terms exist for the standard plan, not the plan asked about | out_of_scope |
| Completeness | Are all parts of a multi-part question covered? | Two of three regions are reported; one is missing | partial |
| Access | May this asker see every required record? | The signed contract is restricted to legal and finance | restricted |
| Determinacy | Does the question identify one answer, or several readings? | Which contract is meant by the contract is unclear | ambiguous |
Why the same question gets different labels for different askers
Answerability depends on who is asking because enterprise search returns only what the asker may see. Microsoft's guidance for Azure AI Search describes the security filter pattern, in which each document carries a field of group identifiers and every query filters on the groups the caller belongs to; the documentation notes the filter is plain string matching and the application must apply it on every query. Elasticsearch document level security attaches a query to a role, so a user sees only documents matching their role's query.
For the task set, that means the same question can be answerable for a finance manager and unanswerable for a support agent, with an identical archive. Write the asker's role or group list into every task, and label from that asker's view of the snapshot. A system that answers the support agent correctly from a restricted document has passed an accuracy check and failed an access check at the same time.
Decide the expected behavior for restricted evidence before labeling. In most business settings the system should decline without confirming that a restricted document exists, and point the asker to a role that can answer. Some organizations prefer an explicit you do not have access message. Either is defensible; what matters is that the task records which one the organization chose, so graders are not inventing policy per item. Leakage testing itself, probing whether restricted content escapes, is a separate suite.
The answerability and missing evidence task set
Each task in the set is one row with the fields below. The required evidence list and the per-item status are the core; everything else exists so a reviewer can reproduce the label from the snapshot without asking the original labeler.
At a glance
The answerability and missing evidence task set
| Field | Content | Illustrative value | Check before the task enters the set |
|---|---|---|---|
| Task ID and label | Unique ID; illustrative or approved real | ANS-0147; illustrative | Reviewer confirms the label |
| Question and intent | The question as a user would type it, plus one sentence on what they need | What is our current refund window for annual plans? | Intent sentence agreed by two labelers |
| Asker and as-of date | Role or group list, and the date the question refers to | Support agent; as of 2026-10-01 | Role exists in the permission model of the snapshot |
| Archive snapshot | Snapshot ID of the corpus and of the permission data | kb-2026-10-01-r2 | Snapshot is frozen and reproducible |
| Required evidence set | Each item the answer needs, with the controlling source named | Current refund policy, annual plan section | Owner of the source agrees it controls |
| Status per item | Found, or one missing-evidence code from the requirement table | superseded: only the 2024 policy is present | Second labeler searches independently |
| Hard negatives | Documents that look relevant but must not be used | 2024 refund policy; monthly plan FAQ | Each one is retrievable by the system under test |
| Answerability label | Answerable, partly answerable, ambiguous, or not answerable for this asker | Not answerable for this asker | Follows mechanically from the item statuses |
| Expected behavior | Answer and cite; answer part and name gap; clarify; decline and route | Decline, say the current policy was not found, route to billing operations | Matches the organization's stated policy |
| Reference answer and citations | For answerable parts only, with record and span references | None | Every cited span supports its claim |
How labelers turn a real question into a task
Labeling runs in a fixed order so two labelers working from the same snapshot reach the same row. The order matters because the expected behavior is derived last, from evidence statuses, rather than decided first from a gut sense of whether the question feels answerable.
Two labelers should complete the evidence search independently before comparing. Disagreements about existence are usually search failures and resolve quickly; disagreements about authority or scope are business questions and go to the source owner, as described below.
- Write the question exactly as a user would ask it, then one sentence on what the user needs to do with the answer. The second sentence is what resolves scope later.
- Fix the asker's role and the as-of date before searching. Searching first and choosing an asker afterward bends the label toward whatever was found.
- List every required evidence item and name, for each, which system or document type would control it in this business, such as the billing policy library or the signed order form in the contract repository.
- Search the snapshot as an administrator for each item and record whether it exists at all, then repeat the search from the asker's permission view.
- For each item found, check authority, currency and scope against the requirement table and assign a status or a missing-evidence code.
- Collect the hard negatives the search turned up: the near-miss documents a system is likely to retrieve and must not rely on.
- Derive the answerability label and expected behavior mechanically from the statuses, then write the reference answer and citations for the supported part only.
Worked task cases from an illustrative company archive
The tasks below come from one invented company archive and show how the same mechanics produce different labels. They are illustrative, not drawn from any real company. The final row is the counterexample a good set needs: a question that looks unanswerable because no single document holds the answer, but that the archive does answer when two sources are combined.
Read the second and third rows together. The archive is identical and the question is identical; only the asker changes, and with it the label, the expected behavior and the correct response. A set that stores one label per question cannot express this, which is why the asker and date sit in the task row rather than in a note.
The SLA row shows why authority is its own requirement. A system that quotes the figure from the sales deck will look helpful and score well on faithfulness, because the deck does state a figure. If the deck is out of date, the business has just made a commitment nobody approved. The label tells the grader that any specific figure in the response is a failure.
| Question | Asker and date | Required evidence status | Label and expected behavior |
|---|---|---|---|
| What is our current refund window for annual plans? | Support agent, 2026-10-01 | Controlling policy superseded; only the 2024 version present | Not answerable; decline, say the current policy is missing, route to billing operations |
| What discount did we give Customer A at renewal? | Support agent, 2026-10-01 | Signed order form exists but is restricted to finance | Not answerable for this asker; decline without confirming the document, route to account owner |
| Same question | Finance manager, 2026-10-01 | Signed order form found and in force | Answerable; answer and cite the order form |
| What were support ticket volumes by region last quarter? | Operations lead, 2026-10-01 | Two of three regional reports present | Partly answerable; give two regions, name the missing one |
| What does the contract say about termination? | Account manager, 2026-10-01 | Three active contracts with this customer | Ambiguous; ask which contract |
| What is the SLA response time for priority 1 tickets? | Support agent, 2026-10-01 | Only a sales deck restates it; the SLA document is not in the index | Not answerable from an authoritative source; give no figure, route to the SLA owner |
| Which open escalations involve products past end of support? | Support lead, 2026-10-01 | Escalation list in the ticket export and end-of-support dates in the product catalog; neither alone answers | Answerable by combining both; answer and cite both sources |
Scoring the set so declining is not a free pass
Report answer quality and declining quality as separate numbers, and never let a system raise one by sacrificing the other. A system that declines every question scores perfectly on the not-answerable tasks and fails every answerable one; a system that always answers does the reverse. The RGB benchmark measures its negative rejection ability with a rejection rate on questions where only noisy documents are supplied, which is the right instinct: test declining on items built to need it, and test answering on items built to allow it.
At a glance
Scoring the set so declining is not a free pass
Score each task against its expected behavior, not against a generic refusal. Useful metrics for the set:
- Accuracy with citation support on answerable and partly answerable tasks, using the reference answer and cited spans.
- Correct declines on not-answerable tasks, counted only when the system also avoids stating an unsupported answer anywhere in the response.
- Over-declining: the share of answerable tasks where the system refused or hedged without giving the supported answer.
- Gap naming: on partly answerable and not-answerable tasks, whether the response identifies what is missing in terms a user could act on, matched against the missing-evidence code.
- Clarification precision: on ambiguous tasks, whether the system asked a question that actually separates the readings, and whether it asked needless questions on unambiguous tasks.
- Access correctness, reported separately: on restricted tasks, whether any content or existence signal from a restricted record appeared in the response.
Keeping labels true when the archive, policies or permissions change
Labels in this set are claims about a snapshot, so any change to the snapshot can flip them. A policy republished, a system newly indexed, a group membership changed or a document reclassified can turn a not-answerable task answerable or the reverse. Tie every task to a corpus snapshot ID and a permission snapshot ID, and re-run the existence, currency and access checks whenever either one changes.
Trigger a relabel on these events: a controlling source is replaced or retired, a new source system is ingested, chunking or metadata extraction changes in a way that alters what is retrievable, permission groups are restructured, or the as-of dates in the tasks drift far enough from the snapshot date that currency checks become meaningless. Retire tasks whose required evidence no longer exists anywhere rather than rewriting them in place, so historical scores stay comparable.
Stop and bring in the source owner when labelers disagree on authority. Two careful labelers can find the same two documents and still disagree on which one controls; that disagreement is a fact about the business, not about the labelers, and only the owner of the policy or system can settle it. Record the owner's decision in the task so the next relabel does not reopen it.
Where real business archives for answerability testing come from
Answerability tasks are only as realistic as the archive behind them. Synthetic corpora rarely contain the messy features that make business questions hard: superseded versions sitting next to current ones, restatements in slides and emails, permission groups that reflect real roles, and questions people actually asked and could not get answered. Help desk tickets that ended in no documentation found, internal search logs with no click, and escalations routed to a policy owner are good sources of real questions, paired with the archive they were asked against and its permission metadata.
Archives like these are what SourceX sources for AI buyers: datasets are found on request at U.S. businesses, not held in stock. Buyers set out the scope, volume, language, format, timing and licensing needs through the request-data form, for example a frozen document archive with version history and access groups plus the questions asked against it.
Frequently asked questions
What is a RAG answerability dataset?
It is an evaluation set in which each question is paired with a frozen archive and labeled for whether that archive can support an answer, for a given asker and date. Good sets also record which evidence is missing and what the system should do about it, so they test declining, partial answers and clarification as well as correct answers.
How do you check that retrieved evidence supports an answer for this asker and date?
Check three things separately. Support: every claim in the answer is stated in a retrieved record. Authority and currency: that record is the controlling source and was in force on the date asked. Authorization: the asker may see every record used. An answer can pass support and fail the other two.
How is this different from a list of unanswerable questions?
A list of unanswerable questions labels questions in isolation. This task set labels a question for a specific asker, date and archive snapshot, and records the missing piece, such as a superseded policy or a restricted contract. The same question can carry different labels on different rows.
Should the system say that a restricted document exists?
Usually not, because confirming existence can itself disclose information, but organizations differ. Decide the policy once, write it into the expected behavior for restricted tasks, and test leakage separately with a permission-focused suite.
How many not-answerable tasks should the set include?
Enough to measure declining and over-declining with useful confidence, and in proportions that reflect the questions users actually ask. Report answerable and not-answerable results separately rather than as one blended score, so a change in the mix cannot hide a regression.
Sources
- SQuAD 2.0 added 53,775 new unanswerable questions written on the same paragraphs as answerable ones because automatically generated unanswerable questions were easy to identify. Source
- The RGB benchmark evaluates retrieval-augmented generation on noise robustness, negative rejection, information integration and counterfactual robustness, measuring negative rejection with a rejection rate when only noisy documents are supplied. Source
- Ragas defines faithfulness as the share of claims in a response that can be inferred from the retrieved context. Source
- Ragas context recall is the share of claims in a reference answer supported by the retrieved context and always requires a reference. Source
- Azure AI Search's security filter pattern stores group identifiers on each document and filters queries by the caller's groups; it is string matching that the application must enforce. Source
- Elasticsearch document level security uses a query on each role to specify which documents users with that role can access. Source
Tell us what data you need
Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.