AI uses for records
RAG document absence verification: testing a no-match claim
By SourceX Editorial · Updated
10 sections
Short answer
Treat a claim that no matching record exists as a statement about a bounded search. Name the corpus, index version, date range and permissions the claim covers, plant known matches inside that boundary, confirm true absences by exhaustive check, and pass the system only when it finds the planted matches and states the boundary whenever it reports none. Score an unbounded none-exists answer as a failure.
Retrieval
RAG document absence
Key takeaways
- A none-found answer is only as good as the boundary it searched; require the system to state it.
- False absences mostly come from retrieval mechanics: filtered approximate search, count caps, refresh lag and permission trimming.
- Pooled relevance judgments assume unjudged documents are not relevant, so absence labels need an exhaustive check.
- Score recall on planted matches, false absences, boundary statements, overclaims and permission-correct answers separately.
- Route absence questions to a keyed lookup on the system of record whenever one exists.
How do you evaluate a claim that no matching record exists?
Evaluate it as a statement about a bounded search, because that is all it can be. A system that says no record matches has searched some corpus, at some index version, within some date range, under some user's permissions, with some query method. The test names that boundary, plants known matches inside it, confirms true absences by exhaustive check, and passes the system only when it finds the planted matches and states the boundary whenever it reports none.
This matters because absence answers drive actions. An accounts payable agent that finds no existing vendor creates a new one; a support agent that finds no prior ticket treats a repeat problem as new; a contract assistant that finds no termination clause tells someone they have no exit. Each wrong absence looks exactly like a right one, and nothing downstream flags it.
The asset on this page is a bounded search completeness and abstention benchmark: an item format, a set of planted cases that break absence claims in known ways, and scores that separate a correct none-found answer from a lucky one. RAG document absence verification is the narrower name for the same job.
How do absence, unanswerable and not-enough-information answers differ?
Keep three answer types apart, because each needs different evidence. An absence answer says nothing matching exists within a stated scope. An unanswerable answer says the question cannot be answered from the provided material. A not-enough-information answer says the evidence found neither supports nor refutes a claim.
Public benchmarks cover the second and third well. SQuAD 2.0 added over 50,000 adversarially written unanswerable questions so systems must decide when a paragraph supports no answer and abstain. FEVER labels claims Supported, Refuted or NotEnoughInfo and requires evidence sentences for the first two but not the third. Neither asks a system to prove that a record is missing from a whole searchable collection, which is the absence case.
The difference shows up in grading. An unanswerable item is graded on whether the system declined. An absence item is graded on whether the system searched the right place completely enough to be entitled to say none, and on whether it said where it looked.
Why does a search return nothing when the record is there?
Most false absences come from the retrieval layer, not the model. Each row below is a documented behavior of a common component, and each becomes a planted case in the benchmark.
| Cause | Documented behavior | What the user sees | Planted case to build |
|---|---|---|---|
| Filtered approximate vector search | pgvector applies filters after an approximate index scan; with HNSW's default ef_search of 40 and a filter matching 10 percent of rows, about 4 rows match on average | Fewer or zero results for a narrow filter | A match that only appears with exact search or iterative scans |
| Approximate recall loss | pgvector says adding an approximate index trades some recall for speed and changes results | A match ranked out of the candidate list | A paraphrased match far from the query embedding |
| Result count cap | Elasticsearch counts hits accurately only up to 10,000 by default and reports a lower bound with relation gte past that | A count that looks final but is not | A query whose true count exceeds the default cap |
| Index refresh lag | Elasticsearch makes changes searchable on refresh, by default every second for indices searched in the last 30 seconds | A just-created record missing | A record written moments before the query |
| Permission trimming | SharePoint search results are always security trimmed to what the user can access | No result for a record that exists but is hidden from this user | A match the asking user is not allowed to see |
| Scope or identifier mismatch | The record lives in an unindexed system or uses a different ID format | No result although a person would find it | A tax ID stored with dashes against a query without them |
What does each bounded search completeness benchmark item need?
Each benchmark item pairs a question with the boundary it must be answered within and the ground truth for that boundary. Fill every field before an item is used; an item without a confirmed boundary cannot score an absence answer.
At a glance
What does each bounded search completeness benchmark item need
| Field | What it holds | Example value (illustrative) | Why it is needed |
|---|---|---|---|
| Query intent | The decision the answer feeds | Before creating a vendor, check none exists with this tax ID | Sets the cost of a false absence |
| Controlling source | The system of record that settles the question | ERP vendor master, not the document index | Separates the index from the truth |
| Search boundary | Corpus, index version, date range and permission principal | Onboarding documents index v12, all dates, AP clerk role | Defines what none means |
| Ground truth | Matches in the boundary, confirmed by exact scan or keyed lookup | One match under a legacy name | Absence labels need exhaustive confirmation |
| Planted matches | Known matches placed to trigger each failure cause | Dash format, restricted folder, written 2 seconds before query | Measures recall where it usually breaks |
| Hard negatives | Near matches that must not be returned as matches | Same name, different tax ID | Stops systems passing by over-matching |
| Expected answer type | Found, none within boundary, or cannot determine | Found, with citation | Scores abstention and over-claiming |
| Required evidence | What a passing answer must cite or state | The matching record or a boundary statement | Makes none-found answers checkable |
| Permission expectation | What may be revealed to this user | Do not reveal records outside the role | Prevents leaks while testing absence |
How do you prove a true absence before grading one?
Prove it with an exhaustive check inside the boundary, not with the judgments you collected for relevance. Pooled relevance judgments, the standard way to build retrieval test collections, assume that documents nobody judged are not relevant. The TREC 2021 overview states that runs are scored assuming all unpooled, unjudged documents are not relevant. That assumption is acceptable for ranking systems against each other and wrong for labeling an item as a confirmed absence.
A bounded corpus makes the exhaustive check possible. Run an exact key lookup against the controlling source, an exact vector scan instead of an approximate index (pgvector performs exact nearest neighbor search with perfect recall by default), and a normalized string search for each identifier variant. Record the methods with the label. If any method finds a candidate, a person adjudicates it before the item is labeled absent.
Keep the ground truth tied to the boundary it was checked against. If the corpus is re-indexed, a permission changes or new records arrive, the absence label is stale until the check runs again, so store the index version and check date with every label and re-run the checks when either changes.
Write down what the exhaustive check could not reach. Scanned images without text, attachments the indexer skipped and records in systems outside the boundary are all reasons to label an item cannot determine rather than absent.
What must a passing none-found answer contain?
A passing answer states the boundary and the method, and never upgrades none found into none exists. Score answers on separate dimensions so a system cannot trade one for another.
| Score | Definition | Pass condition | Failure it catches |
|---|---|---|---|
| Planted match recall | Share of planted in-boundary matches the system finds | All matches the user may see are found | False absence from filters, caps or lag |
| False absence rate | Share of items with a visible match where the system said none | Zero on high-cost intents | Confident wrong none-found |
| Boundary statement rate | Share of none-found answers that name corpus, date range and access scope | Every none-found answer | Unbounded absence claims |
| Overclaim rate | Share of answers saying a record does not exist anywhere | Zero | Upgrading none found to none exists |
| Permission-correct answers | Share of hidden-match items answered without revealing the hidden record | All of them | Leaking restricted records while testing absence |
Worked example (illustrative): the vendor that already existed
Everything here is illustrative. An accounts payable agent receives a request to set up a new vendor and must first check that no vendor with the same tax ID exists. It searches an index of 6,000 onboarding documents through a vector store with an HNSW index and a metadata filter for active vendors.
The benchmark plants four matches. The first is an exact match on a current form, which the agent finds. The second is the same vendor under a legacy name, with the tax ID stored with dashes while the query has none; the agent misses it. The third sits in a folder the AP clerk role cannot see; the agent correctly returns nothing visible. The fourth is a form uploaded two seconds before the query on an index that had not refreshed; the agent misses it.
The second item is the one that matters. Asked about that vendor, the agent answers that no vendor with that tax ID exists and proposes creating one. That answer fails three scores: it misses a visible planted match, it states absence without a boundary and it claims none exists. The passing answer either finds the legacy record or says no match was found in the onboarding index for active vendors and that the ERP vendor master should be checked before creation.
The exception in the same run: for the restricted-folder item, the right answer is not found within your access, without hinting that a hidden record exists. A grader that rewarded finding every planted match would mark that answer wrong and push the system toward leaking.
When can a system say flatly that a record does not exist?
When the question is a keyed lookup against the complete system of record, the system can say the record does not exist. If the tax ID is a unique, validated field in the ERP vendor master, an exact query on it with no permission filter is the controlling check, and its empty result is the answer. A retrieval layer adds nothing there and only adds ways to miss.
Even then, check two things before trusting the empty result: that the field is actually unique and validated at entry, and that the query normalizes the identifier the same way the system stores it. A tax ID kept with dashes in some rows and without them in others turns a keyed lookup back into a search.
The benchmark should include these items too, so systems learn to route absence questions to the keyed source when one exists rather than to semantic search. Score routing as part of the answer: the right tool for an absence question is often not the retriever.
Which absence answers must the record owner sign?
Stop treating it as a retrieval question when the absence will be relied on outside the team. A statement that no contract, complaint, incident or prior claim exists can carry legal, regulatory or audit weight, and the person who owns that record set must confirm the boundary and sign the answer.
Escalate as well when the exhaustive check cannot cover the boundary, when systems disagree about whether a record exists, or when the intent carries a high cost for a false absence, such as duplicate payments, missed safety reports or overlooked termination rights. In those cases the system's job is to report what it searched and hand over.
Where do realistic absence test cases come from?
Real records supply the failure modes illustrative items only guess at: duplicate vendors created after a missed search, tickets reopened as new, documents uploaded under the wrong account. Those events leave traces in merge logs, duplicate resolutions and audit histories, and each can become a planted match with known ground truth. Every real item needs the business owner's approval, with personal and account details removed.
Absence benchmarks built from real records start with a request: SourceX has no ready-made datasets and sources suitable records from U.S. businesses for each one. Teams can describe the scope, volume, language, format, timing and licensing they need, for example vendor master merge histories with the documents behind each merge, on the request-data form.
Frequently asked questions
What should a bounded search completeness and abstention benchmark include?
Each item needs the query intent, the controlling source, a search boundary of corpus, index version, date range and permissions, ground truth confirmed by exhaustive check, planted matches, hard negatives, the expected answer type and the evidence a passing answer must contain. Scores should separate recall on planted matches, false absences, boundary statements, overclaims and permission-correct answers.
Does a none-found result hold for every user and every moment?
Only if the answer is scoped to what the user could see at that index version and time. A none-found result under one role or before a refresh says nothing about other roles or later writes. The answer should state that scope rather than speak for the whole system.
How is absence verification different from unanswerable question testing?
Unanswerable tests check whether a system declines when the provided material cannot answer. Absence tests check whether a system searched the right collection completely enough to say nothing matches, and whether it stated where it looked. The second needs exhaustive ground truth; the first does not.
Can vector search alone confirm that a record does not exist?
Not with an approximate index. Approximate indexes trade recall for speed, and filters applied after the scan can return far fewer rows than exist. Confirm absences with exact search, keyed lookups on the system of record, or both.
What should a RAG system say when a matching record is hidden by permissions?
It should say no matching record was found within the user's access and stop there. Hinting that a restricted record exists is a leak, and claiming none exists anywhere is false. The benchmark should reward exactly that scoped answer.
When should a person confirm an absence instead of the system?
When the answer will be relied on for legal, audit, regulatory or payment decisions, or when the exhaustive check could not cover the whole boundary. The record owner confirms the boundary and signs the statement, and the system's output becomes supporting evidence.
Sources
- pgvector performs exact search with perfect recall by default, approximate indexes trade recall for speed, and filters apply after an approximate scan, so a filter matching 10 percent of rows returns about 4 rows on average with HNSW's default ef_search of 40. Source
- Elasticsearch counts total hits accurately up to 10,000 by default and reports a lower bound with relation gte beyond that. Source
- Elasticsearch makes indexed changes searchable on refresh, by default every second for indices that received a search in the last 30 seconds. Source
- Microsoft says SharePoint search results are always security trimmed, so users only see content they have permission to see. Source
- SQuAD 2.0 adds over 50,000 adversarial unanswerable questions so systems must determine when no answer is supported and abstain. Source
- FEVER labels claims Supported, Refuted or NotEnoughInfo and requires evidence only for Supported and Refuted. Source
- The TREC 2021 overview says runs are evaluated assuming all unpooled, unjudged documents are not relevant. Source
Tell us what data you need
Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.