AI uses for records
Retrieval evidence hierarchy for business search: which source wins
By SourceX Editorial · Updated
10 sections
Short answer
Define an evidence hierarchy for company retrieval tasks per question intent, not per corpus. For each intent, name the controlling source, the sources that may only support it, how to choose among versions, who may see the evidence and when to decline. Then label test items with that worksheet so retrieval is graded on authority, currency and permission, not similarity alone.
Retrieval
Retrieval evidence hierarchy
Key takeaways
- Similarity finds close passages; only a written hierarchy says which passage is allowed to decide the answer.
- Authority tiers are defaults: questions about what happened can be controlled by case records and conversation, not policy.
- Each worksheet row needs a controlling source, required evidence set, hard negatives, version rule, permission rule and abstain condition.
- Record the question date and the asking user's role on every test item, or the evaluation cannot catch stale or unauthorized answers.
- Source owners, not only retrieval engineers, must approve the controlling source and abstain rule for their systems.
How do you define an evidence hierarchy for company retrieval tasks?
A retrieval evidence hierarchy for a business is a written rule, per type of question, that names which source controls the answer, which sources may only support it, and when the system should decline to answer. Build it as a worksheet: list the question intents your search or assistant serves, assign a controlling source to each, add a version rule, a permission rule and an abstain condition, then grade retrieval against that worksheet.
The hierarchy is a decision rule, not a ranking model. It tells the retriever and the grader what correct evidence looks like before anyone measures similarity. A team that skips it ends up arguing case by case about whether a wiki page or a contract clause should have won.
Start small. A first worksheet of ten to twenty question intents is a workable size for one owner review, and the first pass can be drafted from search logs or help desk categories. The worksheet then grows when graders find a question that no row covers, which is itself a useful signal that the assistant is being asked something nobody has defined an answer source for.
Why a similarity score cannot tell you which document is in charge
Similarity measures how close a passage is to a query, not whether the passage is allowed to decide the answer. In company corpora the most similar passage is often a draft, a forwarded email quoting an old term, or a slide that summarized a policy two versions ago.
Common evaluation metrics inherit that blind spot unless the reference data encodes authority. Ragas defines context precision as whether relevant chunks are ranked above irrelevant ones, and context recall as the share of claims in a reference answer that the retrieved context supports. Both are only as good as the label of what counts as relevant, and a superseded price sheet is very relevant to a pricing question while being the wrong evidence.
So the hierarchy has to live in the labels. When the reference for a pricing question names the current effective price record as the required evidence and the old sheet as a hard negative, the same metrics start measuring authority.
Five authority tiers, and why the order flips by question
Most companies can sort their sources into five tiers, but the order is a default that each question intent can override. Asking what our refund policy is and asking what we actually did for this customer need different controlling sources.
Steps
Five authority tiers, and why the order flips by question
- Tier 1, controlling records: the system of record entry that creates the fact, such as an executed contract in a contract repository, an effective-dated price in the ERP, or a closed case in the service desk.
- Tier 2, published derivatives: documents an owner approved for reuse, such as a published policy, a released knowledge article or a current standard operating procedure.
- Tier 3, working documents: drafts, meeting notes, wiki pages without an owner and slide decks that restate other sources.
- Tier 4, conversation: email threads, chat messages and ticket comments, which show practice and context but rarely settle what is authorized.
- Tier 5, outside or generated text: vendor marketing, web pages and earlier model outputs, which should never control an internal answer.
- Override rule: for questions about what happened, such as how an incident was resolved, Tier 4 conversation tied to the case can control and a Tier 2 procedure only supports.
The source authority mapping worksheet
The worksheet is the asset a retrieval engineer and the owner of each source fill in together. Each row is one question intent. The values below are illustrative examples for a mid-size distributor, not a recommended configuration; replace them with your own systems and owners.
| Question intent | Controlling source | Version and date rule | Permission rule | Abstain when |
|---|---|---|---|---|
| Current price for a named customer | Effective-dated price record in the ERP | Use the record valid on the question date; ignore earlier sheets | Sales roles for that account only | No effective record covers the date |
| Refund or return policy | Published policy page with a named owner | Latest published version; drafts excluded | All employees | Two published versions conflict |
| Renewal term in a customer contract | Executed contract and signed amendments | Latest amendment that changes the clause | Legal and account owner | Amendment referenced but not found |
| How a past incident was resolved | Closed case record and its linked comments | The case as closed; later edits flagged | Support staff with case access | Case closed without a resolution note |
| Product specification | Released engineering document | Released revision only; superseded revisions are hard negatives | Engineering and product roles | Released revision missing a required field |
| Who approved an exception | Approval record in the workflow tool | The approval tied to that transaction | Managers in the approval chain | Approval exists only in an unlinked email |
Input dictionary: what each worksheet column must contain
Every worksheet row needs the same fields so that graders can turn it into evaluation items without guessing. The dictionary below is the minimum; teams with regulated content may add a retention field and a legal hold flag.
| Field | Definition | Illustrative entry |
|---|---|---|
| Query intent | A named class of questions with the same evidence needs | Current price for a named customer |
| Controlling source | The one source type that decides the answer | ERP price record |
| Supporting sources | Sources that may add context but never override | Quote emails, sales playbook |
| Required evidence set | Every item an answer must cite to be complete | Price record plus the customer's contract tier |
| Hard negatives | Similar but non-authoritative items the retriever must not use as proof | Last year's price sheet, a draft discount memo |
| Version rule | How to choose among versions and what supersedes what | Valid-on-date; replaced versions excluded |
| Permission rule | Who may receive evidence from the controlling source | Account team for that customer |
| Abstain condition | When the correct output is a refusal or a referral | No record valid on the question date |
Does the evidence authorize this answer for this user at this time?
An answer is supported only if the evidence is authoritative, current for the question date and visible to the person asking. Relevance covers none of the three, so the worksheet tests them separately.
Permission is the condition most often left out of offline tests. Microsoft documents that Microsoft 365 Copilot only surfaces organizational data to which an individual user has at least view permissions, and that grounding respects the user's identity-based access boundary. An evaluation set for a permission-aware assistant needs the asking user's role on every item, or it will reward answers the real system should never give.
Version needs an explicit supersession link rather than a newest-file guess. The DCMI Metadata Terms define isReplacedBy as a related resource that supplants, displaces or supersedes the described resource. Storing that relation, from whatever system holds it, lets the grader mark a superseded document as a hard negative instead of a near miss.
Time is the third test. A question asked about last March should be answered from the record valid last March, which may not be the latest version. Record the question date on every item.
Worked scenario: one pricing question, three plausible sources
The scenario is illustrative and invented for this article. A sales representative asks the assistant what unit price a named customer pays for a pump model this quarter. Retrieval returns three candidates, all highly similar to the question.
- Candidate A: a price sheet attached to an email from last year, quoting the pump at a list price. Tier 4 by location, Tier 3 by content, and superseded.
- Candidate B: a sales wiki page summarizing contract tiers, last edited by someone who has left. Tier 3, no owner, useful for context only.
- Candidate C: the ERP price record for that customer and item, effective from the first of the quarter. Tier 1, controlling, valid on the question date.
- Required evidence set: Candidate C plus the customer's contract tier from the contract repository. An answer that cites C alone is correct but incomplete under the worksheet.
- Permission check: the representative covers this account, so C may be shown. A representative from another region would get an abstention and a referral, even though C is the best match.
- Grade, illustrative: an answer citing A is incorrect even if the number happened to match, because the evidence does not authorize it; an answer citing C and the tier is correct and complete.
Measuring retrieval against the hierarchy instead of relevance alone
Once labels encode authority, standard metrics measure the right thing, and a few additions cover what they miss. Score retrieval, citation and the final answer separately so a failure can be traced to one stage.
For answers, a scale that rewards declining over guessing fits the abstain column. The CRAG benchmark rates responses as perfect, acceptable, missing or incorrect, and its automatic evaluation scores correct answers at 1, missing answers at 0 and incorrect answers at minus 1. For citations, the ALCE benchmark evaluates generated text with citations on fluency, correctness and citation quality, which maps onto whether each claim cites its controlling source. Ragas faithfulness, which counts a response as faithful when all of its claims can be supported by the retrieved context, catches answers that drift beyond what was retrieved.
| Measure | What it checks | How the hierarchy changes it |
|---|---|---|
| Context precision | Relevant chunks ranked above irrelevant ones | Only controlling and supporting sources count as relevant; hard negatives count against |
| Context recall | Reference claims supported by retrieved context | The reference is built from the required evidence set |
| Faithfulness | Answer claims supported by retrieved context | Claims supported only by a hard negative are flagged |
| Citation quality | Whether citations support the claims they sit on | Each key claim must cite the controlling source |
| Correct, missing, incorrect | Answer outcome with a penalty for wrong answers | Abstaining when the worksheet says abstain scores as correct behavior |
Which assumption changes the ranking, and who signs off
Two assumptions move the ranking more than any other: that the controlling system is kept current, and that permissions in the index match permissions in the source. Test both before trusting any score.
If the ERP price record lags real deals because representatives agree terms in email and update the system weeks later, Tier 1 is stale for that intent and the worksheet must say so, for example by requiring the latest approved quote as a second piece of evidence. If the search index copies documents without their access lists, the permission rule is not enforced at all, and a high correctness score hides exposure.
Show uncertainty in the worksheet rather than in prose. Mark each row with who confirmed the controlling source and when, and flag rows where the owner disagreed with the retrieval engineer. A retrieval engineer can build the rows; the owner of each source has to approve the controlling source, the version rule and the abstain condition, because those are business decisions.
Carrying authority labels into licensed retrieval data
Retrieval evaluation sets built from real company records are only reusable if the authority labels travel with them. A buyer assembling such a set needs each document's tier, owner, supersession links, effective dates and access group, along with the rights to use the records for retrieval or evaluation.
When SourceX sources records for a buyer, it handles rights review, contracting, preparation and delivery with the supplying company, so authority labels can be requested as part of the preparation. Nothing is pulled from a shelf: each set is found at U.S. businesses against a buyer's stated scope, volume, language, format, timing and licensing needs. Asking for authority and version metadata in that request is what makes the records usable for the hierarchy described here.
Frequently asked questions
What is an evidence hierarchy in retrieval?
It is a rule that ranks sources by their authority to answer a given type of question, separate from how similar they are to the query. It names the controlling source, the supporting sources, the version rule and when to decline.
Is the newest document always the most authoritative?
No. A newer draft or an email can be more recent than a published policy without replacing it, and a question about a past date needs the version valid then. Use explicit supersession links and effective dates rather than modification time.
How do permissions affect retrieval evaluation?
An answer can be correct and still be wrong for the person asking if they lack access to the evidence. Record the asking user's role on each test item and grade answers that rely on evidence the user cannot see as failures.
Should a company assistant ever refuse to answer?
Yes, when the worksheet's abstain condition is met, such as no record valid on the question date or two published versions in conflict. Scoring schemes that penalize incorrect answers more than missing ones reward that behavior.
Who should own the source authority worksheet?
A retrieval engineer drafts it, and the owner of each source approves the controlling source, version rule and abstain condition for the intents that touch their system. Re-review it when systems, policies or permissions change.
Sources
- Ragas defines context precision as the retriever's ability to rank relevant chunks higher than irrelevant ones for a query. Source
- Ragas defines context recall as the number of claims in the reference supported by the retrieved context divided by the total claims in the reference. Source
- Ragas treats a response as faithful when all of its claims can be supported by the retrieved context. Source
- CRAG rates RAG responses as perfect, acceptable, missing or incorrect, and its automatic evaluation scores correct as 1, missing as 0 and incorrect as -1. Source
- ALCE provides automatic evaluation of LLM generations with citations along fluency, correctness and citation quality. Source
- Microsoft documents that Microsoft 365 Copilot only surfaces organizational data to which individual users have at least view permissions, and that Semantic Index honors the user identity-based access boundary. Source
- DCMI Metadata Terms define isReplacedBy as a related resource that supplants, displaces or supersedes the described resource. Source
Tell us what data you need
Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.