Skip to content

AI uses for records

Retrieval evidence hierarchy for business search: which source wins

By SourceX Editorial · Updated

10 sections

Short answer

Define an evidence hierarchy for company retrieval tasks per question intent, not per corpus. For each intent, name the controlling source, the sources that may only support it, how to choose among versions, who may see the evidence and when to decline. Then label test items with that worksheet so retrieval is graded on authority, currency and permission, not similarity alone.

Retrieval

Retrieval evidence hierarchy

This page covers retrieval evidence hierarchy for business search: which source wins.Illustrative

Key takeaways

  • Similarity finds close passages; only a written hierarchy says which passage is allowed to decide the answer.
  • Authority tiers are defaults: questions about what happened can be controlled by case records and conversation, not policy.
  • Each worksheet row needs a controlling source, required evidence set, hard negatives, version rule, permission rule and abstain condition.
  • Record the question date and the asking user's role on every test item, or the evaluation cannot catch stale or unauthorized answers.
  • Source owners, not only retrieval engineers, must approve the controlling source and abstain rule for their systems.

How do you define an evidence hierarchy for company retrieval tasks?

A retrieval evidence hierarchy for a business is a written rule, per type of question, that names which source controls the answer, which sources may only support it, and when the system should decline to answer. Build it as a worksheet: list the question intents your search or assistant serves, assign a controlling source to each, add a version rule, a permission rule and an abstain condition, then grade retrieval against that worksheet.

The hierarchy is a decision rule, not a ranking model. It tells the retriever and the grader what correct evidence looks like before anyone measures similarity. A team that skips it ends up arguing case by case about whether a wiki page or a contract clause should have won.

Start small. A first worksheet of ten to twenty question intents is a workable size for one owner review, and the first pass can be drafted from search logs or help desk categories. The worksheet then grows when graders find a question that no row covers, which is itself a useful signal that the assistant is being asked something nobody has defined an answer source for.

Why a similarity score cannot tell you which document is in charge

Similarity measures how close a passage is to a query, not whether the passage is allowed to decide the answer. In company corpora the most similar passage is often a draft, a forwarded email quoting an old term, or a slide that summarized a policy two versions ago.

Common evaluation metrics inherit that blind spot unless the reference data encodes authority. Ragas defines context precision as whether relevant chunks are ranked above irrelevant ones, and context recall as the share of claims in a reference answer that the retrieved context supports. Both are only as good as the label of what counts as relevant, and a superseded price sheet is very relevant to a pricing question while being the wrong evidence.

So the hierarchy has to live in the labels. When the reference for a pricing question names the current effective price record as the required evidence and the old sheet as a hard negative, the same metrics start measuring authority.

Five authority tiers, and why the order flips by question

Most companies can sort their sources into five tiers, but the order is a default that each question intent can override. Asking what our refund policy is and asking what we actually did for this customer need different controlling sources.

Steps

Five authority tiers, and why the order flips by question

  • Tier 1, controlling records: the system of record entry that creates the fact, such as an executed contract in a contract repository, an effective-dated price in the ERP, or a closed case in the service desk.
  • Tier 2, published derivatives: documents an owner approved for reuse, such as a published policy, a released knowledge article or a current standard operating procedure.
  • Tier 3, working documents: drafts, meeting notes, wiki pages without an owner and slide decks that restate other sources.
  • Tier 4, conversation: email threads, chat messages and ticket comments, which show practice and context but rarely settle what is authorized.
  • Tier 5, outside or generated text: vendor marketing, web pages and earlier model outputs, which should never control an internal answer.
  • Override rule: for questions about what happened, such as how an incident was resolved, Tier 4 conversation tied to the case can control and a Tier 2 procedure only supports.

The source authority mapping worksheet

The worksheet is the asset a retrieval engineer and the owner of each source fill in together. Each row is one question intent. The values below are illustrative examples for a mid-size distributor, not a recommended configuration; replace them with your own systems and owners.

The source authority mapping worksheet
Question intentControlling sourceVersion and date rulePermission ruleAbstain when
Current price for a named customerEffective-dated price record in the ERPUse the record valid on the question date; ignore earlier sheetsSales roles for that account onlyNo effective record covers the date
Refund or return policyPublished policy page with a named ownerLatest published version; drafts excludedAll employeesTwo published versions conflict
Renewal term in a customer contractExecuted contract and signed amendmentsLatest amendment that changes the clauseLegal and account ownerAmendment referenced but not found
How a past incident was resolvedClosed case record and its linked commentsThe case as closed; later edits flaggedSupport staff with case accessCase closed without a resolution note
Product specificationReleased engineering documentReleased revision only; superseded revisions are hard negativesEngineering and product rolesReleased revision missing a required field
Who approved an exceptionApproval record in the workflow toolThe approval tied to that transactionManagers in the approval chainApproval exists only in an unlinked email

Input dictionary: what each worksheet column must contain

Every worksheet row needs the same fields so that graders can turn it into evaluation items without guessing. The dictionary below is the minimum; teams with regulated content may add a retention field and a legal hold flag.

Input dictionary: what each worksheet column must contain
FieldDefinitionIllustrative entry
Query intentA named class of questions with the same evidence needsCurrent price for a named customer
Controlling sourceThe one source type that decides the answerERP price record
Supporting sourcesSources that may add context but never overrideQuote emails, sales playbook
Required evidence setEvery item an answer must cite to be completePrice record plus the customer's contract tier
Hard negativesSimilar but non-authoritative items the retriever must not use as proofLast year's price sheet, a draft discount memo
Version ruleHow to choose among versions and what supersedes whatValid-on-date; replaced versions excluded
Permission ruleWho may receive evidence from the controlling sourceAccount team for that customer
Abstain conditionWhen the correct output is a refusal or a referralNo record valid on the question date

Does the evidence authorize this answer for this user at this time?

An answer is supported only if the evidence is authoritative, current for the question date and visible to the person asking. Relevance covers none of the three, so the worksheet tests them separately.

Permission is the condition most often left out of offline tests. Microsoft documents that Microsoft 365 Copilot only surfaces organizational data to which an individual user has at least view permissions, and that grounding respects the user's identity-based access boundary. An evaluation set for a permission-aware assistant needs the asking user's role on every item, or it will reward answers the real system should never give.

Version needs an explicit supersession link rather than a newest-file guess. The DCMI Metadata Terms define isReplacedBy as a related resource that supplants, displaces or supersedes the described resource. Storing that relation, from whatever system holds it, lets the grader mark a superseded document as a hard negative instead of a near miss.

Time is the third test. A question asked about last March should be answered from the record valid last March, which may not be the latest version. Record the question date on every item.

Worked scenario: one pricing question, three plausible sources

The scenario is illustrative and invented for this article. A sales representative asks the assistant what unit price a named customer pays for a pump model this quarter. Retrieval returns three candidates, all highly similar to the question.

  • Candidate A: a price sheet attached to an email from last year, quoting the pump at a list price. Tier 4 by location, Tier 3 by content, and superseded.
  • Candidate B: a sales wiki page summarizing contract tiers, last edited by someone who has left. Tier 3, no owner, useful for context only.
  • Candidate C: the ERP price record for that customer and item, effective from the first of the quarter. Tier 1, controlling, valid on the question date.
  • Required evidence set: Candidate C plus the customer's contract tier from the contract repository. An answer that cites C alone is correct but incomplete under the worksheet.
  • Permission check: the representative covers this account, so C may be shown. A representative from another region would get an abstention and a referral, even though C is the best match.
  • Grade, illustrative: an answer citing A is incorrect even if the number happened to match, because the evidence does not authorize it; an answer citing C and the tier is correct and complete.

Measuring retrieval against the hierarchy instead of relevance alone

Once labels encode authority, standard metrics measure the right thing, and a few additions cover what they miss. Score retrieval, citation and the final answer separately so a failure can be traced to one stage.

For answers, a scale that rewards declining over guessing fits the abstain column. The CRAG benchmark rates responses as perfect, acceptable, missing or incorrect, and its automatic evaluation scores correct answers at 1, missing answers at 0 and incorrect answers at minus 1. For citations, the ALCE benchmark evaluates generated text with citations on fluency, correctness and citation quality, which maps onto whether each claim cites its controlling source. Ragas faithfulness, which counts a response as faithful when all of its claims can be supported by the retrieved context, catches answers that drift beyond what was retrieved.

Measuring retrieval against the hierarchy instead of relevance alone
MeasureWhat it checksHow the hierarchy changes it
Context precisionRelevant chunks ranked above irrelevant onesOnly controlling and supporting sources count as relevant; hard negatives count against
Context recallReference claims supported by retrieved contextThe reference is built from the required evidence set
FaithfulnessAnswer claims supported by retrieved contextClaims supported only by a hard negative are flagged
Citation qualityWhether citations support the claims they sit onEach key claim must cite the controlling source
Correct, missing, incorrectAnswer outcome with a penalty for wrong answersAbstaining when the worksheet says abstain scores as correct behavior

Which assumption changes the ranking, and who signs off

Two assumptions move the ranking more than any other: that the controlling system is kept current, and that permissions in the index match permissions in the source. Test both before trusting any score.

If the ERP price record lags real deals because representatives agree terms in email and update the system weeks later, Tier 1 is stale for that intent and the worksheet must say so, for example by requiring the latest approved quote as a second piece of evidence. If the search index copies documents without their access lists, the permission rule is not enforced at all, and a high correctness score hides exposure.

Show uncertainty in the worksheet rather than in prose. Mark each row with who confirmed the controlling source and when, and flag rows where the owner disagreed with the retrieval engineer. A retrieval engineer can build the rows; the owner of each source has to approve the controlling source, the version rule and the abstain condition, because those are business decisions.

Carrying authority labels into licensed retrieval data

Retrieval evaluation sets built from real company records are only reusable if the authority labels travel with them. A buyer assembling such a set needs each document's tier, owner, supersession links, effective dates and access group, along with the rights to use the records for retrieval or evaluation.

When SourceX sources records for a buyer, it handles rights review, contracting, preparation and delivery with the supplying company, so authority labels can be requested as part of the preparation. Nothing is pulled from a shelf: each set is found at U.S. businesses against a buyer's stated scope, volume, language, format, timing and licensing needs. Asking for authority and version metadata in that request is what makes the records usable for the hierarchy described here.

Frequently asked questions

What is an evidence hierarchy in retrieval?

It is a rule that ranks sources by their authority to answer a given type of question, separate from how similar they are to the query. It names the controlling source, the supporting sources, the version rule and when to decline.

Is the newest document always the most authoritative?

No. A newer draft or an email can be more recent than a published policy without replacing it, and a question about a past date needs the version valid then. Use explicit supersession links and effective dates rather than modification time.

How do permissions affect retrieval evaluation?

An answer can be correct and still be wrong for the person asking if they lack access to the evidence. Record the asking user's role on each test item and grade answers that rely on evidence the user cannot see as failures.

Should a company assistant ever refuse to answer?

Yes, when the worksheet's abstain condition is met, such as no record valid on the question date or two published versions in conflict. Scoring schemes that penalize incorrect answers more than missing ones reward that behavior.

Who should own the source authority worksheet?

A retrieval engineer drafts it, and the owner of each source approves the controlling source, version rule and abstain condition for the intents that touch their system. Re-review it when systems, policies or permissions change.

Sources

  • Ragas defines context precision as the retriever's ability to rank relevant chunks higher than irrelevant ones for a query. Source
  • Ragas defines context recall as the number of claims in the reference supported by the retrieved context divided by the total claims in the reference. Source
  • Ragas treats a response as faithful when all of its claims can be supported by the retrieved context. Source
  • CRAG rates RAG responses as perfect, acceptable, missing or incorrect, and its automatic evaluation scores correct as 1, missing as 0 and incorrect as -1. Source
  • ALCE provides automatic evaluation of LLM generations with citations along fluency, correctness and citation quality. Source
  • Microsoft documents that Microsoft 365 Copilot only surfaces organizational data to which individual users have at least view permissions, and that Semantic Index honors the user identity-based access boundary. Source
  • DCMI Metadata Terms define isReplacedBy as a related resource that supplants, displaces or supersedes the described resource. Source

Tell us what data you need

Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.

See if your company qualifies