Skip to content

AI uses for records

RAG source authority dataset: labeling relevant vs authoritative documents

By SourceX Editorial · Updated

9 sections

Short answer

A relevant document is about the question; an authoritative document is the one that governs the answer for this user, scope and date. To tell them apart, label each query-document pair twice: a topical relevance grade, then an authority role based on issuer, approval status, effective dates, scope and access. Score a RAG system on whether it cites the controlling source, and on whether it abstains when none is retrievable.

Retrieval

RAG source authority dataset

This page covers RAG source authority dataset: labeling relevant vs authoritative documents.Illustrative

Key takeaways

  • Relevance says a document is about the question; authority says it governs the answer for this user, scope and date.
  • Five tests decide the controlling source: issuer, approval status, effective period, scope and access.
  • Label relevance first and authority second, so one judgment does not bias the other.
  • Superseded versions, drafts, chat restatements and wrong-scope policies are the hard negatives that matter most.
  • Score controlling-source recall, citation support, staleness, access violations and correct abstention separately.

Relevant versus authoritative: why one score cannot carry both

A relevance label answers whether a document addresses the query. An authority label answers whether the document is entitled to decide the answer. Retrieval systems are trained and judged mostly on the first, so they surface whatever is most similar to the question: a draft, last year's version, a chat thread paraphrasing the policy, or another region's rule. Each can be highly relevant and still wrong to rely on.

Search evaluation already separates these ideas. Google's search quality rater guidelines ask raters for two distinct ratings: Needs Met, which judges how well a result serves the query's intent, and Page Quality, which judges how well the page achieves its purpose. A RAG source authority dataset needs the same split, adapted to enterprise documents, where authority is decided by who issued a document, whether it is in force and whom it applies to.

Government publishing shows how fine the distinction can be. The eCFR, a daily-updated online compilation of the Code of Federal Regulations, describes itself as authoritative but unofficial, and directs readers to the official legal edition for citation. A document can be current, accurate and highly relevant while another document remains the controlling one. Your labels should be able to say exactly that.

This guide gives a label scheme, distractor families, metrics, a worked example, an exception and the limits. It is a practical template; it reports no benchmark results.

What makes a document the controlling source for a question?

The controlling source is the document that an accountable owner would point to if the answer were challenged. In practice five tests decide it, and a document must pass all five for the user and moment of the question; failing any one demotes it to supporting, superseded or out of scope.

Apply the tests in order and stop at the first failure; the order is chosen so the cheapest checks run first. Issuer and status usually come straight from a document management system or knowledge base such as Confluence, SharePoint or a help center, where pages carry an owner and a published or draft state. Effective period and scope often need a person who knows the domain, because many organizations encode them only in the document body, in a sentence such as applies to all U.S. employees from January.

Access is the test teams most often skip during labeling, because labelers usually have broad rights. Label it from the asking user's point of view: a document can be the controlling source for a manager and simply invisible to an analyst, and the dataset should say both.

What makes a document the controlling source for a question?
TestQuestion for the labelerMetadata that answers itTypical failure
IssuerDid a person or body with authority over this subject publish it?Owner, department, approval workflow recordTeam wiki page restating a finance policy
StatusIs it approved and published, not a draft or proposal?Status field, approval date, publication channelDraft revision with tracked changes
Effective periodWas it in force on the date the question is about?Effective and expiry dates; a successor linkPrior version superseded last quarter
ScopeDoes it apply to this user's region, product, plan or role?Applicability tags, audience, jurisdictionPolicy written for the EU subsidiary
AccessMay this user see it?Access list or group identities at query timeManager-only guidance surfaced to staff

The source authority and retrieval relevance label guide

This is the asset: one record per query-document pair, plus one record per query for the required evidence set. Labelers fill the relevance grade first, without looking at authority, then the authority fields, so one judgment does not bias the other. The examples are illustrative.

At a glance

The source authority and retrieval relevance label guide

Effective dates and succession can reuse established metadata. The DCMI Metadata Terms define valid as the date or date range of a resource's validity, with ranges written as ISO 8601 start and end dates separated by a slash, and isReplacedBy as a related resource that supplants the described one. Mapping your document system's fields to these terms keeps labels portable across corpora.

Two rules prevent the most common labeling errors. Never infer an effective period from a file's modified date, because metadata edits, migrations and bulk permission changes all touch that date without changing what is in force. And never label a document controlling for a query unless the labeler can name the user role and as-of date the label assumes; a label without those two facts cannot be checked later.

The source authority and retrieval relevance label guide
FieldAllowed valuesLabeling ruleIllustrative entry
query_intentShort statement plus as-of date, user role, regionWrite the question as the user would mean it, with the date the answer must holdPer-diem limit for a U.S. sales rep traveling in March
relevance_grade0 not relevant, 1 related, 2 partly answers, 3 fully answersJudge topic only; ignore status, date and access3
authority_rolecontrolling, supporting, superseded, draft, informal, out_of_scopeApply the five tests; controlling only if all passsuperseded
valid_periodStart and end dates, open end allowedCopy from the source system; never infer from file dates alone2025-01-01/2025-12-31
successor_idDocument ID or noneLink to the version that replaced itPOL-TRAVEL-v7
access_labelGroups allowed at query timeSnapshot when the query is labeledall-employees
hard_negativeyes or no, with distractor familyyes when relevance is 2 or 3 but role is not controllingyes, prior version
required_evidence_setDocument IDs that together support the answerMinimum set an answer must citePOL-TRAVEL-v7 section 4
abstain_expectedyes or no, with reasonyes when no controlling source is accessible to this userno

Hard negatives that look right: distractor families to include

Include documents that a good retriever will rank highly and a careful employee would still refuse to rely on. Without them, an evaluation measures topical retrieval and calls it accuracy. OWASP's guidance on vector and embedding weaknesses names two of the underlying risks: misaligned access controls that let a system retrieve content the user should not see, and knowledge conflicts when sources contradict each other or retrieved data must supersede what the model already learned.

  • Mine distractors from the real corpus rather than writing them; synthetic near-duplicates are easier to tell apart than genuine old versions.
  • Keep at least one distractor from each family for every domain you evaluate, so a weak spot cannot hide behind a strong average.
  • Record which family each hard negative belongs to, so failures can be reported by family and fixed at the right layer, ranking, filtering or metadata.
Hard negatives that look right: distractor families to include
Distractor familyIllustrative exampleWhy retrieval ranks it highExpected label
Prior versionLast year's travel policy with the old limitNear-identical wording to the current versionsuperseded, hard negative
DraftProposed policy circulated for commentNewest file, often most detaileddraft, hard negative
Informal restatementSlack or Teams thread summarizing the ruleConversational phrasing matches user questionsinformal, supporting at most
Wrong scopeSame policy for another country or business unitSame headings and termsout_of_scope
Unofficial compilationInternal FAQ or a third-party summary of a regulationWritten to answer questions directlysupporting, cite the controlling text
Restricted originalManager-only guidance with the full ruleMost complete answer in the corpuscontrolling for managers, inaccessible for staff

Does the retrieved evidence authorize and support this exact answer for this user and time?

Only when three things hold at once: the cited documents include the controlling source for that user and date, the citations actually support each claim, and every cited document was visible to that user. Score each separately, then report how often all three held together.

Citation support has a well-tested formulation. The ALCE benchmark scores citation recall, whether each statement is fully supported by its cited passages, and citation precision, which flags citations that do not support the statement. Add authority on top: a statement fully supported by a superseded document is supported and still wrong to rely on.

Time needs its own test. The FreshQA benchmark separates questions whose answers never change, change slowly or change quickly, and includes questions with false premises that a system should rebut rather than answer. Enterprise policies behave like slow- and fast-changing facts, so pin an as-of date to each query and expect different answers for different dates.

Access has to be enforced at query time, not only labeled. Microsoft's security filter pattern for Azure AI Search, for example, stores group identities in a filterable field and filters out results that none of the caller's identities may read. A label set that records the expected access outcome lets you test that such a filter works for each user role.

Does the retrieved evidence authorize and support this exact answer for this user and time?
MetricDefinitionFailure it catches
Controlling-source recall at kShare of queries whose controlling document appears in the top k resultsRetriever prefers similar but non-governing documents
Authority precision of citationsShare of cited documents labeled controlling or supportingAnswers that lean on drafts, old versions or chat
Citation supportShare of statements fully supported by their citationsFluent claims that the cited text does not state
Stale citation rateShare of answers citing a document outside its valid periodVersion drift after a policy update
Access violation rateShare of answers citing a document the user may not seeLeaks through retrieval or caching
Correct abstention rateShare of abstain_expected queries where the system declines or routesConfident answers with no legitimate source

Worked example: one expense question, four plausible documents

The case is illustrative and constructed for this guide. A sales representative asks an internal assistant for the daily meal allowance during a trip next month. The corpus holds four relevant documents: the current travel policy, last year's version with a lower figure, a Slack thread where a manager quotes a rounded number, and a draft proposing a change for next year.

All four would earn a relevance grade of 2 or 3. Only the current policy passes issuer, status, effective period, scope and access for this user and date, so it is the controlling source and the only member of the required evidence set. Last year's version and the draft are hard negatives; the Slack thread is informal and may be cited as context but never as the basis of the figure.

An answer that quotes last year's figure and cites the old policy would pass a citation-support check, because the citation does support the statement. It fails authority and staleness. An answer that quotes the draft fails status. An answer that quotes the current figure but cites only the Slack thread is right for the wrong reason and fails authority precision. A single relevance metric scores all four answers as successes; the label guide separates them.

Now change one fact. If the same question is asked with an as-of date next year, after the draft is approved and published, the labels flip: the draft becomes the controlling source and the current policy becomes superseded. The query text is identical; only the as-of date differs. That is why every query in the set carries a date, and why a dataset that stores only question and answer pairs cannot test this behavior.

Finally, ask the question as a contractor who is not in the all-employees group. If the policy is not shared with contractors, the expected behavior is abstention or routing to a manager, and any answer that quotes the figure counts as an access violation even though it is correct for employees.

The exception: no authoritative document exists

Sometimes nothing written governs the answer: the rule lives in practice, in a manager's judgment or in an email that was never turned into policy. Do not promote the best informal source to controlling just to keep the query in the set.

Label such queries abstain_expected or mark the best source as informal and require the answer to say that no written policy was found and who decides. The same applies when the controlling document exists but is inaccessible to the asking user: the correct behavior is to say the answer requires someone with access, not to paraphrase what the restricted document says.

One more exception matters in regulated work. Where an official edition and a convenient compilation both exist, as with the CFR and the eCFR, decide per use case which one counts as controlling for citation and which one is acceptable for day-to-day answers, and write that decision into the guide.

Treat conflicts between two documents that both pass the five tests as a finding, not a labeling problem. Two approved, current, in-scope policies that disagree mean the organization has a governance gap; the evaluation should expect the system to surface the conflict and route it, and the label record should name both documents in the required evidence set.

Where authority labels break down, and who must sign them off

Authority labels expire. A policy update, a reorganization, a new region or a change in access groups can turn yesterday's controlling document into a superseded one, so each label set needs an owner, an as-of date and a refresh trigger tied to corpus, permission and policy changes.

Authority labels also cannot fix a corpus that lacks the evidence. If the source system never recorded approval status or effective dates, labelers end up guessing, and the dataset quietly becomes a relevance set with opinions attached. Check metadata coverage on a sample before committing to a full labeling run, and narrow the scope to the domains where the fields exist.

  • The retrieval engineer owns relevance grades, distractor coverage and metric definitions.
  • A source authority owner for each domain, such as finance policy or HR, signs off on which documents are controlling and for whom.
  • Labels that depend on law or contracts, such as which edition is citable or which customer agreement applies, go to counsel; the dataset should record the decision, not make it.
  • Rerun the authority labels whenever valid periods end, successor links change or access groups are restructured, and retire queries whose as-of date no longer reflects anything users ask.
  • If labelers disagree on authority role for more than a small share of pairs, stop and rewrite the five tests for that domain before labeling more.

Building a RAG source authority dataset from real business records

Most public retrieval benchmarks contain no approval status, effective dates, successor links or access lists, so they cannot test authority at all. A useful RAG source authority dataset comes from document systems where those fields exist and where real version chains and informal restatements sit side by side.

Corpora like this are not something SourceX keeps on a shelf: it looks for U.S. businesses whose records match each buyer request. A retrieval team that needs versioned policies, approval metadata or access-tagged documents can set out the scope, volume, format and licensing terms on the request-data form, and the supplying business approves scope and terms and signs before any data is delivered.

Frequently asked questions

Is a highly relevant document always safe for RAG to cite?

No. Relevance means the document is about the question; it says nothing about whether the document is approved, current, in scope or visible to the user. A superseded policy is usually highly relevant and still the wrong source.

How many relevance grades does a source authority dataset need?

Four grades are enough for most enterprise work: not relevant, related, partly answers and fully answers. Keep authority as a separate field so a document can be fully relevant and still superseded or out of scope.

What metadata do I need before labeling source authority?

At minimum: owner or issuer, approval status, effective and expiry dates, a link to any successor version, applicability tags such as region or role, and access groups at query time. Without these, labelers have to guess authority from file names and dates.

Should a RAG system answer when only informal sources exist?

It should say so. The answer can share what informal sources suggest, but it should state that no written policy was found and name who decides, or route the question; the evaluation should reward that behavior.

How often should authority labels be refreshed?

Whenever the corpus, permissions, policies or effective dates change in a way that affects labeled queries. Tie refreshes to those events rather than a calendar, and record an as-of date on every label.

Sources

  • Google's search quality rater guidelines cover two separate ratings: Needs Met, how useful a result is for the query's intent, and Page Quality, how well a page achieves its purpose. Source
  • The eCFR describes itself as a daily-updated editorial compilation of CFR material and Federal Register amendments, and states that it is not an official legal edition of the CFR. Source
  • DCMI Metadata Terms define valid as a date or date range of validity, written as ISO 8601 dates separated by a slash, and isReplacedBy as a related resource that supplants the described resource. Source
  • Azure AI Search's security filter pattern stores user or group identities in a filterable field and filters out results for which none of the caller's principals have read access. Source
  • OWASP LLM08 lists unauthorized access through misaligned access controls and knowledge conflicts between sources as risks in RAG systems, and recommends permission-aware vector stores. Source
  • The ALCE benchmark evaluates generated text with citations, including citation recall (statements supported by cited passages) and citation precision (irrelevant citations). Source
  • FreshQA groups questions by how quickly their answers change and includes false-premise questions that must be rebutted. Source

Tell us what data you need

Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.

See if your company qualifies