AI uses for records
RAG source authority dataset: labeling relevant vs authoritative documents
By SourceX Editorial · Updated
9 sections
Short answer
A relevant document is about the question; an authoritative document is the one that governs the answer for this user, scope and date. To tell them apart, label each query-document pair twice: a topical relevance grade, then an authority role based on issuer, approval status, effective dates, scope and access. Score a RAG system on whether it cites the controlling source, and on whether it abstains when none is retrievable.
Retrieval
RAG source authority dataset
Key takeaways
- Relevance says a document is about the question; authority says it governs the answer for this user, scope and date.
- Five tests decide the controlling source: issuer, approval status, effective period, scope and access.
- Label relevance first and authority second, so one judgment does not bias the other.
- Superseded versions, drafts, chat restatements and wrong-scope policies are the hard negatives that matter most.
- Score controlling-source recall, citation support, staleness, access violations and correct abstention separately.
Relevant versus authoritative: why one score cannot carry both
A relevance label answers whether a document addresses the query. An authority label answers whether the document is entitled to decide the answer. Retrieval systems are trained and judged mostly on the first, so they surface whatever is most similar to the question: a draft, last year's version, a chat thread paraphrasing the policy, or another region's rule. Each can be highly relevant and still wrong to rely on.
Search evaluation already separates these ideas. Google's search quality rater guidelines ask raters for two distinct ratings: Needs Met, which judges how well a result serves the query's intent, and Page Quality, which judges how well the page achieves its purpose. A RAG source authority dataset needs the same split, adapted to enterprise documents, where authority is decided by who issued a document, whether it is in force and whom it applies to.
Government publishing shows how fine the distinction can be. The eCFR, a daily-updated online compilation of the Code of Federal Regulations, describes itself as authoritative but unofficial, and directs readers to the official legal edition for citation. A document can be current, accurate and highly relevant while another document remains the controlling one. Your labels should be able to say exactly that.
This guide gives a label scheme, distractor families, metrics, a worked example, an exception and the limits. It is a practical template; it reports no benchmark results.
What makes a document the controlling source for a question?
The controlling source is the document that an accountable owner would point to if the answer were challenged. In practice five tests decide it, and a document must pass all five for the user and moment of the question; failing any one demotes it to supporting, superseded or out of scope.
Apply the tests in order and stop at the first failure; the order is chosen so the cheapest checks run first. Issuer and status usually come straight from a document management system or knowledge base such as Confluence, SharePoint or a help center, where pages carry an owner and a published or draft state. Effective period and scope often need a person who knows the domain, because many organizations encode them only in the document body, in a sentence such as applies to all U.S. employees from January.
Access is the test teams most often skip during labeling, because labelers usually have broad rights. Label it from the asking user's point of view: a document can be the controlling source for a manager and simply invisible to an analyst, and the dataset should say both.
| Test | Question for the labeler | Metadata that answers it | Typical failure |
|---|---|---|---|
| Issuer | Did a person or body with authority over this subject publish it? | Owner, department, approval workflow record | Team wiki page restating a finance policy |
| Status | Is it approved and published, not a draft or proposal? | Status field, approval date, publication channel | Draft revision with tracked changes |
| Effective period | Was it in force on the date the question is about? | Effective and expiry dates; a successor link | Prior version superseded last quarter |
| Scope | Does it apply to this user's region, product, plan or role? | Applicability tags, audience, jurisdiction | Policy written for the EU subsidiary |
| Access | May this user see it? | Access list or group identities at query time | Manager-only guidance surfaced to staff |
The source authority and retrieval relevance label guide
This is the asset: one record per query-document pair, plus one record per query for the required evidence set. Labelers fill the relevance grade first, without looking at authority, then the authority fields, so one judgment does not bias the other. The examples are illustrative.
At a glance
The source authority and retrieval relevance label guide
Effective dates and succession can reuse established metadata. The DCMI Metadata Terms define valid as the date or date range of a resource's validity, with ranges written as ISO 8601 start and end dates separated by a slash, and isReplacedBy as a related resource that supplants the described one. Mapping your document system's fields to these terms keeps labels portable across corpora.
Two rules prevent the most common labeling errors. Never infer an effective period from a file's modified date, because metadata edits, migrations and bulk permission changes all touch that date without changing what is in force. And never label a document controlling for a query unless the labeler can name the user role and as-of date the label assumes; a label without those two facts cannot be checked later.
| Field | Allowed values | Labeling rule | Illustrative entry |
|---|---|---|---|
| query_intent | Short statement plus as-of date, user role, region | Write the question as the user would mean it, with the date the answer must hold | Per-diem limit for a U.S. sales rep traveling in March |
| relevance_grade | 0 not relevant, 1 related, 2 partly answers, 3 fully answers | Judge topic only; ignore status, date and access | 3 |
| authority_role | controlling, supporting, superseded, draft, informal, out_of_scope | Apply the five tests; controlling only if all pass | superseded |
| valid_period | Start and end dates, open end allowed | Copy from the source system; never infer from file dates alone | 2025-01-01/2025-12-31 |
| successor_id | Document ID or none | Link to the version that replaced it | POL-TRAVEL-v7 |
| access_label | Groups allowed at query time | Snapshot when the query is labeled | all-employees |
| hard_negative | yes or no, with distractor family | yes when relevance is 2 or 3 but role is not controlling | yes, prior version |
| required_evidence_set | Document IDs that together support the answer | Minimum set an answer must cite | POL-TRAVEL-v7 section 4 |
| abstain_expected | yes or no, with reason | yes when no controlling source is accessible to this user | no |
Hard negatives that look right: distractor families to include
Include documents that a good retriever will rank highly and a careful employee would still refuse to rely on. Without them, an evaluation measures topical retrieval and calls it accuracy. OWASP's guidance on vector and embedding weaknesses names two of the underlying risks: misaligned access controls that let a system retrieve content the user should not see, and knowledge conflicts when sources contradict each other or retrieved data must supersede what the model already learned.
- Mine distractors from the real corpus rather than writing them; synthetic near-duplicates are easier to tell apart than genuine old versions.
- Keep at least one distractor from each family for every domain you evaluate, so a weak spot cannot hide behind a strong average.
- Record which family each hard negative belongs to, so failures can be reported by family and fixed at the right layer, ranking, filtering or metadata.
| Distractor family | Illustrative example | Why retrieval ranks it high | Expected label |
|---|---|---|---|
| Prior version | Last year's travel policy with the old limit | Near-identical wording to the current version | superseded, hard negative |
| Draft | Proposed policy circulated for comment | Newest file, often most detailed | draft, hard negative |
| Informal restatement | Slack or Teams thread summarizing the rule | Conversational phrasing matches user questions | informal, supporting at most |
| Wrong scope | Same policy for another country or business unit | Same headings and terms | out_of_scope |
| Unofficial compilation | Internal FAQ or a third-party summary of a regulation | Written to answer questions directly | supporting, cite the controlling text |
| Restricted original | Manager-only guidance with the full rule | Most complete answer in the corpus | controlling for managers, inaccessible for staff |
Does the retrieved evidence authorize and support this exact answer for this user and time?
Only when three things hold at once: the cited documents include the controlling source for that user and date, the citations actually support each claim, and every cited document was visible to that user. Score each separately, then report how often all three held together.
Citation support has a well-tested formulation. The ALCE benchmark scores citation recall, whether each statement is fully supported by its cited passages, and citation precision, which flags citations that do not support the statement. Add authority on top: a statement fully supported by a superseded document is supported and still wrong to rely on.
Time needs its own test. The FreshQA benchmark separates questions whose answers never change, change slowly or change quickly, and includes questions with false premises that a system should rebut rather than answer. Enterprise policies behave like slow- and fast-changing facts, so pin an as-of date to each query and expect different answers for different dates.
Access has to be enforced at query time, not only labeled. Microsoft's security filter pattern for Azure AI Search, for example, stores group identities in a filterable field and filters out results that none of the caller's identities may read. A label set that records the expected access outcome lets you test that such a filter works for each user role.
| Metric | Definition | Failure it catches |
|---|---|---|
| Controlling-source recall at k | Share of queries whose controlling document appears in the top k results | Retriever prefers similar but non-governing documents |
| Authority precision of citations | Share of cited documents labeled controlling or supporting | Answers that lean on drafts, old versions or chat |
| Citation support | Share of statements fully supported by their citations | Fluent claims that the cited text does not state |
| Stale citation rate | Share of answers citing a document outside its valid period | Version drift after a policy update |
| Access violation rate | Share of answers citing a document the user may not see | Leaks through retrieval or caching |
| Correct abstention rate | Share of abstain_expected queries where the system declines or routes | Confident answers with no legitimate source |
Worked example: one expense question, four plausible documents
The case is illustrative and constructed for this guide. A sales representative asks an internal assistant for the daily meal allowance during a trip next month. The corpus holds four relevant documents: the current travel policy, last year's version with a lower figure, a Slack thread where a manager quotes a rounded number, and a draft proposing a change for next year.
All four would earn a relevance grade of 2 or 3. Only the current policy passes issuer, status, effective period, scope and access for this user and date, so it is the controlling source and the only member of the required evidence set. Last year's version and the draft are hard negatives; the Slack thread is informal and may be cited as context but never as the basis of the figure.
An answer that quotes last year's figure and cites the old policy would pass a citation-support check, because the citation does support the statement. It fails authority and staleness. An answer that quotes the draft fails status. An answer that quotes the current figure but cites only the Slack thread is right for the wrong reason and fails authority precision. A single relevance metric scores all four answers as successes; the label guide separates them.
Now change one fact. If the same question is asked with an as-of date next year, after the draft is approved and published, the labels flip: the draft becomes the controlling source and the current policy becomes superseded. The query text is identical; only the as-of date differs. That is why every query in the set carries a date, and why a dataset that stores only question and answer pairs cannot test this behavior.
Finally, ask the question as a contractor who is not in the all-employees group. If the policy is not shared with contractors, the expected behavior is abstention or routing to a manager, and any answer that quotes the figure counts as an access violation even though it is correct for employees.
The exception: no authoritative document exists
Sometimes nothing written governs the answer: the rule lives in practice, in a manager's judgment or in an email that was never turned into policy. Do not promote the best informal source to controlling just to keep the query in the set.
Label such queries abstain_expected or mark the best source as informal and require the answer to say that no written policy was found and who decides. The same applies when the controlling document exists but is inaccessible to the asking user: the correct behavior is to say the answer requires someone with access, not to paraphrase what the restricted document says.
One more exception matters in regulated work. Where an official edition and a convenient compilation both exist, as with the CFR and the eCFR, decide per use case which one counts as controlling for citation and which one is acceptable for day-to-day answers, and write that decision into the guide.
Treat conflicts between two documents that both pass the five tests as a finding, not a labeling problem. Two approved, current, in-scope policies that disagree mean the organization has a governance gap; the evaluation should expect the system to surface the conflict and route it, and the label record should name both documents in the required evidence set.
Where authority labels break down, and who must sign them off
Authority labels expire. A policy update, a reorganization, a new region or a change in access groups can turn yesterday's controlling document into a superseded one, so each label set needs an owner, an as-of date and a refresh trigger tied to corpus, permission and policy changes.
Authority labels also cannot fix a corpus that lacks the evidence. If the source system never recorded approval status or effective dates, labelers end up guessing, and the dataset quietly becomes a relevance set with opinions attached. Check metadata coverage on a sample before committing to a full labeling run, and narrow the scope to the domains where the fields exist.
- The retrieval engineer owns relevance grades, distractor coverage and metric definitions.
- A source authority owner for each domain, such as finance policy or HR, signs off on which documents are controlling and for whom.
- Labels that depend on law or contracts, such as which edition is citable or which customer agreement applies, go to counsel; the dataset should record the decision, not make it.
- Rerun the authority labels whenever valid periods end, successor links change or access groups are restructured, and retire queries whose as-of date no longer reflects anything users ask.
- If labelers disagree on authority role for more than a small share of pairs, stop and rewrite the five tests for that domain before labeling more.
Building a RAG source authority dataset from real business records
Most public retrieval benchmarks contain no approval status, effective dates, successor links or access lists, so they cannot test authority at all. A useful RAG source authority dataset comes from document systems where those fields exist and where real version chains and informal restatements sit side by side.
Corpora like this are not something SourceX keeps on a shelf: it looks for U.S. businesses whose records match each buyer request. A retrieval team that needs versioned policies, approval metadata or access-tagged documents can set out the scope, volume, format and licensing terms on the request-data form, and the supplying business approves scope and terms and signs before any data is delivered.
Frequently asked questions
Is a highly relevant document always safe for RAG to cite?
No. Relevance means the document is about the question; it says nothing about whether the document is approved, current, in scope or visible to the user. A superseded policy is usually highly relevant and still the wrong source.
How many relevance grades does a source authority dataset need?
Four grades are enough for most enterprise work: not relevant, related, partly answers and fully answers. Keep authority as a separate field so a document can be fully relevant and still superseded or out of scope.
What metadata do I need before labeling source authority?
At minimum: owner or issuer, approval status, effective and expiry dates, a link to any successor version, applicability tags such as region or role, and access groups at query time. Without these, labelers have to guess authority from file names and dates.
Should a RAG system answer when only informal sources exist?
It should say so. The answer can share what informal sources suggest, but it should state that no written policy was found and name who decides, or route the question; the evaluation should reward that behavior.
How often should authority labels be refreshed?
Whenever the corpus, permissions, policies or effective dates change in a way that affects labeled queries. Tie refreshes to those events rather than a calendar, and record an as-of date on every label.
Sources
- Google's search quality rater guidelines cover two separate ratings: Needs Met, how useful a result is for the query's intent, and Page Quality, how well a page achieves its purpose. Source
- The eCFR describes itself as a daily-updated editorial compilation of CFR material and Federal Register amendments, and states that it is not an official legal edition of the CFR. Source
- DCMI Metadata Terms define valid as a date or date range of validity, written as ISO 8601 dates separated by a slash, and isReplacedBy as a related resource that supplants the described resource. Source
- Azure AI Search's security filter pattern stores user or group identities in a filterable field and filters out results for which none of the caller's principals have read access. Source
- OWASP LLM08 lists unauthorized access through misaligned access controls and knowledge conflicts between sources as risks in RAG systems, and recommends permission-aware vector stores. Source
- The ALCE benchmark evaluates generated text with citations, including citation recall (statements supported by cited passages) and citation precision (irrelevant citations). Source
- FreshQA groups questions by how quickly their answers change and includes false-premise questions that must be rebutted. Source
Tell us what data you need
Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.