Skip to content

AI uses for records

RAG access control evaluation data: grading retrieval per user

By SourceX Editorial · Updated

9 sections

Short answer

Evaluate it against gold evidence labeled per user, not per query. For each test item, record who is asking, their entitlements at query time, which documents they may see, and which of those visible documents answer the question. Then score recall and precision only over that visible set, count any forbidden document in the candidates as a failure, and check that the system abstains when only forbidden evidence would answer.

Retrieval

RAG access control

This page covers RAG access control evaluation data: grading retrieval per user.Illustrative

Key takeaways

  • Ground truth for a permission-aware retriever is per user and per time, not per query.
  • Score recall and precision only over documents the user may open, and count forbidden candidates separately.
  • Where the filter runs decides which failures appear: stale groups, top-k starvation or unlabeled documents left public.
  • Mark items where visible evidence is insufficient or misleading, and expect abstention or the controlling source.
  • Re-take the entitlement snapshot with every refresh and plant principals whose access changed.
  • The task set checks enforcement; it cannot tell you whether the source permissions were right.

How do you evaluate retrieval that must exclude inaccessible evidence?

You change the unit of ground truth from a query to a query asked by a specific person at a specific time. The same question about a renewal, a ticket or a policy has different correct evidence for a sales rep, a finance lead and a contractor, so a single relevance list per query cannot grade a permission-aware retriever. It either rewards the system for finding documents the user should never see or punishes it for correctly leaving them out.

A permission scoped retrieval task carries four things a standard retrieval item lacks: the principal, an entitlement snapshot, a visible relevant set and an expected behavior when the visible set is not enough. With those in place, ordinary retrieval metrics become meaningful again, and two new checks appear: forbidden documents in the candidate pool, and correct abstention.

This page covers the retriever and its task specification. Testing whether a generated answer leaks restricted facts through paraphrase or inference is a separate exercise with its own metrics, linked at the end.

Why a single relevance list per query is the wrong gold standard

Recall metrics need a reference set. The Ragas documentation, for example, notes that context recall always requires a reference to compare against, and its context precision metric counts relevant chunks against all chunks retrieved at each rank. If that reference includes a deal-desk memo the user cannot open, perfect access control looks like missed recall. If the reference excludes it for everyone, an administrator who should see it is graded against the wrong target.

OWASP's 2025 entry on vector and embedding weaknesses describes the underlying risk: where several classes of users share one vector database, context can leak between users or queries, and its mitigation is permission-aware stores with strict logical partitioning. An evaluation built on one shared relevance list cannot tell you whether that partitioning works, because it never asks who the query came from.

The permission scoped retrieval task specification

This is the asset: the fields each task item needs so that two graders computing metrics from it get the same numbers. Values in the example column are illustrative.

The permission scoped retrieval task specification
FieldWhat it holdsIllustrative valueWhy the grader needs it
query_intentThe question and what a complete answer must establishCurrent renewal terms for the logistics accountDefines relevance before permissions are applied
principalA concrete test identity with groups and rolesSales rep in the west region groupEntitlements resolve from the identity, never from the prompt
entitlement_snapshotGroup memberships and document access lists, with a timestampDirectory and ACL export taken on the first of the monthPins the visible set so later directory changes do not move the verdict
relevant_allEvery document that answers the intent, ignoring permissionsContract, amendment, approval memo, pricing policyLets you measure what filtering removed
visible_relevantrelevant_all intersected with what the principal may openContract and pricing policyThe recall target for this principal
forbiddenRelevant or near-duplicate documents the principal may not openApproval memo, margin reviewAny appearance in candidates is a failure
controlling_sourceWhich visible document is authoritative and currentSigned amendment, version 3Separates the right document from a similar older one
sufficiencyWhether visible_relevant can support a complete answerPartial: terms visible, approval status notDecides answer, partial answer or abstain
expected_behaviorAnswer, answer with stated gap, or abstain without naming hidden materialAnswer terms; say approval status is not availableGrades the end of the retrieval step
filter_stageWhere trimming happens in the system under testFilter inside the search queryExplains starvation and fail-open results

Where does the permission filter run, and what does each stage break?

The stage at which a system trims results determines which failures you will see, so the task set should be built to expose each one. Microsoft's security filter pattern for Azure AI Search, for instance, stores group identifiers in a filterable field and applies a filter on every query; the same documentation says the search service does not authenticate users or decide what they may see, which leaves identity resolution to the application. Microsoft also documents a newer native option that reads the caller's token at query time, which was still in preview as of October 18, 2026.

Elastic's document level security for content connectors syncs access control documents separately from content. Its documentation states two behaviors worth testing directly: a content document without access control fields has no restrictions on who can view it, and a document whose access control field is present but empty is visible to no one. The first is fail-open, the second fail-closed, and both happen silently.

Where does the permission filter run, and what does each stage break?
Filter stageTypical failureTask items that expose itWhat to measure
Inside the index queryGroup IDs resolved wrongly or stalePrincipals recently added to or removed from a groupForbidden in candidates; visible relevant missing
After retrieval, before the modelTop-k starvation: forbidden hits fill the k slots, then get removedQueries where many forbidden near-duplicates outrank the visible answerVisible recall at k after trimming
At ingestion, from synced ACLsDocuments indexed without ACL fields become publicDocuments planted with missing and with empty ACL fieldsExposure of unlabeled documents
Native token-based trimmingToken scope or tenant mismatchGuest users and cross-tenant identitiesSame metrics, run per identity type
None, relying on the promptModel told not to use certain sourcesAny item with a forbidden documentExpect failure; use as a negative control

Worked example: one question, three people, three correct answers

Illustrative only. A shared workspace indexes six documents about one customer renewal: the signed contract, a later amendment, a draft amendment, a pricing policy, a deal-desk approval memo and a margin review. Four test principals, one of them again after a permission change, ask the same question: what are the current renewal terms?

  • The sales rep row tests ordinary permission-aware recall.
  • The contractor row tests abstention: the retriever should return nothing that answers, and the pipeline should not fill the gap from a forbidden document.
  • The removed-access row tests freshness of the entitlement snapshot.
  • The draft author row is the exception that keeps the set honest: a document can be visible and relevant yet still be the wrong evidence, so permission scoping does not replace authority and version labels.
Worked example: one question, three people, three correct answers
PrincipalMay openGold evidence for this principalExpected behavior
Sales repContract, signed amendment, pricing policySigned amendment, then contractAnswer from the signed amendment; do not mention the approval memo
Deal-desk analystAll except the margin reviewSigned amendment and approval memoAnswer with approval context
Contractor in supportPricing policy onlyNone that answers the questionAbstain; say the terms are not available to them
Sales rep, after access removedPricing policy only, per the newer snapshotNoneAbstain; any contract hit is a stale-ACL failure
Author of the draft amendmentEverything the sales rep sees plus the draftSigned amendment; the draft is visible but not controllingAnswer from the signed version, not the draft

Which metrics should be computed against the scoped gold set?

Compute standard retrieval metrics only over visible_relevant, and add counts that have no tolerance. Keep them in separate columns of the report; folding them into one score lets a good recall number hide an exposure.

At a glance

Which metrics should be computed against the scoped gold set

  • Scoped recall at k: share of visible_relevant documents in the top k after trimming.
  • Scoped precision at k: share of the top k that are in visible_relevant, so visible but irrelevant hits still count against the system.
  • Forbidden candidate count: forbidden documents at any stage before the model sees context. The target is zero, and every hit is investigated by filter stage.
  • Over-trim rate: visible_relevant documents removed by the filter, which catches broken group resolution that leakage tests reward.
  • Controlling-source hit rate: items where the authoritative, current document ranks first among visible results.
  • Abstention accuracy: on insufficient items, the share where the retriever returned no answering evidence and the pipeline abstained without naming hidden material.

When the best evidence is forbidden: the exception to plan for

The hardest items are the ones where a weaker visible document seems to answer. If the approval memo is forbidden but an old email thread the rep can see quotes a superseded discount, a retriever that returns the thread scores well on scoped recall and still produces a wrong answer. Label these items with sufficiency set to misleading, and expect the pipeline to return the controlling visible source or abstain.

Entitlements also drift. OWASP's authorization guidance says permissions should be validated on every request and access denied by default; a test set built on a month-old ACL export checks last month's permissions. Re-take the entitlement snapshot when you refresh the set, and keep a small group of items whose principal's access changed between snapshots so a cached group list shows up as a failure rather than a pass.

Shared assistants raise the stakes. Microsoft states that Microsoft 365 Copilot only surfaces organizational data the individual user has at least view permission for, which means the assistant inherits whatever oversharing already exists in the source permissions. A permission-aware retriever can be correct and still expose a document that was shared too broadly, so the task set can verify enforcement, not whether the access lists themselves are right.

What this specification cannot tell you

A permission scoped task set answers a narrow question well: given these identities and these access lists, does retrieval return the right visible evidence and nothing else? Several things sit outside that question, and a report that uses the set should say so plainly rather than let a high score stand in for a security review.

  • It does not grade generated answers for paraphrased or inferred disclosure; that needs answer-level leakage tests.
  • It cannot judge whether source permissions are correct, only whether the system enforces them.
  • Synthetic org charts miss the irregular inheritance, guest links and deny rules that cause real exposure, so scores on synthetic permission graphs overstate safety.
  • Results hold for the entitlement snapshot used; a new directory sync or connector change needs a rerun.
  • Small sets of insufficient and misleading items give wide uncertainty on abstention accuracy, so report the item counts beside each rate.

Getting corpora whose access lists survived the export

The specification needs documents that arrive with their original access lists, group structure and version history, because those fields produce the visible, forbidden and controlling labels. Many exports drop them, and rebuilding them after the fact means guessing.

For a corpus like this, SourceX does the sourcing per request, from U.S. businesses rather than a catalog, and a buyer states the scope, volume, format and licensing needs through the request form, including which permission and version fields must be kept. The licensing company approves the scope and signs before delivery, and de-identification is done by SourceX through approved processing routes, so identity fields in the ACLs are typically replaced with stable pseudonyms that keep the group structure testable.

Frequently asked questions

What is RAG access control evaluation data?

It is a test set in which each retrieval item names the user asking, their entitlements at that time, and which documents they may and may not see. The gold evidence is computed per user, so the retriever is scored on finding what that person is allowed to use and on leaving the rest out.

Can I reuse an existing RAG benchmark and add a permission filter?

Only partly. Public benchmarks have one relevance list per query and no access lists, so you would need to add principals, synthetic ACLs and per-user gold sets yourself. Real permission structures are more irregular than synthetic ones, which matters for the failures you want to find.

How is this different from testing a RAG system for data leakage?

Leakage tests grade the generated answer for restricted facts, including paraphrase and inference. This specification grades the retrieval step: whether the right visible documents came back, whether any forbidden document reached the candidate pool, and whether the system abstained when it should.

What should happen when a user asks about something they cannot access?

The retriever should return no answering evidence and the pipeline should say the information is not available to them, without confirming that a hidden document exists. Whether naming the existence of a restricted document counts as a failure is a policy decision to record per item.

How often should the entitlement snapshot be refreshed?

Whenever the test set is refreshed, and after any directory sync or connector change. Keep a few items whose principal lost or gained access between snapshots so stale permission caching appears as a measurable failure.

Sources

  • OWASP's 2025 LLM Top 10 warns of context leakage between users sharing a vector database and recommends permission-aware vector stores with strict partitioning. Source
  • OWASP's authorization cheat sheet says to deny by default and validate permissions on every request. Source
  • Microsoft's security filter pattern for Azure AI Search stores group IDs in a filterable field and filters queries by them; the service does not authenticate users or decide what they may view. Source
  • Azure AI Search documents native document-level access enforcement that uses the caller's token at query time; it was still in preview as of October 18, 2026, while security filters are the generally available approach. Source
  • Microsoft states that Microsoft 365 Copilot only surfaces organizational data to which individual users have at least view permissions. Source
  • Elastic's document level security for connectors: documents without access control fields have no restrictions, and a present but empty field makes a document invisible. Source
  • Ragas defines context precision from precision at each rank of the retrieved chunks. Source
  • Ragas notes that context recall always requires a reference to compare against. Source

Tell us what data you need

Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.

See if your company qualifies