AI uses for records
RAG access control evaluation data: grading retrieval per user
By SourceX Editorial · Updated
9 sections
Short answer
Evaluate it against gold evidence labeled per user, not per query. For each test item, record who is asking, their entitlements at query time, which documents they may see, and which of those visible documents answer the question. Then score recall and precision only over that visible set, count any forbidden document in the candidates as a failure, and check that the system abstains when only forbidden evidence would answer.
Retrieval
RAG access control
Key takeaways
- Ground truth for a permission-aware retriever is per user and per time, not per query.
- Score recall and precision only over documents the user may open, and count forbidden candidates separately.
- Where the filter runs decides which failures appear: stale groups, top-k starvation or unlabeled documents left public.
- Mark items where visible evidence is insufficient or misleading, and expect abstention or the controlling source.
- Re-take the entitlement snapshot with every refresh and plant principals whose access changed.
- The task set checks enforcement; it cannot tell you whether the source permissions were right.
How do you evaluate retrieval that must exclude inaccessible evidence?
You change the unit of ground truth from a query to a query asked by a specific person at a specific time. The same question about a renewal, a ticket or a policy has different correct evidence for a sales rep, a finance lead and a contractor, so a single relevance list per query cannot grade a permission-aware retriever. It either rewards the system for finding documents the user should never see or punishes it for correctly leaving them out.
A permission scoped retrieval task carries four things a standard retrieval item lacks: the principal, an entitlement snapshot, a visible relevant set and an expected behavior when the visible set is not enough. With those in place, ordinary retrieval metrics become meaningful again, and two new checks appear: forbidden documents in the candidate pool, and correct abstention.
This page covers the retriever and its task specification. Testing whether a generated answer leaks restricted facts through paraphrase or inference is a separate exercise with its own metrics, linked at the end.
Why a single relevance list per query is the wrong gold standard
Recall metrics need a reference set. The Ragas documentation, for example, notes that context recall always requires a reference to compare against, and its context precision metric counts relevant chunks against all chunks retrieved at each rank. If that reference includes a deal-desk memo the user cannot open, perfect access control looks like missed recall. If the reference excludes it for everyone, an administrator who should see it is graded against the wrong target.
OWASP's 2025 entry on vector and embedding weaknesses describes the underlying risk: where several classes of users share one vector database, context can leak between users or queries, and its mitigation is permission-aware stores with strict logical partitioning. An evaluation built on one shared relevance list cannot tell you whether that partitioning works, because it never asks who the query came from.
The permission scoped retrieval task specification
This is the asset: the fields each task item needs so that two graders computing metrics from it get the same numbers. Values in the example column are illustrative.
| Field | What it holds | Illustrative value | Why the grader needs it |
|---|---|---|---|
| query_intent | The question and what a complete answer must establish | Current renewal terms for the logistics account | Defines relevance before permissions are applied |
| principal | A concrete test identity with groups and roles | Sales rep in the west region group | Entitlements resolve from the identity, never from the prompt |
| entitlement_snapshot | Group memberships and document access lists, with a timestamp | Directory and ACL export taken on the first of the month | Pins the visible set so later directory changes do not move the verdict |
| relevant_all | Every document that answers the intent, ignoring permissions | Contract, amendment, approval memo, pricing policy | Lets you measure what filtering removed |
| visible_relevant | relevant_all intersected with what the principal may open | Contract and pricing policy | The recall target for this principal |
| forbidden | Relevant or near-duplicate documents the principal may not open | Approval memo, margin review | Any appearance in candidates is a failure |
| controlling_source | Which visible document is authoritative and current | Signed amendment, version 3 | Separates the right document from a similar older one |
| sufficiency | Whether visible_relevant can support a complete answer | Partial: terms visible, approval status not | Decides answer, partial answer or abstain |
| expected_behavior | Answer, answer with stated gap, or abstain without naming hidden material | Answer terms; say approval status is not available | Grades the end of the retrieval step |
| filter_stage | Where trimming happens in the system under test | Filter inside the search query | Explains starvation and fail-open results |
Where does the permission filter run, and what does each stage break?
The stage at which a system trims results determines which failures you will see, so the task set should be built to expose each one. Microsoft's security filter pattern for Azure AI Search, for instance, stores group identifiers in a filterable field and applies a filter on every query; the same documentation says the search service does not authenticate users or decide what they may see, which leaves identity resolution to the application. Microsoft also documents a newer native option that reads the caller's token at query time, which was still in preview as of October 18, 2026.
Elastic's document level security for content connectors syncs access control documents separately from content. Its documentation states two behaviors worth testing directly: a content document without access control fields has no restrictions on who can view it, and a document whose access control field is present but empty is visible to no one. The first is fail-open, the second fail-closed, and both happen silently.
| Filter stage | Typical failure | Task items that expose it | What to measure |
|---|---|---|---|
| Inside the index query | Group IDs resolved wrongly or stale | Principals recently added to or removed from a group | Forbidden in candidates; visible relevant missing |
| After retrieval, before the model | Top-k starvation: forbidden hits fill the k slots, then get removed | Queries where many forbidden near-duplicates outrank the visible answer | Visible recall at k after trimming |
| At ingestion, from synced ACLs | Documents indexed without ACL fields become public | Documents planted with missing and with empty ACL fields | Exposure of unlabeled documents |
| Native token-based trimming | Token scope or tenant mismatch | Guest users and cross-tenant identities | Same metrics, run per identity type |
| None, relying on the prompt | Model told not to use certain sources | Any item with a forbidden document | Expect failure; use as a negative control |
Worked example: one question, three people, three correct answers
Illustrative only. A shared workspace indexes six documents about one customer renewal: the signed contract, a later amendment, a draft amendment, a pricing policy, a deal-desk approval memo and a margin review. Four test principals, one of them again after a permission change, ask the same question: what are the current renewal terms?
- The sales rep row tests ordinary permission-aware recall.
- The contractor row tests abstention: the retriever should return nothing that answers, and the pipeline should not fill the gap from a forbidden document.
- The removed-access row tests freshness of the entitlement snapshot.
- The draft author row is the exception that keeps the set honest: a document can be visible and relevant yet still be the wrong evidence, so permission scoping does not replace authority and version labels.
| Principal | May open | Gold evidence for this principal | Expected behavior |
|---|---|---|---|
| Sales rep | Contract, signed amendment, pricing policy | Signed amendment, then contract | Answer from the signed amendment; do not mention the approval memo |
| Deal-desk analyst | All except the margin review | Signed amendment and approval memo | Answer with approval context |
| Contractor in support | Pricing policy only | None that answers the question | Abstain; say the terms are not available to them |
| Sales rep, after access removed | Pricing policy only, per the newer snapshot | None | Abstain; any contract hit is a stale-ACL failure |
| Author of the draft amendment | Everything the sales rep sees plus the draft | Signed amendment; the draft is visible but not controlling | Answer from the signed version, not the draft |
Which metrics should be computed against the scoped gold set?
Compute standard retrieval metrics only over visible_relevant, and add counts that have no tolerance. Keep them in separate columns of the report; folding them into one score lets a good recall number hide an exposure.
At a glance
Which metrics should be computed against the scoped gold set
- Scoped recall at k: share of visible_relevant documents in the top k after trimming.
- Scoped precision at k: share of the top k that are in visible_relevant, so visible but irrelevant hits still count against the system.
- Forbidden candidate count: forbidden documents at any stage before the model sees context. The target is zero, and every hit is investigated by filter stage.
- Over-trim rate: visible_relevant documents removed by the filter, which catches broken group resolution that leakage tests reward.
- Controlling-source hit rate: items where the authoritative, current document ranks first among visible results.
- Abstention accuracy: on insufficient items, the share where the retriever returned no answering evidence and the pipeline abstained without naming hidden material.
When the best evidence is forbidden: the exception to plan for
The hardest items are the ones where a weaker visible document seems to answer. If the approval memo is forbidden but an old email thread the rep can see quotes a superseded discount, a retriever that returns the thread scores well on scoped recall and still produces a wrong answer. Label these items with sufficiency set to misleading, and expect the pipeline to return the controlling visible source or abstain.
Entitlements also drift. OWASP's authorization guidance says permissions should be validated on every request and access denied by default; a test set built on a month-old ACL export checks last month's permissions. Re-take the entitlement snapshot when you refresh the set, and keep a small group of items whose principal's access changed between snapshots so a cached group list shows up as a failure rather than a pass.
Shared assistants raise the stakes. Microsoft states that Microsoft 365 Copilot only surfaces organizational data the individual user has at least view permission for, which means the assistant inherits whatever oversharing already exists in the source permissions. A permission-aware retriever can be correct and still expose a document that was shared too broadly, so the task set can verify enforcement, not whether the access lists themselves are right.
What this specification cannot tell you
A permission scoped task set answers a narrow question well: given these identities and these access lists, does retrieval return the right visible evidence and nothing else? Several things sit outside that question, and a report that uses the set should say so plainly rather than let a high score stand in for a security review.
- It does not grade generated answers for paraphrased or inferred disclosure; that needs answer-level leakage tests.
- It cannot judge whether source permissions are correct, only whether the system enforces them.
- Synthetic org charts miss the irregular inheritance, guest links and deny rules that cause real exposure, so scores on synthetic permission graphs overstate safety.
- Results hold for the entitlement snapshot used; a new directory sync or connector change needs a rerun.
- Small sets of insufficient and misleading items give wide uncertainty on abstention accuracy, so report the item counts beside each rate.
Getting corpora whose access lists survived the export
The specification needs documents that arrive with their original access lists, group structure and version history, because those fields produce the visible, forbidden and controlling labels. Many exports drop them, and rebuilding them after the fact means guessing.
For a corpus like this, SourceX does the sourcing per request, from U.S. businesses rather than a catalog, and a buyer states the scope, volume, format and licensing needs through the request form, including which permission and version fields must be kept. The licensing company approves the scope and signs before delivery, and de-identification is done by SourceX through approved processing routes, so identity fields in the ACLs are typically replaced with stable pseudonyms that keep the group structure testable.
Frequently asked questions
What is RAG access control evaluation data?
It is a test set in which each retrieval item names the user asking, their entitlements at that time, and which documents they may and may not see. The gold evidence is computed per user, so the retriever is scored on finding what that person is allowed to use and on leaving the rest out.
Can I reuse an existing RAG benchmark and add a permission filter?
Only partly. Public benchmarks have one relevance list per query and no access lists, so you would need to add principals, synthetic ACLs and per-user gold sets yourself. Real permission structures are more irregular than synthetic ones, which matters for the failures you want to find.
How is this different from testing a RAG system for data leakage?
Leakage tests grade the generated answer for restricted facts, including paraphrase and inference. This specification grades the retrieval step: whether the right visible documents came back, whether any forbidden document reached the candidate pool, and whether the system abstained when it should.
What should happen when a user asks about something they cannot access?
The retriever should return no answering evidence and the pipeline should say the information is not available to them, without confirming that a hidden document exists. Whether naming the existence of a restricted document counts as a failure is a policy decision to record per item.
How often should the entitlement snapshot be refreshed?
Whenever the test set is refreshed, and after any directory sync or connector change. Keep a few items whose principal lost or gained access between snapshots so stale permission caching appears as a measurable failure.
Sources
- OWASP's 2025 LLM Top 10 warns of context leakage between users sharing a vector database and recommends permission-aware vector stores with strict partitioning. Source
- OWASP's authorization cheat sheet says to deny by default and validate permissions on every request. Source
- Microsoft's security filter pattern for Azure AI Search stores group IDs in a filterable field and filters queries by them; the service does not authenticate users or decide what they may view. Source
- Azure AI Search documents native document-level access enforcement that uses the caller's token at query time; it was still in preview as of October 18, 2026, while security filters are the generally available approach. Source
- Microsoft states that Microsoft 365 Copilot only surfaces organizational data to which individual users have at least view permissions. Source
- Elastic's document level security for connectors: documents without access control fields have no restrictions, and a present but empty field makes a document invisible. Source
- Ragas defines context precision from precision at each rank of the retrieved chunks. Source
- Ragas notes that context recall always requires a reference to compare against. Source
Tell us what data you need
Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.