AI uses for records
Permission-aware retrieval dataset: 5 records that catch forbidden hits
By SourceX Editorial · Updated
10 sections
Short answer
A permission-aware retrieval dataset can count a relevant but unauthorized result as a failure only when each item records who asked, the access model and its version, the relationship state at query time, an expected denial, and a grader that applies the decision before relevance. OpenFGA's conformance tests keep all five. EnterpriseBench's release keeps the requester and its attributes, and nothing in its rubric marks a forbidden but relevant answer wrong.
Retrieval
Permission aware retrieval
Key takeaways
- Five records turn a forbidden hit into a failure: requester, versioned access model, state at query time, expected denial and a grader that checks the decision first.
- In OpenFGA's tests at commit cbc9bd4, 191 of 473 yes-or-no checks expect a denial and 112 of 354 list assertions expect nothing back.
- Hard denials matter more than denial counts: 179 of OpenFGA's 191 ask about objects already in the test's stored relationships.
- EnterpriseBench's release at commit 9e41e5c names the requester in all 451 tasks but holds no expected denial, and 23 tool docstrings promise checks that no code performs.
- Audit a dataset's files before trusting its paper: six checks on the files show whether a permission score can mean anything.
What must a permission-aware retrieval dataset record before a relevant result can fail?
Five things, attached to every item: the requester, the authorization model with its version, the relationship or access-list state at the moment of the query, an expected decision that includes denials, and a grading rule that applies that decision before any relevance score. Drop any one and a result that answers the question perfectly but should never have been shown is either scored as a success or cannot be scored at all.
This page tests that claim on two public artifacts at pinned commits rather than on a design of our own. One is an authorization engine's conformance suite, where denial is the whole point; the other is an enterprise agent benchmark whose paper puts access control at the center of its design. Counting what each one actually stores shows which of the five records survive into a released dataset and which remain in prose.
A companion page sets out a full per-user task specification and the retrieval metrics to compute from it. This one asks the prior question a buyer faces when handed someone else's test set: does the evidence needed to fail a forbidden hit exist in the files at all?
Five records, counted in two public test sets
OpenFGA's conformance tests keep all five records on every assertion; EnterpriseBench's released tasks keep the requester and the attributes a decision would need, and none of the rest. The counts come from openfga/openfga at commit cbc9bd4, its consolidated and attribute-based test files, and from ast-fri/EnterpriseBench at commit 9e41e5c, read by a script that runs no code from either repository.
In OpenFGA's two files, all 198 test stages declare the model they test and 160 also write relationship tuples. Of 516 check assertions, 282 expect access, 191 expect a denial and 43 expect an error code, so 40.4 percent of the assertions with a yes-or-no answer are denials. A further 354 assertions ask which objects a user can list, and 112 of them, 31.6 percent, expect an empty list. In the attribute-based file, 113 checks also pass request context, 40 of them a timestamp, because a time-bound grant can be valid at one moment and expired the next: one test allows a user to view a document at 09:00 and denies the same view at 11:00 on the same day.
EnterpriseBench's task file holds 451 tasks, every one naming the requesting employee, 133 distinct employees in all. Its employee table gives all 1,260 employees a level code and a reports-to field, and all 750 repositories in its code workspace name an owner, which is enough to derive a decision. Yet no task's ground truth contains an expected denial under the pattern we applied, no file path in the repository names access rules, and the judge's five score levels mention relevance five times and access never.
Each record mapped to what the two repositories hold:
| Record | OpenFGA tests (cbc9bd4) | EnterpriseBench release (9e41e5c) | Without it |
|---|---|---|---|
| Requester | All 516 checks and 354 list assertions name a user | 451 of 451 tasks name an employee | No way to say whose permission applies |
| Access model and version | 198 of 198 stages declare a model | Paper describes JSON rules per source; no rule file in the release | The verdict cannot be reproduced |
| State at query time | 160 of 198 stages write tuples; 113 checks pass context | Employee levels, managers and repository owners, with no snapshot date | A revoked grant still looks valid |
| Expected decision, denials included | 191 deny and 282 allow assertions | 0 tasks with an expected denial | A forbidden hit has nothing to fail against |
| Decision applied before relevance | Each assertion passes or fails on the decision alone | Judge: 5 of 5 levels mention relevance, 0 mention access | A relevant but forbidden answer can score top marks |
Why a hard deny proves more than a deny count
A denial tests a permission boundary only when the forbidden object exists and someone else may see it. Asking whether a user can read a document nobody can read is passed by an empty index; asking whether a user can read a document a colleague can read is the relevant-but-unauthorized case a retrieval system actually meets in a shared workspace.
OpenFGA's suite rarely asks about an object with no stored state. We call a denial hard when the object it asks about appears as the object of a relationship tuple written in the same stage or an earlier stage of the same test, or in the assertion's own contextual tuples. By that rule 179 of the 191 denials, 93.7 percent, are hard, and only 12 ask about an object with no stored relationship at all. The rule shows that the object is already in the test's stored relationships, not that another user may see it.
A retrieval dataset needs the same property, stated the same way. For each expected denial, record the document that would answer the question, a principal who may see it, and the principal who asks and may not. A set whose forbidden documents are irrelevant to the query, or visible to nobody, measures nothing about the boundary however many denials it reports.
What EnterpriseBench's paper describes, and what its release lets you re-score
The paper describes access control in detail; the public release at commit 9e41e5c does not carry the records needed to re-score it. The camera-ready paper stored in the repository describes a role-based system with permissions tied to organizational levels 9 through 14, access rules kept in JSON for each data source, an access control layer that filters retrieved content before it reaches the user, and an ablation in which attempts at tasks the requester lacks permission for are counted as failures.
The released tooling tells a narrower story. The tool file that the evaluation client loads defines 76 functions. The docstrings of 23 promise a check, with phrases such as verifying access permissions or if access is allowed, and none of those 23 contains a denial path: no raise statement and no string reporting a refusal. One email-reading function carries a comment saying access control and permission checks should be added as needed. The judge then scores a trajectory from 1 to 5 on relevance and on matching the reference's entities, and is told to evaluate only what the generated and reference trajectories have in common.
None of this shows the paper's results are wrong; its experiments may have used code that is not in this commit, whose message reads redesigned training tasks. It shows what a buyer can and cannot verify from the release. Re-run from this commit, a trajectory that also opened a conversation its requester should never see would be judged on relevance and entity match, and the extra step falls outside what the judge is told to compare. MITRE's CWE-862 names the general weakness: a product that does not perform an authorization check when an actor attempts to access a resource or perform an action.
Testing the boundary with safe stand-ins for people and documents
You can test a permission boundary without exposing what it protects, as long as the stand-ins keep the structure that decides access. OpenFGA's consolidated suite uses invented users, such as user:aardvark and user:badger, and objects with bare identifiers, yet its denials still exercise group membership, inheritance and conditions, because those live in the tuples and the model rather than in the names.
Three rules carry the idea to a dataset built from business records. Replace people with stable pseudonyms that keep group memberships and reporting lines intact. Keep document identifiers, owners and sharing grants as relationship records rather than as prose inside the documents. And plant marker documents that are relevant to a test query but visible only to a named group, so any appearance of a marker in another principal's results is an unambiguous failure.
Time belongs in the representation too. OpenFGA's documentation on testing models shows a grant valid for one hour, written as a grant time and a duration, and a check that supplies a current time; the same pair of fields lets a dataset ask whether a grant had lapsed when a query ran. Its search guidance adds a practical warning: the authorization service stores no object metadata, so filtering and sorting happen in the application's own database, and a local index built from the change feed can lag a revocation unless each result is checked again. Pin the time of the relationship snapshot and the time of each query as separate fields.
The evidence-grounding test: six checks before trusting a score
Run six checks on any permission-aware retrieval dataset before trusting a score computed from it. Each passes or fails on the dataset itself, before any system is evaluated, and each corresponds to one of the counts above.
At a glance
The evidence-grounding test: six checks before trusting a score
- Requester resolves: every item names a principal that exists in the identity snapshot, with groups and attributes as of the query time.
- Model pinned: every item cites the authorization model or access-list export it was labeled against, by version or content hash.
- State dated: the relationship snapshot carries a timestamp, and each query time is recorded as a separate field.
- Denials present and hard: the share of items expecting a denial is reported, and most denials concern documents that answer the query and that some other principal may open.
- Empty answers expected: some list-style items expect no permitted results at all, so a system that always returns something fails them.
- Decision before relevance: the grader fails any item with a forbidden document in the candidates or the answer, whatever its relevance, and reports errors apart from denials.
When the relevant text is stale or insufficient rather than forbidden
The strongest alternative explanation for a bad result is that the evidence was outdated or incomplete rather than off-limits, and a dataset has to label the three apart or every failure looks like a permission bug. A document can be forbidden to this requester, superseded by a later version, or permitted but too thin to answer; each calls for its own expected behavior.
Label forbidden items with the principal who may see the document, superseded items with the version in force on the query date, and insufficient items with what is missing. A system that withholds a forbidden memo and says it cannot answer then passes; a system that answers from a superseded but visible draft fails for authority, not access; and a system that guesses from partial visible evidence fails for grounding. Merging the three inflates permission failure counts and hides the version and answerability problems that have their own test designs.
What these counts do not show
They describe two repositories at two commits and nothing more. OpenFGA's suite tests an authorization engine's decisions, not a retriever's ranking, so it shows what a well-formed denial record looks like rather than how any search system behaves. EnterpriseBench's data are synthetic, and a later commit could add the missing rules and labels.
We ran no model and measured no leakage rate. The phrase patterns that found no expected denials could miss a denial worded some other way, so the methods note lists every pattern for a reviewer to widen. Nothing here is a statement about the security of any product, SourceX's included: an evaluation dataset can show that a system enforced the access records it was given, not that those records were right.
Methods note: two commits and the rules the script applied
Counts were produced on 2026-10-28 by a script that reads files from openfga/openfga at commit cbc9bd4ce4be7b70a503d468c1d0abb647301e6e (assets/tests/consolidated_1_1_tests.yaml and abac_tests.yaml) and from ast-fri/EnterpriseBench at commit 9e41e5c1159c7ffd8359e943b1e4d296d3e91b4c (the task file, the tool file, the evaluation script, the employee and repository tables and the repository's file list). Python files are parsed as syntax trees and nothing from either repository is executed. OpenFGA's documentation was read from openfga/openfga.dev at commit b7fd424.
Rules as applied: a check assertion is error when it carries an error code, and otherwise allow when its expectation is true and deny when false; a deny is hard when its object appears as a tuple object in the same or an earlier stage of the same test, or among its contextual tuples; a list-objects assertion without an error code is empty when its expectation is null or an empty list. An expected denial in EnterpriseBench is any whole-word match in a task's ground truth for access denied, denied, deny, refuse, refused, refusal, unauthorized, not authorized (both with s or z), not permitted, not allowed, unanswerable, lack or lacks permission, or lack or lacks the necessary permission. A tool docstring promises a check when it contains access, accessible, permission, permissions, authorized, authorization, allowed, access rights or access controls as whole words; a function has a denial path when its body, docstring excluded, holds a raise statement or a string containing denied, not authorized, unauthorized, permission, forbidden or access. Rule files are paths whose lowercased text contains access, acl, permission or rbac, excluding the policy documents folder; none matched. Percentages come from exact fractions, rounded once. Saved outputs hold counts and function names only, with no employee records.
Sourcing business records that keep their access history
A dataset that passes the six checks needs records whose permissions travelled with them: access-list exports, group memberships with dates, sharing grants and version histories, alongside the documents themselves. Most exports drop those fields, and rebuilding them afterwards is guesswork.
SourceX licenses business data directly from companies and, for an AI developer, sources datasets against a specific request rather than holding them in stock. A request sets out scope, volume, language, format, timing and licensing, and it can name the permission and history fields that must survive preparation. No record changes hands until the company licensing it has agreed the scope and terms in writing, and de-identification runs through approved processing routes, so the request should also say how names inside access records are to be replaced, for example with stable pseudonyms that keep group structure testable. Nothing here is a security claim about any system; what a delivered dataset proves is set by the acceptance test the buyer writes into the request.
Frequently asked questions
What is a permission-aware retrieval dataset?
It is a retrieval test set in which every item records who is asking, which access model and relationship state apply at that moment, and which results that person may and may not see. With those records a grader can fail a result that is relevant but forbidden. Without them, relevance is the only thing measured.
Can a public benchmark test RAG access control?
Only if its released files carry the requester, the access rules with a version, the state at query time and expected denials, and its grader uses them. Check the files, not the paper: at commit 9e41e5c, EnterpriseBench's paper describes access control but its release holds no expected denials and its judge scores relevance and entity match.
How many denial cases should a permission test set contain?
There is no fixed number, but report the share and its quality. In OpenFGA's conformance tests at commit cbc9bd4, 191 of 473 yes-or-no check assertions expect a denial, and 179 of those ask about objects already in the test's stored relationships. A set with few denials, or denials on documents nobody can see, tests little.
Why use synthetic identities in permission tests?
Because the decision depends on structure, not names. Stable pseudonyms that keep group membership, reporting lines and sharing grants exercise the same boundary as real names, and they keep personal data out of the test set. OpenFGA's own suite uses invented users throughout.
Does passing a permission-aware retrieval test prove a system is secure?
No. It shows the system enforced the access records in the test set at the snapshot time. It cannot show those records were correct, that live permissions match the snapshot, or that generated answers do not leak restricted facts by inference.
Sources
- OpenFGA's consolidated conformance tests pair each stage's model and relationship tuples with check and list-objects assertions that expect allow, deny, an error code or an empty list. Source
- OpenFGA's attribute-based tests add request context, such as a timestamp, to check assertions for conditional grants, including a view allowed at 09:00 and denied at 11:00. Source
- OpenFGA's testing documentation shows a grant valid for a set duration and a check that supplies the current time as context. Source
- OpenFGA's search guidance says the service stores no object metadata, so filtering and sorting run in the application database, and an index built from the changes endpoint should be re-checked for revoked permissions. Source
- EnterpriseBench's task file at commit 9e41e5c holds 451 tasks, each naming the requesting employee, with ground truth and subtasks but no expected denials. Source
- EnterpriseBench's tool file defines 76 functions; 23 docstrings promise access checks, and none of those functions contains a denial path. Source
- EnterpriseBench's judge rubric scores trajectories from 1 to 5 on relevance and entity match and evaluates only what generated and reference trajectories share. Source
- The EnterpriseBench paper describes role-based access by levels 9 through 14, JSON access rules per data source, an access control layer on retrieval and an access-control ablation. Source
- MITRE's CWE-862 defines missing authorization as a product not performing an authorization check when an actor attempts to access a resource or perform an action. Source
- SourceX's request page says it licenses business data directly from companies, asks buyers for scope, volume, language, format, timing and licensing, and sources datasets on request. Source
- SourceX's how-it-works page says the company approves the scope and terms and signs before any data is delivered, and that data is de-identified through approved processing routes. Source
Tell us what data you need
Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.