Skip to content

Systems and records

IT permission revocation dataset: did offboarding remove access in every system?

By SourceX Editorial · Updated

10 sections

Short answer

To verify that offboarding removed access across all systems, check the end state of every access path the person held, not the offboarding ticket: the directory account, sessions and tokens, accounts local to each app, API keys and SSH keys, cloud role sessions, shared credentials and copies such as repository forks. Record when each path closed and what proves it. Those records also form an IT permission revocation dataset for testing agents.

Evaluation split

IT permission revocation

This page covers IT permission revocation dataset: did offboarding remove access in every system.Illustrative

Key takeaways

  • A closed offboarding ticket records intent; only per-system evidence shows that access ended.
  • Disabling the identity provider account does not end application sessions, issued tokens or app-local accounts on its own.
  • Each access path needs its own check, its own evidence and a time after which the check is meaningful.
  • A revocation propagation evaluation scores coverage of access paths, the status given for each, timing and false all-clear answers.
  • Offboarding records must be stripped of secrets and host details before they are used for evaluation or licensed.

How do you verify that employee offboarding removed access across all systems?

You verify it path by path, against evidence from each system, after a waiting period long enough for existing sessions and tokens to lapse. Start from a list of every way the person could reach company data, check the state of each one, and record what proves it: an audit log entry, an admin console status, a failed sign-in, a key that no longer appears in the key list. The offboarding ticket tells you what someone intended to do; it does not tell you what the systems did.

The gap between the two is the subject of this page. Most offboarding misses are not a forgotten step in the identity provider; they are access paths that never ran through it, such as a local admin account in a vendor portal, a personal access token, a still-valid session in a browser or a fork of a private repository held by another organization.

The same records that answer the verification question also make a strong evaluation set for AI agents that claim to audit access. Below are the reasons access lingers, the baseline that sets expectations, a catalog of access paths, a field-level case specification, a grading rule, an illustrative case and the scrubbing rules that keep such a dataset safe to share.

Why does disabling the directory account leave access behind?

Disabling the account stops new sign-ins through the identity provider, but credentials already issued keep working until they expire or the issuing system revokes them. Microsoft's Entra documentation says access tokens it issues last one hour by default, notes there can be a period between starting revocation and access actually ending, and says Entra cannot directly revoke a session token that an application issued itself; the application has to do that under its own rules.

Cloud consoles have the same property. AWS documents for IAM Identity Center that a disabled user with an active access portal session can continue to assume new roles until their access is removed, and advises keeping a deny policy in place for at least 12 hours so active role sessions cannot regain their actions.

Provisioning protocols do not settle the question either. In the SCIM core schema, the active attribute is the user's administrative status, and the definitive meaning of that attribute is determined by the service provider. Sending active false to an application tells you the message was accepted; whether that application suspended the user, ended sessions or removed rights has to be checked in that application.

What does a control baseline require when someone leaves?

NIST SP 800-53 sets the expectation that access ends within a period the organization defines, and that the organization can prove it. Control PS-4, Personnel Termination, asks the organization to disable system access within a defined time, terminate or revoke the person's authenticators and credentials, retrieve security-related property, and retain access to information and systems the person controlled. Control AC-2, Account Management, asks that account managers be notified when users are terminated or transferred, that shared or group account authenticators be changed when someone leaves the group, and that account management be aligned with termination and transfer processes.

Two points from those controls shape the evaluation. The deadline is organization-defined, so each case record must carry the team's own target per system rather than a universal number. And revocation sits next to retention: PS-4 asks the organization to keep access to what the person controlled, so a correct offboarding often suspends and transfers rather than deletes, and an agent that recommends deleting a mailbox under legal hold has failed even if access ended.

Which access paths does the evaluation have to check?

Check every path in the catalog below that the person could have held, and treat a path the team cannot inspect as unverified, not as closed. GitHub's documentation is a good reminder of how far paths reach: removed organization members may keep local copies, people can keep access to forks that other organizations control, and enterprise members who still belong to another organization keep access to internal repositories.

Build the list of paths from the systems, not from memory. Single sign-on application assignments, the software spend ledger, the expense system and vendor contracts each reveal tools that the offboarding runbook may not name, and a case record should note which of those sources were checked.

Which access paths does the evaluation have to check?
Access pathExampleHow it usually closesEvidence that it closed
Directory or identity provider accountEntra ID, Okta or Google Workspace userDisable or suspend, then remove group membershipsAdmin status and an audit log entry with time and actor
Issued sessions and refresh tokensBrowser sessions in a CRM, mobile app refresh tokensSession revocation in the identity provider and in each appRevocation event, then a failed request after token lifetime
App-local accounts outside single sign-onVendor portal admin, accounting system userDeactivate in the app itselfApp user list export showing inactive status
API tokens and personal access tokensHelp desk API key, source control tokenRevoke or rotate in the issuing systemToken absent from the key list or marked revoked
SSH keys and certificatesServer authorized keys, deploy keysRemove keys, revoke certificatesConfiguration diff or certificate revocation record
Cloud role sessionsConsole or command line role sessionsRemove assignments, end sessions, apply a deny policyAssignment removal plus denied calls during the session window
Shared and group credentialsTeam vault entries, shared service loginsRotate every secret the person could readRotation timestamps later than the departure
Copies and delegated accessRepository forks, shared drives, mailbox delegationRemove collaborators and delegations; request deletion of copiesCollaborator lists and delegation settings after removal
Devices and physical tokensLaptop, hardware key, badgeRetrieve, wipe or deregisterAsset record and device management status

The revocation propagation evaluation specification

Each evaluation case is one departure, reconstructed from the records that existed at the time, with a question such as: at a given time after departure, which access paths remain open, and what proves it. The gold answer is a status per path with its evidence. The fields below make that answer checkable without exposing anything an attacker could use.

Freeze the system inventory as it stood on the departure date. An agent graded against today's inventory will be blamed for missing a system that did not exist then, or credited for one the company has since retired.

The revocation propagation evaluation specification
FieldWhat it holdsValidation checkWhy it matters
case_id, event_type, event_timePseudonymous ID, termination or transfer, departure timeEvent type from a closed listTransfers keep some access by design
system_inventoryEvery system in scope on that date, with how users sign inEach system tagged federated, provisioned or localLocal accounts are where misses hide
entitlements_beforeRoles, groups, tokens and keys held before departure, by typeCounts and types only; no secret valuesDefines what had to close
revocation_actionsEach action with system, actor role, time and methodTimes in one time zone; actor is a role, not a nameLets graders check order and timing
propagation_windowsToken and session lifetimes per system as configured thenSource of each value recordedA check before the window ends proves little
verification_evidenceLog entries, status exports, failed sign-ins per pathEvery path has evidence or is marked unverifiedThe gold answer cites these
residual_accessPaths still open at the question time, and when they closedEach residual path has a closing time or remains openThe core of the label
retention_constraintsLegal hold, mailbox retention, data transfer ownerPresent for every case, even if noneStops agents from recommending deletion
runbook_versionThe offboarding procedure in force on the dateVersion and date recordedSeparates process gaps from execution gaps
label_provenanceTwo reviewers, adjudication note, reviewer roleDisagreements resolved and recordedMakes the gold answer auditable

What evidence shows the access decision beyond a closed ticket?

The decision is explained by the chronology across systems, not by the ticket's final state. A closed ticket and a disabled directory account can sit next to an API token used two days later; only the system logs show that. Keep, for each case, the request that started offboarding and who approved it, each system's action log, the configuration that set token and session lifetimes, any exception approved in writing such as delayed removal for a handover, and the runbook version.

Exceptions belong in the record, not out of it. A departing engineer kept on a support rotation for a week, a contractor converted to an employee under a new identity, or a manager transferred to another unit are all cases where some access was meant to remain. An evaluation that labels every remaining path as a failure teaches agents to recommend blanket removal, which in a transfer breaks the person's new job.

How should an agent that audits offboarding be graded?

Grade coverage, correctness per path, timing and restraint separately, and treat a false all-clear as the most serious error. An agent that says access is fully removed when one token remains has done more harm than one that says it cannot verify three systems.

At a glance

How should an agent that audits offboarding be graded

Ask each case at two or more times, for example just after the revocation actions and after the longest session lifetime in scope. An agent that gives the same answer at both points is not reasoning about propagation, and the comparison shows it more clearly than any single score.

Report results by access path type as well as overall. Agents tend to do well on directory accounts, which are visible through one admin console, and poorly on app-local accounts and tokens, which require looking in each system. That split tells a team which part of its own offboarding an agent could safely help with and which part still needs a person.

  • Coverage: share of in-scope access paths the agent checked or explicitly marked unverifiable.
  • Per-path status: open, closed or unverified, compared with the gold label, with the evidence the agent cites.
  • Timing: whether the agent accounted for token and session lifetimes before calling a path closed.
  • False all-clear: any case where the agent reported complete removal while a gold residual path was open, reported on its own.
  • Restraint: no recommendation to delete data under a retention constraint, and no secret values in the agent's output.

Illustrative case: a departing analyst and a token nobody listed

This case is invented for illustration. An analyst leaves on a Friday. At 17:00 the identity provider account is disabled and group memberships removed; the ticket is closed at 17:20. The analyst had single sign-on access to a CRM and a business intelligence tool, a local account in a vendor billing portal created before single sign-on was set up, and a personal API token for the help desk used by a reporting script.

Asked at 17:30 whether access is fully removed, the gold answer is no. The CRM web session may still be valid until its own lifetime ends, the vendor portal account was never touched because it is not federated, and the help desk token is still active because tokens are issued by the help desk, not the identity provider. Asked again on Monday, the CRM session has lapsed and a failed sign-in is logged, the portal account was deactivated Saturday by the vendor admin, and the token remains active, so the gold answer lists one residual path with the help desk key list as evidence.

The exception: had the event been a transfer to the finance team rather than a departure, the billing portal account would be expected to stay, and an agent recommending its removal would be marked wrong.

Stripping a revocation history before anyone else sees it

Remove every secret value, hostname, internal IP address, account name and key fingerprint before an offboarding record leaves the security team, and keep the structure that makes it useful: the system type, the access path type, the times, the actions and whether each path was open or closed. Replace people with stable pseudonymous IDs and roles. Never include a residual access path that is still open today; close it first or drop the case.

Stop and bring in a security reviewer when a case involves a suspected insider incident, an open investigation, privileged infrastructure accounts or a system whose configuration would reveal how to get around controls. Those cases may still teach something, but the decision to use them belongs to the security owner, and many should never leave the company.

Offboarding histories as licensable records

Years of scrubbed offboarding and access review records are the kind of operational history AI developers use to test identity and IT agents, because the failures are real and the outcomes were checked. Having the records is not the same as having the right to license them: employee data, vendor terms and internal security policies all need review, and de-identifying a log does not by itself create a licensing right.

An IT team weighing that use can work with SourceX, which takes selected operational records through rights review, preparation and delivery to AI developers, and no access history leaves the company until it has approved the scope and terms and signed. On eligibility, SourceX works with U.S., Canadian and EU companies that had at least 50 full-time employees at their peak, not counting contractors; companies in the UK, Australia, New Zealand, India, Singapore, South Africa, Latin America, the UAE and Saudi Arabia can be considered when at least 70% of the offered data is in English, as confirmed on the actual dataset.

Frequently asked questions

Is disabling the user in the identity provider enough to end access?

Not by itself. It blocks new sign-ins through the identity provider, but sessions and tokens already issued can keep working until they expire or each application revokes them, and accounts that never used single sign-on are untouched. Check each system after the relevant session and token lifetimes have passed.

How long after departure should access be verified?

Check twice: once right after the revocation actions to confirm they ran, and again after the longest configured session or token lifetime in scope. NIST SP 800-53 leaves the deadline for disabling access to the organization, so set and record a target per system.

What is a revocation propagation evaluation?

It is a set of past departures, each reconstructed with the systems, entitlements, revocation actions, timing and evidence, used to test whether an agent can say which access paths remained open at a given time. It is graded on coverage, per-path status, timing and false all-clear answers.

Should transfers be treated the same as terminations?

No. A transfer keeps some access by design and changes the rest, and NIST treats personnel transfer as its own control. Label which access should remain for the new role so that an agent is not rewarded for removing everything.

Can offboarding logs be shared outside the company?

Only after secrets, hostnames and personal details are removed, every residual path is closed, and the security owner and counsel have approved the use. Even then, the right to license employee and vendor records has to be confirmed separately.

Sources

  • NIST SP 800-53 PS-4 and AC-2 set termination and account management expectations: disable access within a defined period, revoke credentials, retain access to the person's information, and align account management with termination and transfer. Source
  • Microsoft Entra access tokens last one hour by default, revocation can take effect after a delay, and Entra cannot directly revoke an application's own session token. Source
  • In AWS IAM Identity Center a disabled user with an active portal session can keep assuming roles until access is removed; AWS advises a deny policy kept for at least 12 hours. Source
  • Removed GitHub organization members may keep local copies and access to forks controlled by other organizations, and may keep internal repository access through another organization in the enterprise. Source
  • The SCIM active attribute is the user's administrative status, and its definitive meaning is set by the service provider. Source

See if your company qualifies

A short company assessment. No data uploads are needed.

See if your company qualifies