Skip to content

AI uses for records

Agent outdated policy failure case: how to test for expired rules

By SourceX Editorial · Updated

9 sections

Short answer

To test whether an agent follows an expired policy, build cases where both the superseded rule and the rule in force are reachable, fix the as-of date of every request, and grade the agent on the system state it leaves behind and the policy version it relied on, not on how reasonable its reply sounds. A failing case names the stale version, the step where it entered and the record that proves the wrong outcome.

Agent run

Agent outdated policy

This page covers agent outdated policy failure case: how to test for expired rules.Illustrative

Key takeaways

  • An expired-policy failure quotes a real rule accurately; the error is which version governed on the date that mattered.
  • Modification timestamps are not effective dates, and most policy stores do not record when a rule stops governing.
  • Grade the records the agent created or changed against an expected state written from the governing version, not the tone of its reply.
  • Every case needs a counterexample where the older rule is still correct, or an always-pick-newest shortcut passes.
  • Prove each case with linked records: version history, retrieval trace, audit log and the eventual outcome.

Why does an agent apply a policy that has already expired?

Because nothing it reads tells it the rule has lapsed. Most knowledge bases, file shares and tool responses carry a last-modified time, a title and a body, but no machine-readable statement of the dates a rule governs, so an older version that matches the customer's words looks just as authoritative as the current one.

The plumbing makes this easy to miss. The Model Context Protocol lets a server annotate a resource with lastModified, an ISO 8601 timestamp of the last change, and lets clients subscribe to update notifications. A modification time says when a file was edited, not whether its content still governs. Schema.org draws the distinction explicitly: expires marks the date content is no longer useful or available, while validFrom and validUntil mark when an item becomes valid and stops being valid. Few internal policy stores fill fields like these.

Old versions also stay findable on purpose. Salesforce documents that Knowledge keeps archived article versions, retains versions attached to cases beyond the default version limit, and flags a case's attached version as outdated once a newer version is published. That history is exactly what an auditor wants and exactly what an agent with broad search access can surface as if it were current. OWASP's 2025 guidance on vector and embedding weaknesses names a related failure: conflicting sources in a shared store, and a model that cannot set aside older knowledge in favor of newer retrieved data.

What counts as a dated authority failure, and how is it different from a wrong answer?

A dated authority failure is a case where the agent took or recommended an action that a real rule allowed on some date, but not on the date that mattered. The answer is internally consistent and often quotes the source accurately; the error is in which version governed.

That makes it different from a hallucination, where the rule never existed, and from a retrieval miss, where nothing relevant came back. It needs its own label because the fix is different: better ranking will not help if both versions are equally relevant, and better prompting will not help if the effective dates are missing from the data. The patterns below cover most of what teams see in service, finance and operations workflows.

What counts as a dated authority failure, and how is it different from a wrong answer?
Failure patternWhat the agent relied onWhy the output looks rightEvidence that exposes it
Superseded version retrievedAn archived or replaced policy document with a closer wording matchIt quotes a real company rule accuratelyVersion history showing the document was replaced before the request date
Right version, wrong as-of dateThe current rule applied to an event that happened under an older oneIt cites the newest policy, which reviewers expectOrder, contract or claim date that falls before the change took effect
Rule frozen in instructionsA limit or window pasted into the system prompt or tool description months earlierNo retrieval step to inspect, so nothing looks stalePrompt or configuration history compared with the policy change log
Carried-over memoryA rule stated earlier in the same thread or a previous sessionIt is consistent with the conversationTimestamp of the earlier statement against the policy's effective date
Lower authority winsAn FAQ, macro or email that was never updated after the policy changedAgents and customers both use the FAQ wordingAuthority ranking of sources and the date each was last reconciled
Rule applied too earlyAn announced change that had not yet taken effectIt reflects the newest published textEffective-from date later than the request date

What should the illustrative dated authority failure packet include?

It should let a second reviewer reach the same verdict without reading the transcript first. Every field below exists to answer one question the grader will ask: what the agent knew, which rule governed, what it changed, and how anyone can confirm the outcome. The example values are illustrative and invented for this page; they do not describe a real company or customer.

At a glance

What should the illustrative dated authority failure packet include

  • Counterexample: a paired case where the older version is the correct one, for example an order placed before the change under a rule that protects existing orders.
  • Linked evidence: identifiers for the policy history entry, the retrieval or tool trace, the audit log row and the outcome record, each with its own timestamp.
  • Grader notes: the single sentence that states which condition failed, written before anyone looks at the agent's reply.
What should the illustrative dated authority failure packet include?
FieldWhat to recordIllustrative valueWhy the grader needs it
Case labelReal-approved or illustrative, plus the packet version and dateIllustrative, packet v1, 2026-10-18Readers must know whether any of it happened
As-of timestampThe moment the request was made and the event it concernsRequest on day 41 after deliveryDefines which rule version governs
Starting stateAccount, order or case fields before the agent actedOrder delivered, no prior return, standard tierLets the grader diff the end state
Authority setEvery rule version the agent could reach, with effective-from and effective-toReturns policy v3 (60-day window, ended) and v4 (30-day window, in force)Shows the stale version was reachable
Controlling versionThe version that governed on the as-of date and the clause that appliesv4, clause 2.1The basis for the expected decision
Failure stepThe tool call, retrieval or message where the stale rule enteredSearch returned v3 first; agent quoted itSeparates retrieval fault from reasoning fault
Resulting stateRecords the agent created or changedReturn authorization issued, prepaid label sentWhat actually went wrong in the business
Expected stateRecords a correct handling would have producedNo authorization; exception request routed to a personThe target the grader compares against
SeverityReversible or not, customer-facing or not, and who must fix itReversible within a day, customer-facingWeights the failure in a score
Recovery checkThe record that confirms the error was caught or correctedAuthorization voided; customer notifiedProves the outcome, not just the intent

Illustrative worked case: a return approved under a lapsed window

This case is illustrative. It shows how a packet reads end to end; it is not a record from any company.

A retailer shortened its return window from 60 days to 30 days. Version 4 of the returns policy took effect on the first of the month and version 3 was archived the same day, but remained searchable for staff. A customer whose order was delivered after the change writes in on day 41 asking to return an unopened item.

  • Starting state: order delivered after the effective date of v4, no prior returns, no exception flag on the account.
  • Linked evidence: the knowledge base shows v3 archived and v4 published on the effective date; the retrieval trace shows the search for return window returned v3 above v4 because v3 had a longer, closer-matching passage.
  • Agent decision: it quoted the 60-day window from v3, confirmed the item qualified, asked the customer to confirm, then called the create-return tool and sent a label.
  • Final message: polite, accurate to v3, and marked the conversation resolved.
  • Verified outcome: the warehouse received the item; a staff reviewer later flagged the authorization as outside policy and logged a one-time exception so the customer was not penalized.
  • Grader verdict: fail. The governing condition was a 30-day window under v4; the request on day 41 should have produced a decline or a routed exception request, not an authorization.

Which date comparison failed behind the agent's plausible reply?

The failed condition is always a comparison between a date in the business record and a date range in the governing rule. In the worked case it is delivery date plus window, measured under the version in force on the request date. Stating that condition in one line before grading is what keeps reviewers from being persuaded by a fluent reply.

Grade the end state, not the message. The original tau-bench benchmark, which pairs a simulated customer with a tool-using agent bound by written domain policies, checks whether the database the agent leaves behind matches the state produced by the annotated correct actions, and only then checks required pieces of information in the agent's replies. A return label that should not exist fails the case no matter how the conversation ended. The same benchmark's retail policy also requires explicit user confirmation before consequential database updates, which is a useful reminder that a confirmation step does not make an action authorized.

Repeat each case. The tau-bench leaderboard reports Pass^1 through Pass^4 for each agent configuration, measuring consistency across repeated attempts rather than a single lucky run. Expired-policy failures are often intermittent because retrieval ranking shifts with wording, so a single pass hides the problem.

No single record proves it. You need one record for each link in the chain: the rule that governed, what the agent could see, what it did, and what happened afterward. If any link is reconstructed from memory or from the agent's own summary, the case is unverified and should be marked that way.

What record proves each link from the expired rule to the final outcome?
What must be provenRecord that proves itWhere it usually livesWeak substitute to reject
Which version governed on the as-of datePolicy version history with publish, archive and effective datesKnowledge base or document management version logThe file's last-modified date
Which versions the agent could reachRetrieval or tool trace listing document and version identifiersAgent run log or search service logThe citation in the final message
What the agent changedAudit log row or API response for each writeOrder, billing, CRM or helpdesk audit trailConversation status set to resolved
When the triggering event happenedDelivery, contract, claim or invoice timestampFulfillment, ERP or policy administration systemDate the customer stated in chat
What finally happenedLater reversal, refund, exception, complaint or reopenFinance ledger, case history, exception registerAbsence of a complaint
Who approved any exceptionApproval record naming the role and reasonApproval workflow or case notes with authorA note saying exception granted with no author

How do you build a set of expired-policy cases from real records?

Start from the policy change log, not from the conversations. Every rule change creates a boundary, and the cases worth testing sit within days of it: requests that straddle an effective date, events that happened under one version and were handled under the next, and announcements that preceded the change.

The NIST AI Risk Management Framework says accuracy measurements should always be paired with clearly defined and realistic test sets. For dated authority, realistic means the set reflects how often your rules actually change and how long old versions stay reachable, not a sprinkling of synthetic traps.

  • List rule changes from the last two to three years with effective dates, the clause that changed and where each old version is still stored.
  • For each change, pull requests within a short band either side of the effective date and rebuild the account state as of the request time from audit or field history.
  • Write the expected state from the governing version before running any agent, and have a second person confirm it.
  • Pair every expired-rule case with a counterexample where the older rule still governs, so an agent that always prefers the newest document does not pass by accident.
  • Run each case several times with light rewording of the customer message and record the version retrieved on each run.
  • Score severity separately from pass or fail, so an irreversible payment outside policy is not averaged away by many harmless wording slips.
  • Keep cases from recent changes out of any training or prompt examples, so the test measures behavior rather than recall of the test itself.

When is the older rule still the right one?

More often than test designers assume. Contracts, warranties, subscriptions and insurance policies commonly keep the terms in force when they were signed, so a customer who bought before a change may be governed by the old version for months or years after it was archived.

Scope matters too. Two versions can both be current for different regions, account tiers or product lines, and a change to one does not retire the other. An agent that reads effective dates but ignores scope will fail in a different direction, applying a tighter rule to customers it does not cover.

This is why the packet requires a counterexample. A test set made only of cases where newest is correct rewards a shortcut: always pick the most recent document. A balanced set makes the agent show that it read the as-of date, the scope and the grandfathering clause, which is the behavior you actually want in production.

Where can real dated policy cases come from?

Real cases need three things most public datasets lack: a policy history with effective dates, the operational records that show what happened, and the outcome that confirms whether handling was right. Many companies hold all three across a knowledge base, a helpdesk and a transaction system, but the content belongs to the business, the customer details in it are personal data, and using it for evaluation requires a license from the company, not just system access or de-identification.

Should the company decide to license that history, SourceX takes care of the rights review, contract, preparation and delivery. Evaluation teams do not buy such cases from stock: SourceX sources them on request from U.S. businesses, so a team can set out the policy domains, date range, volume and format it needs on the request form. Where real cases are not available, label every case illustrative and keep the packet structure the same, so real cases can replace them later without changing the grader.

Which dates must a temporal policy case keep apart, and which one governs?

Five dates and one rule: when the policy was announced, the window in which it governs, when the triggering event happened, when the request arrived and when the agent actually acted, plus the clause wording that says which of those dates counts. The packet's as-of timestamp folds the request and the event it concerns into one field; splitting them, and adding the governing-date key, tells a grader which date actually counts.

Rule writers already keep these apart. The Office of the Federal Register's tutorial lists effective dates and applicability and compliance dates as separate entries under a rule's DATES caption. Company policies are looser, and the wording does the work: orders placed on or after a date, deliveries after a date and requests received after a date give three different answers for the same customer.

Execution date is the one most test sets miss. Agents queue work, retry failed tool calls and run batch jobs, so a request received under version 3 can be carried out after version 4 takes effect. Whether version 3 still governs depends on the clause, and the case must record the answer before grading.

Which dates must a temporal policy case keep apart, and which one governs?
Date roleIllustrative fieldUsually recorded inFailure when it stands in for another date
Announced or publishedpolicy.published_atPolicy register, change noticeRule applied when announced, before it takes effect
Effective windowpolicy.effective_from and effective_toPolicy text and version historySuperseded version applied after the change
Triggering eventorder.placed_at or order.delivered_atERP, fulfillment or contract systemNew rule applied to an event it does not cover
Request receivedcase.created_atHelpdesk or workflow systemCustomer's stated date used instead of the record
Executionaction.executed_atAgent run log or job schedulerQueued or retried action judged under the wrong version
Governing-date keyclause.governing_date = placed_atClause wording, quotedAgent and grader use different dates and both look right

What should a temporal policy application dataset request specify?

A task-to-data requirements brief with seven lines, each paired with a check a supplier and a reviewer can both run. Shares and counts are editable assumptions for a first evaluation set, not tested thresholds. No validated policy-date case pack or measured results exist yet under this page's protocol.

The integrity line can be enforced instead of audited. PostgreSQL 18's CREATE TABLE documentation describes a WITHOUT OVERLAPS option that checks the last key column, such as a range of dates or timestamps, for overlaps instead of equality, which it notes is sometimes called a temporal key. A key on clause, scope and effective range then rejects a second version whose window overlaps the first.

Score consistency the way Anthropic's engineering post of January 9, 2026 defines it: pass^k is the probability that all k trials succeed. That is the right bar when one wrong-version refund in k runs is a production incident.

What should a temporal policy application dataset request specify?
Requirement lineSpecification (editable assumption)Acceptance check
Task and decisionOne policy domain, such as returns or price changes, and the end state the agent must produceExpected end state written from the governing version before any run
Evidence boundaryInputs limited to versions published on or before the execution date and records dated before itNo input document carries a date after the execution timestamp
Boundary strataSix strata: before the change; after it; event before, request after; request before, execution after; announced, not yet effective; two versions current for different scopesEach stratum holds at least 10 percent of cases
Governing-date labelThe date the clause keys on, quoted from the clauseTwo reviewers agree on the label; disagreements go to the policy owner
Version store integrityEffective windows per clause and scopeNo two versions of one clause in one scope overlap
Success testEnd-state match plus the version identifier the agent relied onPasses in all k reworded runs, reported as pass^k
ExclusionsClauses whose governing date is ambiguous in the text itselfLogged and sent to the policy owner, never labeled by guesswork

Does the failure need more examples, or a fix to retrieval, metadata, tools or process?

Run four cheap tests on the failing cases before requesting data. Only a failure that survives all four, with the right version and explicit dates in hand, points to a gap that task-specific examples could close.

Check the grader as well. The same Anthropic post notes that a 0 percent pass rate across many trials, such as 0 percent at pass@100, is most often a sign of a broken task. If no configuration ever passes the straddling strata, suspect a wrong governing-date label before an incapable agent.

When all four tests keep data in play, send the request with the brief above, the failing cases as the first evidence packet and the test results attached, so a supplier can see exactly which date reasoning is missing.

Does the failure need more examples, or a fix to retrieval, metadata, tools or process?
TestWhat changesResult that points away from dataResult that keeps data in play
Oracle versionSupply only the governing versionAgent passes: retrieval or ranking is the bottleneckAgent still fails with the right version
Explicit datesSupply both versions with effective windows, plus event, request and execution dates as structured fieldsAgent passes: add the dates to the store and to tool outputsAgent fails even with every date in front of it
Tool reachCheck that some tool returns the event and execution datesNo tool returns them: a tooling gap no record can fixThe dates are reachable
Human baselineTwo staff members label the same cases from the recordsThey often disagree: the business applied the rule inconsistently, so fix the process or policy firstThey agree

Sources for this section

  • The Office of the Federal Register's tutorial lists effective dates, applicability and compliance dates, and comment dates as entries under a rule's DATES caption. Source
  • PostgreSQL 18's CREATE TABLE documentation describes WITHOUT OVERLAPS, which checks the last key column for overlaps instead of equality and is sometimes called a temporal key when that column is a date or timestamp range. Source
  • Anthropic's engineering post (January 9, 2026) defines pass^k as the probability that all k trials succeed and says a 0 percent pass rate across many trials (0 percent pass@100) most often signals a broken task. Source

Frequently asked questions

How do you test whether an agent follows an expired policy?

Build cases where both an expired version and the version in force are reachable, fix the as-of date of each request, and write the expected end state from the governing version before running the agent. Grade on the records the agent created or changed and the version it relied on. Repeat each case several times, because stale retrieval is often intermittent.

Is an expired-policy failure the same as a hallucination?

No. In a hallucination the rule does not exist; in an expired-policy failure the agent quotes a real rule accurately, but one that did not govern on the date that mattered. The fix is usually effective-date metadata, version-aware retrieval and as-of reasoning, not a better prompt.

Why not just delete old policy versions?

Old versions are needed for audits, disputes and customers whose contracts still fall under earlier terms. Systems such as Salesforce Knowledge keep archived versions and versions attached to cases for that reason. The safer approach is to mark each version's effective window and scope, and limit what the agent can retrieve by default.

What makes a good counterexample in a dated authority test set?

A case where the older rule is correct, such as an order placed before a policy change that protects existing orders, or a customer in a region the change did not cover. Counterexamples stop an agent from passing by always choosing the newest document.

Can these test cases be built from real company records?

Yes, when the company licenses them for that purpose. You need the policy version history, the operational records and the outcome records, with customer details de-identified. System access or de-identification alone does not give the right to use records for AI evaluation; that comes from an agreement with the business.

Sources

  • The Model Context Protocol specification (2026-07-28) lets servers annotate resources with lastModified, an ISO 8601 timestamp of the last modification, and lets clients subscribe to resource update notifications through subscriptions/listen. Source
  • Schema.org defines expires as the date content expires and is no longer useful or available. Source
  • Schema.org defines validFrom as the date an item becomes valid, alongside validUntil for the date it is no longer valid. Source
  • Salesforce Knowledge keeps draft, published and archived article versions, stores versions attached to cases beyond the default version limit, and flags a case's attached version as outdated when a new version is published. Source
  • OWASP LLM08:2025 describes cross-context leaks and knowledge conflicts in shared vector stores, including a model failing to supersede old knowledge with newly retrieved data. Source
  • tau-bench scores a task by comparing the final database state with the state produced by the annotated ground-truth actions, then checking required outputs in the agent's replies. Source
  • The tau-bench retail agent policy requires listing action details and obtaining explicit user confirmation before consequential database updates. Source
  • tau-bench emulates conversations between a simulated user and a language agent with domain API tools and policy guidelines, and reports Pass^1 to Pass^4 on its leaderboard. Source
  • The NIST AI Risk Management Framework 1.0 states that accuracy measurements should always be paired with clearly defined and realistic test sets. Source

Tell us what data you need

Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.

See if your company qualifies