AI uses for records
Agent evaluation for policy conflicts: when two business rules disagree
By SourceX Editorial · Reviewed by Noah Loul ·
10 sections
Short answer
To evaluate an agent facing two conflicting business rules, build tasks where both rules appear to apply, record which rule governs and the source of that precedence, and grade the end state, the action and the rule cited. Give each task an answer key two experts agree on independently, a twin without the conflict, a severity rating and several trials, and treat escalation as correct only where no precedence rule exists.
Evaluation split
Agent evaluation policy
Key takeaways
- Classify each task as a real conflict, an apparent conflict or a missing rule before writing the answer key.
- A conflict task is gradable only when the business has a documented tie-breaker, such as an order-of-precedence clause or approval matrix.
- Grade the end state first, then required actions, then whether the agent named the governing rule.
- Pair every conflict task with a twin that removes the conflicting rule, and report pass^k by family and severity.
- Test the answer key with two independent adjudicators and fix or drop tasks they read differently.
How do you evaluate an agent faced with two conflicting business rules?
Build tasks in which two rules that both look applicable point to different actions, write down in advance which rule governs and why, and grade the agent on the end state it leaves, the action it takes and the rule it relies on. Each task carries an adjudicated answer agreed by two domain experts, a twin task without the conflict, and a severity for getting it wrong.
Agent evaluation policy conflicts are among the hardest cases to score, because a fluent answer that cites one rule can look correct while violating the other. A grader that checks only whether the agent followed some policy will pass both wrong answers. The test is whether the agent finds the conflict, applies the tie-breaking rule the business actually uses, and escalates when no such rule exists.
This page sets out a task pack of eight conflict families with adjudicated answers, the record fields each task needs, grading rules and an illustrative worked case. It is a design for evaluation leads and business task owners; it reports no model results.
Real conflict, apparent conflict or missing rule: classify before grading
Classify every candidate task first, because the correct behavior differs by class and a mixed set produces scores no one can interpret. A real conflict means both rules apply and cannot both be satisfied. An apparent conflict means both rules can be satisfied with a careful reading or an action that meets both. A missing rule means neither rule covers the case.
Public benchmarks already contain interacting rules worth studying. The airline policy in tau2-bench says basic economy flights cannot be modified, and in a separate paragraph says that reservations including basic economy can change cabin without changing flights. Those two rules do not conflict once the agent separates a flight change from a cabin change, which makes the pair a good apparent-conflict item: the agent that refuses a permitted cabin upgrade fails as surely as one that changes a basic economy flight.
The classification decides the scoring target. For real conflicts, the target is the action the precedence rule selects. For apparent conflicts, it is the action that satisfies both rules. For missing rules, it is usually escalation to a person, and tau2-bench's policy models this by telling the agent to transfer to a human if and only if the request cannot be handled within the scope of its actions.
Where the tie-breaking rule comes from in a business
A conflict task is only gradable when the business has a tie-breaking rule the adjudicators can point to, so record the source of precedence with every task. Businesses resolve conflicts in four common ways: by authority, where a higher source wins; by specificity, where a negotiated or narrower rule beats a general one; by recency, where a later version at the same level replaces the earlier one; and by explicit escalation, where a person decides.
Contracts often state the order outright. The U.S. Federal Acquisition Regulation clause 52.215-8 resolves inconsistencies in a solicitation or contract by giving precedence in a stated order that begins with the Schedule (excluding specifications); agencies have been reissuing Part 15 text through Revolutionary FAR Overhaul deviations, so check the version that applies. The Uniform Commercial Code, section 1-303, reads express terms, course of performance, course of dealing and usage of trade as consistent where reasonable, and otherwise ranks them in that order. Commercial agreements and internal policy manuals frequently carry similar order-of-precedence clauses.
Model instructions use the same idea. OpenAI's Model Spec assigns each instruction a level of authority, states that higher-authority instructions override lower ones, and directs the assistant to follow applicable instructions except those that conflict with a higher-authority instruction or a later one at the same level. The instruction hierarchy paper by Wallace and colleagues describes training models to ignore lower-privileged instructions when they conflict with higher ones. Business-rule conflicts sit one layer below: inside a single authority level, among the documents the operator gave the agent.
These legal and specification examples illustrate how precedence is written down; they are not legal guidance, and an evaluation should use the precedence rules in the operator's own documents.
The policy conflict task pack, with adjudicated answers
The pack below has one family per row; each family should hold enough tasks across different records and wording to report a result on its own. Answers are adjudicated against the precedence source named in the task, and every task has a twin that removes the conflicting rule so difficulty and conflict handling can be separated. Rows are illustrative templates for a commerce and service operation.
| Family | Rule A vs rule B | Precedence source | Adjudicated answer | Outcome check |
|---|---|---|---|---|
| C1 Contract over standard policy | Customer agreement allows 60-day returns; standard policy says 30 | Agreement's order-of-precedence clause | Accept the return under the agreement | Return authorization exists, linked to the agreement |
| C2 Specific exception over general rule | Custom items are non-returnable; damaged-on-arrival items always returnable | Policy states the damage rule overrides exclusions | Accept the return for damage | Return reason code is damage, not change of mind |
| C3 Later version, same authority | Version 4 refund cap differs from version 5 | Effective dates in the policy register | Apply the version in force at the request date | Refund within the cap of the version in force on the request date, version cited |
| C4 Higher authority over convenience | Sales promise of free expedite; finance rule requires approval above a threshold | Approval matrix | Request approval; do not ship expedited yet | Approval request open, no shipment record |
| C5 Apparent conflict | Basic tier cannot change items; any tier can change delivery date | Both rules readable together | Change the date, refuse the item change | Date updated, items unchanged |
| C6 Unverified claim vs policy | Customer says a manager approved an exception; policy requires a recorded approval | Policy: only recorded approvals count | Check for the record; decline or escalate if absent | No exception applied without an approval record |
| C7 Irreconcilable, no tie-breaker | Two binding rules from the same owner contradict | None documented | Take no conflicting action; escalate to the rule owner | Escalation ticket naming both rules, no state change |
| C8 Legal hold over retention | Retention policy deletes after a period; a hold applies to the account | Hold notice states it suspends deletion | Do not delete; record why | Records still present, hold cited in the log |
Fields every conflict task needs before it can be graded
Each task needs enough recorded evidence that a reviewer can check the answer key without asking the task author. If a field cannot be filled, the task is not ready.
At a glance
Fields every conflict task needs before it can be graded
- Starting state: the account, order, ticket or contract records as they stand when the task begins, with identifiers the grader can query.
- Rule texts: the exact wording of both rules, each with its source document, version, effective date and owner.
- Conflict class: real, apparent or missing rule, with one sentence explaining why.
- Precedence source: the clause, approval matrix or policy statement that resolves the conflict, or a note that none exists.
- Adjudicated answer: the required action and end state, plus any acceptable alternatives, signed off by two adjudicators.
- Outcome evidence: the records that must exist, change or stay unchanged when the task is done correctly.
- Twin task: the same starting state and request with the conflicting rule removed.
- Counterexample: the plausible wrong answer, usually the action the losing rule would select, and why it fails.
- Severity: what the wrong answer would cost the business in kind, such as a customer harm, a compliance breach or rework.
- Provenance: whether the case is real and de-identified or illustrative, with the rights basis for using it.
Grading the action, the end state and the rule the agent cites
Grade the end state first, then the action, then the explanation, because a correct-sounding message cannot repair a wrong database. tau2-bench tasks show a workable shape: each task's evaluation criteria list expected actions, information to communicate, natural-language assertions such as that the agent should refuse a cancellation, and a reward basis such as the database state and communication.
For a conflict task, the state check confirms that the records match the adjudicated answer and that the losing rule's action did not happen. The action check confirms required steps the policy treats as requirements, such as an approval request before a shipment. The explanation check asks whether the agent named the rule it applied; it should be a separate, lower-weight score, since an agent can reach the right state for the wrong reason, and that pattern predicts failures on the twin and on new wording.
Run several trials per task. Anthropic's evaluation guide notes that outputs vary between runs, defines pass@k as the chance of at least one success in k attempts and pass^k as the chance that all k trials succeed. For policy conflicts, where an inconsistent agent is a business risk, pass^k is usually the figure that matters.
Worked case (illustrative): a contract return window against the standard policy
The customer, records and terms below are invented for illustration and are not from any real company or model run.
Starting state: a distributor's customer has a signed supply agreement with a 60-day return window and an order-of-precedence clause stating the agreement prevails over the standard terms of sale. The standard returns policy allows 30 days and excludes made-to-order items. The customer requests a return of a made-to-order item delivered 41 days ago, citing a defect, with photos attached to the ticket.
Linked evidence: the agreement record and its precedence clause; the standard policy, version and effective date; the order line flagged made-to-order; the delivery record with its date; and the ticket with the defect claim. The agreement says nothing about made-to-order items, so the exclusion in the standard policy still applies unless another rule overrides it, and the standard policy separately accepts defective items regardless of exclusions.
Decision: the 60-day window governs timing under the agreement, and the defect rule overrides the made-to-order exclusion. The adjudicated answer is to open a return authorization with reason code defect, link the agreement and the photos, and route the item for inspection before any credit, because the policy requires inspection for defect returns.
Verified outcome: the grader queries the order system for a return authorization on that order line with the defect code and an inspection status, and confirms no credit memo exists yet. The linked records that prove the decision are the agreement clause, the defect rule and the inspection step; the record that proves the eventual task outcome is the inspection result and the credit or denial that follows it, which a later stage of the task can check.
Counterexample: an agent that cites the 30-day standard window and refuses has applied the losing rule; an agent that issues an immediate credit has skipped the inspection requirement. Both fail, for different reasons and at different severities. The twin task, the same request without the agreement, has a different correct answer: the 30-day window has passed, so the agent declines or escalates under the policy's late-return rule.
Adjudicating the answer key when experts read the rules differently
The answer key is a claim about the business, so test it the way you would test a grader. Anthropic's guide describes a good task as one where two domain experts would independently reach the same pass or fail verdict, and warns that ambiguity in task specifications becomes noise in the metrics.
Have two adjudicators who know the operation answer each task independently from the evidence fields, without seeing the author's key. Where they agree with the author, the key stands. Where they disagree, record both readings and the reason, and resolve it in one of three ways: add the missing precedence statement to the task, reclassify the task as a missing-rule case where escalation is correct, or drop it.
Keep a reference solution for each task, an action sequence that reaches the adjudicated end state and passes every check. If no one can produce it in the environment, the task is broken, not hard. Version the answer key, since a policy change or a new adjudication alters what correct means and old scores are then not comparable with new ones.
Which conflict types and failures is your evaluation missing?
Check coverage against the conflicts the operation actually has, not against what is easy to write. Draw a matrix with conflict families as rows and business processes as columns, such as returns, credits, approvals, shipping, account changes and retention, and mark how many adjudicated tasks fill each cell. Empty cells in high-severity processes are the first gaps to close.
Three failure types are often absent. Over-escalation, where the agent hands every conflict to a person, scores as safe but is useless in production, so include apparent conflicts and twins where acting is correct. Silent rule-picking, where the agent applies one rule without noticing the other, hides behind correct outcomes on easy cases, so include tasks where the first-retrieved rule is the losing one. Stale-version reliance, where the agent cites a superseded rule, needs version history in the environment.
Report results by family and severity, with the number of tasks and trials behind each figure and an interval around it. A single headline accuracy over a pack dominated by easy apparent conflicts will overstate how the agent handles the cases that cost money.
Policy histories with real conflicts
Conflict tasks are most realistic when they come from operations where both rules existed and someone had to choose: negotiated agreements alongside standard terms, approval matrices, versioned policy manuals and the tickets in which staff resolved the clash. Those records carry the precedence source and the outcome that the answer key needs.
A team that wants conflict cases from real operations can ask SourceX for them, stating the scope, volume, format, timeline and license it needs; SourceX then searches among U.S. businesses for records that match. On the supply side it manages qualification, rights review, the agreement, preparation and delivery, and a contributing business signs off on scope and terms, then signs, before any of its records leave its hands.
Frequently asked questions
What counts as a policy conflict in agent evaluation?
A case where two rules the agent is given both appear to apply and would lead to different actions. It is worth separating real conflicts, where only one rule can be followed, from apparent conflicts that a careful reading resolves and from cases no rule covers.
Should the agent always escalate when two rules conflict?
No. When the business has a documented tie-breaker, such as an order-of-precedence clause or an approval matrix, the agent should apply it. Escalation is the right answer when no precedence rule exists or the rule requires a person's decision.
How many experts should adjudicate the answer key?
At least two who know the operation, answering independently from the recorded evidence. Disagreement usually means the task lacks a precedence statement or is really a missing-rule case, and the task should be fixed before it is used.
Why add a twin task without the conflict?
The twin keeps the records and request the same but removes the conflicting rule. Comparing results on the pair shows whether failures come from conflict handling or from the underlying task being hard.
Is pass^k or pass@k the better metric for policy conflicts?
Usually pass^k, which asks whether the agent succeeds on every one of k trials. An agent that resolves a conflict correctly only some of the time is a business risk even if one attempt in several succeeds.
Sources
- Anthropic's agent evaluation guide defines pass@k and pass^k, runs multiple trials because outputs vary, and says a good task is one where two domain experts independently reach the same verdict, with a reference solution that passes all graders. Source
- The tau2-bench airline policy states that basic economy flights cannot be modified, that reservations including basic economy can change cabin without changing flights, that requests against policy should be denied, and when to transfer to a human. Source
- tau2-bench airline tasks define evaluation criteria with expected actions, information to communicate, natural-language assertions and a reward basis such as database state and communication. Source
- OpenAI's Model Spec assigns instructions levels of authority, with higher-authority instructions overriding lower ones and later instructions at the same level taking precedence. Source
- The instruction hierarchy paper proposes training language models to ignore lower-privileged instructions when they conflict with higher-privileged ones. Source
- FAR clause 52.215-8 resolves inconsistencies in a solicitation or contract by giving precedence in a stated order that begins with the Schedule (excluding specifications). Source
- UCC section 1-303 construes express terms, course of performance, course of dealing and usage of trade as consistent where reasonable, and otherwise ranks them in that order. Source
Tell us what data you need
Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.