Skip to content

AI uses for records

Enterprise agent constraint benchmark: testing rules that interact

By SourceX Editorial · Updated

10 sections

Short answer

A benchmark can test interacting business constraints by building tasks from closed cases where the right action depends on several policy rules at once, then grading the final system state, each rule separately, and the interaction gap: how much more often an agent fails when rules combine than its single-rule accuracy predicts. This page proposes the expert-authored constraint suite and protocol; it reports no results.

Agent run

Enterprise agent constraint

This page covers enterprise agent constraint benchmark: testing rules that interact.Illustrative

Key takeaways

  • Interacting constraints are rules that change whether or how other rules apply, such as an override, a date window or an authority limit.
  • Grade the end state, each constraint, and the interaction gap between combined and single-rule performance.
  • Pair every multi-rule task with single-rule siblings so the interaction gap can be estimated.
  • Hold out companies, task families and composition types separately and report the gap on each.
  • Report forbidden irreversible actions as their own rate, next to unnecessary escalations.

How could a benchmark test interacting business constraints?

Build tasks from closed business cases in which the correct action depends on several rules at once, such as a refund that turns on membership tier, purchase date, payment method and a prior exception, and grade the agent on the final system state, on each rule separately, and on whether it fails more often when rules combine than single-rule accuracy predicts. That last measure, the interaction gap, is what distinguishes this benchmark from a list of instructions.

Domain experts author the suite from real policies and records, a second expert solves every item independently, and each multi-rule task comes with single-rule sibling tasks so the gap can be computed. Companies, task families and composition types are held out separately so a result can be checked for transfer.

This page proposes the suite and its analysis plan. It is a protocol: no results are reported, and example counts are illustrative.

Why business rules interact instead of simply adding up

Business rules interact when one rule changes whether, or how, another applies. A precedence rule overrides a default, an eligibility rule depends on a date window, a cap applies across several records, and an approval limit decides who may act at all. An agent that checks each rule in isolation can satisfy every one of them on paper and still take the wrong action.

A public example shows the shape. The tau-bench airline policy sets the free checked bag allowance by both the booking user's membership tier and each passenger's cabin, so the right answer depends on two facts read together. Its cancellation rule allows any reservation to be cancelled within 24 hours of booking or when the airline cancelled the flight; otherwise economy and basic economy can be cancelled only with travel insurance and a qualifying reason, while business can always be cancelled. The policy also warns that the API does not check these rules for the agent.

That warning is the enterprise reality. In a ticketing system, an ERP or a billing platform, the API will usually accept an action the policy forbids. The benchmark therefore has to grade the decision against the policy, not just whether the tool call succeeded.

What instruction-following benchmarks already measure, and where they stop

Instruction-following benchmarks have moved from single checkable instructions toward composed ones, but they score text responses, not actions on business records. IFEval uses instructions whose compliance a program can verify, and its evaluation code reports both a strict and a loose check of each response.

FollowBench adds one constraint at a time to an initial instruction, from one to five constraints, to find the level at which a model stops complying. ComplexBench goes further and names composition types, And, Chain, Selection and Nested, and scores an answer by aggregating per-constraint checks through the dependency structure among them. InFoBench decomposes each instruction into simpler criteria and reports the Decomposed Requirements Following Ratio.

Three things are missing for enterprise work. The constraints in those suites are mostly about the form or content of a response, not about eligibility, authority and timing in business records. The tasks are not grounded in a system state the agent must read and change. And none of them measures how much worse a model does when constraints interact than its per-constraint accuracy would predict, which is the failure a business actually sees.

The proposed constraint suite: eight constraint classes

The suite organizes constraints into eight classes that recur across finance operations, customer service, procurement, field service and HR administration. Each item combines two to five constraints from at least two classes, and the classes double as the dimensions for reporting results.

The proposed constraint suite: eight constraint classes
Constraint classIllustrative business exampleHow it is verifiedTypical interaction
Eligibility thresholdReturn allowed only for orders under a size limitCompare record fields with the thresholdChanged by tier or contract exceptions
Temporal windowCredit allowed within a set number of days of invoiceDate arithmetic on record timestampsWindow extended by a prior approved exception
Precedence and overrideContract terms override the standard policyIdentify the governing document for the accountDecides which threshold applies at all
Aggregate capNo more than a set number of credits per account per quarterCount prior qualifying recordsDepends on which prior records count
Authority limitAgent may approve up to a set amount; above it, a manager must approveCompare action size with the role's limitForces escalation even when eligible
Cross-record consistencyShipping address must match the account of recordJoin two records and compare fieldsBlocks an otherwise valid action
Exception and carve-outRegulated products are never refundable to store creditCheck the product category flagOverrides tier benefits
SequencingHold must be released before a payment is reissuedInspect the order of state changesWrong order makes a correct end state invalid

One task item, field by field

Every task is stored as one item with the policy, the state, the constraints and the gold outcome kept separate, so graders can check each piece and items can be retired without breaking others.

One task item, field by field
FieldContentWho writes itCheck before inclusion
Policy excerptThe rules that apply, quoted from the source policy or contractAuthoring expertA second expert confirms nothing relevant is omitted
State snapshotRecords the agent can read, frozen at the decision timeAuthoring expert from the closed caseRights record allows benchmark use
Constraint listEach applicable constraint with an identifier and classAuthoring expertEvery constraint is needed for the gold outcome
Interaction graphWhich constraints enable, override or depend on othersAuthoring expertMatches one of the registered composition types
Gold end stateThe exact records that should change and howTwo experts independentlyBoth solutions agree, or the item is revised or dropped
Per-constraint goldWhether and how each constraint is satisfied in the gold pathTwo expertsEach constraint can be checked by rule or rubric
Single-rule siblingsVariants of the task where only one constraint is activeAuthoring expertEach sibling has its own verified gold state
Holdout tagsCompany, task family, composition type, release batchBenchmark leadTags assigned before any model run

Grading the end state, each constraint and the interaction gap

Grade three things and report them separately. Task success is all-or-nothing: the final system state matches the gold end state and no forbidden action occurred. Constraint satisfaction is the share of an item's constraints the agent's path satisfies, checked by rule where possible and by rubric otherwise, in the spirit of decomposed scoring.

The interaction gap is the new measure. For each multi-constraint item, multiply the agent's success rates on the item's single-rule siblings to get the success expected if the rules were independent, then subtract the observed success on the full item. A gap near zero means the agent handles rules as well together as apart; a large gap means it breaks on combinations. Report the gap by composition type, because a model can handle conjunctions well and still fail on overrides.

A worked example, illustrative only, shows how the pieces fit. A customer asks for a store credit on a returned order. Three constraints apply: the return must fall within a set number of days of delivery, the account's enterprise contract extends that window, and the account has already used its quarterly credit allowance. The gold path is to approve the return under the contract window but route the credit for manager approval because the cap is reached. An agent that reads the standard policy alone rejects the return; one that reads the contract but not the credit history issues a credit it was not allowed to issue.

Suppose, illustratively, that an agent succeeds on 9 of 10 trials of each single-rule sibling but on 5 of 10 trials of the full item. Independence would predict about 73 percent success, so the observed 50 percent gives an interaction gap of roughly 23 percentage points on that item. One item proves nothing; the gap becomes informative only when averaged across many items of the same composition type, with an interval.

Some failures cost more than others. Taking a forbidden irreversible action, such as issuing a payment above the agent's authority, should be reported as its own rate rather than averaged into task success. Escalating a case that the agent could have resolved is a lesser error and is reported next to it, so a model cannot look safe by escalating everything.

Would a constraint result transfer to companies and task families nobody tuned on?

A result is worth publishing only if it holds on companies, task families and composition types the developers never saw. Hold out entire companies, so policies and record conventions are new; entire task families, such as returns when development used billing disputes; and at least one composition type, such as nested overrides.

Holdouts also need a fair construction rule. If the development split's policies were written by the same experts who wrote the held-out items, phrasing habits can leak across splits. Assign different authoring experts to held-out companies where possible, record who authored each item, and test whether performance differs by author as a diagnostic.

Report the interaction gap separately for each holdout. A model whose gap shrinks on the development split but not on held-out companies has probably learned one company's conventions rather than how to combine rules. Keep the held-out items private, release them only in batches after a fixed date, and record which model versions were evaluated before each release, so contamination can be bounded.

Baselines, trial counts and clustered intervals for the constraint suite

Three baselines make the numbers interpretable. A rule engine that encodes the formalizable constraints gives an upper reference for items that can be written as code. A per-rule checker that verifies each constraint separately and then acts shows what ignoring interactions costs. Expert humans working the same items under the same tools give accuracy and time to compare against.

Report each model by exact name, version and evaluation date. Items share companies and task families, so standard errors should be clustered at the task family or company level, and model comparisons should use paired differences on the same items, as recommended in the statistical literature on evaluations. Run several trials per item and report the spread, because an agent that succeeds once in several attempts is not reliable on a business process.

The sample size follows from the smallest interaction gap difference worth reporting and the clustering. As an illustration of the reporting structure only: an illustrative suite of 40 task families across 6 companies, with 2 companies held out, would report every metric for the development and holdout splits with 95 percent intervals. Those counts are not a target; the power analysis sets them.

Before the first model run: registration and known blind spots

Register the following before any model is run, publish them with a version and date, and treat later changes as amendments with reasons.

At a glance

Before the first model run: registration and known blind spots

  • The constraint taxonomy, composition types and the rule for which items count as interacting.
  • The item schema and the agreement rule for two-expert gold states.
  • Holdout assignments by company, task family and composition type.
  • The primary metric, task success on held-out companies, and every secondary metric.
  • The interaction gap formula and how sibling success rates are estimated.
  • Baselines, model versions and the trial count per item.
  • Clustering level, interval method and the smallest effect of interest.
  • Release conditions for private items and the contamination log.
  • Known blind spot: rules that require judgment, such as good faith or reasonable effort, cannot be graded by rule and need a rubric with reported agreement.
  • Known blind spot: frozen snapshots remove the back-and-forth of real cases, so results say nothing about an agent's ability to gather missing facts from people.

Where the closed cases for a constraint suite come from

The suite is only as good as its cases, and those come from closed business records: tickets, orders, invoices, approvals and the policies in force when the decisions were made. Each case needs a rights record that allows benchmark use, de-identification that preserves the fields the constraints depend on, and a policy version matched to the case date.

A benchmark team that needs rights-cleared closed cases can ask SourceX to find them: it keeps no inventory and instead locates records at U.S. businesses once a buyer has described the need. The team can describe the record types, volume, format, timing and license terms it needs, including whether items may be published or must stay private, through the request form.

Frequently asked questions

What is an enterprise agent constraint benchmark?

It is a test set of business tasks in which the correct action depends on several policy rules at once, graded on the final system state and on each rule. The design here also measures the interaction gap, the extra failure that appears when rules combine.

How is this different from an instruction-following benchmark?

Instruction-following benchmarks mostly check the form and content of a text answer. A constraint benchmark for agents checks actions on business records against policies about eligibility, timing, authority and precedence, where the system will often accept a forbidden action.

Why do single-rule sibling tasks matter?

They let you estimate how the agent would perform if the rules were independent. Comparing that estimate with the agent's success on the combined task isolates failures caused by the interaction itself.

Should an agent that escalates too often be penalized?

Yes, though less than an agent that takes a forbidden action. Reporting unnecessary escalations next to forbidden actions stops a model from looking safe simply by handing every case to a person.

Has this constraint suite been run on any models?

No. This page sets out the proposed suite, grading and analysis plan only. Any counts in examples are illustrative, and results would require rights-cleared cases, two-expert gold states and a registered analysis.

Sources

  • The tau-bench airline policy sets free checked bags by membership tier and cabin, limits cancellations by booking time, cause, cabin and insurance, and states that the API does not check these rules for the agent. Source
  • IFEval's evaluation code tests instruction following with both a strict and a loose check of each response. Source
  • FollowBench incrementally adds a single constraint to an initial instruction at each level, from one to five constraints. Source
  • ComplexBench defines And, Chain, Selection and Nested composition types and aggregates constraint checks through the dependency structure. Source
  • InFoBench decomposes each instruction into simpler criteria and scores models with the Decomposed Requirements Following Ratio. Source
  • Adding Error Bars to Evals recommends clustered standard errors for related questions and paired, question-level differences when comparing models. Source

Tell us what data you need

Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.

See if your company qualifies