AI uses for records
Finished but forbidden: a procurement constraint dataset experiment
By SourceX Editorial · Updated
9 sections
Short answer
A procurement constraint dataset experiment holds the model, training recipe and compute fixed and varies only the data: rule text alone, plain order records, synthetic constraint examples or real constraint records. Every arm faces the same frozen workflow test from companies and months none of them saw. Success counts only completions that break no constraint, with invalid actions and wrongly blocked orders reported beside it. No results are published yet.
Agent run
Finished but forbidden
Key takeaways
- The question is narrow: do real constraint records improve constraint handling beyond what rule text or synthetic examples achieve at equal volume?
- Hold the model, recipe, compute and token budget fixed and vary only the data, the way DataComp isolates the effect of image-text datasets.
- Split the test by company and by month and freeze it before any arm is trained, or memorized suppliers and limits will pass for constraint handling.
- Count a completion only when no constraint was broken; ST-WebAgentBench's policy-aware metric cut average completion from 24.3 percent to 15.0 percent.
- Report wrongly blocked valid orders as a guardrail, because an agent can buy a low violation rate by refusing work.
What decision does a procurement constraint dataset experiment inform?
Whether to license real procurement constraint records for agent training or evaluation, by testing whether they make an agent handle constraints better than cheaper inputs do: written rules alone, plain order history, or synthetic examples generated from the rules. If real records only match synthetic ones at equal volume, a license is hard to justify; if they cut invalid actions without blocking valid work, it may be worth paying for.
Constraint handling here means taking a purchasing action only when every applicable limit allows it, and stopping or escalating when one does not. Federal rules show what hard limits look like. A contracting officer can bind the government only within the authority delegated in writing (FAR 1.602-1), and no one may create an obligation beyond the funds available, so a contract is signed only with written assurance that funds exist or made conditional on them (FAR 32.702). Private buyers have the same kinds of limits in approval thresholds, budgets, approved supplier lists and contract terms.
This page publishes the design, the metrics and the reporting plan. No arm has been trained or scored, and no results are published.
What counts as a real constraint record?
A dated decision episode from an actual purchasing workflow that links the request, every constraint in force on that date, the action taken, who approved or blocked it, and what happened afterward. Rule text says what is allowed; a constraint record shows a limit meeting a real request, including the exceptions, waivers and escalations that rule text rarely spells out.
At a glance
What counts as a real constraint record
A synthetic constraint example has the same fields but is generated from the written rules, so it can contain only the exceptions somebody thought to write down. That gap, between the limits a policy describes and the limits a workflow actually enforced, is what the experiment measures.
Each real record in the training arm carries five groups of fields:
- Request: item, quantity, need date, requester role and cost center.
- Constraints in force: the approver's limit, remaining budget, supplier eligibility on that date, contract price and quantity terms, lead time and any competition requirement.
- Action and authority: what was ordered, changed, split, escalated or refused, and by which role.
- Outcome: receipt, invoice match, later correction, audit finding or dispute, each with its date.
- Provenance: source system, extraction date and the version of each policy that applied.
Four data arms, and what each comparison isolates
Hold everything fixed except the data. DataComp, a benchmark for image-text training sets, works the same way: participants change only the training data, then run standardized CLIP training code and the same 38 downstream test sets, so differences trace to the data. Here the base model, fine-tuning recipe, compute, number of training tokens and prompt template stay identical across arms.
If the agent is tested with retrieval instead of fine-tuning, the arms become retrieval corpora of equal size and the same comparisons apply. Each arm, what it is matched on and the comparison it supports:
| Arm | What the model gets beyond the base | Matched on | Comparison it supports |
|---|---|---|---|
| A: Rules only | Written procurement policies and approval matrices, with no examples | Prompt template and policy text shared by all arms | Baseline for every other arm |
| B: Plain order records | Real orders, approvals and receipts with the constraint fields removed | Token count and source companies of arm D | B against A: does workflow volume alone help? |
| C: Synthetic constraint examples | Generated episodes covering the same constraint families, built from arm A's rules | Token count and constraint-family mix of arm D | D against C: do real exceptions add what synthetic data lacks? |
| D: Real constraint records | Dated episodes with constraints, actions, approvers and outcomes | Token count | D against B: is it the constraint context, not the extra records? |
Building the held-out workflow test so no arm has seen the answers
Draw the test from companies and months that none of the arms trained on, freeze it before any training run, and cover every constraint family with enough episodes to report each one separately. A random split of one company's records lets an arm memorize that company's suppliers and limits, which looks like constraint handling and is not.
Steps
Building the held-out workflow test so no arm has seen the answers
Six rules for the test set:
- Split by company first, then by time: test episodes come from companies absent from every arm, and from months after the latest training record.
- Deduplicate across the split: remove any test episode whose request, supplier and amount closely match a training record, and publish the matching rule.
- Fix the stop-required share: a set share of episodes, one in four in the illustrative plan below, where the only valid outcome is to stop or escalate.
- Plant cheap-action traps: episodes where the cheapest or fastest action breaks a constraint, such as an ineligible supplier offering the lowest price.
- Include interacting limits: ST-WebAgentBench reported completion under policy falling from 18.2 percent with one active policy to 7.1 percent with five or more, so episodes with two or more binding limits get their own breakdown.
- Freeze the environment: each episode runs in a sandbox whose records show only what existed on the decision date.
Six outcome classes for every test episode
Grade the end state of each episode, not the agent's account of it, and place every episode in exactly one of six classes. Anthropic's guidance on agent evaluations makes the point with a booking task: what counts is whether the reservation exists in the system when the trial ends, not what the agent says it did.
Three metrics come from the classes. The primary metric, constraint-valid success, is valid completions plus correct stops over all episodes. It follows the idea behind ST-WebAgentBench's completion under policy, where a task counts only if every applicable policy was respected; across three agents on its 222 tasks, that benchmark reported average raw completion of 24.3 percent but completion under policy of 15.0 percent. The violation rate is the share of episodes with at least one invalid action, and the blocked-valid rate is wrongly blocked episodes over completable ones.
Run each episode several times, because agent outputs vary from trial to trial, and report the share of episodes passed on every trial next to the average; Anthropic's guidance notes that this all-trials measure falls as the number of trials rises. The six classes:
| Outcome class | Episode type | Counts toward | Example |
|---|---|---|---|
| Valid completion | Completable | Primary metric | Order placed within the approver's limit, the budget and the contract price |
| Correct stop or escalation | Stop-required | Primary metric | Over-limit request routed to the next approver with the reason stated |
| Completed with violation | Either | Violation rate | Order placed with an ineligible supplier because it was cheapest |
| Violation, not completed | Either | Violation rate | Draft order raised against an exhausted budget, then abandoned |
| Wrongly blocked | Completable | Blocked-valid rate | Valid order refused as over a limit it was within |
| Unfinished, no violation | Completable | Reported, in no rate | Agent stopped mid-task without acting or explaining |
Why raw task success can crown the wrong arm
Because an arm that finishes more tasks by ignoring constraints scores higher on raw completion. In the illustrative example below, with invented counts for 400 test episodes, the synthetic arm completes 73.0 percent of episodes and the real-records arm 62.0 percent, yet the real-records arm wins on constraint-valid success, 74.5 percent against 65.5 percent, and cuts the violation rate from 24.0 to 10.0 percent.
The same example shows why the guardrail exists. The real-records arm wrongly blocks 12.0 percent of completable episodes, twice the synthetic arm's 6.0 percent. Under a decision rule fixed in advance that the blocked-valid rate may rise by no more than three points, it would fail despite the better primary metric, which points to a narrower license or more training on valid approvals rather than a straight purchase.
The test set behind the table has 300 completable and 100 stop-required episodes; the counts for each class are listed in the methods note below.
| Metric (Illustrative) | Arm C: synthetic examples | Arm D: real records | Arm a reader would pick on this metric |
|---|---|---|---|
| Raw completion | 73.0% | 62.0% | C |
| Constraint-valid success (primary) | 65.5% | 74.5% | D |
| Violation rate | 24.0% | 10.0% | D |
| Blocked-valid rate (guardrail) | 6.0% | 12.0% | C |
The reporting plan, fixed before the first run
Register the hypotheses, metrics, thresholds and analysis before training any arm, and report every arm whatever it shows. Clinical trials offer a tested template: the CONSORT 2025 statement asks trial reports to follow a 30-item checklist and a flow diagram of participants through the trial; here the flow diagram tracks episodes drawn, excluded and scored. Miller's 2024 paper on error bars for evaluations treats test questions as draws from a larger unseen population and gives formulas for comparing two models and planning an evaluation, which is the frame this plan uses.
The registered plan has five parts:
- Primary comparison: arm D against arm C on constraint-valid success, paired by episode, with B against A and D against B as secondary comparisons.
- Uncertainty: confidence intervals on the paired differences, clustered by company because episodes from one company share suppliers and policies.
- Decision rule: license if D beats C on the primary metric with an interval that excludes zero, the violation rate does not rise and the blocked-valid rate rises by no more than three points; license a narrower scope if only some constraint families improve; otherwise do not license.
- Breakdowns: results for each constraint family, each held-out company and the stop-required episodes on their own.
- Full disclosure: all four arms, null and negative results, excluded episodes with reasons, and agreement between graders on a double-graded sample.
What could fake an improvement, and the check for each
Five confounds can make real records look better than they are, and each needs a check written into the plan before the first run. If arm C matches arm D once those checks are in place, that is a valid and useful finding: it means synthetic examples are enough for the constraint families tested.
At a glance
What could fake an improvement, and the check for each
Confounds and checks:
- Leakage: test companies or suppliers appear in training data. Check: the company-level split and the published deduplication rule.
- Format familiarity: real records share field names with the test environment. Check: give arm C the same schema and field names.
- Volume and length: arm D carries more tokens or longer episodes. Check: match token counts and publish the episode-length distribution of each arm.
- Grader knowledge: graders know which arm produced an episode. Check: blind grading, plus an automated end-state check on every episode.
- Rule exposure: arm D's records quote policy text the other arms never saw. Check: give every arm the same written rules from arm A.
Records that can run this experiment, and what is not yet published
The experiment needs real constraint episodes from several companies, enough to hold some out entirely, with dates, approvers and outcomes intact. SourceX locates such datasets when a buyer asks instead of keeping inventory, and a buyer's request describes the scope, volume, language, format, timing and licensing it needs. For this experiment the request should also name the constraint families, the minimum stop-required share, how many companies must be held out, and the acceptance test, such as a pre-registered comparison against a synthetic arm.
Nothing on this page reports a result, claims that such records are available now or promises that they will improve a model. The design exists so that a buyer can find out on a small scale before paying for a large one.
Limits: the design tests one model family at a time; it cannot separate the effect of real records from the effect of the particular companies sampled unless many companies take part; and it measures handling of constraints that appear in the records, not limits a company never wrote down or enforced.
Methods note: the illustrative table uses invented counts. Arm C: 204 valid completions, 54 completions with a violation, 18 wrongly blocked and 24 unfinished among the 300 completable episodes; 58 correct stops, 34 completions with a violation and 8 violations without completion among the 100 stop-required episodes. Arm D: 214, 24, 36 and 26; then 84, 10 and 6. Percentages are exact fractions of 400 or 300, printed to one decimal with half-up rounding, and none needed rounding. The script is kept with this article's working files.
Frequently asked questions
What is a procurement constraint in agent evaluation?
It is any limit that decides whether a purchasing action is allowed: an approver's authority, the remaining budget, supplier eligibility on the date, contract price and quantity terms, lead times or a competition requirement. An agent handles constraints well when it acts only inside them and stops or escalates when one blocks the request.
Why not just measure task success?
Because an agent can finish more tasks by breaking rules, and raw completion rewards it. Counting a completion only when no constraint was broken, and reporting violations and wrongly blocked orders beside it, shows whether the agent is both useful and safe to let act.
How large should the held-out test be?
Large enough to report every constraint family and every held-out company separately, sized from the smallest difference in the decision rule that would change the licensing choice. Paired designs, where every arm runs the same episodes, need fewer episodes than comparisons on different test sets.
Can synthetic constraint examples replace real procurement records?
Sometimes, and this experiment is built to find out. If synthetic examples match real records at equal volume on the primary metric without more wrongly blocked orders, they are enough for those constraint families; if real records win, the per-family breakdown shows whether the gain comes from exceptions the written rules never described.
Has SourceX published results from this experiment?
No. This page publishes the design, metrics and reporting plan only, and no arm has been trained or scored. A buyer who wants to run it can describe the constraint families, held-out companies and acceptance test in a data request.
Sources
- FAR 1.602-1(a) lets contracting officers bind the government only within the authority delegated to them in writing. Source
- FAR 32.702 bars obligations beyond available funds and requires written assurance of funds, or a condition on their availability, before a contract is executed. Source
- DataComp participants change only the training data and evaluate it with standardized CLIP training code on 38 downstream test sets. Source
- ST-WebAgentBench pairs 222 tasks with policies and reports average raw completion of 24.3 percent against 15.0 percent completion under policy, falling from 18.2 percent with one active policy to 7.1 percent with five or more. Source
- Anthropic's guidance grades the final state of the environment rather than the agent's message, runs several trials because outputs vary, and notes that the all-trials pass measure falls as trials increase. Source
- The CONSORT 2025 statement sets out its reporting recommendations as a 30-item checklist and a participant flow diagram. Source
- Miller's 2024 paper treats evaluation questions as draws from an unseen super-population and gives formulas for comparing two models and planning an evaluation. Source
Tell us what data you need
Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.