AI uses for records
Cross-system correction benchmark: how to test one fix across five systems
By SourceX Editorial · Updated
10 sections
Short answer
This is a protocol, not a result. To test whether an agent can carry one correction across five business systems, start from a real fix, such as a merged customer or a wrong tax ID, and grade the final state of every system. Score both required changes and the records that must stay untouched, such as posted invoices and closed tickets. Hold out whole companies and report intervals. No results are reported here.
Evaluation split
Cross system correction
Key takeaways
- Grade final state in all five systems, not the agent's narrative of what it did.
- A correct propagation edits some records, annotates others and leaves history alone; the suite has to test all three.
- Invariants such as balanced ledgers, untouched closed periods and resolvable references catch the damage a goal-state diff misses.
- Hold out whole companies and whole task families, and report Wilson intervals plus repeat-trial reliability.
- This page publishes the method only; no measurements have been collected.
How could a benchmark test one correction across five business systems?
Give an agent one confirmed correction and access to five systems of record, then compare the final state of all five against an expert-built target state and a set of invariants. The correction might be a customer merge in the CRM, a corrected tax ID or a wrong unit of measure. The agent passes only if every system that should change does, in the form that system allows, and nothing else moves.
That last condition is what makes the task hard and worth measuring. In real companies a correction does not mean the same edit everywhere. The CRM record is updated, the posted invoice is left alone and offset with a credit memo, the support ticket from last year keeps the old value as history, and the data warehouse needs a dated adjustment rather than an overwrite.
This page is a protocol. It describes the task suite, the invariant checks, the sampling and holdout design, the metrics and a pre-registration checklist. No agent has been run against it here, and no results, scores or rankings are reported.
What do AppWorld and tau-bench already check, and what do they leave out?
Existing agent benchmarks already grade final state rather than transcripts, and this design borrows that idea directly. AppWorld describes its evaluation as database-state-based unit tests that check whether the agent completed the task without causing collateral damage. Its README says evaluation is state-based, not process-based, so any route to the right end state counts.
tau-bench sets an agent with domain API tools and written policy guidelines against a user simulated by a language model, and its paper grades by comparing the database at the end of the conversation with an annotated goal state. Its README reports results as Pass^1 through Pass^4, a reliability view across repeated trials.
What neither was built to isolate is propagation: one upstream fact changes and the agent must find every dependent record across several independently owned systems, each with its own rules about what may be edited. The cases in this protocol come from business correction histories, where the right answer depends on posting status, period locks and retention rules that consumer apps rarely have.
The five-system world: what the correction should do in each system
Build the environment as five separate stores with their own identifiers, permissions and edit rules, seeded from de-identified records of a real company. The table shows the default roles; the systems named are examples of the category, not requirements, and the behavior column is the rule the target state encodes.
| System | Examples of the category | What a correct propagation does | What must stay unchanged |
|---|---|---|---|
| CRM | Salesforce, HubSpot | Updates the account or contact, or merges duplicates and records the surviving ID | Activity history, consent fields and owner assignments not tied to the correction |
| ERP and accounting | NetSuite, Dynamics 365, QuickBooks | Updates the customer master; offsets posted documents with credit memos or adjustments | Posted invoices, journal lines in closed periods, audit trail |
| Billing or subscription | Chargebee, Zuora, Stripe Billing | Changes future invoices and renewal terms only | Issued invoices and payment records |
| Support or ITSM | Zendesk, ServiceNow, Jira Service Management | Re-links open tickets to the surviving customer and adds an internal note | Closed ticket content and historical field values |
| Reporting store | A warehouse table fed nightly | Writes a dated correction row or flags affected rows for restatement | Prior snapshots used in signed-off reports |
The propagation task suite
Each task starts from one correction with a known source of truth and a target state for all five systems. Represent the case as linked objects and events, in the spirit of OCEL 2.0, where one event can relate to several objects with a qualifier describing each object's role and objects can relate to each other without an event. That makes it explicit which invoice belongs to which customer, which ticket mentions which order and which row came from which posting.
Every family below includes a trap: a record that looks like it should change but must not, or a dependency that is easy to miss. Traps matter because a naive agent that rewrites every match will pass a goal-state check on the records it was supposed to change and still fail the business.
| Task family | Trigger correction | Required downstream actions | Built-in trap |
|---|---|---|---|
| Duplicate customer merge | Two CRM accounts are the same company | Merge in CRM, re-point ERP customer, billing account and open tickets to the surviving ID | CRM merge issues a new ID that other systems still reference by the old one |
| Legal name or tax ID fix | Customer's registered name or tax ID was wrong | Update master records and future invoices; note the change on open documents | Issued invoices must keep the name they were issued under |
| Billing address change | Address was entered wrong at onboarding | Update CRM, ERP and billing; recompute tax only on unissued invoices | Shipping addresses on fulfilled orders are history, not errors |
| Wrong price on a posted invoice | One line was billed at the wrong rate | Issue a credit memo or adjustment in ERP and billing; note the ticket | Editing the posted invoice in place breaks the ledger |
| Unit of measure error | An item was sold by case but recorded by unit | Fix the item master and open orders; flag affected warehouse rows | Closed-period quantities must not be rewritten |
| Contact reassignment | A contact moved to a different customer account | Move the contact in CRM; re-route open tickets; keep past tickets with the old account | Past invoices and tickets belong to the account at the time |
| Correction of a correction | Yesterday's merge was wrong and must be reversed | Undo dependent changes in every system without losing edits made since | Restoring a snapshot wipes legitimate later changes |
| No-op decoy | A reported error that the source of truth shows is already correct | Confirm and change nothing | Any write counts as damage |
Invariant checks the grader runs on final state
Run invariants over all five stores after every attempt, in addition to the target-state diff. A diff tells you whether the expected fields changed; invariants tell you whether the agent broke something no one listed. Compensating transaction guidance from Microsoft makes the same point about distributed undo: work must be undone in every data store, and simply restoring an original state can overwrite changes made by others in the meantime.
Each invariant below is scored pass or fail per task, with the failing records logged so reviewers can tell a real violation from a grader bug.
- Reference integrity: every foreign key in every system resolves to an existing record after the run, including references to a merged-away ID.
- Single surviving identity: one customer maps to one surviving ID per system, and a crosswalk records the mapping from old IDs.
- Ledger balance: total debits equal total credits in the accounting store, and every adjustment links to the document it offsets.
- Closed periods untouched: no posting date, amount or quantity in a locked period changes.
- Issued documents immutable: issued invoices, credit memos and payment records keep their original content and numbers.
- History preserved: closed tickets and historical field values keep the value they had at the time, with any note added rather than edited in.
- Scope bound: no record outside the correction's dependency graph changes, counted from a full before-and-after diff.
- Idempotence: running the same correction twice leaves the same final state as running it once.
- Cross-system agreement: the corrected attribute matches the source of truth in every system where it should, and the reporting store carries a dated correction marker.
Population, sampling and holdout: keeping the suite honest
Draw cases from correction histories of real companies, with rights cleared for evaluation use and records de-identified before any task is built. Good sources are CRM merge logs, credit memo histories, master data change logs and ticket notes that record a fix. The population statement should name the industries, company size bands, systems and years the cases come from, so readers know what the result can and cannot speak for.
Sample by task family and by company, with a cap on how many tasks any one company contributes, so a single unusual ERP configuration cannot dominate the score. Record the sampling frame and the denominator: how many candidate corrections were found, how many were excluded and why.
For held-out validity, split by company and by task family, not by task. Keep at least one whole company and at least one whole family out of any development set. A result that only holds on companies whose quirks the agent has seen is a memorization result. AppWorld's practice of releasing only evaluation programs for its test splits, and its warning that training on package contents can influence scores, are good precedents for keeping the held-out cases private.
| Design element | Rule | Why it matters |
|---|---|---|
| Unit of sampling | Correction event, stratified by family and company | Keeps common corrections from crowding out rare ones |
| Per-company cap | A fixed maximum share of tasks per company, set before sampling | Limits the weight of one configuration |
| Holdout | Whole companies and whole task families held back | Tests transfer, not recall of a known setup |
| Ground truth | Two domain reviewers build each target state; disagreements adjudicated | Target states encode judgment about what may change |
| Exclusions | Logged with a reason code | Readers can see what the denominator leaves out |
Metrics, baselines and how to report uncertainty
Report a strict task pass rate first: the share of tasks where the target-state diff matches and every invariant holds. Then report the parts separately so a reader can see why tasks fail: required-change recall, collateral changes per task, invariant violation rate by invariant, and decoy write rate.
Repeat each task several times per agent configuration and report reliability across trials. The tau-bench paper defines pass^k as the probability that all k independent trials of a task succeed; for propagation work, where one bad run can corrupt a ledger, that view matters more than the best of several attempts.
Baselines should include a scripted rule-based propagator that knows the system rules, a naive agent that edits every matching record, and the same model with and without access to the cross-system crosswalk. Name model versions, tool definitions and dates.
For each rate, report a binomial confidence interval using the Wilson method, which NIST's Dataplot reference lists for proportions, and also a company-clustered interval, since tasks from one company are not independent. If held-out intervals are too wide to separate two configurations, say so rather than ranking them.
Pre-registration checklist before the first run
Freeze these decisions in a dated document before any agent touches the held-out set. Changes after that point are reported as deviations.
At a glance
Pre-registration checklist before the first run
Checklist · 10 items
- Research question, in one sentence, naming propagation across systems of record.
- Population: industries, size bands, systems, years and the rights basis for evaluation use.
- Task families, trap types and the number of tasks per family and per company.
- Target-state construction rules and the reviewer adjudication process.
- The full invariant list and how each one is computed.
- Holdout companies and families, and who has access to them.
- Primary metric, secondary metrics and the number of repeat trials.
- Baselines, model versions, tool definitions and time limits.
- Interval method, clustering unit and the rule for declaring two configurations different.
- Release conditions: what is published, what stays private and when tasks are retired.
0 of 10 ticked. Tick each item as you go.
What a passing score would and would not show
A strong held-out score would show that an agent can follow written system rules across several stores and avoid the obvious overwrite mistakes on the companies and families tested. It would not show the agent is safe to run unsupervised on production systems, where permissions, concurrency and partial outages add failure modes a sandbox does not reproduce.
The design has other limits worth stating up front. Five systems is a simplification; many companies run more, with overlapping masters. Target states encode reviewer judgment, and some corrections have more than one acceptable answer, which the grader must allow explicitly. De-identification can remove the very cues, such as names that differ only by suffix, that make merge decisions hard. Any published result should carry these caveats next to the number.
Where rights-cleared correction histories come from
The raw material for this benchmark is unglamorous: merge logs, credit memos, master data change histories and the ticket notes where someone explained a fix. Few public datasets contain them, and they need rights review and de-identification before an evaluation team can build tasks from them.
SourceX handles qualification, rights review, contracting, preparation and delivery for businesses licensing selected records, and it sources correction histories on request from U.S. businesses rather than keeping them in stock. An evaluation team describes the systems, volume, format, timing and licensing terms it needs, and every supplying company approves scope and terms and signs before its records are delivered.
Frequently asked questions
What is a cross-system correction benchmark?
It is an evaluation in which an agent receives one confirmed business correction and must apply it across several systems of record, such as a CRM, an ERP, billing and a support desk. Grading compares the final state of every system with an expert target state and a list of invariants. It measures both the changes made and the records correctly left alone.
Why not grade the agent's explanation of what it changed?
An agent can describe the right steps and still write the wrong records, or the right records in the wrong form. Final-state grading, as used by AppWorld and tau-bench, checks what actually happened in the data. Invariants then catch side effects that a target-state diff does not list.
Why include tasks where the right answer is to change nothing?
No-op decoys catch agents that write whenever they are asked to. In business systems an unnecessary write can reopen a closed period or create a duplicate adjustment. Reporting the decoy write rate separately keeps an eager agent from looking stronger than it is.
How many tasks does a cross-system correction benchmark need?
Enough per task family and per held-out company for the intervals to separate the configurations you want to compare. Set the number before running, using the interval width you need, and report it with the population. If the held-out set is too small to tell two agents apart, the honest report says that.
Can synthetic companies replace real correction histories?
Synthetic worlds are useful for development and for testing the grader. Real correction histories add the configurations, partial fixes and edge cases that people actually create, which is what the held-out set should measure. Real records need rights clearance and de-identification before use.
Sources
- AppWorld grades agents with database-state-based unit tests that check task completion without collateral damage, and releases only evaluation programs for its test splits. Source
- tau-bench pairs an agent holding domain API tools and policy guidelines with a language-model user simulator and reports Pass^1 to Pass^4. Source
- The tau-bench paper grades by comparing end-of-conversation database state with an annotated goal state and defines pass^k as the probability that all k trials succeed. Source
- Undoing a multi-store operation must reach every data store, restoring an original state can overwrite concurrent changes, and steps should be idempotent. Source
- By default a HubSpot merge gives the resulting record a new record ID, keeps a Merged Record IDs property and cannot be undone; accounts in a Primary ID Preservation beta keep the primary record's ID instead. Source
- OCEL 2.0 relates events to several objects through qualified relationships and allows object-to-object relations independent of events. Source
- NIST's Dataplot reference documents the Wilson method for confidence intervals on a proportion. Source
Tell us what data you need
Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.