AI uses for records
Cross-company agent generalization benchmark with a similarity audit
By SourceX Editorial · Updated
9 sections
Short answer
This is a design, not a result. A cross-company agent generalization benchmark tests whether an agent system works at companies it has never seen. Develop on some permissioned organizations, seal others completely, and before scoring rate each sealed task's similarity to development on text, templates, systems, task family and difficulty. Report transfer per company and per similarity band. The question is whether the agent transfers, not whether more source companies make better training data.
Agent run
Cross company agent
Key takeaways
- Hold out whole organizations, not records: a random split lets an agent pass by recognizing a company's templates and rules.
- Development includes prompt and tool iteration, so a sealed company must stay unseen by the team's people as well as by training.
- An overlap audit finds each sealed task's nearest development neighbor; transfer claims hold only within the similarity bands actually tested.
- A context arm, with and without the company's own written policies, sizes how much performance depends on supplier-specific context.
- With companies as the unit of uncertainty, the number of sealed companies sets the precision, and every per-company failure belongs in the report.
What does a cross-company agent generalization benchmark test?
It tests whether an agent built and tuned on some companies' work performs at companies it has never met, after accounting for how similar those companies are to the ones it learned from. That is the question a buyer faces at deployment, and a score earned on familiar companies cannot answer it. Whether training data from ten companies is worth more than data from one is a different, data-purchase question, which the source diversity experiment protocol answers; this benchmark scores the agent a buyer would deploy.
Generalization limits are a documented risk item, not an academic nicety. NIST's AI Risk Management Framework states under MEASURE 2.5 that the AI system to be deployed is demonstrated to be valid and reliable and that limitations of generalizability beyond the conditions under which it was developed are documented. The framework's Playbook asks evaluators to assess the external validity of all measurements.
This page is the protocol for the Cross-Company Transfer Challenge: an organization-held-out split, an overlap audit and a transfer-performance table. It reports no results. Every illustrative entry is a placeholder, and the challenge remains a proposed study until it is actually run.
Why a random split across companies flatters the agent
Records from the same company share templates, codes, policies and people, so a random split puts near-twins on both sides and rewards recognition rather than transfer. scikit-learn's documentation states the general rule: when samples come in dependent groups, measuring generalization to unseen groups requires every validation sample to come from groups absent from the paired training fold. Its GroupKFold splitter exists to catch a flexible model that learns group-specific features and fails on new groups.
For agents, the leak is wider than training data. scikit-learn defines data leakage as using information that would not be available at prediction time and says test data should never be used to make choices about the model. An agent team makes such choices constantly: prompt edits, tool descriptions, retrieval settings and retry rules, often tuned by reading failure transcripts. If those transcripts came from a company in the test set, that company is no longer unseen. Anthropic's agent evaluation guide adds a runtime version of the problem, noting that shared state between trials can inflate scores, as when Claude gained an advantage on some tasks by reading git history left by earlier trials.
Public agent benchmarks are rarely built to answer this question. Tau-bench varies domains, with airline, retail and telecom each defined by one written policy and a simulated customer. TheAgentCompany runs its 175 task images inside one set of self-hosted workplace services: GitLab, Plane, ownCloud and RocketChat. GDPval spans 44 occupations. Each design varies tasks, occupations or domains; none of the three is organized around holding out whole companies.
Defining comparable task families across permissioned organizations
A task family is comparable across companies when it shares the trigger, the class of system and the kind of verifiable outcome, even though each company's policy differs. Define families from the record as it stood when the task arose, before its resolution, and freeze the definitions before any company is assigned to a split.
At a glance
Defining comparable task families across permissioned organizations
Each organization supplies its records under its own approval for evaluation use and for publication of aggregated, de-identified results. Companies appear in a release under codes, never names, unless they approve otherwise. Assign family identifiers before splitting, a step the guide to enterprise benchmark task family IDs covers in detail.
Illustrative example, not a study: a warranty-claim family drawn from six equipment distributors. Four run the same field-service product and two run different ones; all six approve claims against a written warranty policy, but their limits for automatic approval differ. Trigger, system class and outcome match, so the family is comparable across all six. The approval limit is the local rule, and an agent that carries one distributor's limit to another has memorized rather than transferred.
- Trigger: what starts the task, such as a disputed invoice, a warranty claim or a schedule change request.
- System class: the kind of system of record involved, such as an ERP, a ticketing tool or a field-service platform, with product and version recorded.
- Outcome: what a grader can check afterward, such as a posted credit, a ticket closed with the right code or an approved change order.
- Local rule: the company-specific threshold or policy that decides the right answer, recorded separately so the audit can see it.
- Eligibility: a family enters the challenge only if it occurs at enough organizations to place some in development and some in the sealed set.
The organization-held-out split
Split by organization first, then by task family inside the development set, never the other way round. The table is the split specification to register before anyone builds or tunes the agent.
Development and sealed companies need the same license terms, including benchmark and publication use. A sealed company that can be evaluated but not reported leaves holes in the results table that nobody can explain.
| Component | Design choice | Why | What the report states |
|---|---|---|---|
| Development organizations | Companies whose records the agent team may train on, prompt against and read failures from | Everything the team learns comes from here | Number of companies, families and tasks, with date ranges |
| Validation organization | A company outside both development and the sealed set, used to choose prompts, settings and checkpoints | Choosing settings on the test set turns transfer into recall | Which settings were chosen on it, and how |
| Sealed organizations | Companies picked by a recorded random draw, stratified by size and system class | Hand-picking produces an easy or hard test without anyone intending it | The draw procedure, the strata and any replacements |
| Custody | Sealed records held by a party outside the agent team; the team submits a frozen agent | Reading sealed failures is development by another name | Who held the records, and every access event |
| Exposure log | A list of every person and system that touched sealed data | Leaks are usually human, not technical | The log, attested before results are opened |
| Rounds | A second round re-draws the sealed set after the first round is published | One draw can be lucky | Results per round, never pooled without saying so |
The overlap audit: how close is each sealed company to the development set?
Before any score is opened, find each sealed task's nearest neighbor in the development set on five dimensions and give it a similarity band. Transfer to a company running the same software, templates and policy family as a development company is a weaker claim than transfer to one that shares none of them.
Duplication is common enough to check every time. The authors of Google Research's deduplication tool report finding a single 50-word sequence repeated 60,000 times in the C4 web corpus, and that language models trained on well-deduplicated data are less likely to show memorization. Business records repeat in their own ways, through shared vendors, franchise templates and forwarded threads. The datasketch library's MinHash and MinHash LSH estimate Jaccard similarity and answer threshold queries in sub-linear time, one practical way to run the audit across a large pool.
Band rules turn the audit into a design decision. A sealed task is near when it shares its system and template with a development company, mid when it shares the family and system class but not the template, and far when it shares only the family. Exact and near-duplicate tasks are removed and listed, never quietly scored.
| Audit dimension | How it is measured | What it guards against |
|---|---|---|
| Near-duplicate text | MinHash estimates of Jaccard similarity, with LSH to find candidate pairs | The same document or thread appearing at two companies |
| Template and form reuse | Shared field layouts, form identifiers and boilerplate passages | Vendor or franchise templates that make two companies look alike |
| System familiarity | Same product and version of the system of record | An agent that learned one product's screens and codes |
| Task family and local rule | Same family, with the same or a different threshold or policy | Success that comes from a memorized rule rather than the workflow |
| Difficulty | Expert time estimates or difficulty ratings made before scoring | Sealed tasks that are simply easier or harder |
Measuring transfer gains, failures and company-specific context
Measure transfer as the gap between the agent's score on unseen cases from development companies and its score at sealed companies, computed within each similarity band and at matched difficulty. A raw gap mixes transfer with the plain fact that some sealed companies are harder. Matching means comparing tasks with the same audit difficulty rating across arms, or reweighting one arm to the other's difficulty mix, and reporting the raw and matched gaps side by side.
Score outcomes, not narration. Anthropic's guide defines the outcome as the final state of the environment: for a flight-booking agent, what counts is whether the reservation exists, not whether the agent says the flight is booked. Log every sealed failure with a cause code, whether a development company's rule applied where it is wrong, an unfamiliar system field, missing context, an ambiguous task or a grader error. The cause codes, more than the average, tell a buyer what would break at its own company.
- Arm A, development companies, unseen cases: the familiarity reference.
- Arm B, sealed companies with no company context: the agent meets each company cold.
- Arm C, sealed companies with their written context: the same tasks with the company's policies, glossary and field definitions supplied at run time. The B-to-C gap sizes how much performance depends on supplier-specific context.
- Repeated trials on every task, so agent randomness is separated from company effects, with pass^k reported beside the per-trial rate.
The transfer-performance table: what a results release must contain
One row per sealed company, never only an average. The template defines each column; its entries are illustrative placeholders, not results.
Uncertainty follows the company. Anthropic's statistical guidance recommends clustering standard errors on the unit of randomization and reports clustered errors on popular evaluations more than three times the naive size. Here the unit is the sealed company, so precision depends mainly on how many companies are sealed, not how many tasks each contributes. Run a power analysis on the company count before recruiting, report the worst sealed company next to the average, and resample companies first and tasks within them second when building intervals.
| Report column | Definition | Illustrative entry, not a result |
|---|---|---|
| Company code | Anonymized identifier for the sealed organization | S-03 |
| Similarity band mix | Share of the company's tasks in the near, mid and far bands | Mostly far; its system differs from every development company |
| Tasks and exclusions | Tasks scored, tasks excluded and the reason for each exclusion | Counted per family, with exclusion codes |
| Transfer gap | Arm A score minus Arm B score within band, with an interval | To be measured |
| Context effect | Arm C minus Arm B on the same tasks, as a paired difference | To be measured |
| Consistency | pass^k across repeated trials per task | To be measured |
| Dominant failure cause | The most frequent cause code among failures | A development company's rule applied where it is wrong |
| Verdict | Transfers, transfers with context, or does not transfer, within the bands tested | Assigned only after intervals are computed |
What the results can support, and what stays unique to the sampled companies
A passing result supports transfer for that agent version, those task families and the similarity bands that were actually populated. It does not support transfer to a far band nobody sampled, to system products absent from the pool, to later periods or to company sizes outside the strata.
Publish negative results as prominently as positive ones. A company where the agent fails to transfer, with its cause codes, is the most useful row for a buyer deciding where to deploy. Keep measured results visibly separate from this design in every release: the design exists now, the results do not, and any figure quoted before the challenge runs is illustrative.
Register these items before the first sealed task is scored:
- The question, the primary metric and the smallest transfer gap that would change a deployment decision.
- The organization pool, eligibility rules, the random draw and its strata.
- Task family definitions, band rules and audit thresholds.
- Arms, trials per task, the grader and its validation.
- The interval method, the company-level power analysis and the rule for reporting the worst company.
- Rights confirmed for evaluation use and for publishing de-identified, aggregated results.
Sourcing permissioned organizations for the challenge
The binding constraint is rarely the statistics. It is assembling enough comparable companies, each with closed tasks, recorded outcomes and permission for evaluation use, that some can be sealed without leaving development too thin.
SourceX takes requests for multi-company datasets like this and sources them from U.S. businesses rather than holding stock. A request names scope, volume, language, format, timing and licensing needs, which for this challenge include evaluation use, sealed custody and publication of de-identified results. As a general guide, buyers look for each company to bring years of consistent history, two or more being typical, and a single system holding tens of thousands of records or more; smaller sets can still qualify when the work is rare, and every case gets its own deal-by-deal assessment.
Each participating company approves its scope and terms and signs before any data is delivered. Nothing here implies that a particular company, record type or result is available; the request is where the acceptance test for the challenge gets written down.
Frequently asked questions
Why hold out whole companies instead of random records?
Records from one company share templates, codes, policies and people, so a random split puts near-twins in both training and test. The agent can then pass by recognizing the company rather than handling the task. Holding out whole organizations is the only way to measure performance at a company the agent has never seen.
How many held-out companies does a transfer benchmark need?
Enough to detect the smallest transfer gap that would change a deployment decision, which a power analysis should set before recruiting. Because companies are the unit of uncertainty, adding tasks per company helps far less than adding companies. A challenge with very few sealed companies can report per-company results but cannot support a general transfer claim.
What is a similarity band in a transfer benchmark?
It labels how close a sealed task is to the development set: near when it shares system and template with a development company, mid when it shares family and system class, far when it shares only the task family. Reporting by band stops an easy, near-identical company from standing in for real generalization.
Is giving the agent a company's written policies at test time cheating?
Not if it is reported as its own arm. Running the same sealed tasks with and without the company's policies, glossary and field definitions measures how much performance depends on supplier-specific context, which tells a buyer what it must supply at deployment.
Has SourceX run the Cross-Company Transfer Challenge?
No. This page publishes the design only; no challenge has been run and no results exist. Every entry in the transfer table is an illustrative placeholder.
Sources
- scikit-learn's cross-validation guide: grouped data breaks the independence assumption; testing generalization to unseen groups needs validation groups absent from training; GroupKFold keeps a group out of both sides so a model that learns group-specific features is caught. Source
- scikit-learn defines data leakage as using information unavailable at prediction time, which yields overly optimistic estimates, and says test data should never be used to make choices about the model. Source
- NIST AI RMF 1.0 MEASURE 2.5: the AI system to be deployed is demonstrated to be valid and reliable, and limitations of generalizability beyond the conditions under which it was developed are documented. Source
- NIST's AI RMF Playbook asks evaluators to assess the external validity of all measurements, the degree to which measurements taken in one context can generalize to others. Source
- Anthropic recommends clustering standard errors on the unit of randomization, reports clustered standard errors on popular evaluations more than three times naive ones, and recommends power analysis. Source
- Anthropic's agent evaluation guide: shared state between trials can inflate performance (Claude read git history from earlier trials); the outcome is the final environment state, such as whether a reservation exists; pass^k measures consistency across trials. Source
- Google Research's deduplication repository reports a single 50-word sequence repeated 60,000 times in C4 and that models trained on well-deduplicated data are less likely to show memorization. Source
- datasketch's MinHash estimates Jaccard similarity, and MinHash LSH supports sub-linear threshold queries on Jaccard similarity. Source
- Tau-bench is a simulation framework for customer service agents in which each domain, such as airline, retail and telecom, specifies a policy, tools and tasks, with a simulated user. Source
- TheAgentCompany benchmark runs GitLab, Plane, ownCloud and RocketChat services and lists 175 task images. Source
- OpenAI's GDPval spans 44 occupations selected from the top nine industries contributing to U.S. GDP. Source
Tell us what data you need
Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.