Skip to content

AI uses for records

Cross-company agent generalization benchmark with a similarity audit

By SourceX Editorial · Updated

9 sections

Short answer

This is a design, not a result. A cross-company agent generalization benchmark tests whether an agent system works at companies it has never seen. Develop on some permissioned organizations, seal others completely, and before scoring rate each sealed task's similarity to development on text, templates, systems, task family and difficulty. Report transfer per company and per similarity band. The question is whether the agent transfers, not whether more source companies make better training data.

Agent run

Cross company agent

This page covers cross-company agent generalization benchmark with a similarity audit.Illustrative

Key takeaways

  • Hold out whole organizations, not records: a random split lets an agent pass by recognizing a company's templates and rules.
  • Development includes prompt and tool iteration, so a sealed company must stay unseen by the team's people as well as by training.
  • An overlap audit finds each sealed task's nearest development neighbor; transfer claims hold only within the similarity bands actually tested.
  • A context arm, with and without the company's own written policies, sizes how much performance depends on supplier-specific context.
  • With companies as the unit of uncertainty, the number of sealed companies sets the precision, and every per-company failure belongs in the report.

What does a cross-company agent generalization benchmark test?

It tests whether an agent built and tuned on some companies' work performs at companies it has never met, after accounting for how similar those companies are to the ones it learned from. That is the question a buyer faces at deployment, and a score earned on familiar companies cannot answer it. Whether training data from ten companies is worth more than data from one is a different, data-purchase question, which the source diversity experiment protocol answers; this benchmark scores the agent a buyer would deploy.

Generalization limits are a documented risk item, not an academic nicety. NIST's AI Risk Management Framework states under MEASURE 2.5 that the AI system to be deployed is demonstrated to be valid and reliable and that limitations of generalizability beyond the conditions under which it was developed are documented. The framework's Playbook asks evaluators to assess the external validity of all measurements.

This page is the protocol for the Cross-Company Transfer Challenge: an organization-held-out split, an overlap audit and a transfer-performance table. It reports no results. Every illustrative entry is a placeholder, and the challenge remains a proposed study until it is actually run.

Why a random split across companies flatters the agent

Records from the same company share templates, codes, policies and people, so a random split puts near-twins on both sides and rewards recognition rather than transfer. scikit-learn's documentation states the general rule: when samples come in dependent groups, measuring generalization to unseen groups requires every validation sample to come from groups absent from the paired training fold. Its GroupKFold splitter exists to catch a flexible model that learns group-specific features and fails on new groups.

For agents, the leak is wider than training data. scikit-learn defines data leakage as using information that would not be available at prediction time and says test data should never be used to make choices about the model. An agent team makes such choices constantly: prompt edits, tool descriptions, retrieval settings and retry rules, often tuned by reading failure transcripts. If those transcripts came from a company in the test set, that company is no longer unseen. Anthropic's agent evaluation guide adds a runtime version of the problem, noting that shared state between trials can inflate scores, as when Claude gained an advantage on some tasks by reading git history left by earlier trials.

Public agent benchmarks are rarely built to answer this question. Tau-bench varies domains, with airline, retail and telecom each defined by one written policy and a simulated customer. TheAgentCompany runs its 175 task images inside one set of self-hosted workplace services: GitLab, Plane, ownCloud and RocketChat. GDPval spans 44 occupations. Each design varies tasks, occupations or domains; none of the three is organized around holding out whole companies.

Defining comparable task families across permissioned organizations

A task family is comparable across companies when it shares the trigger, the class of system and the kind of verifiable outcome, even though each company's policy differs. Define families from the record as it stood when the task arose, before its resolution, and freeze the definitions before any company is assigned to a split.

At a glance

Defining comparable task families across permissioned organizations

Each organization supplies its records under its own approval for evaluation use and for publication of aggregated, de-identified results. Companies appear in a release under codes, never names, unless they approve otherwise. Assign family identifiers before splitting, a step the guide to enterprise benchmark task family IDs covers in detail.

Illustrative example, not a study: a warranty-claim family drawn from six equipment distributors. Four run the same field-service product and two run different ones; all six approve claims against a written warranty policy, but their limits for automatic approval differ. Trigger, system class and outcome match, so the family is comparable across all six. The approval limit is the local rule, and an agent that carries one distributor's limit to another has memorized rather than transferred.

  • Trigger: what starts the task, such as a disputed invoice, a warranty claim or a schedule change request.
  • System class: the kind of system of record involved, such as an ERP, a ticketing tool or a field-service platform, with product and version recorded.
  • Outcome: what a grader can check afterward, such as a posted credit, a ticket closed with the right code or an approved change order.
  • Local rule: the company-specific threshold or policy that decides the right answer, recorded separately so the audit can see it.
  • Eligibility: a family enters the challenge only if it occurs at enough organizations to place some in development and some in the sealed set.

The organization-held-out split

Split by organization first, then by task family inside the development set, never the other way round. The table is the split specification to register before anyone builds or tunes the agent.

Development and sealed companies need the same license terms, including benchmark and publication use. A sealed company that can be evaluated but not reported leaves holes in the results table that nobody can explain.

The organization-held-out split
ComponentDesign choiceWhyWhat the report states
Development organizationsCompanies whose records the agent team may train on, prompt against and read failures fromEverything the team learns comes from hereNumber of companies, families and tasks, with date ranges
Validation organizationA company outside both development and the sealed set, used to choose prompts, settings and checkpointsChoosing settings on the test set turns transfer into recallWhich settings were chosen on it, and how
Sealed organizationsCompanies picked by a recorded random draw, stratified by size and system classHand-picking produces an easy or hard test without anyone intending itThe draw procedure, the strata and any replacements
CustodySealed records held by a party outside the agent team; the team submits a frozen agentReading sealed failures is development by another nameWho held the records, and every access event
Exposure logA list of every person and system that touched sealed dataLeaks are usually human, not technicalThe log, attested before results are opened
RoundsA second round re-draws the sealed set after the first round is publishedOne draw can be luckyResults per round, never pooled without saying so

The overlap audit: how close is each sealed company to the development set?

Before any score is opened, find each sealed task's nearest neighbor in the development set on five dimensions and give it a similarity band. Transfer to a company running the same software, templates and policy family as a development company is a weaker claim than transfer to one that shares none of them.

Duplication is common enough to check every time. The authors of Google Research's deduplication tool report finding a single 50-word sequence repeated 60,000 times in the C4 web corpus, and that language models trained on well-deduplicated data are less likely to show memorization. Business records repeat in their own ways, through shared vendors, franchise templates and forwarded threads. The datasketch library's MinHash and MinHash LSH estimate Jaccard similarity and answer threshold queries in sub-linear time, one practical way to run the audit across a large pool.

Band rules turn the audit into a design decision. A sealed task is near when it shares its system and template with a development company, mid when it shares the family and system class but not the template, and far when it shares only the family. Exact and near-duplicate tasks are removed and listed, never quietly scored.

The overlap audit: how close is each sealed company to the development set?
Audit dimensionHow it is measuredWhat it guards against
Near-duplicate textMinHash estimates of Jaccard similarity, with LSH to find candidate pairsThe same document or thread appearing at two companies
Template and form reuseShared field layouts, form identifiers and boilerplate passagesVendor or franchise templates that make two companies look alike
System familiaritySame product and version of the system of recordAn agent that learned one product's screens and codes
Task family and local ruleSame family, with the same or a different threshold or policySuccess that comes from a memorized rule rather than the workflow
DifficultyExpert time estimates or difficulty ratings made before scoringSealed tasks that are simply easier or harder
A flow from organizations split into development and sealed groups, through a five-dimension overlap audit, to a transfer report that keeps near, mid and far similarity bands separate.
The Cross-Company Transfer Challenge design. Illustrative: no challenge has been run and no results are shown.

Measuring transfer gains, failures and company-specific context

Measure transfer as the gap between the agent's score on unseen cases from development companies and its score at sealed companies, computed within each similarity band and at matched difficulty. A raw gap mixes transfer with the plain fact that some sealed companies are harder. Matching means comparing tasks with the same audit difficulty rating across arms, or reweighting one arm to the other's difficulty mix, and reporting the raw and matched gaps side by side.

Score outcomes, not narration. Anthropic's guide defines the outcome as the final state of the environment: for a flight-booking agent, what counts is whether the reservation exists, not whether the agent says the flight is booked. Log every sealed failure with a cause code, whether a development company's rule applied where it is wrong, an unfamiliar system field, missing context, an ambiguous task or a grader error. The cause codes, more than the average, tell a buyer what would break at its own company.

  • Arm A, development companies, unseen cases: the familiarity reference.
  • Arm B, sealed companies with no company context: the agent meets each company cold.
  • Arm C, sealed companies with their written context: the same tasks with the company's policies, glossary and field definitions supplied at run time. The B-to-C gap sizes how much performance depends on supplier-specific context.
  • Repeated trials on every task, so agent randomness is separated from company effects, with pass^k reported beside the per-trial rate.

The transfer-performance table: what a results release must contain

One row per sealed company, never only an average. The template defines each column; its entries are illustrative placeholders, not results.

Uncertainty follows the company. Anthropic's statistical guidance recommends clustering standard errors on the unit of randomization and reports clustered errors on popular evaluations more than three times the naive size. Here the unit is the sealed company, so precision depends mainly on how many companies are sealed, not how many tasks each contributes. Run a power analysis on the company count before recruiting, report the worst sealed company next to the average, and resample companies first and tasks within them second when building intervals.

The transfer-performance table: what a results release must contain
Report columnDefinitionIllustrative entry, not a result
Company codeAnonymized identifier for the sealed organizationS-03
Similarity band mixShare of the company's tasks in the near, mid and far bandsMostly far; its system differs from every development company
Tasks and exclusionsTasks scored, tasks excluded and the reason for each exclusionCounted per family, with exclusion codes
Transfer gapArm A score minus Arm B score within band, with an intervalTo be measured
Context effectArm C minus Arm B on the same tasks, as a paired differenceTo be measured
Consistencypass^k across repeated trials per taskTo be measured
Dominant failure causeThe most frequent cause code among failuresA development company's rule applied where it is wrong
VerdictTransfers, transfers with context, or does not transfer, within the bands testedAssigned only after intervals are computed

What the results can support, and what stays unique to the sampled companies

A passing result supports transfer for that agent version, those task families and the similarity bands that were actually populated. It does not support transfer to a far band nobody sampled, to system products absent from the pool, to later periods or to company sizes outside the strata.

Publish negative results as prominently as positive ones. A company where the agent fails to transfer, with its cause codes, is the most useful row for a buyer deciding where to deploy. Keep measured results visibly separate from this design in every release: the design exists now, the results do not, and any figure quoted before the challenge runs is illustrative.

Register these items before the first sealed task is scored:

  • The question, the primary metric and the smallest transfer gap that would change a deployment decision.
  • The organization pool, eligibility rules, the random draw and its strata.
  • Task family definitions, band rules and audit thresholds.
  • Arms, trials per task, the grader and its validation.
  • The interval method, the company-level power analysis and the rule for reporting the worst company.
  • Rights confirmed for evaluation use and for publishing de-identified, aggregated results.

Sourcing permissioned organizations for the challenge

The binding constraint is rarely the statistics. It is assembling enough comparable companies, each with closed tasks, recorded outcomes and permission for evaluation use, that some can be sealed without leaving development too thin.

SourceX takes requests for multi-company datasets like this and sources them from U.S. businesses rather than holding stock. A request names scope, volume, language, format, timing and licensing needs, which for this challenge include evaluation use, sealed custody and publication of de-identified results. As a general guide, buyers look for each company to bring years of consistent history, two or more being typical, and a single system holding tens of thousands of records or more; smaller sets can still qualify when the work is rare, and every case gets its own deal-by-deal assessment.

Each participating company approves its scope and terms and signs before any data is delivered. Nothing here implies that a particular company, record type or result is available; the request is where the acceptance test for the challenge gets written down.

Frequently asked questions

Why hold out whole companies instead of random records?

Records from one company share templates, codes, policies and people, so a random split puts near-twins in both training and test. The agent can then pass by recognizing the company rather than handling the task. Holding out whole organizations is the only way to measure performance at a company the agent has never seen.

How many held-out companies does a transfer benchmark need?

Enough to detect the smallest transfer gap that would change a deployment decision, which a power analysis should set before recruiting. Because companies are the unit of uncertainty, adding tasks per company helps far less than adding companies. A challenge with very few sealed companies can report per-company results but cannot support a general transfer claim.

What is a similarity band in a transfer benchmark?

It labels how close a sealed task is to the development set: near when it shares system and template with a development company, mid when it shares family and system class, far when it shares only the task family. Reporting by band stops an easy, near-identical company from standing in for real generalization.

Is giving the agent a company's written policies at test time cheating?

Not if it is reported as its own arm. Running the same sealed tasks with and without the company's policies, glossary and field definitions measures how much performance depends on supplier-specific context, which tells a buyer what it must supply at deployment.

Has SourceX run the Cross-Company Transfer Challenge?

No. This page publishes the design only; no challenge has been run and no results exist. Every entry in the transfer table is an illustrative placeholder.

Sources

  • scikit-learn's cross-validation guide: grouped data breaks the independence assumption; testing generalization to unseen groups needs validation groups absent from training; GroupKFold keeps a group out of both sides so a model that learns group-specific features is caught. Source
  • scikit-learn defines data leakage as using information unavailable at prediction time, which yields overly optimistic estimates, and says test data should never be used to make choices about the model. Source
  • NIST AI RMF 1.0 MEASURE 2.5: the AI system to be deployed is demonstrated to be valid and reliable, and limitations of generalizability beyond the conditions under which it was developed are documented. Source
  • NIST's AI RMF Playbook asks evaluators to assess the external validity of all measurements, the degree to which measurements taken in one context can generalize to others. Source
  • Anthropic recommends clustering standard errors on the unit of randomization, reports clustered standard errors on popular evaluations more than three times naive ones, and recommends power analysis. Source
  • Anthropic's agent evaluation guide: shared state between trials can inflate performance (Claude read git history from earlier trials); the outcome is the final environment state, such as whether a reservation exists; pass^k measures consistency across trials. Source
  • Google Research's deduplication repository reports a single 50-word sequence repeated 60,000 times in C4 and that models trained on well-deduplicated data are less likely to show memorization. Source
  • datasketch's MinHash estimates Jaccard similarity, and MinHash LSH supports sub-linear threshold queries on Jaccard similarity. Source
  • Tau-bench is a simulation framework for customer service agents in which each domain, such as airline, retail and telecom, specifies a policy, tools and tasks, with a simulated user. Source
  • TheAgentCompany benchmark runs GitLab, Plane, ownCloud and RocketChat services and lists 175 task images. Source
  • OpenAI's GDPval spans 44 occupations selected from the top nine industries contributing to U.S. GDP. Source

Tell us what data you need

Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.

See if your company qualifies