Skip to content

AI data market

AI evaluation infrastructure economics, tested on 129 open evals

By SourceX Editorial · Reviewed by Noah Loul ·

10 sections

Short answer

Only one layer of AI evaluation infrastructure economics behaves like software: the workflow that runs and compares evals while customers pay for model calls. In the open Inspect Evals library, 76 of 129 evals are built with a sandbox or a second model per sample, 116 fetch items from outside the repository, and 76 of 111 migrated evals changed comparability within eight months. Hosted runs, fresh content and human grading carry recurring cost.

Evaluation split

AI evaluation infrastructure

This page covers AI evaluation infrastructure economics, tested on 129 open evals.Illustrative

Key takeaways

  • Evaluation infrastructure has four layers, harness, run compute, content and human grading, and only the harness can carry software margins on its own.
  • In a count of 129 open evals, 58.9 percent are built with a container or a second model per sample; all 21 coding evals are, against 4 of 20 reasoning evals.
  • Cloud price lists treat evaluation as inference plus a charge per human task, which sets a ceiling on what independent vendors can charge for automated scoring.
  • Open licensing and consolidation, such as an MIT-licensed eval library and Langfuse joining ClickHouse, push the harness toward being a feature rather than a product.
  • Content needs recurring labor: 76 of 111 evals migrated to a versioning scheme in February 2026 had a comparability-breaking change by late October.

What would software-like economics mean for evaluation infrastructure?

It would mean that serving one more customer, or running one more evaluation, costs almost nothing beyond what the customer already pays for, and that the product does not wear out between sales. Evaluation infrastructure meets that bar in only one of its four layers, so the investment question is which layer a company actually sells.

The four layers carry different costs. The harness and workflow software defines tasks, runs them, logs results and compares versions. Run-time compute pays for the model under test, any judge or simulated user, and any container a task needs. Content is the items, rubrics and reference answers. Human grading is people scoring outputs. Software economics are available in the first layer, inference-like costs live in the second, and recurring labor sits in the third and fourth.

This article tests those claims against public evidence rather than vendor positioning: a count of how 129 open evaluations are built, the price lists two cloud providers publish for evaluation, the licensing of open evaluation tools, and the record of how often open tasks change in ways that break comparability.

A count of 129 open evals: where the per-sample cost sits

In the open Inspect Evals library, 76 of 129 evaluations, 58.9 percent, are built to add a container sandbox, a second model or both on top of each call to the model under test. Only 53 evaluations, 41.1 percent, need nothing per sample beyond the model call itself.

Inspect Evals is maintained by Generality Labs, a London nonprofit, and was founded in 2024 with contributions from the UK AI Security Institute, Arcadia Impact and the Vector Institute. Each evaluation folder carries an eval.yaml metadata file. A script written for this article read those files and the evaluations' Python source as text at commit cc193dc of 27 October 2026, without running any of the library's code.

The split differs sharply by subject. All 21 coding evaluations need a sandbox or a second model, as do 16 of 21 safeguards evaluations and 10 of 13 cybersecurity evaluations, while only 4 of 20 reasoning evaluations and 1 of 7 mathematics evaluations do. Agentic, coding and safety work is exactly where each run costs more than a model call. The four cost classes:

A count of 129 open evals: where the per-sample cost sits
Cost classEvaluationsShare of 129What the evaluation adds per sample
Model call only5341.1%Nothing beyond the model under test
Second model, no sandbox3527.1%A judge, grader, simulated user or opponent model call
Sandbox, no second model3124.0%A container to execute code or tools
Sandbox and second model107.8%Both a container and an extra model call
Mosaic chart dividing 129 open evaluations by whether they need a sandbox and a second model: 53 model call only, 35 add a second model, 31 add a sandbox and 10 add both, beside three shares: 59 percent add per-sample cost, 90 percent load content from outside the repository, 68 percent of migrated evals changed comparability.
Inspect Evals at commit cc193dc, 27 October 2026; counts from the article's script and stated rules.

Who pays for the run? What cloud price lists reveal

Cloud providers price evaluation as inference plus labor, which tells investors where the variable cost sits. Amazon Bedrock's pricing page works through a human-based evaluation in which the charges are the input and output tokens for each model evaluated plus a charge per human task, with one task for each prompt-response set per worker.

Amazon SageMaker's pricing page says serverless model evaluation has no instance charges and is billed on the input and output tokens processed. For human evaluation with the customer's own workforce, it adds a charge per completed human evaluation task to the instance costs, with no separate workforce charge because the customer supplies the people; AWS-managed expert evaluation is priced on a custom basis. Bedrock's documentation adds that an LLM-as-a-judge evaluation needs two models, a generator and an evaluator, and skips the generation step when the customer supplies its own responses.

Google's Vertex AI pricing follows the same logic: model-based metrics incur only the prediction costs of the underlying autorater model, while computation-based metrics such as exact match are priced per thousand characters. For an independent evaluation company, the lesson is blunt. If it absorbs model calls, judges and sandboxes inside a flat subscription, its cost of revenue behaves like an inference business. If customers bring their own model keys and compute, the margin can look like software, but the price of the service falls toward what the clouds charge for scoring.

The harness is turning free; the periphery is what sells

The workflow layer is under steady price pressure from open source, so the defensible revenue tends to sit in hosting, enterprise controls and integration rather than in the harness itself. Inspect Evals is released under the MIT License, with copyright held by the UK AI Security Institute, and 56 further evaluations are listed in its register while hosted in their authors' own repositories.

Langfuse, an open source platform for developing, monitoring, evaluating and debugging AI applications, describes itself in its enterprise license as an open core project: the core is permissively licensed under MIT, and certain peripheral features are commercially licensed. Its README says the project has been part of ClickHouse since January 2026, and both license files now name ClickHouse, Inc. as copyright holder. Open harnesses plus a database company owning a popular evaluation tool point to the same conclusion: the harness is a feature that larger platforms absorb.

That does not make the layer worthless. It makes it a distribution channel. A company that gives away the harness can sell hosted runs, private content or human grading to the teams that adopt it, and investors should value it by which of those it converts users into.

Content wears out: comparability changes and refresh cycles

Evaluation content needs recurring work, and the open record shows how often. Inspect Evals moved its evaluations to a versioning scheme in February 2026, in which the first number rises only for changes that can make results incomparable, such as bug fixes that affect scores, scoring logic, dataset or prompt changes. Of the 111 evaluations migrated then, 110 of them on 16 February, 76, about 68 percent, had at least one such change by 27 October 2026, 160 in all.

The changes are ordinary maintenance, not crises. In September 2026 the GPQA evaluation began shuffling answer choices with a fixed seed so every build presents the same exam, a change its changelog marks as a comparability bump. In June 2026 the O-NET evaluation was pointed at a new public copy of its dataset after the original Hugging Face repository became unavailable, and its comparability version rose because byte-level equivalence could not be verified. Content hosted outside the library carries that risk widely: 116 of the 129 evaluations, 89.9 percent, fetch their items from outside the repository.

Contamination adds a second clock. LiveBench, a benchmark designed around contamination, draws questions from recent sources and adds and updates them monthly, and its authors retitled it from contamination-free to contamination-limited between paper versions. A business whose product is the test set signs up for that refresh cost permanently, which makes content closer to a subscription to expert labor than to a software release.

Evaluation business models compared on scope-normalized metrics

The comparison normalizes each model to the unit a customer consumes, a traced call, a scored sample, a licensed test item or a graded output, and asks what each additional unit costs the seller. The verdict column is an editorial judgment from the evidence above, not a measured margin. Metrics to request appear in the third column:

Evaluation business models compared on scope-normalized metrics
ModelUnit of scopeMarginal cost per unit, and metric to requestWho supplies labor and owns contentSoftware-like?
Open harness with paid supportSupported deploymentSupport hours per deploymentMaintainers and contributors; content stays with original authorsWeak: the core is free
Hosted evaluation workflow, customer pays model callsTraced call or scored sampleHosting per million records; share of runs on customer keysCustomer writes tests; vendor supplies toolingYes, while customers fund compute
Managed evaluation runs, vendor pays computeScored sampleModel, judge and sandbox cost per scored sampleVendor runs jobs; content often publicPartly: inference-like cost of revenue
Private benchmark or test set licenseLicensed test item per refreshExpert hours to replace retired items per quarterVendor's experts write items; vendor or source owner holds rightsOnly if one refresh serves many buyers
Expert human evaluationGraded outputRater minutes per graded outputVendor or customer workforceNo: services priced per task
Cloud-bundled evaluationTokens or characters scoredBundled into inference pricingCloud supplies tooling; customer supplies dataYes for the cloud, a ceiling for everyone else

Is a hosted evaluation platform a services firm with a dashboard?

Often it is, and the test is how much bespoke labor each new customer requires before the first useful score. The strongest alternative explanation for a services-heavy profile is temporary: a young platform writes rubrics and test sets for early customers until self-serve tools mature, then the labor falls away.

The evidence separates those stories. Track rubric design and test-writing hours per new customer by signing cohort; a platform should show those hours falling cohort over cohort. Track the human-graded share of scored samples; if it rises with revenue, the company is growing a grading workforce. And split revenue by layer, because a single platform label can cover hosted software, managed runs and expert grading at once. Even AWS lists expert evaluation run by its own managed team as custom-priced, beside its metered automatic evaluation, which shows that the services layer persists inside the largest platforms.

What could break the software case for evaluation infrastructure

Five developments could undo a software thesis for an evaluation company, and each can be checked in public documents or the company's own records:

  • Cloud bundling. When Bedrock, SageMaker and Vertex AI price automated scoring as inference or per character, independent vendors compete against a line item, not a product.
  • Open source substitution. An MIT-licensed library with 129 maintained evaluations and a public register lowers what anyone will pay for task implementations.
  • Consolidation. Langfuse joining ClickHouse shows that evaluation tooling can become a feature of a larger data platform, which caps standalone pricing and exit options.
  • Content decay. Comparability changes in 76 of 111 migrated evaluations within about eight months, plus monthly refresh at LiveBench, mean that content revenue requires content labor.
  • Falling judge costs cut both ways. Cheaper model calls improve the margins of managed runs, but they also lower the price ceiling that clouds set for scoring.

Diligence questions for evaluation infrastructure, SourceX included

SourceX is not an evaluation infrastructure company, but its supply sits in the content layer: companies license selected business records through it, and it runs sourcing and qualification, rights review, contracting, data preparation, delivery and payment for those deals. Datasets for AI buyers, including evaluation buyers, are sourced on request rather than held in stock, and a supplier approves scope and terms and signs before any data is delivered.

Private records that never appeared online are one answer to the contamination problem, which is why the content layer matters to this market. SourceX publishes no revenue, margin or volume figures and shares company and commercial metrics with investors directly; materials for its raise from accredited investors under Rule 506(c) are available on request through its investor page. Questions to ask any company in evaluation infrastructure, and SourceX where the content layer is concerned:

  • What share of scored samples last quarter ran on customer-paid model calls, and what share on yours?
  • What did model, judge and sandbox calls cost per thousand scored samples, by cost class?
  • How many expert hours did it take to replace retired or contaminated items per test set per quarter?
  • What share of revenue came from human grading, and is that share rising or falling?
  • Which content do you own, which do you license, and which is fetched from outside sources at run time?
  • How many hours of rubric and test design did each new customer cohort need before its first score?

Methods note: repository, commit, counting rules and limits

Repository: github.com/UKGovernmentBEIS/inspect_evals at commit cc193dc367363affb3d4b3b1a0fb1dbb459cb830, dated 27 October 2026, cloned without running any code. Population: every file matching src/inspect_evals/<name>/eval.yaml, 129 evaluations with 239 tasks and 567,426 declared samples; challenge-level metadata files deeper in the tree are excluded.

Counting rules, exactly as the script applies them. Sandbox: eval.yaml has a non-empty metadata.sandbox list. Second model: any Python file in the evaluation's folder contains the text get_model(, model_graded_qa or model_graded_fact; this marks code presence, not proof that every run calls it. Content from outside the repository: eval.yaml lists at least one external asset. Comparability changes: version N minus a base, where the base is the N of the changelog entry that records the February 2026 migration (dated 13 or 16 February), or 1 for evaluations created later. A cross-check that searches the Python code for the text sandbox=, sandbox( or SandboxEnvironmentSpec found 41 evaluations, with two names differing from the metadata list; other reasonable patterns give 39 or 40.

Price-list evidence comes from Amazon's Bedrock and SageMaker pricing pages, Bedrock documentation and Google's Vertex AI pricing page, checked on 27 October 2026; prices change, and no amounts are reproduced. Not measured: real run costs, vendor margins, or how often each evaluation is actually run. The script is kept with the working files for reviewers. No SourceX data is used.

Frequently asked questions

Is AI evaluation infrastructure a software business or a services business?

It depends on the layer. Workflow software that versions, runs and compares tests can scale like software when customers pay for their own model calls. Managed runs carry inference-like costs, and test content and human grading need recurring expert labor, which behaves like services.

What drives the marginal cost of running an AI evaluation?

The model under test is only the start. Many evaluations add a second model as a judge, grader or simulated user, and agentic or coding tasks add a container per sample. In a count of 129 open evaluations, 58.9 percent added one or both.

Why do open benchmarks lose value over time?

Public items can leak into training data, scoring code gets fixed, and hosted datasets move or disappear, all of which can break comparability. Benchmarks built to resist contamination, such as LiveBench, refresh questions regularly, which turns content into a recurring cost rather than a one-time build.

How do cloud providers price model evaluation?

Amazon bills SageMaker serverless evaluation on the tokens processed and charges per completed task for human evaluation, with custom pricing for its own managed expert team. Google prices model-based metrics as the prediction cost of the judge model and computation-based metrics per thousand characters. Both treat scoring as a metered extension of inference.

What should investors ask an evaluation platform about its gross margin?

Ask who pays for model, judge and sandbox calls, what share of scored samples ran on the customer's own keys, and how the human-graded share of revenue is moving. A margin earned while customers fund compute is not comparable with one that absorbs it.

Sources

  • In Inspect Evals at commit cc193dc (27 October 2026), 41 of 129 evaluations declare a sandbox, 45 contain a second-model call, 116 list external assets, and 76 of 111 evaluations migrated to versioning in February 2026 have a later comparability version. Source
  • Inspect Evals is maintained by Generality Labs, a London nonprofit, founded in 2024 with contributions from the UK AI Security Institute, Arcadia Impact and the Vector Institute; the repository is MIT licensed with copyright held by the UK AI Security Institute. Source
  • Inspect Evals' task versioning bumps the first number for changes that can make results incomparable, including bug fixes affecting results, scoring logic, dataset and prompt changes. Source
  • Inspect Evals' changelog for 4 June 2026 records the O-NET dataset repointed after the original repository became unavailable, with a comparability bump because byte-level equivalence could not be verified. Source
  • Langfuse's enterprise license describes an open core project whose core is MIT licensed while peripheral features are commercially licensed, with copyright held by ClickHouse, Inc. Source
  • Langfuse's README describes an open source platform to develop, monitor, evaluate and debug AI applications and says the project has been part of ClickHouse since January 2026. Source
  • Amazon Bedrock's pricing page charges a human-based model evaluation job for model inference tokens plus a charge per human task, one task per prompt-response set per worker. Source
  • Amazon SageMaker's pricing page bills serverless model evaluation on tokens processed, charges human evaluation with the customer's workforce per completed task with no workforce charge, and prices AWS-managed expert evaluation on a custom basis. Source
  • Bedrock documentation says an LLM-as-a-judge evaluation requires a generator model and an evaluator model and skips the model invoke step when the customer supplies its own responses. Source
  • Google's Vertex AI pricing says model-based evaluation metrics incur only the prediction costs of the autorater model, while computation-based metrics are priced per thousand characters. Source
  • The LiveBench paper describes a contamination-limited benchmark drawing on recent sources with questions added and updated monthly; its first version was titled contamination-free. Source

Request SourceX investor materials

SourceX shares company and commercial metrics with investors directly. Any offering is made only to verified accredited investors.

See if your company qualifies