Skip to content

Leadership and readiness

An AI benchmark investment committee template ending in three verdicts

By SourceX Editorial · Reviewed by Noah Loul ·

10 sections

Short answer

An AI benchmark investment committee template should end every score in one of three verdicts: credit it, discount it or keep it out of the memo. Nine questions cover the claim, task population, comparison, test exposure, grading, uncertainty, cost, reproduction and disclosure, each answered by a named artifact. A gap smaller than the test's noise floor counts as no difference: on HumanEval's 164 problems, about 11 points.

Evaluation split

AI benchmark investment

This page covers an AI benchmark investment committee template ending in three verdicts.Illustrative

Key takeaways

  • A benchmark score earns a place in a memo only after it is tied to the sentence in the investment case it supports; most scores support a narrower sentence than the pitch uses.
  • Nine template lines pair a question for the company with a required artifact, a red flag and an effect on the verdict; four of them are fatal when unanswered.
  • Noise floors grow quickly as tests shrink: at the worst case for two independent runs, about 12.7 points on ARC-AGI-2's 120 evaluation tasks, 10.8 on HumanEval's 164 problems and 3.8 on GSM8K's 1,319 test problems.
  • Regulators have acted on untested or loosely defined AI performance claims (the FTC's DoNotPay order, the SEC's Presto Automation order), and an audit of Chatbot Arena reports 27 private variants tested by one provider before a release.
  • Fatal gaps exclude a score, fixable gaps discount it, and credit requires item-level results checked by the committee's reviewer or an independent reproduction.

What does an investment committee need to decide about a benchmark score?

Whether the score supports a sentence in the investment case, and how much weight that sentence can bear. A committee rarely needs to know if a model is good in general. It needs to know whether this score, on this test, supports a claim the valuation leans on: higher accuracy on a buyer's task, a lead over a named rival, or proof that proprietary data made the difference.

The research record says to expect gaps between score and claim. BetterBench, a 2024 NeurIPS paper by Reuel and colleagues, assessed 24 benchmarks against 46 best practices and found large differences in quality, with most benchmarks neither reporting statistical significance nor allowing results to be easily replicated. A 2025 review of 445 language model benchmarks from leading conferences, by a 29-person team (Bean and colleagues), found that almost every paper had a weakness somewhere across how it defined the phenomenon, the task, the metric and the claims drawn from them.

Standards work is converging on a similar order. NIST's January 2026 initial public draft, NIST AI 800-2, groups voluntary practices for automated benchmark evaluations into three stages: defining the measurement target, implementing and running the evaluation, and analyzing and reporting results. The template here follows that order and adds the step a committee owns, deciding what the result is worth in the memo.

The thesis a reader can test: as capability claims move into pitch decks and valuations, the scarce input is evidence that survives diligence, not another leaderboard position. Companies that can hand over item-level results, pinned test conditions and an outside reproduction should be easier to underwrite than those that cannot, and the template is a way to see which kind a company is.

The nine-line benchmark diligence template

Each line pairs a question for the company with the artifact that answers it, the red flag that should stop the discussion and the line's effect on the verdict. Lines 1 to 4 decide whether the number means anything; lines 5 to 9 decide how much it is worth.

Ask for the artifacts before the meeting, not during it. A company that can produce them in a few days has usually built its evaluation to be checked; one that needs weeks is telling the committee something about how the score was made.

The nine-line benchmark diligence template
LineQuestion for the companyEvidence requiredRed flagIf unanswered
1 Claim mapWhich sentence in the case does this score support, for which buyer task?A written claim tied to the test's task description and the buyer workflowThe score is cited for a capability the test never measuresExclude
2 Task populationHow many items, drawn how, from which domains, dates and version?Item count, sampling rule, domain mix, collection dates, versionA percentage with no item count or versionExclude
3 Comparison held fixedWhat changed between the two arms, and what stayed constant?Run log naming model, prompt, tools, budget and data for both armsArms that differ in more than the factor being creditedExclude
4 Test exposureCould the system, its training data or its developers have seen the items?Decontamination method, post-cutoff or held-out items, access and scoring logRepeated tuning against the evaluation scoreExclude
5 GradingWho or what graded, against what reference, with what agreement?Rubric, grader type, agreement rates, human audit sampleThe company grades its own outputs with no auditDiscount
6 UncertaintyIs the gap larger than the test's noise floor?Item-level results, repeated runs, a paired intervalA gap below the floor presented as a winDiscount
7 Cost and attemptsWhat did each result cost in attempts, tokens, time and human help?Attempts allowed, token or compute log, human intervention rateBest of many attempts compared with a single attemptDiscount
8 ReproductionCan an outside party rerun it and get the same number?Code, environment and version pins, or a third-party rerunResults only the company can produceDiscount
9 DisclosureWhich runs, variants and tests went unreported?Every variant and test tried in the period, failures includedOnly the best variant or the best test appearsDiscount

How large must a gap be before it counts?

Larger than most decks assume. A score on a test of n items carries sampling error that shrinks only with the square root of n, so the committee should fix a noise floor for each test before reading any comparison on it.

We counted the items in four public test sets from their own repositories and computed the worst-case 95 percent floor, at 50 percent accuracy, for a single score and for the gap between two systems run independently. A paired comparison on the same items is usually tighter, so the last column shows the paired floor under an illustrative assumption that the two systems disagree on 20 percent of items. All floors are in percentage points.

A simulation check, two equally good systems answering each test 200,000 times with seed 20261027, produced gaps at or above the two-run floor in about 4.5 to 5.4 percent of trials, close to the 5 percent the formula implies. The rule that follows: a gap below the floor is recorded as no measured difference unless the company supplies item-level results for a paired test.

Scoring rules can change the effective count. ARC-AGI-2's readme counts a task as solved only when every test input in it is answered correctly, with two trials allowed per input, so line 7 should record attempts as well as items.

How large must a gap be before it counts?
Test set (repository)ItemsOne score, plus or minusGap between two independent runsPaired gap at 20% disagreement
ARC-AGI-2 public evaluation (arcprize)1208.912.78.0
HumanEval (OpenAI)1647.710.86.8
MBPP test split, task IDs 11 to 510 (Google Research)5004.46.23.9
GSM8K test (OpenAI)1,3192.73.82.4

The decision rule: credit, discount or exclude

Apply the lines in order and stop at the first fatal gap. A score is excluded from the memo if any of lines 1 to 4 lacks its artifact, because without a claim map, a task population, a controlled comparison and an answer on test exposure the number has no defined meaning.

A score is discounted if lines 1 to 4 pass but any of lines 5 to 9 fails. A discounted score may appear in the memo as context, labeled with its open lines, but it carries no weight in the valuation case and cannot support a claim about a data or model moat.

A score is credited only if all nine lines pass and the gap clears the noise floor, either on item-level results the committee's reviewer has checked or in an independent reproduction. Even then, write the claim narrowly: credited for this task population and this comparison, not for the product in general.

Every verdict enters the memo with its open questions listed as unresolved risks, so a later committee can see what was assumed and revisit it when new evidence arrives. The figure shows the gates in order.

A gate diagram in which a benchmark score passes four fatal checks that exit to Exclude, then five discount checks that exit to Discount, and reaches Credit only after clearing the noise floor with item-level results or an outside rerun.
The verdict rule: four fatal lines, five discount lines, then the noise floor.

Which red flags have regulators and auditors already named?

Three public matters map directly onto the template's red flags. Each is a reason to ask for the artifact rather than accept the summary, and none depends on the technical sophistication of the company involved.

  • No testing behind a capability claim (lines 1 and 5). Announcing Operation AI Comply in September 2024, the FTC alleged that DoNotPay had not tested whether its AI chatbot's output matched the level of a human lawyer and had not retained attorneys to test it. The final order, announced February 11, 2025, bars the company from claiming its service performs like a real lawyer without sufficient evidence.
  • A headline rate defined in the company's favor (lines 2 and 7). In January 2025 the SEC found that Presto Automation falsely claimed its own AI drive-thru product eliminated the need for human order-taking when the vast majority of orders through that version required human intervention, and that it misleadingly disclosed its rate of orders completed without human intervention. Presto settled without admitting or denying the findings.
  • Selective disclosure of variants (line 9). The Leaderboard Illusion audit of Chatbot Arena by Singh and colleagues (2025) reports that a few providers tested many variants privately and released the best, including 27 private variants tested by Meta before the Llama 4 release; the authors' simulations found that testing 10 variants raised the maximum score found by about 100 Arena points.

Was it the asset, or the model, prompt, budget or grader?

Assume the score moved for a reason other than the asset under review until a controlled comparison shows otherwise. Model version, prompt, tools, inference budget, attempt count and grader each move scores on their own, and a pitch that credits proprietary data, a training method or a new team has to show that the other factors held still.

When data is the claim, ask for a data ablation: the same model and training recipe with and without the dataset, scored on a test the data never touched, with several training seeds if the gap is small. When inference is involved, ask for matched budgets, the same attempts and tokens per item in both arms. Hold the grader fixed across arms and audit it against people on a sample.

Repeated tuning against an evaluation set is a quieter form of test exposure. ARC-AGI-2's own readme warns developers not to leak information from the evaluation set into an algorithm, for example by repeatedly modifying it while using its evaluation score as feedback. Line 4 should therefore ask how often the evaluation set was scored during development, not only whether its items appeared in training data.

Other analyses in this SourceX series work through individual lines on public data and can serve as worked examples: "A 4.9-point swing with no new data: underwriting benchmark performance as a data moat" for line 3, "Human grading and AI benchmark reliability: a recount of 51,200 grades" for line 5, and "Agent benchmark environment reproducibility: an 89-task audit for investors" for line 8.

Replication checklist for the committee's technical reviewer

A reviewer with a few days and the company's cooperation can check most of the template. Work through the steps in order and record each result in the memo's appendix, including the steps that could not be completed.

Checklist · 8 items

  1. Obtain the item list and its version, and confirm the item count matches the task population line.
  2. Obtain item-level results for both arms and recompute the headline score exactly.
  3. Compute the paired difference and its interval on shared items, and compare it with the noise floor for that test.
  4. Rerun a random sample of items under the company's pinned settings, with the sampling seed written in the appendix, and compare item by item.
  5. Swap the grader on a sample, people for a model or the reverse, and report agreement.
  6. Check attempt and token logs for equal budgets across arms.
  7. Request every variant and test tried in the reporting period and compare the reported one with the rest.
  8. List the lines that could not be checked and why; they become unresolved risks in the memo.

0 of 8 ticked. Tick each item as you go.

What could make this template mislead a committee?

Four failure modes are worth naming before the template is adopted, and a fifth limit applies to any benchmark review.

  • It can exclude real progress. A young company with a genuine advance may lack the logs to pass lines 3 and 4. The template then says the evidence is missing, not that the advance is false, and the memo should say exactly that.
  • It can be gamed. A company that knows the template can produce tidy artifacts for a narrow test while the commercial claim rests elsewhere. Line 1 is the defense, and it works only if the reviewer reads the buyer workflow as well as the benchmark.
  • A benchmark is not deployment. A credited score on a well-run test says nothing about adoption, reliability under real workloads or willingness to pay; those need customer evidence the template does not cover.
  • The noise floor assumes independent items. Items drawn in clusters, such as many tasks from one codebase or one customer, carry less information than their count suggests, so the true floor is higher than the table shows. Ask how the items cluster.
  • No template detects fabrication alone. The FTC and SEC matters above turned on facts outside any benchmark file, so diligence still needs customer calls, data room access and the usual verification.

Running the template on SourceX and other data suppliers

The template applies to SourceX exactly as it applies to any company claiming its data improves models. SourceX turns business records that a company selects into licensed AI datasets and handles the work between the two ends: qualifying the supplier, reviewing rights, contracting, preparing the data, delivering it and paying the company. There is no SourceX benchmark score in this article to put through the template; when any supplier, SourceX included, says a dataset improved a model, ask for lines 3, 4 and 6 first: the ablation, the exposure answer and the item-level results.

SourceX is raising from accredited investors under Rule 506(c) and shares investor materials on request. An investor evaluating it can request those materials through the investor page and bring a specific diligence question, such as how a dataset's effect on a model would be measured and which of the nine lines that measurement would satisfy.

Methods note: counts, simulation and the limits of this template

The template, the verdict rule and the red-flag mapping were written for this article on 2026-10-27 as editorial tools. They draw on BetterBench (2024), the construct-validity review by Bean and colleagues (2025), NIST's AI 800-2 initial public draft (January 2026), the Leaderboard Illusion audit (2025), the FTC's DoNotPay actions (2024 and 2025) and the SEC's Presto Automation order (January 2025).

Item counts come from each repository's own files: HumanEval as the non-empty lines of data/HumanEval.jsonl.gz at openai/human-eval commit b9453e8; GSM8K as the non-empty lines of grade_school_math/data/test.jsonl at openai/grade-school-math commit 3101c7d; ARC-AGI-2 as the JSON files in data/evaluation at arcprize/ARC-AGI-2 commit f3283f7; MBPP as the records with task IDs 11 to 510 in mbpp/mbpp.jsonl at google-research commit 700de96, the test split its readme specifies.

Floors use the normal approximation at 50 percent accuracy: 1.96 times the square root of 0.25/n for one score, of 0.5/n for two independent runs, and of d/n for a paired comparison in which the systems disagree on a share d of items, set to 0.20 for illustration. The simulation draws binomial scores from one numpy default generator, seed 20261027, 200,000 trials per test, taking the tests in the table's order and, for each test, all of system A's scores before system B's; in that draw order the shares are 4.5 to 5.3 percent, and other orders give about 4.5 to 5.4. The script reads data files only and is kept for reviewers.

Not measured: how often pitch decks or data rooms satisfy each line, how many claims each verdict would catch, and any SourceX result. The template has not been validated against investment outcomes.

Frequently asked questions

What should an AI benchmark investment committee template include?

A claim map, the task population, the comparison held fixed, test exposure, grading, uncertainty, cost and attempts, reproduction and disclosure, each paired with the artifact that answers it and a red flag. It should end in a verdict, credit, discount or exclude, so the memo records what the score is worth and what remains unresolved.

How many test items does a benchmark need before a score gap matters?

It depends on the gap you need to detect. At 50 percent accuracy, the worst case for two independent runs, the 95 percent floor is about 10.8 points for 164 items and 3.8 points for 1,319 items. Paired, item-level results on the same items usually lower the floor, which is why the template asks for them.

Can a private benchmark pass this template?

Yes, if the company can still answer each line: an item count and sampling rule, a controlled comparison, an exposure answer and a reproduction by a trusted third party who can see the items. Secrecy changes who can check the work; it does not remove the need for the artifacts.

What is a data ablation, and when should investors ask for one?

A data ablation trains or tunes the same model with the same recipe twice, once with the dataset and once without, and scores both on a test the data never touched. Ask for one whenever a company credits its data for a gain, because model, prompt and budget changes can produce similar gains on their own.

Does a credited benchmark score justify a higher valuation?

Not by itself. A credited score supports a narrow claim about one task population and one comparison. Valuation still rests on customer evidence, adoption, retention and economics, which a benchmark does not measure.

Sources

  • BetterBench (2024) assessed 24 AI benchmarks against 46 best practices and found most did not report statistical significance or allow easy replication. Source
  • A 2025 review of 445 LLM benchmarks by a 29-person team found weaknesses in nearly every paper across phenomena, tasks, metrics and claims. Source
  • NIST's January 2026 draft, NIST AI 800-2, organizes automated benchmark evaluation practices into defining the measurement target, running the evaluation, and analyzing and reporting results. Source
  • The Leaderboard Illusion reports undisclosed private testing of model variants on Chatbot Arena, including 27 private variants tested by Meta before Llama 4. Source
  • In September 2024 the FTC alleged DoNotPay had not tested whether its AI chatbot's output matched a human lawyer's. Source
  • The FTC's final DoNotPay order, announced February 11, 2025, bars claims that the service performs like a real lawyer without sufficient evidence. Source
  • The SEC found in January 2025 that Presto Automation made false claims about its AI product removing the need for human order-taking and misleadingly disclosed its non-intervention rate. Source
  • ARC-AGI-2 has 120 public evaluation tasks, counts a task as solved only when all test inputs are correct with two trials each, and warns against tuning on the evaluation score. Source
  • OpenAI's HumanEval data file holds 164 problems (SourceX count). Source
  • MBPP's readme specifies task IDs 11 to 510, 500 problems, as the test split. Source
  • GSM8K's test file holds 1,319 problems (SourceX count). Source

Request SourceX investor materials

SourceX shares company and commercial metrics with investors directly. Any offering is made only to verified accredited investors.

See if your company qualifies