Skip to content

Leadership and readiness

Private AI benchmark credibility depends on custody, not secrecy

By SourceX Editorial · Reviewed by Noah Loul ·

10 sections

Short answer

A private AI benchmark can be independently credible when someone other than the system's builder holds the test items, runs the scoring and rations access, and when a written method and a public sample let outsiders check the design. Secrecy alone proves nothing: on 100 hidden tasks, reporting the best of 20 equally good variants lifts the expected score by about nine points with no real gain.

Evaluation split

Private AI benchmark

This page covers private AI benchmark credibility depends on custody, not secrecy.Illustrative

Key takeaways

  • Secrecy shields a test set from contamination but removes outside checking, so credibility moves to whoever holds the items and runs the scoring.
  • Mature designs split custody: NIST runs submitted algorithms on sequestered data, and MLCommons keeps 12,000 AILuminate prompts private while publishing practice and demo sets.
  • Selection is the quiet risk: by exact calculation, the best of 20 identical variants on 100 hidden tasks scores 9.0 points above its true rate, and 2.9 points on 1,000 tasks.
  • Conflicts are disclosed, not absent: Epoch AI says OpenAI owns the core FrontierMath problems and sees all but a holdout, and private curators can depend on the firms they rank.
  • Before a private score enters a memo, ask for the custodian, the run log, a public sample built the same way and a rerun or fresh holdout by someone independent.

When is a private AI benchmark independently credible?

When the score could have come out differently without the builder being able to steer it. That takes four things: custody of the test items by someone other than the team being judged, scoring run by that custodian or another independent party, rationed access so nobody can tune against the hidden set, and a published method with a public sample so outsiders can judge whether the test measures work that matters.

Private AI benchmark credibility has become an investor question because hidden scores now appear in fundraising decks, launch posts and data-licensing pitches. A company that reports a lead on its internal benchmark is asking readers to trust an instrument none of them can inspect. The productive response is neither to dismiss the number nor to repeat it, but to ask who held the key and who turned it.

This page compares the arrangements that make hidden scores checkable, computes how far selection alone can lift one, and turns both into a worksheet and a reproduction checklist for diligence. Why teams build private evaluation sets at all, and how to restrict access before a launch, are covered in other SourceX guides linked below.

What secrecy buys a benchmark, and what it gives up

Secrecy buys protection from contamination: items a model never saw in training measure skill rather than recall. It gives up an outsider's ability to inspect the items, the answer key and the grading. A public benchmark makes the opposite trade, which is why neither kind settles a question alone.

Kaggle's competition design shows the trade cleanly. Its public leaderboard, visible during a competition, is based on a sample of the test data; the private leaderboard, computed on the remainder, sets the final ranking, and Kaggle warns entrants that a high public score does not ensure a high private one and advises against chasing the public board.

The deeper problem is reuse. Dwork and colleagues described adaptive data analysis, in which each new analysis is chosen after seeing earlier results on the same data, and proposed a holdout that can be reused many times while keeping its validity (Science, 2015). A private benchmark that the team it judges can query hundreds of times is the situation they set out to fix, not a safe default.

Six arrangements that keep test items hidden and scores checkable

MLCommons publishes enough to check the design without exposing the test. Its AILuminate v1.0 benchmark holds 24,000 human-written prompts, 12,000 kept private for official benchmarking and 12,000 offered to members as practice prompts it describes as statistically nearly identical to the official set, plus a public demo file that is a 10 percent sample of the practice set. Our count of the English (en_US) demo file at commit 769cc2b finds 1,200 prompts: 100 in each of 11 hazard categories, 100 spread over four specialized-advice categories, and an even split of 600 skilled and 600 unskilled personas.

The table compares six documented arrangements on who holds the hidden items, who scores, what an outsider can check and where each is weakest. None is perfect, and each offers a control a company's own private benchmark can borrow.

Six arrangements that keep test items hidden and scores checkable
ArrangementWho holds the hidden itemsWho runs the scoringWhat an outsider can checkWeak point
NIST face recognition evaluation (FRTE)NIST, on sequestered datasetsNIST runs algorithms developers submit through its published APIPublic reports and the published APIDevelopers choose what and when to submit
MLCommons AILuminate v1.0MLCommons: 12,000 private promptsMLCommons, with a tuned ensemble of evaluator modelsPractice prompts for members, 1,200 public demo prompts, the hazard taxonomyGrading by models, not people
Kaggle competitionsThe platform: the private share of the test dataThe platform, with the private ranking revealed at the closePublic leaderboard on a sample; final private rankingThe visible board invites chasing
ARC Prize verified scoresARC Prize: semi-private and private task setsARC Prize verifies on the semi-private setPublic tasks and the public versus semi-private gapLeakage risk kept verification off the private set
Epoch AI FrontierMathEpoch, under OpenAI's ownership of the core problemsEpochPublished methods and holdout disclosuresThe funder can see most problems
Scale SEAL leaderboardsScale's curated private datasetsScale, with expert assessorsRankings and per-leaderboard methodsIndependence rests on the curator's own description

How far can choosing the best run lift a hidden score?

Further than most readers expect, and the lift is invisible from outside. If a team tries several variants of a system on its private set, such as different prompts, seeds, sampling settings or data mixes, and reports the best, the reported score rises even when every variant is equally good. We computed the expected lift exactly, assuming each variant passes each task independently at the same true rate of 60 percent.

On 100 hidden tasks, the best of 5 variants lands 5.7 points above the true rate and the best of 20 lands 9.0 points above it, close to the 9.6-point half-width of a 95 percent interval for a single run on that many tasks. On 1,000 tasks the best of 20 still gains 2.9 points. Real variants fail many of the same tasks, so these figures are a ceiling, but the direction holds: the more tries and the fewer tasks, the more of a headline score is selection.

This is the strongest alternative explanation for any private gain, put in numbers. A rise credited to new data can come from a changed model, a new prompt or a larger inference budget, and it can also come from keeping the luckiest of many runs. Only the custodian's log of how many runs were scored can rule selection out.

The lift columns give the expected gain in percentage points, and the first column gives one run's noise band for comparison; the calculation and script are described in the last section.

How far can choosing the best run lift a hidden score?
Hidden tasksOne run: 95% interval, +/- pointsBest of 5: lift in pointsBest of 20: lift in pointsBest of 50: lift in points
5013.68.012.715.2
1009.65.79.010.8
3005.53.35.26.3
1,0003.01.82.93.5
A heat map of expected score lift from reporting the best of 5, 20 or 50 equally good variants on 50, 100, 300 or 1,000 hidden tasks, from 1.8 points at 1,000 tasks and 5 tries to 15.2 points at 50 tasks and 50 tries, with 9.0 points at 100 tasks and 20 tries highlighted.
Expected lift from selection alone when every variant's true pass rate is 60 percent. Exact binomial calculation, SourceX, October 2026.

A worksheet for underwriting a score nobody else can see

Score each line from documents, not from the company's description of them. A claim that fails custody or scoring cannot be independently credible however strong the rest looks, and a claim that passes every line still supports only the comparison it was designed to make.

A worksheet for underwriting a score nobody else can see
LineWhat to requestCredible answerRed flag
CustodyWho holds the items and answer key, and who can open themA named party outside the team being judged, with an access listEngineers on the model team can read the files
ScoringWho ran the scoring that produced the claim, and whereThe custodian ran it, or an outside party reran itNumbers pasted from the builder's own notebook
Run countEvery scored run on the hidden set in the claim period, datedA short log, or the selection lift accounted forOnly the best run survives
Public sampleA sample built the same way as the hidden setReleased with the same strata and gradingNo sample, or one drawn differently
MethodTask population, grading rule, inference budget, model and prompt versionsWritten and versioned before resultsMethod drafted after the result
ComparisonBaselines run under the same harness and budgetSame tasks, budget and grader for every systemA rival's number copied from another setup
ConflictsFunding, ownership and data-supply links between evaluator and evaluatedDisclosed in writing, with any holdout termsSponsor access discovered later
RefreshPlan for retiring exposed items and adding new onesDated refreshes, with scores on fresh itemsOne set used for years with no fresh check

Five ways to reproduce a score without seeing the test set

Reproduction does not require publishing the items. Each option keeps the hidden set hidden and still lets someone other than the builder produce or check the number.

At a glance

Five ways to reproduce a score without seeing the test set

  • Custodian rerun: the company submits the system, or access to it, and the custodian scores it on the hidden set and reports directly to the investor. NIST's face recognition program works this way, and ARC Prize verified its 2024 competition submissions on a 100-task semi-private set.
  • Parity check: compare the score on the public sample with the score on the hidden set. ARC Prize's 2024 technical report treated a gap of more than 10 percentage points between semi-private and public results as a sign of overfitting.
  • Fresh holdout: the custodian writes or sources new items after the claim and scores the system once. A score that holds on items that did not exist while the system was tuned is the strongest evidence a private benchmark can offer.
  • Blind regrade: an outside reviewer grades a random sample of the system's outputs without knowing which system produced them, and agreement with the official grades is reported.
  • Confidential audit: an independent reviewer under a nondisclosure agreement inspects the items, answer key, run log and harness, then signs a short report of what was checked and what was not.

Was it the data, the model, the prompt or the budget?

A private gain says nothing about its cause until the comparison isolates one change. For a claim that licensed data improved a model, that means the same base model, prompt, harness and inference budget, with only the training data different, run more than once on the hidden set by the custodian.

Each competing cause leaves a trace an investor can ask for: a model version in the run log, a prompt file hash, a token or compute budget per task, and the training data manifest. If any of them changed between the before and after runs, the gain is shared among the changes and the data's part is unknown. A private benchmark makes this harder to check, not less necessary.

What could make a well-run private benchmark mislead

Even a well-designed private benchmark can mislead. These are the risks that would make the credibility test above too generous, or too harsh.

  • Conflicted custodians. Epoch AI says OpenAI commissioned and owns the core FrontierMath problems and can see problems and solutions apart from a holdout, terms Epoch acknowledged it had not explained clearly at first; its own description now says that exclusive access has reduced confidence in FrontierMath results for OpenAI models.
  • Curators with other business. Bansal and Maini argue that private evaluators can depend on the firms they evaluate as clients, and that expert annotators' preferences can tilt results toward models trained on the curator's own data (2025). The argument applies to any curator that also sells data or services to the developers it ranks. Scale, for instance, describes its SEAL leaderboards as using curated private datasets that cannot be gamed, a claim about design that outside reproduction would have to confirm.
  • Model graders. AILuminate scores responses with an ensemble of evaluator models. A private benchmark graded by models inherits their errors, and outsiders cannot audit those errors without the items.
  • Over-discounting. The selection table is a ceiling. Roelofs and colleagues found little evidence of substantial overfitting to holdout sets across more than 100 Kaggle competitions (NeurIPS 2019), so a carefully reused private set can stay informative.
  • Claims regulators test. In April 2025 the FTC announced an order requiring Workado to back up its AI detection claims; it alleged that an advertised 98 percent accuracy was unsupported when independent testing on general-purpose content found 53 percent. An internal score is not that support unless someone independent can stand behind it.
  • Wrong tasks. A credible private score on work unlike the buyer's is still weak evidence for the buyer's case; the task population has to match the decision.

How the worksheet applies to a data supplier such as SourceX

The worksheet applies to any company that cites hidden evaluations, including suppliers of training and evaluation data. SourceX runs sourcing, rights review, contracting, preparation, delivery and payment for companies that license selected business records to AI developers. This page reports no SourceX evaluation results, and SourceX shares its company and commercial metrics with investors directly rather than publicly.

An investor looking at SourceX, or at any data supplier, can put the same eight lines to every evaluation a buyer ran on licensed data: who held the test items, who scored them, how many runs were tried and whether a fresh holdout confirmed the result. SourceX is raising from accredited investors under Rule 506(c), and its investor materials are available on request through the investors page, where a specific question about evaluation evidence is a reasonable place to start.

How this analysis was built, and where it stops

Sources: the arrangements in the comparison table come from each operator's own documentation, reports and announcements, checked on October 27, 2026 through searches limited to their domains, and from MLCommons' AILuminate repository read directly at commit 769cc2b. The table describes stated designs; we audited none of them.

Selection calculation: for n hidden tasks and K variants that each pass every task independently with probability 0.60, the expected best score is the sum over j from 0 to n minus 1 of one minus F(j) raised to the power K, divided by n, where F is the binomial distribution function. Values are exact, not simulated, so no random seed is involved. The single-run interval is 1.96 times the square root of 0.24 divided by n. The script, private_bench_math.py, is kept with this draft's working files.

AILuminate count: rows of the English (en_US) demo prompt file grouped by its hazard and persona columns; prompt text was not read. Not measured: how often real companies rerun their private benchmarks, the actual correlation between variants, or the accuracy of any operator's grading.

Frequently asked questions

Why do AI developers keep some benchmarks private?

Mainly to keep test items out of training data, so a score measures skill rather than recall. Kaggle's private leaderboard and MLCommons' 12,000 private AILuminate prompts work this way. The cost is that outsiders cannot inspect the items, so credibility shifts to whoever holds and scores them.

Can a third party verify a benchmark score without seeing the test set?

Yes, if the custodian reruns the system or an independent reviewer audits the items under confidentiality. NIST runs submitted face recognition algorithms on data developers never receive, and ARC Prize verifies scores on a semi-private set. A comparison of public-sample and hidden-set scores adds a further check.

How many tasks does a private benchmark need?

Enough that the claimed gap is larger than both sampling noise and selection. On 100 tasks at a 60 percent pass rate, one run's 95 percent interval is about plus or minus 9.6 points, and the best of 20 equal variants gains about 9 points by chance. At 1,000 tasks those figures shrink to about 3 points.

Is a leaderboard run by a data vendor independent?

Not automatically. A vendor that also sells training data or services to the firms it ranks has a conflict that researchers have flagged, even when its process is careful. Ask for written disclosure of those relationships, access rules for the test items and whether any outside party has reproduced the scores.

Does SourceX publish evaluation results for its datasets?

This page reports none, and SourceX provides company and commercial metrics to investors directly rather than publicly. Investors can request materials at sourcex.si/investors and ask how any evaluation of licensed data was run, using the worksheet above. Any offering is open only to accredited investors whose status has been verified.

Sources

  • NIST's FRTE 1:1 evaluation is ongoing, accepts algorithm submissions at any time and runs algorithms that implement NIST's published API. Source
  • NIST describes its face recognition benchmarks as independent and black box, with sequestered datasets. Source
  • MLCommons' AILuminate v1.0 has 12,000 private and 12,000 practice prompts and a public 1,200-prompt demo file, scored with a tuned ensemble of evaluator models. Source
  • Kaggle's public leaderboard uses a sample of test data, the private leaderboard the remainder for final ranking, and Kaggle warns against chasing the public board. Source
  • ARC Prize's 2024 report says submissions were verified on a 100-task semi-private set because of leakage risk, and treats a gap above 10 points versus public results as overfitting. Source
  • Epoch AI says OpenAI commissioned and owns the core FrontierMath problems and can see problems and solutions apart from a holdout set. Source
  • Epoch AI says OpenAI's exclusive access to FrontierMath has diminished confidence in results for OpenAI models. Source
  • Scale describes SEAL Leaderboards as ranking frontier models with curated private datasets assessed by verified domain experts. Source
  • Bansal and Maini argue that private evaluators face conflicts of interest and annotator bias toward models trained on their own data. Source
  • Dwork and colleagues proposed a reusable holdout that keeps validity under adaptive data analysis (Science, 2015). Source
  • Roelofs and colleagues found little evidence of substantial overfitting to holdout sets across more than 100 Kaggle competitions. Source
  • The FTC's April 2025 Workado order concerned an AI detection accuracy claim the agency alleged was unsupported by competent evidence. Source

Request SourceX investor materials

SourceX shares company and commercial metrics with investors directly. Any offering is made only to verified accredited investors.

See if your company qualifies