Leadership and readiness
Private AI benchmark credibility depends on custody, not secrecy
By SourceX Editorial · Reviewed by Noah Loul ·
10 sections
Short answer
A private AI benchmark can be independently credible when someone other than the system's builder holds the test items, runs the scoring and rations access, and when a written method and a public sample let outsiders check the design. Secrecy alone proves nothing: on 100 hidden tasks, reporting the best of 20 equally good variants lifts the expected score by about nine points with no real gain.
Evaluation split
Private AI benchmark
Key takeaways
- Secrecy shields a test set from contamination but removes outside checking, so credibility moves to whoever holds the items and runs the scoring.
- Mature designs split custody: NIST runs submitted algorithms on sequestered data, and MLCommons keeps 12,000 AILuminate prompts private while publishing practice and demo sets.
- Selection is the quiet risk: by exact calculation, the best of 20 identical variants on 100 hidden tasks scores 9.0 points above its true rate, and 2.9 points on 1,000 tasks.
- Conflicts are disclosed, not absent: Epoch AI says OpenAI owns the core FrontierMath problems and sees all but a holdout, and private curators can depend on the firms they rank.
- Before a private score enters a memo, ask for the custodian, the run log, a public sample built the same way and a rerun or fresh holdout by someone independent.
When is a private AI benchmark independently credible?
When the score could have come out differently without the builder being able to steer it. That takes four things: custody of the test items by someone other than the team being judged, scoring run by that custodian or another independent party, rationed access so nobody can tune against the hidden set, and a published method with a public sample so outsiders can judge whether the test measures work that matters.
Private AI benchmark credibility has become an investor question because hidden scores now appear in fundraising decks, launch posts and data-licensing pitches. A company that reports a lead on its internal benchmark is asking readers to trust an instrument none of them can inspect. The productive response is neither to dismiss the number nor to repeat it, but to ask who held the key and who turned it.
This page compares the arrangements that make hidden scores checkable, computes how far selection alone can lift one, and turns both into a worksheet and a reproduction checklist for diligence. Why teams build private evaluation sets at all, and how to restrict access before a launch, are covered in other SourceX guides linked below.
What secrecy buys a benchmark, and what it gives up
Secrecy buys protection from contamination: items a model never saw in training measure skill rather than recall. It gives up an outsider's ability to inspect the items, the answer key and the grading. A public benchmark makes the opposite trade, which is why neither kind settles a question alone.
Kaggle's competition design shows the trade cleanly. Its public leaderboard, visible during a competition, is based on a sample of the test data; the private leaderboard, computed on the remainder, sets the final ranking, and Kaggle warns entrants that a high public score does not ensure a high private one and advises against chasing the public board.
The deeper problem is reuse. Dwork and colleagues described adaptive data analysis, in which each new analysis is chosen after seeing earlier results on the same data, and proposed a holdout that can be reused many times while keeping its validity (Science, 2015). A private benchmark that the team it judges can query hundreds of times is the situation they set out to fix, not a safe default.
Six arrangements that keep test items hidden and scores checkable
MLCommons publishes enough to check the design without exposing the test. Its AILuminate v1.0 benchmark holds 24,000 human-written prompts, 12,000 kept private for official benchmarking and 12,000 offered to members as practice prompts it describes as statistically nearly identical to the official set, plus a public demo file that is a 10 percent sample of the practice set. Our count of the English (en_US) demo file at commit 769cc2b finds 1,200 prompts: 100 in each of 11 hazard categories, 100 spread over four specialized-advice categories, and an even split of 600 skilled and 600 unskilled personas.
The table compares six documented arrangements on who holds the hidden items, who scores, what an outsider can check and where each is weakest. None is perfect, and each offers a control a company's own private benchmark can borrow.
| Arrangement | Who holds the hidden items | Who runs the scoring | What an outsider can check | Weak point |
|---|---|---|---|---|
| NIST face recognition evaluation (FRTE) | NIST, on sequestered datasets | NIST runs algorithms developers submit through its published API | Public reports and the published API | Developers choose what and when to submit |
| MLCommons AILuminate v1.0 | MLCommons: 12,000 private prompts | MLCommons, with a tuned ensemble of evaluator models | Practice prompts for members, 1,200 public demo prompts, the hazard taxonomy | Grading by models, not people |
| Kaggle competitions | The platform: the private share of the test data | The platform, with the private ranking revealed at the close | Public leaderboard on a sample; final private ranking | The visible board invites chasing |
| ARC Prize verified scores | ARC Prize: semi-private and private task sets | ARC Prize verifies on the semi-private set | Public tasks and the public versus semi-private gap | Leakage risk kept verification off the private set |
| Epoch AI FrontierMath | Epoch, under OpenAI's ownership of the core problems | Epoch | Published methods and holdout disclosures | The funder can see most problems |
| Scale SEAL leaderboards | Scale's curated private datasets | Scale, with expert assessors | Rankings and per-leaderboard methods | Independence rests on the curator's own description |
How far can choosing the best run lift a hidden score?
Further than most readers expect, and the lift is invisible from outside. If a team tries several variants of a system on its private set, such as different prompts, seeds, sampling settings or data mixes, and reports the best, the reported score rises even when every variant is equally good. We computed the expected lift exactly, assuming each variant passes each task independently at the same true rate of 60 percent.
On 100 hidden tasks, the best of 5 variants lands 5.7 points above the true rate and the best of 20 lands 9.0 points above it, close to the 9.6-point half-width of a 95 percent interval for a single run on that many tasks. On 1,000 tasks the best of 20 still gains 2.9 points. Real variants fail many of the same tasks, so these figures are a ceiling, but the direction holds: the more tries and the fewer tasks, the more of a headline score is selection.
This is the strongest alternative explanation for any private gain, put in numbers. A rise credited to new data can come from a changed model, a new prompt or a larger inference budget, and it can also come from keeping the luckiest of many runs. Only the custodian's log of how many runs were scored can rule selection out.
The lift columns give the expected gain in percentage points, and the first column gives one run's noise band for comparison; the calculation and script are described in the last section.
| Hidden tasks | One run: 95% interval, +/- points | Best of 5: lift in points | Best of 20: lift in points | Best of 50: lift in points |
|---|---|---|---|---|
| 50 | 13.6 | 8.0 | 12.7 | 15.2 |
| 100 | 9.6 | 5.7 | 9.0 | 10.8 |
| 300 | 5.5 | 3.3 | 5.2 | 6.3 |
| 1,000 | 3.0 | 1.8 | 2.9 | 3.5 |
A worksheet for underwriting a score nobody else can see
Score each line from documents, not from the company's description of them. A claim that fails custody or scoring cannot be independently credible however strong the rest looks, and a claim that passes every line still supports only the comparison it was designed to make.
| Line | What to request | Credible answer | Red flag |
|---|---|---|---|
| Custody | Who holds the items and answer key, and who can open them | A named party outside the team being judged, with an access list | Engineers on the model team can read the files |
| Scoring | Who ran the scoring that produced the claim, and where | The custodian ran it, or an outside party reran it | Numbers pasted from the builder's own notebook |
| Run count | Every scored run on the hidden set in the claim period, dated | A short log, or the selection lift accounted for | Only the best run survives |
| Public sample | A sample built the same way as the hidden set | Released with the same strata and grading | No sample, or one drawn differently |
| Method | Task population, grading rule, inference budget, model and prompt versions | Written and versioned before results | Method drafted after the result |
| Comparison | Baselines run under the same harness and budget | Same tasks, budget and grader for every system | A rival's number copied from another setup |
| Conflicts | Funding, ownership and data-supply links between evaluator and evaluated | Disclosed in writing, with any holdout terms | Sponsor access discovered later |
| Refresh | Plan for retiring exposed items and adding new ones | Dated refreshes, with scores on fresh items | One set used for years with no fresh check |
Five ways to reproduce a score without seeing the test set
Reproduction does not require publishing the items. Each option keeps the hidden set hidden and still lets someone other than the builder produce or check the number.
At a glance
Five ways to reproduce a score without seeing the test set
- Custodian rerun: the company submits the system, or access to it, and the custodian scores it on the hidden set and reports directly to the investor. NIST's face recognition program works this way, and ARC Prize verified its 2024 competition submissions on a 100-task semi-private set.
- Parity check: compare the score on the public sample with the score on the hidden set. ARC Prize's 2024 technical report treated a gap of more than 10 percentage points between semi-private and public results as a sign of overfitting.
- Fresh holdout: the custodian writes or sources new items after the claim and scores the system once. A score that holds on items that did not exist while the system was tuned is the strongest evidence a private benchmark can offer.
- Blind regrade: an outside reviewer grades a random sample of the system's outputs without knowing which system produced them, and agreement with the official grades is reported.
- Confidential audit: an independent reviewer under a nondisclosure agreement inspects the items, answer key, run log and harness, then signs a short report of what was checked and what was not.
Was it the data, the model, the prompt or the budget?
A private gain says nothing about its cause until the comparison isolates one change. For a claim that licensed data improved a model, that means the same base model, prompt, harness and inference budget, with only the training data different, run more than once on the hidden set by the custodian.
Each competing cause leaves a trace an investor can ask for: a model version in the run log, a prompt file hash, a token or compute budget per task, and the training data manifest. If any of them changed between the before and after runs, the gain is shared among the changes and the data's part is unknown. A private benchmark makes this harder to check, not less necessary.
What could make a well-run private benchmark mislead
Even a well-designed private benchmark can mislead. These are the risks that would make the credibility test above too generous, or too harsh.
- Conflicted custodians. Epoch AI says OpenAI commissioned and owns the core FrontierMath problems and can see problems and solutions apart from a holdout, terms Epoch acknowledged it had not explained clearly at first; its own description now says that exclusive access has reduced confidence in FrontierMath results for OpenAI models.
- Curators with other business. Bansal and Maini argue that private evaluators can depend on the firms they evaluate as clients, and that expert annotators' preferences can tilt results toward models trained on the curator's own data (2025). The argument applies to any curator that also sells data or services to the developers it ranks. Scale, for instance, describes its SEAL leaderboards as using curated private datasets that cannot be gamed, a claim about design that outside reproduction would have to confirm.
- Model graders. AILuminate scores responses with an ensemble of evaluator models. A private benchmark graded by models inherits their errors, and outsiders cannot audit those errors without the items.
- Over-discounting. The selection table is a ceiling. Roelofs and colleagues found little evidence of substantial overfitting to holdout sets across more than 100 Kaggle competitions (NeurIPS 2019), so a carefully reused private set can stay informative.
- Claims regulators test. In April 2025 the FTC announced an order requiring Workado to back up its AI detection claims; it alleged that an advertised 98 percent accuracy was unsupported when independent testing on general-purpose content found 53 percent. An internal score is not that support unless someone independent can stand behind it.
- Wrong tasks. A credible private score on work unlike the buyer's is still weak evidence for the buyer's case; the task population has to match the decision.
How the worksheet applies to a data supplier such as SourceX
The worksheet applies to any company that cites hidden evaluations, including suppliers of training and evaluation data. SourceX runs sourcing, rights review, contracting, preparation, delivery and payment for companies that license selected business records to AI developers. This page reports no SourceX evaluation results, and SourceX shares its company and commercial metrics with investors directly rather than publicly.
An investor looking at SourceX, or at any data supplier, can put the same eight lines to every evaluation a buyer ran on licensed data: who held the test items, who scored them, how many runs were tried and whether a fresh holdout confirmed the result. SourceX is raising from accredited investors under Rule 506(c), and its investor materials are available on request through the investors page, where a specific question about evaluation evidence is a reasonable place to start.
How this analysis was built, and where it stops
Sources: the arrangements in the comparison table come from each operator's own documentation, reports and announcements, checked on October 27, 2026 through searches limited to their domains, and from MLCommons' AILuminate repository read directly at commit 769cc2b. The table describes stated designs; we audited none of them.
Selection calculation: for n hidden tasks and K variants that each pass every task independently with probability 0.60, the expected best score is the sum over j from 0 to n minus 1 of one minus F(j) raised to the power K, divided by n, where F is the binomial distribution function. Values are exact, not simulated, so no random seed is involved. The single-run interval is 1.96 times the square root of 0.24 divided by n. The script, private_bench_math.py, is kept with this draft's working files.
AILuminate count: rows of the English (en_US) demo prompt file grouped by its hazard and persona columns; prompt text was not read. Not measured: how often real companies rerun their private benchmarks, the actual correlation between variants, or the accuracy of any operator's grading.
Frequently asked questions
Why do AI developers keep some benchmarks private?
Mainly to keep test items out of training data, so a score measures skill rather than recall. Kaggle's private leaderboard and MLCommons' 12,000 private AILuminate prompts work this way. The cost is that outsiders cannot inspect the items, so credibility shifts to whoever holds and scores them.
Can a third party verify a benchmark score without seeing the test set?
Yes, if the custodian reruns the system or an independent reviewer audits the items under confidentiality. NIST runs submitted face recognition algorithms on data developers never receive, and ARC Prize verifies scores on a semi-private set. A comparison of public-sample and hidden-set scores adds a further check.
How many tasks does a private benchmark need?
Enough that the claimed gap is larger than both sampling noise and selection. On 100 tasks at a 60 percent pass rate, one run's 95 percent interval is about plus or minus 9.6 points, and the best of 20 equal variants gains about 9 points by chance. At 1,000 tasks those figures shrink to about 3 points.
Is a leaderboard run by a data vendor independent?
Not automatically. A vendor that also sells training data or services to the firms it ranks has a conflict that researchers have flagged, even when its process is careful. Ask for written disclosure of those relationships, access rules for the test items and whether any outside party has reproduced the scores.
Does SourceX publish evaluation results for its datasets?
This page reports none, and SourceX provides company and commercial metrics to investors directly rather than publicly. Investors can request materials at sourcex.si/investors and ask how any evaluation of licensed data was run, using the worksheet above. Any offering is open only to accredited investors whose status has been verified.
Sources
- NIST's FRTE 1:1 evaluation is ongoing, accepts algorithm submissions at any time and runs algorithms that implement NIST's published API. Source
- NIST describes its face recognition benchmarks as independent and black box, with sequestered datasets. Source
- MLCommons' AILuminate v1.0 has 12,000 private and 12,000 practice prompts and a public 1,200-prompt demo file, scored with a tuned ensemble of evaluator models. Source
- Kaggle's public leaderboard uses a sample of test data, the private leaderboard the remainder for final ranking, and Kaggle warns against chasing the public board. Source
- ARC Prize's 2024 report says submissions were verified on a 100-task semi-private set because of leakage risk, and treats a gap above 10 points versus public results as overfitting. Source
- Epoch AI says OpenAI commissioned and owns the core FrontierMath problems and can see problems and solutions apart from a holdout set. Source
- Epoch AI says OpenAI's exclusive access to FrontierMath has diminished confidence in results for OpenAI models. Source
- Scale describes SEAL Leaderboards as ranking frontier models with curated private datasets assessed by verified domain experts. Source
- Bansal and Maini argue that private evaluators face conflicts of interest and annotator bias toward models trained on their own data. Source
- Dwork and colleagues proposed a reusable holdout that keeps validity under adaptive data analysis (Science, 2015). Source
- Roelofs and colleagues found little evidence of substantial overfitting to holdout sets across more than 100 Kaggle competitions. Source
- The FTC's April 2025 Workado order concerned an AI detection accuracy claim the agency alleged was unsupported by competent evidence. Source
Request SourceX investor materials
SourceX shares company and commercial metrics with investors directly. Any offering is made only to verified accredited investors.