Skip to content

Leadership and readiness

Why AI benchmark baseline selection can halve a field of winners

By SourceX Editorial · Reviewed by Noah Loul ·

10 sections

Short answer

AI benchmark baseline selection decides which story a gain can support. In timm's public ImageNet results, 140 of 146 similarly sized models beat ResNet-50 as torchvision first trained it, but only 69 beat the same architecture on torchvision's newer recipe, and the gain from pretraining on 940 million outside images shrank from 5.0 points to 0.3. Underwrite a claim at the strongest baseline it survives, not the one it names.

Evaluation split

AI benchmark baseline

This page covers why AI benchmark baseline selection can halve a field of winners.Illustrative

Key takeaways

  • A benchmark gain means nothing commercial until the baseline is named by version, date, recipe and evaluation settings.
  • In timm's ImageNet results, 95.9% of similar-size ImageNet-1k models beat torchvision's original ResNet-50 and 47.3% beat its newer recipe.
  • Outside-data ResNet-50s gained 5.0 and 3.1 points over the old baseline but 0.3 and minus 1.6 over the retrained one.
  • Published reversals in medicine and forecasting show prompted general models and simple linear models overturning specialist narratives.
  • Credit each claim only with the narrative of the highest rung it clears on a six-rung baseline ladder, from stale incumbent to incumbent process.

What does AI benchmark baseline selection decide in an investment case?

It decides which story a benchmark gain is allowed to tell. The same measured improvement can stand for a new method, a better training recipe, more data or nothing at all, depending on what it was compared against, so the baseline, not the headline number, limits how much of an investment narrative a result can carry.

Diligence on AI benchmarks usually starts with the score and the test set. This page starts one row earlier, with the comparison. A company that says its model beats a well-known system has made two claims: that it scored higher, and that the system it beat is the right yardstick. The second claim is rarely examined, and it is where narratives about proprietary data, model advantages and defensibility most often overreach.

What follows is a public recount showing how far a single baseline choice can move a verdict, published cases from medicine and forecasting in which a stronger or simpler baseline reversed a field's conclusion, a six-rung ladder that maps each kind of comparison to the narrative it can support, and an underwriting worksheet with a replication checklist.

How far can one baseline choice move a verdict? A recount of 146 ImageNet models

Far enough to halve the field of winners. In timm's published ImageNet results, 140 of 146 similarly sized models beat ResNet-50 as torchvision originally trained it, but only 69 beat the same architecture trained with torchvision's newer recipe. The architecture, the parameter count and the test set stayed the same; only the baseline's training recipe changed.

The data is public and dated. timm, the PyTorch image-model library Hugging Face maintains on GitHub, keeps a results file listing each pretrained weight's top-1 accuracy on the 50,000-image ImageNet-1k validation set, with its parameter count and evaluation size. At commit 558fb25 (October 26, 2026) the file holds 1,556 rows, 783 of them evaluated at 224 pixels, the size both baselines use.

Torchvision ships two sets of ResNet-50 weights. Its source code describes the first, IMAGENET1K_V1 at 76.130 percent top-1, as closely reproducing the original paper with a simple training recipe, and the second, IMAGENET1K_V2 at 80.858 percent, as improving on the paper with torchvision's new training recipe; V2 is now the library's default. timm's own evaluation of the two gives 76.156 and 80.846 percent. Both have about 25.6 million parameters.

This is not a quirk of one library. In ResNet strikes back (2021), Wightman, Touvron and Jégou report that a vanilla ResNet-50 reached 80.4 percent top-1 at 224 pixels with no extra data or distillation under their training procedure, and that ResNet-50 baselines quoted in the literature ranged from 75.2 to 79.5 percent.

The comparison population is every weight in the file evaluated at 224 pixels, within 25 percent of ResNet-50's parameter count, trained on ImageNet-1k alone by its tag, and not itself a ResNet-50: 146 weights from 117 architectures. Against the V1 baseline the median weight gains 4.6 points; against V2 the median is minus 0.1. Tighter and looser size bands tell the same story, as the table shows:

How far can one baseline choice move a verdict? A recount of 146 ImageNet models
Comparison populationWeightsBeat torchvision V1 (76.156%)Beat torchvision V2 (80.846%)
Within 25% of ResNet-50's size, ImageNet-1k only (main)146140 (95.9%)69 (47.3%)
Within 10% of its size, ImageNet-1k only7474 (100%)32 (43.2%)
Within 50% of its size, ImageNet-1k only193179 (92.7%)84 (43.5%)
Within 25%, extra-data and distilled weights included173166 (96.0%)93 (53.8%)
Two-panel chart: bars show 95.9 percent of 146 similar-size ImageNet-1k models beating torchvision's original ResNet-50 against 47.3 percent beating its newer recipe; slope lines show two outside-data ResNet-50s falling from gains of 5.0 and 3.1 points to 0.3 and minus 1.6.
Source: SourceX recount of timm's ImageNet results, commit 558fb25, October 2026. Top-1 accuracy at 224 pixels.

When the baseline moves, so does the case for extra data

A data advantage measured against a stale baseline can mostly vanish against a current one. The same results file holds two ResNet-50s pretrained on outside image collections before ImageNet fine-tuning, and their apparent gains depend almost entirely on which torchvision baseline they are read against.

Facebook Research's semi-weakly supervised ResNet-50 was pretrained, its repository says, using 940 million public images whose hashtags matched ImageNet classes, with the student model trained on 64 million images its teacher selected; the README called its 81.2 percent state of the art for ResNet-50 as of September 2019. timm measures it at 81.146 percent at 224 pixels: 5.0 points above V1 and 0.3 above V2. The companion semi-supervised ResNet-50, pretrained on part of the unlabeled YFCC100M collection, scores 79.252 percent, 3.1 points above V1 and 1.6 points below V2, which used no outside images at all.

Recipe changes alone moved scores as much as the outside data did. Across the seven architectures in the file that carry both torchvision weight sets, moving from the original recipe to the new one raised top-1 accuracy by 3.1 to 4.7 points, median 3.7, on the same architecture and the same ImageNet-1k images. Evaluation size is another quiet lever: timm lists the identical resnet50.a1_in1k weights at 80.382 percent at 224 pixels and 81.240 percent at 288.

For an investor the lesson is direct. A pitch that credits proprietary data with a five-point gain over a named model is making a recipe claim and a data claim at once, and until the baseline has been retrained on a current recipe at the same evaluation settings, the data's share of the gain is unknown. In this public example, most of it belonged to the recipe. The four ResNet-50s side by side:

When the baseline moves, so does the case for extra data
ResNet-50 weight (top-1 at 224 px)Outside data before ImageNetGain over V1Gain over V2
Semi-weakly supervised (81.146%)940 million hashtag-matched public images+5.0 points+0.3 points
Semi-supervised (79.252%)Unlabeled subset of YFCC100M+3.1 points-1.6 points
torchvision V2 recipe (80.846%)None+4.7 pointsBaseline
ResNet strikes back A1 recipe (80.382%)None+4.2 points-0.5 points

Which published reversals show a stronger baseline overturning a narrative?

Two are especially useful in diligence: a prompted general model overtaking a domain-tuned one in medicine, and a one-layer linear model beating a generation of complex forecasters. Each overturned a specialization story as soon as someone ran a stronger or a simpler baseline.

Medicine supplies the cleanest public sequence. Google's Med-PaLM 2 paper (May 2023) reported up to 86.5 percent on MedQA and credited a stronger base model, medical domain fine-tuning and prompting strategies including ensemble refinement. Six months later, Microsoft researchers reported that GPT-4 steered by Medprompt, a set of general-purpose prompting techniques with no expert-curated medical content, set state-of-the-art results on all nine MultiMedQA datasets, cut the MedQA error rate by 27 percent against the best specialist methods, passed 90 percent, and outperformed Med-PaLM 2 with an order of magnitude fewer model calls.

Then the yardstick moved again. A follow-up paper, From Medprompt to o1, reports that OpenAI's o1-preview, without advanced prompting, outperformed GPT-4 with Medprompt. Each step overturned the previous narrative: domain tuning beats general models, prompting beats domain tuning, a newer general model beats prompting. An investor deck written at any one of those moments would have been accurate on its date and misleading a year later.

Forecasting shows the simple-baseline version. In a 2022 preprint, Zeng and coauthors tested a one-layer linear model against the Transformer-based forecasters that had led recent papers and reported that, across nine widely used benchmark datasets, the linear model beat the complex models in every case, often by 20 to 50 percent. A literature that compares new architectures mainly with one another can look like steady progress against a field that never faced a simple yardstick.

Two diligence moves follow. Date the baseline: ask when the comparison ran, which general models existed on that date, and whether the strongest received prompting, retrieval and inference budgets comparable to the company's own system. Then check the simple rung: for forecasting, pricing, churn or demand products, ask how much of the gain survives a well-tuned conventional model on the same features. If little does, the narrative shifts from model advantage to data access or workflow integration, and each needs its own evidence.

The baseline ladder: which narrative each comparison can support

Rank baselines by how hard they are to beat, and credit a claim only with the narrative of the highest rung it clears. A gain over a stale baseline supports no commercial story; a gain that survives a data-swap control is the first that supports a data narrative.

The rungs are cumulative, so a company should clear them from the bottom. A gap at a rung is a question for the next meeting rather than an automatic rejection: some products win on cost, latency or ease of deployment even when accuracy stalls at rung three, and then the thesis has to rest on those measures. The ladder was built for this article from the recount and the published cases above:

The baseline ladder: which narrative each comparison can support
RungBaseline to requestA win supportsNot evidence when
1. Stale or default incumbentThe named model as first published, or run at library defaultsOnly that the method worksA newer recipe or version of the incumbent exists, as it did for ResNet-50
2. Simplest credible methodA linear model, rules or a conventional model on the same inputsThat added complexity earns its costThe simple baseline was skipped or left untuned
3. Retuned incumbentThe same model family on its current best recipe, with an equal tuning budgetA method advantage over what already existsTuning budgets or evaluation settings differ between the arms
4. Current general model at matched effortToday's strongest general model with comparable prompting, retrieval and inference budgetThat specialization beats what any customer can rentThe general model is an old version or ran at defaults
5. Data-swap controlThe same pipeline trained on public or substitute dataThat the data, not the recipe, carries the gainThe model, prompt or compute changed along with the data
6. Incumbent processThe human or software process the customer runs todayThat the gain is worth paying forCost, time and error severity went unmeasured

Benchmark underwriting worksheet and replication checklist

Fill in one worksheet per claim and answer each line with a document, a configuration file or a rerun, never a slide. A claim that cannot name its baseline's version and date has not reached the first rung, however large the gain.

At a glance

Benchmark underwriting worksheet and replication checklist

Score the claim at the highest rung it survives, write down the margin and its uncertainty there, and carry only that rung's narrative into the memo. The replication checklist an analyst can run on any claim:

Checklist · 8 items

  1. Baseline identity: model name, version or weight tag, release date, and whether the number was rerun or copied from a paper.
  2. Date match: the strongest public baseline available on the date of the claim, and whether it was tested.
  3. Tuning parity: hours, trials and hyperparameter searches given to each arm, logged in the same units.
  4. Evaluation parity: identical test items, preprocessing, input size and scoring code; the a1 weights' 0.86-point move from 224 to 288 pixels shows why.
  5. Inference parity: calls, samples, tokens and ensembling per item for every arm, since Medprompt's result came from how GPT-4 was prompted rather than from new training.
  6. Outside data and distillation: every source used before fine-tuning, disclosed for the baseline as well as for the claim.
  7. Rung reached: the highest ladder rung the gain survives, with the margin and an interval around it.
  8. Independent rerun: whether someone outside the company reproduced the baseline number from the released configuration.

0 of 8 ticked. Tick each item as you go.

Could the gain be real even against a weak baseline?

Yes. A weak baseline does not make a result false; it leaves the result unattributed. The method may be genuinely better, but the evidence cannot say by how much until the comparison is rerun at a higher rung, and an investment case cannot price an unknown margin.

The opposite failure is just as common: a gain credited to data that came from a newer base model, a better prompt or a larger inference budget arriving at the same time. Rung five exists for that reason. It holds the pipeline fixed and swaps only the data, so whatever remains of the gain belongs to the data. In the recount, the outside-data ResNet-50s look like data wins against V1; against V2, the recipe accounts for nearly all of their lead.

Both failures point to the same request. Ask for the comparison at every rung the company claims, with the settings that differ between arms listed, and treat any rung without a rerun as unproven rather than as passed.

What could make baseline-first underwriting wrong

Five ways the approach can mislead, each worth stating in the memo beside any conclusion drawn from it:

  • Baselines go stale quickly. The medical sequence turned over twice in about a year and a half, judging by the papers' arXiv dates, so every rung needs a rerun date and a trigger for the next one.
  • Strong baselines can be over-tuned. A baseline tuned on the test set misleads as much as a straw one; tuning should use a separate split, for the baseline and the claim alike.
  • Accuracy is not the only thesis. A product that loses at rung four can still win on cost, latency, privacy or fit with a customer's systems, and then the investment case rests on those measures and their own evidence.
  • Public benchmarks are not enterprise tasks. The recount covers ImageNet classification, and timm's README notes that the validation set guides model selection during training, so it is not a true test set; the size of baseline effects on business tasks is not measured here.
  • Simple-baseline results describe their date. The linear forecaster's win covers the models and benchmarks of 2022, so the simple rung must be rerun against whatever the claim competes with now.

Questions for a data company about its baselines, SourceX included

Ask every company that credits data for a model result which rung its evidence reached, and ask SourceX the same. SourceX turns business records that companies choose to license into datasets for AI developers, handling sourcing and qualification, rights review, contracting, preparation, delivery and payment; the supplying company approves the scope and terms and signs before any records reach SourceX.

SourceX has not published controlled comparisons of licensed data against substitute data, so it has no rung-five result to show here. It shares company and commercial metrics with investors directly rather than in articles. SourceX is raising from accredited investors under Rule 506(c), and investor materials are available on request through its investor page. Questions to put to any data company:

  • Which baseline, by version and date, does each headline result beat, and was it rerun or copied from a paper?
  • What was the strongest general-purpose model on the claim's date, and how did it score at matched prompting and inference budget?
  • Has the same pipeline been trained on public or substitute data, and how much of the gain remained?
  • How much tuning did the baseline receive compared with the company's own system?
  • What does the customer's current process achieve on the same tasks, in time, cost and errors?
  • When will each comparison be rerun as new models ship, and who decides?

Methods note: how the ImageNet baselines were recounted

Data: results/results-imagenet.csv from github.com/huggingface/pytorch-image-models at commit 558fb25 (October 26, 2026), read with Python's csv module by the script baseline_ladder.py, kept with this article's working files for reviewers. The script reads that one file, runs no repository code and uses no randomness. Baselines are resnet50.tv_in1k and resnet50.tv2_in1k at 224 pixels; their descriptions come from torchvision's resnet.py at commit a034671 (October 23, 2026).

Rules as the script applies them: only rows evaluated at 224 pixels count; a weight is one row, an architecture plus its pretrained tag; a tag counts as extra data when any underscore-separated token is in12k, in21k, in22k, ig1b, yfcc100m, jft, laion2b, laion400m, openai, m38m or merged, and as distilled when any token contains dist or equals ssld; the main population is non-ResNet-50 rows with 19.17 to 31.95 million parameters (0.75 to 1.25 times 25.56 million) that count as ImageNet-1k only; beats means a strictly higher top-1. Recipe gains compare tv_in1k and tv2_in1k rows of the same architecture at 224 pixels.

Other evidence: the Facebook Research semi-supervised ImageNet README (commit a189fd0), timm's ResNet documentation and results README, and the arXiv records for ResNet strikes back, Med-PaLM 2, Medprompt, its o1 follow-up and the linear forecasting study, all checked on October 27, 2026. Not measured: whether any real investment pitch relied on a stale baseline, the size of baseline effects on enterprise tasks, and how validation-set selection inflates any single score.

Frequently asked questions

What is a baseline in an AI benchmark?

A baseline is the system a new result is compared against on the same test. Its choice decides what a gain means: beating an old or default version shows the method works, while beating a retuned incumbent or today's strongest general model at matched effort says something about competitive position.

What makes an AI baseline a straw man?

A baseline is a straw man when it is weaker than what a competent team could deploy on the date of the claim: an old version, default settings, an untuned recipe or a smaller inference budget. In timm's ImageNet results, 96 percent of similar-size models beat ResNet-50 as torchvision first trained it, and 47 percent beat it on the newer recipe.

Should an AI startup compare its model with frontier general models?

If customers could rent a general model for the same task, yes, at matched prompting, retrieval and inference budget, with the model version and date recorded. Medicine shows why: GPT-4 with general prompting overtook a domain-tuned model on MedQA, and a newer general model later overtook the prompted one.

How can an investor tell whether data or the training recipe produced a gain?

Ask for a data-swap control: the same pipeline, settings and compute trained on public or substitute data, so that only the data differs. In timm's results, a ResNet-50 pretrained with a 940-million-image public collection led the original baseline by 5.0 points but the retrained one by only 0.3.

Does SourceX publish benchmark comparisons for licensed data?

No. SourceX has not published controlled comparisons of licensed data against substitute data, and it shares company and commercial metrics with investors directly. Investors can request materials through sourcex.si/investors and ask which comparisons exist and how each baseline was chosen.

Sources

  • timm's ImageNet results file at commit 558fb25 lists top-1 accuracy, parameter count and evaluation size for 1,556 pretrained weights; the SourceX recount of it finds 140 of 146 similar-size ImageNet-1k weights beating torchvision's original ResNet-50 and 69 beating its newer recipe. Source
  • timm's results README describes the ImageNet-1k validation set as 50,000 images and notes that model selection during training uses it, so it is not a true test set. Source
  • timm's ResNet documentation lists the torchvision weights (tv_in1k, tv2_in1k) under pytorch/vision, the a1 weights under ResNet strikes back and the semi-supervised weights under Facebook Research. Source
  • torchvision describes ResNet-50 IMAGENET1K_V1 (76.130% top-1) as closely reproducing the paper with a simple recipe and IMAGENET1K_V2 (80.858%) as improving on it with a new training recipe; V2 is the default. Source
  • Facebook Research's README says its semi-weakly supervised models were pretrained using 940 million hashtag-matched public images, with students trained on 64 million teacher-selected images, and calls ResNet-50's 81.2% state of the art as of September 2019; semi-supervised models used a YFCC100M subset. Source
  • ResNet strikes back (2021) reports a vanilla ResNet-50 at 80.4% top-1 at 224 pixels without extra data or distillation, against literature baselines of 75.2% to 79.5%. Source
  • The Med-PaLM 2 paper reports up to 86.5% on MedQA from a stronger base model, medical domain fine-tuning and prompting strategies including ensemble refinement. Source
  • Medprompt (November 2023): GPT-4 with general-purpose prompting set state-of-the-art results on all nine MultiMedQA datasets, cut MedQA error 27% versus the best specialist methods, passed 90%, and outperformed Med-PaLM 2 with an order of magnitude fewer calls. Source
  • The follow-up paper From Medprompt to o1 reports that o1-preview without advanced prompting outperformed GPT-4 with Medprompt. Source
  • Zeng and coauthors report that a one-layer linear model outperformed Transformer-based long-term forecasting models in all cases across nine benchmark datasets, often by 20% to 50%. Source

Request SourceX investor materials

SourceX shares company and commercial metrics with investors directly. Any offering is made only to verified accredited investors.

See if your company qualifies