Third-party eval data vendors: independence, conflicts and verification

Why private eval curators carry a structural conflict

A vendor that sells training data to model developers and also builds or runs the hidden test set those developers are judged on has a conflict of interest, even when nobody acts in bad faith. Bansal and Maini make this argument about private data curators, noting that open benchmarks give full transparency while private ones ask users to trust the curator's process [1]. Their paper also flags a subtler issue: the same annotator pool, guidelines and stylistic preferences can shape both a vendor's training data and its grading, so models trained on that vendor's data may score well partly because they share its taste [1].

This combination exists in the market. Some data vendors advertise custom training data and custom evaluation datasets from one catalog [2]. That is not disqualifying, but it moves the burden of proof onto the vendor. Your procurement question is not "is this vendor honest" but "what evidence shows the eval items never touched any training pipeline, ours or another client's?"

The conflict matters most in three situations:

• Model selection. You are comparing foundation models, and one of them was fine-tuned on data from the same vendor that wrote your test set (see running a model bake-off with your own eval set).
• Your own training data. You buy training data and eval data in the same domain from the same supplier, so leakage would inflate your own scores.
• Vendor-run leaderboards. A vendor publishes rankings on a hidden set while selling data to some of the ranked labs.

How contamination turns a vendor score into a training artifact

When eval items or close paraphrases leak into training data, score gains stop measuring capability and start measuring exposure. OpenAI stopped reporting SWE-bench Verified in February 2026, stating that contamination meant improvements increasingly reflected training-time exposure to the benchmark [4]. LiveBench's authors describe the same failure for static test sets and respond by refreshing questions over time [8].

A private vendor set is not immune; it simply moves the leakage path from the public web to the vendor's own operations. Typical leak routes include:

• An item written for an eval engagement gets recycled into a training "seed set" for another client.
• Annotators who wrote gold answers later produce SFT or preference data in the same domain from memory, with templates and phrasings intact.
• The vendor sends eval prompts to a hosted model API for scoring, and the provider's data retention policy allows those prompts to be used later.
• Near-duplicates survive because the vendor deduplicates only on exact hashes, not on n-gram or embedding similarity.

Controls for the first and last routes belong in your contamination-resistant evaluation design, and the access-control side is covered in keeping a private eval set private. Contamination testing on delivered sets is covered in the owner guide on contamination checks for licensed evaluation data.

Separation controls to require in writing

Ask for separation controls the vendor can attest to and you can audit, not general assurances. The strongest set combines organizational, technical and contractual controls.

Organizational. Different teams, and ideally different annotator pools, for eval items and training data in the same domain. Ask how the vendor prevents an annotator from moving between an eval project and a training project for the same capability within a set cooling-off window.

Technical. Every eval item should carry a unique ID, a creation timestamp and a content hash, stored in a registry the training-data side cannot write to. Ask whether the vendor screens outbound training deliveries against that registry using exact hashes, MinHash or n-gram overlap, and embedding similarity, and what thresholds it uses. Canary strings embedded in eval files let you detect later ingestion.

Contractual. An item non-reuse attestation: the vendor warrants that items delivered to you were not previously sold, will not be resold, and will not be used to create training data for anyone. Pair this with an evaluation-only license on the buyer side so obligations run both ways, plus an audit right over the item registry.

Verifying scores: run grading where you control it

The cleanest way to reduce dependence on a vendor is to take delivery of the items and gold labels and run scoring yourself. Some vendors are built around this model, delivering held-out sets that the buyer evaluates on its own infrastructure [3]. If the vendor must host the set, for example to keep it out of your fine-tuning pipelines, ask for per-item outputs, grader prompts and rubric versions so you can re-score a sample independently.

Treat a vendor-reported score as a claim to reproduce. A minimal verification run looks like this:

1. Pull a stratified sample of 100 to 300 items across slices (see sizing an eval set for statistical power for how large it must be to detect the deltas you care about).
2. Re-run the model with the exact decoding settings the vendor used (temperature, max tokens, system prompt, tool configuration).
3. Re-grade with your own grader, either human reviewers or a pinned LLM judge with a versioned rubric.
4. Compare per-item pass/fail, not just aggregate accuracy. Disagreement concentrated in one slice usually points to a rubric ambiguity or a mislabeled gold answer.

Label methodology and agreement statistics to demand

A vendor's gold labels are only as good as its labeling process, and published test sets show how far that can drift. Northcutt and colleagues estimated an average label error rate of at least 3.3% across test sets of 10 widely used datasets, including at least 6% of the ImageNet validation set [5]. On a small private set, a few percent of wrong gold answers can flip a model comparison.

Ask for the methodology as a document, not a sales summary. It should state rater qualifications (credentials, domain tenure, screening tests), the number of independent labels per item, adjudication rules, and inter-rater agreement per slice. Krippendorff's alpha suits this because it handles multiple raters, missing ratings and nominal, ordinal or interval scales [6]; Cohen's kappa is acceptable for two-rater designs. Then run your own gold-label audit on the delivered set before you trust it.

Red flags include a single annotator per item with no adjudication, agreement reported only as raw percent, agreement computed on an easy calibration batch rather than production items, and rubrics that changed mid-project without versioning.

Eval vendor due-diligence questionnaire

Adapt a general data-provider DDQ, such as the FISD Alternative Data Council template, which added generative AI questions in its 2024 edition [7], with eval-specific sections. SourceX's data provider due diligence questionnaire covers rights, security and delivery; the questions below cover independence.

Illustrative example: invented to show structure; it does not describe an available dataset.

Score each section pass, conditional or fail, and route conditional answers to your supplier risk tiering process. Broader supplier quality questions sit in the owner guide on evaluating data supplier quality.

When to switch to independent or self-built eval data

If a vendor cannot answer the commercial-conflict and separation rows credibly, the practical fix is to change the source of the test set rather than negotiate harder. Options include commissioning a set from a vendor that does not sell training data in your domain, building a golden dataset from your own business records, or licensing real operational records from a company outside the model-training supply chain under evaluation-only terms. If you want operational records sourced for you, describe the evaluation data to SourceX in terms of records and fields rather than named companies; allowed uses are agreed in each license.

Sets built from real business outcomes, such as resolved support tickets or completed contract redlines, have a structural advantage: the gold label is what actually happened, not an annotator's preference, which reduces the shared-annotator bias the private-curator paper describes [1]. The tradeoff is de-identification work and rights review, both of which a reputable source should handle before delivery. Independent evaluation organizations face their own version of these questions, covered in data rights for independent AI evaluators. For the wider map of benchmarks and private sets, start at the evaluation datasets hub.

Sourcing independent evaluation data from operational records

SourceX sources operational datasets, such as support and sales histories, engineering records, and finance and legal workflows, from US companies on request and manages licensing and ongoing purchases. Each dataset is rights-reviewed and delivered under a license that defines records, uses, term and delivery, with personal details removed or replaced before delivery, and SourceX does not train models. Describe the evaluation data you need at SourceX for buyers.