Definitions and comparisons
AI dataset vendor comparison: accepted utility beats catalog size
By SourceX Editorial · Reviewed by Noah Loul ·
9 sections
Short answer
Compare dataset vendors on accepted utility: the units that match your task, are licensed for your intended use, pass your acceptance test on a sample and arrive as specified, divided by total cost. Catalog size measures inventory, not fit. Give every vendor the same written test and sampling method, hold license scope constant, and choose on accepted units per unit of cost.
Checklist
AI dataset vendor comparison
Key takeaways
- Catalog size counts inventory; accepted utility counts what passes your task, license and acceptance test per unit of cost.
- Filtering, duplication and license errors can each reverse a ranking by size, as public research on DataComp, the C4 corpus and dataset licensing shows.
- Write the acceptance test, unit and sampling method before talking to vendors, and apply them identically to every offer.
- Hold permitted use, model scope, term and exclusivity constant before comparing any price.
- Record the decision with gate shares, intervals, unknowns and the delivery results that would reopen it.
How should an AI team compare dataset vendors?
By accepted utility, not by catalog size. Accepted utility is the number of units, whether records, conversations or task cases, that match your task, are licensed for your intended use, pass your acceptance test on a sample and arrive in the format and version you specified, divided by everything the purchase costs you.
Catalog size answers a different question, how much inventory a vendor holds, and it arrives with the vendor's own definitions of a record, a domain and a quality check. Two vendors with very different catalog counts can deliver the same number of usable units for your task, or the smaller one can deliver more. An AI dataset vendor comparison that starts from the bottom of the funnel, what survives your test, cannot be won by the largest brochure.
This page gives a five-gate funnel for any offer, a worked comparison of three illustrative vendors, the conditions under which catalog size does matter and a one-page decision record. It complements the evidence-dated vendor grid on this site, which handles how to record each vendor claim; this page handles how to turn claims into one comparable number for your use.
Why catalog size is a weak signal of usable data
Because size counts inventory before the three filters that decide value: relevance to your task, duplication and license fit. Published research shows each of them can reverse a ranking by size.
Filtering can beat volume. DataComp, a 2023 benchmark that held training code and compute fixed while comparing candidate datasets, reported that a filtered subset of about 1.4 billion image-text pairs from a 12.8 billion pair pool trained a CLIP ViT-L/14 model to 79.2% zero-shot ImageNet accuracy, 3.7 points above OpenAI's CLIP ViT-L/14 trained with the same procedure and compute.
Duplicates inflate counts. Katherine Lee and colleagues found a single 61-word English sentence repeated more than 60,000 times in the C4 web corpus (2021), and reported that models trained on deduplicated data emitted memorized text about ten times less often and needed fewer training steps to reach the same or better accuracy.
License labels are often wrong. Published in 2024, the Data Provenance Initiative's review of more than 1,800 text datasets found widely used hosting sites omitting licenses more than 70% of the time and carrying license error rates above 50%, so a catalog's rights labels need checking before a single record counts toward your total.
Security badges and selected case studies share the weakness from another angle: they describe a vendor's controls or another customer's project, not the records you would receive for your task.
Write the acceptance test before the first vendor call
Define the unit, the intended use and the pass rule first, so that every vendor is measured against the same bar. A test written after seeing samples drifts toward the sample you liked.
At a glance
Write the acceptance test before the first vendor call
Borrow the structure of a model evaluation. Anthropic's engineering guidance on agent evaluations treats a task as one test with its inputs and success criteria spelled out, runs several trials per task because outputs vary, and grades the outcome left in the environment rather than the agent's account of it. For a dataset, the equivalent is a written test with fixed inputs, a pass rule a reviewer can apply without the vendor in the room and a sample size fixed in advance.
Choose quality measures for the task, not the vendor's favorites. ISO/IEC 5259-2:2024 sets out a data quality model and measurable characteristics for analytics and machine learning, building on ISO/IEC 25012 and ISO 8000. Use it as a vocabulary for deciding which characteristics your test checks, such as completeness of the fields your task needs or accuracy of outcome labels.
Vet the supplier against the same use case. NIST's Generative AI Profile (NIST AI 600-1, July 2024) lists value chain and component integration among generative AI risks, including data that was improperly obtained and improper vetting of suppliers, and it suggests a supplier risk assessment based on the use case. The acceptance test and the rights review belong in one procedure, not two.
A usable acceptance test fixes six things:
- Unit: what one record, conversation or task case is, and which fields it must carry.
- Intended use: training, fine-tuning, evaluation or retrieval, and for which models.
- Pass rule: the checks a unit must pass, written so a reviewer can apply them without knowing the vendor.
- Sample: size and drawing method, fixed in advance and identical for every vendor.
- Reviewers: who grades, how disagreements are settled and what share is graded twice.
- Threshold: the acceptance rate and accepted volume below which an offer drops out.
The accepted-utility funnel: five gates every offer passes
Run every offer through the same five gates and record the surviving share at each, with its evidence. Multiplying the catalog claim by the surviving shares estimates accepted units; dividing by total cost gives the figure to compare.
Two rules keep the funnel honest. Measure each gate on the vendor's evidence for your scope, not its whole catalog. And when a vendor cannot supply evidence for a gate, record the share as unknown and keep the offer out of the ranking until it does, rather than guessing.
Delivery evidence can often be checked mechanically. A machine-readable format such as Croissant describes each file with its encoding format and a SHA-256 checksum and maps columns to typed fields, so a buyer can confirm that delivered files match the version that was sampled. The five gates, with what to request at each:
| Gate | Question | Evidence to request | Common failure |
|---|---|---|---|
| 1. Task match | What share of the catalog fits your unit, domain, period and language? | Counts by the fields that define your scope, from the vendor's own manifest | Counts given for a whole category rather than your slice |
| 2. License fit | What share is licensed for your intended use, models and term? | The license or term sheet naming permitted uses and exclusions for that share | Rights labels copied from upstream sources without verification |
| 3. Sample acceptance | What share of a fixed random sample passes your written test? | A sample drawn by a method you set, graded without knowing the vendor | Vendor-picked samples or showcase examples |
| 4. Delivery fit | Can the accepted units arrive in your format, version and timeline? | A manifest, file checksums and a version identifier tied to the sample | Delivered files differ from the sampled version |
| 5. Total cost | What does each accepted unit cost once review, integration and rework are added? | License price plus your own estimate of review, integration and rework effort | List prices per record compared across different scopes |
Hold license scope constant before comparing prices
Compare offers only at the same permitted use, model scope, term and exclusivity, because each of those changes what a record is worth to you. A price per record for evaluation-only use and a price for training use are prices for different products.
Normalize in one of two ways: ask every vendor to quote the same scope, which is cleaner, or have each vendor restate its quote at your scope in writing. Do not adjust prices yourself on assumptions about what a vendor would charge for a broader license; mark the cell unknown instead.
Exclusivity deserves its own line. An exclusive grant stops the licensor from licensing the same records to others for its term, which can be worth paying for when the dataset feeds a product feature, and rarely changes what you receive for an evaluation set. Price it separately so it does not hide inside a per-record comparison.
Keep one more distinction in view. Access to records, a privacy basis for using them and the authority to license them are three different things, and de-identification does not create the third. Gate 2 asks for the document that grants the right, and your counsel reads it against your intended use.
Worked comparison: three illustrative vendors, one acceptance test
Illustrative: the vendors and numbers in this example are invented to show the arithmetic and describe no real company. An AI team needs support conversations with resolution outcomes to fine-tune an assistant, and needs at least 500,000 accepted conversations. All three vendors quoted the same training scope, and each sample was 400 conversations drawn by the team's own method.
Vendor A has the largest catalog and the fewest accepted conversations per unit of cost. Vendor B delivers the most accepted conversations per unit of cost and clears the volume floor on its own. Vendor C has the highest acceptance rate but cannot reach 500,000 accepted conversations alone, so it is a candidate for a second source or for an evaluation set rather than the main purchase.
What the team did next shows how the funnel narrows the work. It ran a pilot on Vendor B's sampled conversations for its fine-tuning task, asked Vendor C whether its conversations could be licensed for evaluation use instead, and asked Vendor A for task-match counts on the exact slice before spending more review time. None of those steps needed a larger catalog; each needed a narrower question.
Sample size limits precision. With 400 graded units, a 75% acceptance rate carries a 95% interval of roughly plus or minus 4 points, so gaps of a few points between vendors are noise, while a gap like Vendor A's 40% against Vendor B's 75% is not. Relative total cost is an index with Vendor A set to 1.0, not a price. The table traces each vendor through the gates.
| Measure | Vendor A | Vendor B | Vendor C |
|---|---|---|---|
| Catalog claim (conversations) | 40,000,000 | 3,000,000 | 600,000 |
| Task match | 5% (2,000,000) | 60% (1,800,000) | 85% (510,000) |
| License fit for training | 50% (1,000,000) | 90% (1,620,000) | 100% (510,000) |
| Sample acceptance (400 graded) | 40% | 75% | 85% |
| Estimated accepted conversations | 400,000 | 1,215,000 | 433,500 |
| Relative total cost (A = 1.0) | 1.0 | 1.6 | 0.9 |
| Accepted conversations per cost unit | 400,000 | about 759,000 | about 482,000 |
When catalog size does matter
Size matters when the acceptance test is cheap and the need is breadth: coverage of many languages, rare cases or long time spans, or a refresh schedule that draws on a deep pool. A large catalog then raises the odds that gate 1 yields enough units, but every unit still has to pass gates 2 to 5.
It also matters as a capacity signal for later orders. A vendor whose accepted volume already sits near your floor has little room for a second order on the same scope, which is worth recording even when it wins the first comparison. In the worked example, Vendor C's 433,500 estimated accepted conversations sit below a 500,000 floor, so it has no headroom at all.
What size never replaces is acceptance evidence for your scope. A catalog count, a security certification or another customer's case study can each be accurate and still say nothing about whether the records you would receive pass your test.
A quick check before giving size any weight: ask what would happen to the decision if the catalog were half as large. If the answer is nothing, because your accepted volume comes from one narrow slice, size is not a criterion for this purchase.
A one-page purchase decision record
Record the decision on one page that a colleague could audit a year later: what was tested, what passed, what it cost and what would change the choice. The page needs six fields:
At a glance
A one-page purchase decision record
- Intended use, unit definition and the acceptance test version, with its date.
- Vendors compared, the scope each quoted and how scope was normalized.
- Surviving share at each gate, with sample sizes, intervals and any unknowns.
- Total cost components: license, review, integration and expected rework.
- The decision, the conditions attached to it and the evidence still missing.
- Reversal triggers: results from the first delivery that would reopen the comparison.
Applying the same funnel to SourceX
Apply the funnel to every vendor, SourceX included. SourceX does not hold datasets in stock: it sources them on request from U.S. businesses, and the buyer's request form is where scope and volume, language and format, timing and licensing needs are set out. There is no catalog count to compare, so the comparison starts at gate 1 with the scope you submit.
SourceX's general guide describes what buyers usually look for in a source: one system holding tens of thousands of records or more, backed by years of consistent history, typically two or more; smaller sets are considered when the work is rare, and every case is assessed deal by deal. Large deliveries stay in the supplying company's own storage or ship on encrypted drives, which bears on gate 4.
None of that is a statement that a request will be filled or that records will pass a given test. Send your acceptance test with the request, including the intended use, so that candidate data can be judged against it from the start.
Frequently asked questions
What is accepted utility in dataset procurement?
Accepted utility is the number of units that match your task, are licensed for your intended use, pass your acceptance test on a fixed sample and arrive as specified, divided by the total cost of the purchase. It turns very different vendor catalogs into one comparable figure for your use.
Is a bigger dataset catalog better for AI training?
Not by itself. DataComp (2023) showed a filtered subset of a much larger pool training a better model at the same compute, and duplicated records can inflate counts. A large catalog helps only when enough of it survives task match, license fit and your acceptance test.
How large should an acceptance sample be?
Large enough that the interval is narrower than the differences you care about. With 400 graded units, a 75% acceptance rate has a 95% interval of roughly plus or minus 4 points. Smaller samples suit early screening but cannot separate vendors whose rates are close.
How do you compare dataset prices across different license scopes?
Do not compare them directly. Ask every vendor to quote the same permitted use, model scope, term and exclusivity, or have each vendor restate its quote at your scope in writing. Mark anything a vendor will not restate as unknown.
Does SourceX have a dataset catalog?
No. Datasets at SourceX are sourced from U.S. businesses once a buyer asks for them; nothing is held in stock. The request-data form collects what a buyer needs on scope, volume, language, format, timing and licensing, and a buyer can include an acceptance test there so candidate data is judged against it.
Sources
- DataComp (2023) fixed training code and compute; a filtered 1.4 billion pair subset of a 12.8 billion pool trained a CLIP ViT-L/14 to 79.2% zero-shot ImageNet accuracy, 3.7 points above OpenAI's CLIP ViT-L/14 at the same compute. Source
- Lee et al. (2021) found a 61-word sentence repeated over 60,000 times in C4; deduplicated data cut memorized output about tenfold and needed fewer steps for the same or better accuracy. Source
- The Data Provenance Initiative's 2024 audit of more than 1,800 text datasets found license omission above 70% and error rates above 50% on popular hosting sites. Source
- Anthropic's guidance on agent evaluations defines a task by its inputs and success criteria, runs multiple trials because outputs vary, and grades the final environment state rather than the agent's claim. Source
- ISO/IEC 5259-2:2024 defines a data quality model and measurable characteristics for analytics and machine learning, building on ISO/IEC 25012 and ISO 8000. Source
- NIST AI 600-1 (July 2024) lists value chain and component integration as a generative AI risk, including improperly obtained data and improper supplier vetting, and suggests use-case based supplier risk assessment. Source
- Croissant describes each dataset file with its encoding format and a SHA-256 checksum and maps columns to typed fields. Source
Tell us what data you need
Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.