Definitions and comparisons
Enterprise dataset procurement scorecard with a weight flip test
By SourceX Editorial · Reviewed by Noah Loul ·
9 sections
Short answer
An enterprise dataset procurement scorecard for context-rich business records should do three things: apply floors on rights and task fit that no total can override, weight six measured criteria for the intended use, and test whether the winner survives a 10-point change in any weight. Score from a sample drawn by your method, keep price outside the score, and set the trade-off rule before bids arrive.
Process
Enterprise dataset
Key takeaways
- Context-rich records earn their value from links, outcomes and history, so the scorecard measures those directly on a sample.
- Floors come first: a vendor that cannot document rights for your use is out, however high its total.
- Weights depend on the intended use; evaluation data leans harder on attached outcomes than training data does.
- Publish the weights and the price rule before bids, as U.S. federal source selection requires for evaluation factors.
- Run the flip test: if a 10-point change in one weight changes the winner, decide that trade-off openly instead of trusting the total.
What should an enterprise dataset procurement scorecard decide?
It should decide which vendor's offer best serves one stated intended use, at a known total cost, and show whether that answer rests on a judgment call about weights. A scorecard that produces a number without naming the assumption that drives it only hides the decision.
This version is built for context-rich business data: records whose value comes from what surrounds each item. A support ticket matters to an agent developer because of its audit trail, the customer's contract tier in the CRM, the refund posted in billing and the policy version in force when the reply went out. Strip those links and the same ticket becomes a short piece of text that many sources could supply.
The evidence the decision needs is concrete: a sample drawn by your method and measured for task fit, link integrity, outcome attachment and history depth; documents showing who can grant rights and for which uses; and a delivery record. A vendor's own description of those things is a claim to check, not a number to enter.
Where generic vendor scorecards lose the context
Generic scorecards rate volume, format, price and security, and none of those shows whether the links and history survived export. Context usually disappears through ordinary, documented system behavior rather than anyone's carelessness.
Two vendors' documentation shows how. In Zendesk, each update to a ticket stores an audit, and change events record a field's previous and new value; but the endpoint that lists audits across all tickets leaves out archived tickets, and Zendesk directs developers to the per-ticket endpoint to retrieve audits for an archived ticket. An export built on the convenient route can drop the change history of exactly the older, resolved cases a buyer wants. In Salesforce, field history is kept for up to 18 months, and up to 24 months through the API, unless the customer runs Field Audit Trail, which can retain it until deleted. A vendor offering five years of opportunity records may hold only two years of how those records changed.
Neither limit is a platform defect. Both are reasons to score history depth and link integrity separately from record counts, and to ask how each table was exported. The datasheet proposal by Gebru and colleagues makes the general case: every dataset should come with a document covering its motivation, composition, collection process and recommended uses, which is where those export answers belong.
Floors before weights: rights and task fit no total can buy back
Set a minimum score on the criteria where a shortfall cannot be offset, and remove any vendor below a floor before adding anything up. Without floors, a vendor that cannot grant your intended use can still top the table on completeness.
Rights evidence needs the highest floor because rights problems are both common and discovered late. MIT Media Lab's account of the Data Provenance Initiative audit reports that more than 70 percent of the datasets it examined carried unspecified licenses, and that the correct licenses were often more restrictive than those the repositories had assigned. For private business records the equivalent documents are the supplier's authority to license, the scope of its agreement and any customer contracts or platform terms that limit outside use.
Completeness has a legal anchor for some buyers. Article 10 of the EU AI Act requires training, validation and testing data for high-risk systems to be relevant, sufficiently representative and, to the best extent possible, free of errors and complete in view of the intended purpose. The last phrase is the one procurement should copy: completeness is judged against a use, which is why the scorecard in the next section is weighted by use rather than fixed.
Six criteria, two weight profiles, one floor each
Score each criterion from 0 to 4 on evidence from a sample you drew, multiply by its weight and divide by 4, so that each profile totals 100. The training profile favors task coverage and delivery; the evaluation profile moves weight to attached outcomes and link integrity, because an evaluation case without a verified outcome cannot grade anything.
Fixed definitions keep scores comparable across vendors. Link integrity is the share of sampled records whose keys resolve to the related tables delivered. Outcome attachment is the share carrying a final outcome and its date. History depth is the months of change history present, divided by the months the use needs. Delivery reliability covers version identifiers, a manifest with counts and a documented correction process.
Methods note: the criteria, weights, floors and score anchors are an editorial framework built for this article on 2026-10-26 from the cited documentation and regulations. They have not been calibrated against model results or delivery outcomes; adjust them to your use before scoring, and record which version you applied.
| Criterion | Measured on the sample as | Training weight | Evaluation weight | Floor (0 to 4) |
|---|---|---|---|---|
| Task fit | Share of sampled cases that match the task's case types and pass your acceptance test | 28 | 24 | 2 |
| Rights evidence | Documented authority, a license scope covering your use, listed restrictions | 24 | 20 | 3 |
| Link integrity | Share of records whose keys resolve to the related tables delivered | 12 | 16 | 1 |
| Outcome attachment | Share of records carrying a final outcome and its date | 12 | 20 | 1 |
| History depth | Months of change history present over months the use needs | 8 | 8 | 1 |
| Delivery reliability | Version identifier, manifest with counts, documented correction process | 16 | 12 | 1 |
Fix the weights and the price rule before the first bid
Write the weights, the floors and the rule for trading score against price before any vendor responds, then score only on what you wrote. Weights chosen after the answers arrive tend to reward whichever vendor someone already preferred.
U.S. federal source selection makes this a rule. FAR 15.304 requires the factors and significant subfactors that affect award, and their relative importance, to be stated clearly in the solicitation, and requires the solicitation to say whether the non-cost factors combined are significantly more important than, about equal to, or significantly less important than cost or price. A private buyer is not bound by the FAR, but the discipline transfers directly.
Keep price out of the weighted total. Compare the total cost of acquisition, including preparation, integration and the cost of rejected deliveries, against the score using the trade-off rule you stated: for example, that a vendor scoring at least 5 points higher justifies a higher price up to a limit the budget owner sets. Once a pilot produces a cost per accepted record, use it in place of the estimate.
The flip test: does the winner survive a 10-point weight change?
Move 10 points from one criterion to another, one move at a time, and count how many moves change the winner among vendors that clear every floor. If many do, the total is not deciding anything; the trade-off between the criteria involved is, and it should be decided in the open.
Illustrative example: three anonymized vendors scored on a sample against the table above. Vendor A has strong task fit and delivery but thin context, scoring 4 on task fit, 3 on rights, 1 on link integrity, 1 on outcome attachment, 2 on history and 4 on delivery. Vendor B has deeper context and weaker delivery: 3, 4, 3, 3, 2 and 1. Vendor C has the richest context but scores 2 on rights evidence: 3, 2, 4, 4, 4 and 3.
Vendor C posts the highest total under both profiles, 77 for training and 81 for evaluation, and is removed by the rights floor. Under the training profile A leads B by 72 to 71, and 11 of the 25 possible single 10-point moves hand the lead to B. Under the evaluation profile B leads A by 72 to 64, and only 3 of the 25 moves reverse it, each one shifting weight into delivery reliability.
The reading differs by use. For evaluation data, B is the answer unless delivery reliability matters about twice as much as written, a question B's correction record can settle. For training data the score is a tie: the choice turns on whether attached outcomes and links matter more to the model owner than A's delivery record, and that owner should sign it. As a working rule, treat any profile where more than one move in five flips the winner as a tie.
What a large catalog, a certificate or a case study cannot score
None of them can fill a cell in this scorecard, because each measures something other than the offered records against your use. They can help build a shortlist; they cannot rank offers.
A catalog count says how many records exist, not how many keep their links after export. A security certificate describes controls over systems, not the fit or rights of a dataset. A case study reports a selected result on someone else's task. Even curated public benchmarks carry defects: one audit of 10 widely used test sets estimated an average of at least 3.3% label errors (Northcutt and colleagues, arXiv 2103.14749). Only a sample drawn by your method and measured against your definitions shows what you would receive.
The evidence should also reach whoever signs. NIST's Generative AI Profile (NIST AI 600-1) recommends sharing pre-deployment test results with the people who hold release approval authority. In procurement terms, the scorecard, the sample measurements and the flip test go to the budget owner with the recommendation, not a single number on a slide.
Writing up the result: buy, pilot or pass, with the limits stated
Close the scorecard with a short written decision: which vendor, for which use, under which conditions, and what evidence would change the answer. One page is enough if it carries the numbers and names the trade-off.
At a glance
Writing up the result: buy, pilot or pass, with the limits stated
State the limits plainly. A sample score estimates the offer; it does not measure model improvement, which only a pilot on your task can show. Weights are judgments and floors are policy choices, and a vendor that scored well on a sample can still deliver a full set that drifts from it, so the acceptance test has to apply to every delivery.
The write-up should carry six items:
- Decision: buy, pilot with named conditions, or pass, for one stated intended use.
- Scores: each vendor's six criterion scores, floors applied, and both totals if two uses were considered.
- Flip test: the number of 10-point moves that change the winner, and the trade-off that settles a tie.
- Price rule: the stated score-versus-cost rule and the total cost of acquisition for each finalist.
- Conditions: the acceptance test, the correction process and the evidence each delivery must carry.
- Next evidence: the pilot measurement that would confirm or reverse the decision, and who owns it.
Sending a weighted requirement to an on-request supplier
Put the intended use, the acceptance test, the weights and the floors in the request itself, so suppliers know what will be measured before they offer anything. Sourcing on request works best when the buyer's definitions travel with the request.
SourceX does not sell from stock: it sources each dataset from U.S. businesses only after a buyer has asked, and the request-data form records the buyer's scope, volume, language, format and timing next to its licensing needs, which is where weights and floors can travel. With suppliers, SourceX qualifies each source it finds, reviews its rights, handles contracting, prepares the data and handles delivery and payment, and no data reaches SourceX before the supplying company has accepted the scope and terms and signed its agreement, the document trail the rights floor asks for.
A buyer can attach this scorecard to a requirement. Whether a candidate dataset exists, and how it would score, is established case by case; nothing here implies stock, an acceptance result or a model improvement.
Frequently asked questions
What criteria belong in an enterprise dataset procurement scorecard?
Task fit, rights evidence and delivery reliability belong in every version. For context-rich business records, add link integrity, outcome attachment and history depth, each measured on a sample you drew. Keep price outside the weighted total and compare it using a rule you set in advance.
Why not let the highest total score win?
Because a high total can hide a disqualifying gap, such as rights that do not cover your use, and because small weight changes can reorder close vendors. Apply floors first, then check whether the winner survives 10-point weight changes before treating the total as a decision.
How do you measure completeness in linked business records?
Measure the links and the history, not only the row count: the share of records whose keys resolve to delivered related tables, the share with a final outcome and date, and the months of change history present against the months your use needs. Ask how each table was exported, since export routes and retention limits can drop history.
Should training and evaluation purchases use the same weights?
No. Evaluation cases need verified outcomes to grade anything, so outcome attachment and link integrity deserve more weight than in a training purchase, where coverage of case types and dependable delivery matter more. Score both profiles if a dataset may serve both uses.
How much should rights evidence count against price?
Rights evidence works best as a floor rather than a weight to trade: below the floor, no price makes the offer usable. Above it, deeper documentation can earn extra points. Counsel should review the license scope against the intended use before signing.
Sources
- FAR 15.304 requires factors and significant subfactors affecting award, and their relative importance, to be stated clearly in the solicitation, along with how non-cost factors compare with cost or price. Source
- Zendesk stores an audit for each ticket update, with change events recording previous and new values; the list-all-audits endpoint excludes archived tickets, whose audits come from the per-ticket endpoint. Source
- Salesforce retains field history for up to 18 months, and up to 24 months via the API, unless Field Audit Trail is used to keep it until deleted. Source
- Datasheets for Datasets proposes that every dataset be accompanied by a datasheet documenting its motivation, composition, collection process and recommended uses. Source
- MIT Media Lab reports that more than 70 percent of datasets in the Data Provenance Initiative audit carried unspecified licenses and that correct licenses were often more restrictive than those repositories assigned. Source
- Article 10 of the EU AI Act requires training, validation and testing data to be relevant, sufficiently representative and, to the best extent possible, free of errors and complete in view of the intended purpose. Source
- An audit of 10 widely used benchmark test sets estimated an average of at least 3.3% label errors. Source
- NIST AI 600-1 includes an action to share pre-deployment testing results with relevant AI actors, such as those with system release approval authority. Source
Tell us what data you need
Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.