Should-Cost Modeling for a Dataset Purchase

What a should-cost model tells you, and what it cannot

A should-cost model answers "what would an efficient supplier spend to deliver this exact specification," not "what is this data worth to us." Federal acquisition practice draws the same line: price analysis compares a total price against other prices, while cost analysis examines the individual cost elements behind it, and should-cost reviews ask what a program ought to cost if run efficiently [1]. Data buyers can borrow that discipline without borrowing the regulation.

The model is strongest where labor dominates: commissioned collection, annotation, preference ranking and evaluation sets. It is weakest for rare operational records, such as a decade of resolved support tickets or engineering change orders, where the supplier's cost to export is small and the price reflects scarcity. Pair it with a value estimate (see estimating a dataset's value before you buy) and with a relative check across bids (see normalizing vendor quotes to cost per usable record).

Cost components to build up line by line

Every dataset should-cost draws on the same ten or so buckets, and most quotes that look high are hiding one large bucket. In-house annotation cost breaks further into workforce, tooling, QA and ramp-up [2]. The table below turns those into model rows with the driver you should ask about for each.

Illustrative example: invented to show structure; it does not describe an available dataset.

Worked example: an hours-based model for a commissioned labeling set

The fastest credible should-cost is built in hours, then priced with a loaded rate you can defend. The example below models 20,000 support conversations labeled for intent, resolution outcome and escalation reason, with dual labeling on a share of items. Every input is invented to show the arithmetic; replace each with your own pilot timing or the supplier's disclosed figures.

Illustrative example: invented to show structure; it does not describe an available dataset.

INPUTS (assumed, replace with pilot data)
  items                         = 20,000 conversations
  minutes_per_item_label        = 3.0
  dual_label_share              = 0.15   (QA overlap)
  adjudication_min_per_dispute  = 4.0, dispute_rate = 0.20 of overlapped items
  rework_rate                   = 0.05
  deid_review_sample            = 1,000 items x 1.5 min
  extraction_engineer_hours     = 60
  rights_review_hours           = 25
  pm_and_rampup_hours           = 80
  packaging_hours               = 30

HOURS
  labeling      = 20,000 x 3.0 / 60                    = 1,000.0
  QA overlap    = 20,000 x 0.15 x 3.0 / 60             =   150.0
  adjudication  = 20,000 x 0.15 x 0.20 x 4.0 / 60      =    40.0
  rework        = 20,000 x 0.05 x 3.0 / 60             =    50.0
  de-id review  = 1,000 x 1.5 / 60                     =    25.0
  annotator subtotal                                   = 1,265.0
  specialist subtotal (extract + rights + PM + pack)   =   195.0

SHOULD-COST = (annotator_hours x annotator_rate)
            + (specialist_hours x blended_specialist_rate)
            + tooling + transfer
            then x (1 + margin)

Two things usually fall out of this exercise. First, labeling minutes per item dominate, so a 30-second error in that assumption shifts about 167 annotator hours, more than the rights review and packaging lines combined. Second, the QA overlap and rework lines are where quotes most often diverge, which is why your acceptance criteria for licensed training data should be fixed before you price.

Licensed existing records: cost is a floor, not the price

For data that already exists in a company's systems, the should-cost captures the supplier's preparation work, but the price also reflects scarcity, competitive sensitivity and the supplier's internal approval burden. A dataset of resolved insurance claims or field-service notes may cost a supplier little to export, yet carry real legal review, de-identification and executive sign-off costs that a pure labor model misses. Treat the model as a reasonableness floor: if a quote is below it, ask what step is being skipped, typically rights review or de-identification QA.

When the quote is far above the floor, the gap is a value and scarcity premium, and the right tool is a value estimate and, if you have no alternative supplier, a sole-source justification for proprietary data. For the qualitative drivers behind licensed-data pricing, see what drives the price of licensed enterprise data. If you are still deciding which route to take, compare custom collection against licensing existing records first.

Calibrating the inputs so the model survives a challenge

A should-cost model is only as defensible as its per-item timings, so calibrate them from a pilot rather than vendor rate cards. Time a small batch against the real guideline, with the real tool, by annotators at the skill level the task requires; ask for the same sample a supplier would use to scope (see requesting a training data sample).

Label granularity is the largest single lever. Research on chest-radiograph detection found that finer-grained annotations cost more to produce and that the extra cost should be weighed against the detection benefit it actually buys [3]. Preference and ranking data has its own economics: one RLHF study notes that preference annotation carries a direct monetary cost while the economic value of the resulting dataset is rarely assessed [4]. Curated instruction data is cheap in volume but expensive per item; LIMA reached competitive results from 1,000 examples, and its authors note that curating them was labor-intensive [5].

Delivery mechanics belong in the model too. If data moves out of an S3 bucket configured as Requester Pays, the buyer pays request and download charges while the owner pays storage [6]. A Snowflake Secure Data Sharing arrangement gives read-only access without copying data between accounts, which changes the transfer line but not the preparation work [7].

Using the model to negotiate on the biggest drivers

Negotiate the two or three rows that carry most of the cost, not the headline number. Share your model structure (not necessarily your rates) and ask the supplier to fill in the same rows; a cost-element comparison exposes whether the gap is rate, productivity, QA scope or margin [1].

Practical levers that move the total without lowering quality:

• Granularity: label spans only where the model needs them; use document-level labels elsewhere.
• QA overlap: set dual-labeling to what your agreement metric actually requires, then hold the supplier to it.
• Rework terms: tie rework to acceptance criteria so the cost is not prepriced at a pessimistic rate.
• Volume: re-check the record count against how much training data you actually need; buying 40 percent fewer records beats a 10 percent rate cut.
• Delivery: agree the format and channel early so packaging and egress are not repriced later.

Carry the agreed figures into your AI training data budget and your total cost of ownership for licensed training data, which adds the internal costs a supplier quote never shows.

Common failure modes in dataset should-cost models

Most broken models fail the same way: they price the happy path and omit the people who say no. Watch for these:

1. Zero legal line. Rights and consent review is treated as free because "the company owns the data."
2. De-identification as a script cost only. The script is cheap; reviewing free text, attachments and edge cases is not.
3. No ramp-up. Calibration rounds and guideline revisions are real hours, especially in a narrow domain [2].
4. Ignoring rejection. A model with no rework line assumes perfect first-pass quality.
5. Pricing scarcity as labor. Applying a labor model to rare records and concluding the supplier is overcharging, when the premium is for access, not effort.
6. Mixing units. Comparing a per-record quote to a per-hour model without converting through items per hour.

How SourceX approaches pricing on sourced datasets

SourceX sources operational datasets from US companies on request and manages the commercial process, including licensing agreements and ongoing purchases; categories are examples, not inventory, and a request does not guarantee a match. Each dataset is rights-reviewed for ownership and consents, personal details are removed or replaced before delivery with the method recorded and a sample checked (no method is perfect), and the license defines records, uses, term and delivery. SourceX does not publish prices; terms are agreed per deal, and nothing is contracted until a supplier agrees. You can describe the data you need to SourceX and bring your should-cost structure to that conversation. More procurement guides sit in the procurement hub and the wider AI data library.

Test a dataset price against your should-cost estimate

SourceX looks for US businesses that hold the data you describe, assesses data and licensing permissions, and agrees pricing and allowed uses in a license per deal. Diligence materials on source, rights, preparation and allowed use are prepared per dataset. Describe the dataset you want to price.