Deal economics
What does it cost to prepare a business dataset? A cost study protocol
By SourceX Editorial · Updated
10 sections
Short answer
No reliable public figure exists yet for what it costs to prepare a real business dataset for licensing, because projects are not measured the same way. This page publishes a cost study protocol and a ten-line cost dictionary, from scoping and rights review to delivery and rework, with rules for loaded hourly rates, drivers, anonymity and uncertainty. It reports no costs or results.
Value drivers
Data licensing preparation
Key takeaways
- This page is a protocol and a cost dictionary; it reports no costs or results.
- The unit is one preparation project, and abandoned projects are logged so cost is not understated.
- Every hour or dollar falls on exactly one of ten dictionary lines, from scoping to rework.
- Internal hours are valued at a loaded rate that includes benefits, not salary alone.
- Results will be ranges by line and driver with company-clustered intervals, published only for groups large enough to protect participants.
- Preparation cost says nothing about value, price or the right to license.
What does it cost to prepare a real business dataset for licensing?
There is no reliable public figure yet, because the cost depends on which records, which systems and how much review the data needs, and nobody has measured it the same way twice. This page publishes a study protocol and a cost dictionary so that the answer can be measured consistently. It reports no costs and no results.
The study's question is: for a business that prepares one dataset from its own operational records for licensing to an AI developer, how much internal time and external spend does each preparation stage take, and which characteristics of the data explain the differences? The answer a reader should get, once results exist, is a range by stage and by driver, not a single average.
Until then, the cost dictionary below is useful on its own. A company can use it to plan and track its own preparation effort, and two companies using it will be counting the same things.
Figures that circulate today usually mix different things: one quote covers only a vendor's labeling invoice, another counts engineering time but not legal review, a third includes the cost of a failed first attempt. None is wrong on its own terms, but they cannot be compared or added up. A defensible answer needs the same boundaries, the same units and a disclosed sample, which is what the protocol fixes.
What is the unit of analysis and who is in the study?
The unit is a preparation project: the work to produce one dataset from one or more source systems for delivery under one license agreement. A company that prepared three datasets contributes three projects, and the analysis accounts for projects clustering within companies.
The target population is companies that completed at least one such project inside a stated window. The frame, its source and its coverage gaps are disclosed in the way survey research discloses a sample frame. Companies that started but abandoned a project are recorded separately, because leaving them out would understate cost: abandoned projects carry real spend and often fail at the most expensive stage.
Recruitment is by open invitation with a fixed screening form. Each participant gives informed consent for its cost data to be used in aggregate. The study team does not pay participants in a way that depends on their reported costs.
- In scope: projects that reached a signed agreement or a delivery, plus abandoned projects logged separately.
- In scope: internal staff time, contractors, vendor services, software and storage used for the project.
- Out of scope: the license fee or any revenue, which this study does not collect.
- Out of scope: ongoing refresh deliveries after the first, unless the protocol adds a second wave for them.
The cost dictionary: what counts in each preparation stage
The cost dictionary is a work breakdown for dataset preparation. Its rule, borrowed from public cost-estimating practice, is that every hour or dollar belongs to exactly one line, so nothing is left out and nothing is counted twice. Each line has a definition, what it includes, what it excludes and a unit of measure.
| Line | Definition | Includes | Excludes |
|---|---|---|---|
| 1. Scoping and inventory | Deciding which records, fields and years are offered | System inventory, field lists, sample pulls, scoping meetings | Sales calls before a scoping decision |
| 2. Rights review | Checking that the company can license the records | Contract review of customer and vendor terms, counsel hours, ownership questions | General legal retainer not tied to the project |
| 3. Privacy review and de-identification | Removing or transforming personal and sensitive data | Classification of fields, de-identification design and execution, disclosure-risk testing | Routine security operations |
| 4. Extraction | Getting records out of source systems | Export jobs, API scripts, history and audit-log pulls, engineer time | Platform subscriptions the company pays anyway |
| 5. Transformation | Making records usable and consistent | Schema mapping, joins across systems, deduplication, format conversion | Analytics work for internal use |
| 6. Quality assurance | Checking completeness and correctness | Validation rules, spot checks, fixes, re-runs | Quality work done before the project started |
| 7. Documentation | Describing the dataset for the licensee | Data dictionary, datasheet, known limits, change log | Marketing material |
| 8. Delivery | Moving data to the licensee | Transfer setup, encrypted media, storage during transfer, access control | Long-term hosting after delivery |
| 9. Project management | Coordinating the work | Planning, status meetings, approvals by executives | Time already counted on another line |
| 10. Rework and abandonment | Work repeated or discarded | Re-extraction after a scope change, work on an abandoned project | Planned iterations already on lines 4 to 6 |
How are hours turned into cost?
Internal time is the largest and least visible cost, so the protocol collects hours by role and line first and converts to money second. Converting at salary alone understates cost, because employers also pay benefits. The Bureau of Labor Statistics' Employer Costs for Employee Compensation program measures employer cost per hour worked for wages and for benefits, and its definition is the model for the loaded rate: total compensation cost divided by hours actually worked.
Participants provide their own loaded hourly rate per role when they have one. When they do not, the study applies a published loaded rate for the closest occupation group and region, named and dated in the results, and reports the share of hours valued that way. External spend is taken from invoices and recorded in the currency and year it was paid, then converted to a single base year with a disclosed index.
Many companies do not log time by project, so hours often have to be reconstructed after the fact. The protocol uses a structured interview for this: the participant walks through each dictionary line, names the people involved, and estimates hours from calendars, tickets, commit history and email threads rather than from memory alone. Each reconstructed figure carries a quality flag, and the analysis reports logged and reconstructed hours side by side, because recall tends to compress long, intermittent work such as rights review into a smaller number than it was.
Prospective projects are better evidence. Where a participant is starting a project during the study window, the team provides the dictionary as a time-tracking template up front, and those projects are analyzed as a separate, higher-quality stratum.
| Field | Unit | How it is collected | Quality flag |
|---|---|---|---|
| Hours by role and line | Hours | Timesheets or project tracker exports where available | Logged, reconstructed or estimated |
| Loaded hourly rate | Currency per hour | Company payroll figure or published fallback | Company rate or fallback rate |
| Vendor and contractor spend | Currency | Invoices allocated to a dictionary line | Invoiced or estimated |
| Software and storage | Currency | Incremental spend caused by the project only | Incremental or allocated |
| Calendar duration | Days | Start and end dates per line | Recorded or recalled |
Which drivers does the study record?
Cost only becomes useful when it can be tied to features of the data. The study records drivers that a company can see before it starts, so results can later be used for planning rather than only description.
Each driver is defined in advance with categories, so participants classify their own project the same way. Free-text explanations are collected too, but analysis uses the coded categories.
An illustrative case shows why drivers matter; it is invented and implies no result. Two companies each license several years of field-service work orders. The first exports structured fields only, from one cloud system, with technicians identified by role. The second includes technician notes and customer emails from two systems that must be joined on a shared job identifier. Even with similar record counts, the second project puts far more effort on privacy review and transformation, and the dictionary makes that visible line by line instead of hiding it in a single total.
- Number of source systems and whether they are cloud applications, on-premises databases or file shares.
- Record count and years of history in the delivered dataset, as ranges.
- Whether the dataset includes free text such as notes, emails or ticket comments, which usually drives privacy review effort.
- Categories of sensitive data present, such as financial account data or health information, without recording any values.
- De-identification approach, such as suppression, generalization or pseudonymization, and whether an outside party performed it.
- Whether field change history or audit logs were included, not just current records.
- Share of work done by internal staff versus vendors.
- Company size band and NAICS sector, for grouping only.
How is the study itself kept anonymous?
Cost data is commercially sensitive, so the published results must not let a reader identify a participant. The protocol treats this as a disclosure-control problem, the same kind NIST's de-identification guidance addresses for datasets about individuals and establishments.
Practical rules: raw submissions are held by the study team only; company names are replaced with random identifiers on receipt; results are published as ranges and medians for groups that meet a minimum number of companies fixed before collection; small groups are merged or suppressed; and a reviewer outside the analysis team checks every table for disclosure risk before release. Participants see their own data and the aggregate, never another participant's rows.
Drivers need the same care as costs. A table that shows one company in a narrow sector with an unusual system mix can identify it even without a name, so driver categories are coarse by design and cross-tabulations are checked against the minimum group size. NIST's guidance also describes governance options such as a disclosure review board; a small study can meet the same aim with a written release rule and a reviewer who signs off on every published table.
How are skewed project costs summarized without misleading anyone?
Cost data is skewed: a few projects with legal disputes or major rework can cost far more than the rest. The protocol therefore reports medians and interquartile ranges by line, not means alone, and gives bootstrap confidence intervals that resample companies, not projects, so clustering is respected.
The baseline is the dictionary itself: every result is reported per line, so a reader can compare like with like. Normalized views, such as cost per million records or per source system, are shown only next to the raw totals, because normalization can make large, simple projects look cheap and small, messy ones look expensive.
Following cost-estimating practice, the analysis includes a sensitivity check that varies the main assumptions, such as the fallback loaded rate and the treatment of estimated hours, and reports how much the ranges move. Results based on recalled rather than logged hours are shown separately as a robustness check.
What will the cost results leave unanswered?
The study would not show what a dataset is worth or what a license pays; those are different questions and the study does not collect them. It would not show the cost for a company unlike the participants, and early waves will over-represent companies that already finished a project, which tends to mean better-organized records.
It would also not show cost as a reason to license or not. A company's decision depends on rights, privacy obligations and fit with buyer needs, and those come before cost. Results describe effort, not obligations, and any line that involves law, such as rights review, is general information rather than advice for a specific company.
Finally, the study measures cost at one point in time. Tooling for extraction, de-identification and documentation changes quickly, so results carry their collection dates and should be refreshed in new waves rather than treated as permanent. A later wave that shows lower effort on a line may reflect better tools, a different mix of participants, or both, and the protocol records enough about each project to tell those apart.
- Not supported: a single typical cost for preparing a business dataset.
- Not supported: claims about license prices, revenue or return on effort.
- Supported: dated ranges of hours and spend by dictionary line and driver, with intervals and stated coverage.
Pre-registration checklist for a preparation cost study
Register the protocol publicly before the first submission arrives, so the confirmatory analysis is fixed and any later exploration is labeled as such. Report deviations with dates and reasons.
At a glance
Pre-registration checklist for a preparation cost study
Checklist · 10 items
- Research question and the planning decisions results are meant to support.
- Population, frame source, recruitment method and coverage gaps.
- Collection window and the window in which included projects finished.
- The cost dictionary version, with line definitions frozen.
- Loaded-rate rules, fallback source and base-year conversion index.
- Driver definitions and categories.
- Minimum group size for publication and the suppression rule.
- Summary statistics, interval method and clustering approach.
- Sensitivity and robustness checks to be run.
- Datasheet for the study dataset, covering motivation, composition, collection process and recommended uses.
0 of 10 ticked. Tick each item as you go.
How to use the dictionary now, and where SourceX fits
Any company considering a license can use the dictionary today as a planning sheet: list the lines, estimate hours by role, and mark which lines an outside party would carry. Tracking actual hours against it during the project produces the evidence a finance team will want later, for example when deciding how preparation costs are treated in the accounts.
Where a company does go ahead, SourceX handles rights review, contracting, data preparation, delivery and payment for selected business records licensed as AI datasets. Two of its standing terms touch dictionary lines 3 and 8 directly: SourceX performs de-identification through approved processing routes, and large deliveries either remain in the seller's own storage or ship on encrypted drives. Whether a company qualifies is settled by a brief assessment with nothing uploaded.
Frequently asked questions
Does this page say how much dataset preparation costs?
No. It publishes the study protocol and the cost dictionary only. Results will be published after data collection under the pre-registered plan, as ranges by line and driver with intervals.
What is a cost dictionary?
It is a fixed list of cost lines, each with a definition, what it includes and what it excludes, so every hour or dollar falls on exactly one line. It lets two companies, or two projects at one company, be compared on the same basis.
Why count internal hours instead of only invoices?
Most preparation work is done by staff who are already on payroll, so invoices miss much of the effort. Hours by role, converted at a loaded rate that includes benefits, capture that hidden cost.
How are participants protected?
Names are replaced with random identifiers, results are published only for groups above a minimum size fixed in advance, small groups are merged or suppressed, and an outside reviewer checks tables for disclosure risk before release.
Is de-identification part of preparation cost?
Yes. Classifying fields, designing and running de-identification, and testing disclosure risk form one dictionary line. Free text and sensitive categories are recorded as drivers because they tend to change the effort on that line.
Does preparing a dataset give a company the right to license it?
No. Preparation and de-identification do not create a right to license. Ownership, customer and vendor contract terms, and privacy permissions are reviewed separately, and counsel reviews the specific facts.
Sources
- GAO's Cost Estimating and Assessment Guide calls for a product-oriented work breakdown structure detailed enough that cost elements are neither omitted nor double-counted, plus sensitivity and risk and uncertainty analysis. Source
- The BLS Employer Costs for Employee Compensation program measures employer cost per hour worked for wages and salaries and for benefits, dividing total cost by hours worked. Source
- NIST SP 800-188 gives guidance on de-identifying datasets to limit disclosure risk to individuals and establishments while supporting meaningful analysis, including governance such as disclosure review boards. Source
- Preregistration separates hypothesis-generating (exploratory) research from hypothesis-testing (confirmatory) research, and planning the analysis in advance improves quality and transparency. Source
- AAPOR's disclosure standards ask researchers to state dates of data collection, the sample frame and its source, coverage gaps, and sample sizes. Source
- Datasheets for Datasets proposes that every dataset be documented with its motivation, composition, collection process and recommended uses. Source
See if your company qualifies
A short company assessment. No data uploads are needed.