Deal economics
Dataset rework acceptance cost: count every cycle, not just the yes
By SourceX Editorial · Reviewed by Noah Loul ·
9 sections
Short answer
Dataset rework acceptance cost is everything both sides spend between the first failed acceptance test and the final decision: supplier rework, buyer re-review and retest, calendar days and any concession accepted instead of a fix. Keep one record per submission with its version, failing cases, reason code and hours, route each failure before fixing it, and label the outcome honestly: first-pass, reworked, conceded or narrowed.
Value drivers
Dataset rework acceptance
Key takeaways
- An accepted status hides how many cycles a delivery took; first pass yield, as ASQ defines it, counts only units that pass without being rerun or retested.
- Rework costs the acceptance side too: re-review hours, retest runs and delay, which federal supply contracts let the buyer charge back after a rejection.
- Not every failed cycle is rework: a new buyer requirement is scope work, a wrong gold label is a test fix, and rights or terms problems have no technical fix.
- Four labels replace a bare accepted status: first-pass, accepted after rework, accepted with concession and accepted on narrowed scope.
- Count by submission cohort and treat open deliveries as censored, so slow and failed deliveries do not drop out of the rate.
When can a dataset delivery be called successful?
Only when the final version passes the agreed acceptance test and the record shows how it got there: how many submissions it took, why each failed, how much time each side spent and what was conceded. A delivery accepted on its third cycle and one accepted on its first share a status but not a result.
Manufacturing quality settled this long ago. The ASQ quality glossary defines first pass yield as the percentage of units that complete a process and meet quality guidelines without being scrapped, rerun, retested, returned or diverted into an offline repair area. Dataset teams rarely track the equivalent. Delivery trackers keep the last status, so three rounds of fixes collapse into one green check and the next estimate starts from a cost that never happened.
This guide defines the dataset rework acceptance cost and gives four tools to capture it: a cycle record, a routing table for each failed cycle, four labels that replace a bare accepted status, and a worked comparison of two deliveries.
What rework costs the acceptance side, not just the supplier
Every failed cycle also costs the buyer: someone re-reviews the delivery, reruns the acceptance test and waits. Counting only supplier hours understates the cost of a reworked delivery and misprices the next one.
Cost-of-quality accounting supplies the categories. ASQ splits quality costs into four groups: internal failure costs for defects found before the customer receives the product, external failure costs for defects found after, appraisal costs for checking conformance and prevention costs for keeping the other three low. A defect the supplier's own checks catch is an internal failure; the same defect caught by the buyer's acceptance test is an external failure, and it brings appraisal cost on both sides with it.
Mature procurement treats retesting as a real, chargeable cost. The standard federal inspection clause for fixed-price supply contracts, FAR 52.246-2, lets the contracting officer charge the contractor for any additional cost of inspection or test when a prior rejection makes reinspection or retest necessary. Dataset agreements rarely say anything similar, so retest cost lands wherever it happens to fall.
The buyer-side items worth logging per cycle are review hours, compute or annotator time to rerun the test, the days the downstream training or evaluation plan slipped, and time spent disputing the result. Hours and days are enough; converting them to money is a separate decision for each organization.
The cycle record: what each submission must leave behind
Open one row per submission, not per delivery, and keep every row after the final acceptance. The rows are the operating record that later decisions read: estimates, supplier comparisons, renewal terms and the next request.
Versioning makes the rows comparable. The Croissant metadata specification recommends semantic versioning for datasets, says the major version should change when existing data has been edited, removed or shuffled across splits, and pairs the version with file checksums. Hugging Face's hub client can download files at a specific tag or commit hash, which lets a retest run on exactly the files that were submitted.
The eight fields, with the decision each one supports:
| Field | What to record | Decision it supports | Example (Illustrative) |
|---|---|---|---|
| Cycle and version | Submission number, dataset version and file checksums | Proves the retest ran on the submitted files | Cycle 2, version 2.0.0 |
| Test version | The acceptance test and grader version used | Separates a data change from a test change | Acceptance test v1.1 |
| Failing cases | Case IDs, the count and the number of cases tested | Sizes the failure and shows whether it shrank | 9 of 400 cases |
| Reason code | One code: scope, measurement, rights or terms, task fit, data defect | Routes the cycle to the right remedy | Scope: field added after signing |
| Found by | Supplier pre-check or buyer acceptance test | Separates internal from external failure | Buyer acceptance test |
| Hours by side | Supplier rework hours and buyer review and retest hours | Prices the cycle for both parties | 16 supplier, 6 buyer |
| Remedy and approver | Repair, change order, test fix, concession, narrowed scope or stop, and who approved it | Shows who decided and on what authority | Change order approved by the buyer's data lead |
| Dates | Submission, review start and decision dates | Measures calendar time per cycle | Submitted day 18, decided day 29 |
Routing a failed cycle before anyone starts fixing
Route every failed cycle through six questions in order and stop at the first yes. The order checks the paper and the test before touching the data, because rework applied to a scope gap or a test error adds a cycle without bringing acceptance any closer.
The routing reconciles three sources of truth that are easy to confuse: the approved scope both sides signed, the supplier's measured quality on its own checks, and the buyer-specific acceptance evidence from the buyer's test. A failure that shows up in only one of the three is usually not a data defect.
A no at all six steps means the failure is not yet explained; escalate to the people who own the agreement before another cycle starts. The six steps:
| Step | Question (yes means the cause is here) | Route | Log the cycle as |
|---|---|---|---|
| 1. Scope | Is the failing requirement outside the signed scope or the agreed test version? | Change order, priced customization or narrowed scope | Scope work, not rework |
| 2. Measurement | Does the failure vanish when the agreed test is rerun on the pinned version, or trace to a wrong gold label? | Fix the test or the labels; the data stays as submitted | Test error |
| 3. Rights and terms | Is the objection about permissions, license terms or price rather than the records? | Renegotiate or stop; no rework | Rights or terms |
| 4. Task fit | Do correct records still lack the decisions or outcomes the buyer's task needs? | Re-scope to a task the records support, or stop | Task mismatch |
| 5. Data defect | Does a measured defect in the delivered records explain the failing cases? | Repair at the stage that created it, issue a new major version, rerun the full test | Rework |
| 6. Cycle cap | Has the agreed limit on resubmissions been reached? | Written concession with consideration, narrowed scope or end the delivery | Concession or closure |
Four labels that replace a bare accepted status
Replace accepted with one of four labels, plus not accepted, and report each label's count against every delivery submitted in the period. The labels keep the record honest about what the buyer actually received.
At a glance
Four labels that replace a bare accepted status
Federal acquisition rules draw the same line between a conforming delivery and a conceded one. FAR 46.407 says critical or major nonconformances are normally rejected, allows acceptance when the contracting officer finds it in the government's interest, and requires the contract to be modified for an equitable price reduction or other consideration when such nonconformances are accepted. A concession is a different outcome from a pass, at a different price.
The labels yield the metrics a delivery owner should report: first-pass acceptance rate, mean cycles to acceptance, defect rework hours per accepted delivery on each side, concession rate, and calendar days from first submission to final decision. Count each by submission cohort, meaning deliveries first submitted in a period, and treat deliveries still open on the reporting date as censored instead of dropping them, or slow deliveries vanish and the operation looks faster than it is.
The labels, in the order they should be preferred:
- First-pass accepted: the first submitted version passed the agreed test. Only these count toward first pass yield.
- Accepted after rework: a later version passed after one or more defect repairs. Record the cycle count and the hours on each side.
- Accepted with concession: some failing cases were accepted in writing instead of fixed, with the consideration and the approver named.
- Accepted on narrowed scope: the buyer accepted less than was signed, such as fewer record types or a shorter period, after a scope or task-fit finding.
- Not accepted: the delivery ended without acceptance. Keep its cycle rows; they are evidence for the next qualification decision.
Reconciling measured quality with the buyer's own evidence
When the supplier's quality checks pass and the buyer's acceptance test fails, assume either side's measurement may be wrong until the failing cases have been inspected one by one.
Even well-studied test sets carry errors. Northcutt, Athalye and Mueller estimated an average of at least 3.3 percent label errors across the test sets of 10 widely used datasets, including at least 6 percent of the ImageNet validation set, and showed that corrected labels could change which model ranked higher. A private acceptance set built quickly for one delivery deserves the same suspicion: a wrong gold label turns a correct record into a failing case, and step 2 of the routing exists to catch it.
Documentation closes the loop. The maintenance questions in Datasheets for Datasets ask whether there is an erratum, whether the dataset will be updated to correct labeling errors or to add or delete instances, and whether older versions will continue to be supported. Answering them for every cycle tells the buyer which records changed between versions and which earlier versions remain valid for comparison.
The next evidence worth collecting is case-level agreement between the supplier's checks and the buyer's test on a shared sample, so both sides know which measurement to trust before the next cycle opens.
Cycles that rework cannot end: rights, terms and task fit
Some rejections have no technical fix, and every repair cycle spent on them is cost with no path to acceptance. Route them out at steps 3 and 4 and record them under their real reason.
Rights problems are common enough to check first. Published in Nature Machine Intelligence in 2024, with MIT Media Lab researchers among its authors, the Data Provenance Initiative's license study covered more than 1,800 text datasets and reported that widely used hosting sites omitted licenses at rates above 70 percent and mislabeled them at rates above 50 percent, and that the correct license was often stricter than the assigned one. No cleaning step changes what a license allows.
Commercial terms can arrive dressed as quality complaints. A buyer that wants a lower price or a narrower license may cite defects it would otherwise waive; if the same failing cases passed in an earlier cycle, or the objection moves every round, log the cycle as terms and take it to the people who own the agreement. Task fit fails differently: records can be correct and complete and still miss the decisions or outcomes the buyer's model must learn, which no amount of cleaning adds.
These cycles still belong in the record. They show which qualification checks failed upstream, and they keep rework metrics honest by separating repairable defects from deliveries that were never going to pass.
Illustrative: two deliveries with the same accepted status
Two illustrative support-ticket deliveries, each tested against the same 400-case acceptance test, both end with an accepted status. Their cycle records tell different stories.
Delivery A passed on its first submission. The buyer spent 6 hours reviewing it, and it was accepted 9 days after submission. Label: first-pass accepted.
Delivery B took three cycles. Cycle 1 failed 38 of 400 cases because resolution codes were missing from part of the export, a measured defect inside the signed scope; the supplier spent 30 hours repairing it at the export stage and the buyer 7 hours on review. Cycle 2 failed 9 of 400 cases on a field the buyer's team added after signing; routed at step 1, it became a change order, with 16 supplier hours and 6 buyer hours. Cycle 3 passed 397 of 400 cases, and the buyer accepted the remaining 3, which depended on a field the source system never captured, under a written concession after 5 hours of review.
Delivery B's record shows 46 supplier hours, 30 of them defect rework and 16 change-order work, 18 buyer hours across three reviews, and 41 calendar days from first submission to acceptance. Label: accepted after rework, three cycles, with a concession. A tracker that showed only accepted would plan the next delivery like A and would hide that one of B's three cycles was paid scope work, not a supplier failure.
Putting the cycle record into a data request
Ask for the cycle record before the first delivery: name the acceptance test and its version, the reason codes, the cycle limit and how retest time is handled, so both sides log the same facts from cycle 1.
SourceX keeps no stock of datasets; it goes looking among U.S. businesses once a buyer's request arrives, and its request-data form captures the buyer's scope, the volume and language wanted, the file format, the timing and the licensing needs. Because the supplying company signs the agreement, having approved scope and terms, before any record is handed over, step 1 of the routing always has a signed document to check. Include the intended use and an acceptance test in the request; whether a dataset is accepted, or improves a model, depends on the records and the test, and neither outcome is assured in advance.
Limits of this method: the cycle record measures effort and outcomes per delivery; it does not show whether an accepted dataset improved a model, and its hour counts are only as good as both sides' logging. SourceX has not published cycle, rework or acceptance figures, and none are implied here.
Frequently asked questions
What is dataset rework acceptance cost?
It is the total effort and delay between a dataset's first failed acceptance test and the final decision: supplier repair hours, buyer re-review and retest hours, calendar days and any concession accepted instead of a fix. Measure it per delivery across every submission cycle, not only on the version that finally passed.
Should a delivery accepted after rework count as a success?
It counts as accepted, but not as first-pass. Report it under its own label with the number of cycles and the hours on each side, so rework does not disappear into an overall acceptance rate.
Who pays for retesting a resubmitted dataset?
Whoever the agreement names; most dataset agreements are silent, so the buyer absorbs it. Federal supply contracts under FAR 52.246-2 let the government charge the contractor for reinspection or retest made necessary by a prior rejection, a model a data agreement can adapt with counsel's review.
What is the difference between rework and a change order?
Rework repairs a defect against the signed specification; a change order adds or changes a requirement after signing. Logging a change order as rework blames the supplier for scope growth and overstates its defect rate.
How many resubmission cycles should a dataset agreement allow?
There is no standard number. Agree a cap before delivery and say what happens when it is reached: a written concession with consideration, a narrowed scope or the end of the delivery. Without a cap, cycles can continue past the point where either side gains.
Sources
- ASQ defines first pass yield as the percentage of units that complete a process and meet quality guidelines without being scrapped, rerun, retested, returned or diverted into an offline repair area. Source
- ASQ divides quality costs into internal failure, external failure, appraisal and prevention costs, with internal and external failure split by whether the customer has received the product. Source
- FAR 52.246-2 lets the contracting officer charge the contractor for additional inspection or test costs when a prior rejection makes reinspection or retest necessary. Source
- FAR 46.407 says critical or major nonconformances are normally rejected and that accepting them requires a contract modification for an equitable price reduction or other consideration. Source
- The Croissant 1.1 specification versions datasets with a version plus file checksums and increments the major version when existing data is edited, removed or shuffled across splits. Source
- The Hugging Face hub client downloads files at a particular branch, tag or commit hash through its revision parameter. Source
- Northcutt, Athalye and Mueller estimated at least 3.3 percent label errors on average across 10 widely used test sets, at least 6 percent in the ImageNet validation set, and showed corrected labels can change model rankings. Source
- Datasheets for Datasets asks maintenance questions about errata, updates that correct labeling errors or add and delete instances, and continued support for older versions. Source
- An audit of more than 1,800 text datasets found license omission rates above 70 percent and error rates above 50 percent on widely used hosting sites, with correct licenses often more restrictive. Source
Tell us what data you need
Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.