Privacy and preparation
Dataset buyer feedback taxonomy: turning complaints into verifiable repairs
By SourceX Editorial · Reviewed by Noah Loul ·
10 sections
Short answer
To turn buyer feedback into a verifiable dataset repair, classify each item as a defect, a utility gap, a rights issue or a scope request, then rewrite it as a testable claim about named records. Trace those records to their source and to every derived file and recipient, fix the earliest stage that caused the problem, prove it with a check that failed before, and release a new version with a recipient log.
Value drivers
Dataset buyer feedback
Key takeaways
- Only defects are repairs to the data; utility gaps, rights issues and scope requests each need a different owner and response.
- Rewrite every complaint as a testable claim about specific records before anyone changes anything.
- Rights and privacy issues come first, because each further copy or training run makes them harder to undo.
- Lineage from record ID to source, derived files and recipients is what answers who is affected by a change.
- A repair is done when a check that failed before now passes, the version number reflects the change and the buyer has confirmed it.
How do you turn buyer feedback into a specific, verifiable dataset repair?
Sort each piece of feedback into one of four classes (defect, utility, rights or scope), rewrite it as a testable statement about named records, trace those records back to their source and forward to every derived file and recipient, fix the problem at the earliest stage that caused it, and prove the fix with a check that would have failed before. Then release a new version and log who received what.
The classification step matters most because each class has a different owner and a different kind of fix. A defect is the data being wrong against its own source or specification. A utility gap is the data being correct but less useful for the buyer's task than hoped. A rights issue is a question about whether the records may be used or shared at all. A scope request asks for something the agreement never covered. Treating all four as "fix the data" sends rights questions to engineers and turns scope requests into unpaid rework.
The rest of this page gives the taxonomy, the issue record that carries each item from complaint to closure, severity rules, a worked example and the cases where feedback should not become a repair at all.
Why does "the data is bad" need a class before anyone touches it?
Because data problems compound and surface late, a fix aimed at the wrong cause tends to create a second problem downstream. Sambasivan and colleagues, interviewing 53 AI practitioners, described data cascades as compounding events with negative downstream effects from data issues, and found them pervasive, often invisible and delayed, but avoidable.
Buyer feedback usually arrives as a symptom: a model trained on the delivery underperforms on refunds, or an evaluator notices the same ticket twice. The symptom can come from a defect (duplicates), a utility gap (too few refund cases), or a rights decision (refund disputes were deliberately excluded). Each needs a different response, and only one of them is a repair to the data.
A shared vocabulary also helps the buyer. When the supplier replies "classified as utility, coverage subtype, not a defect against the specification", the conversation moves to whether a new collection is worth commissioning, rather than an argument about whether the delivery failed.
The defect, utility, rights and scope feedback taxonomy
This is the asset, version 1.0 dated 2026-10-18. Defect subtypes follow the data quality characteristics in ISO/IEC 25012, which groups fifteen characteristics into inherent and system-dependent views; the subtypes below are the ones buyer feedback most often names. Each row gives what buyers typically say, the usual repair and the check that proves it.
| Class and subtype | What the buyer usually says | Typical repair | Proof the repair worked |
|---|---|---|---|
| Defect: accuracy | Labels or field values do not match the source record | Correct at the extraction or labeling stage, re-derive affected files | Re-check a fresh sample of affected records against source; error rate below agreed threshold |
| Defect: completeness | Fields are blank or related records are missing | Fix the join or filter that dropped them; backfill from source | Null and orphan counts per field match the specification |
| Defect: consistency | Duplicates, contradictions between files, mixed codes | Deduplicate on a stable key; map codes to one scheme | Duplicate-key and cross-file contradiction checks return zero |
| Defect: currentness or format | Stale snapshot, wrong time zone, broken encoding | Re-extract or re-serialize; no change to content | Checksums and schema validation pass; content diff is empty |
| Utility: coverage or distribution | Too few cases of a type the buyer cares about | Not a repair; propose a new collection or a different slice | Buyer confirms the counts by type before any new work |
| Utility: granularity | Codes are too coarse to train on | Not a repair unless finer source codes exist and were in scope | Finer codes trace to source fields, not to after-the-fact relabeling |
| Rights: permission or content | A third party's content, personal data or restricted records appear | Stop further distribution of affected records; remove or replace; escalate to the rights owner | Affected records absent from the new version; recipient notices logged |
| Scope: outside the agreement | Requests new fields, more volume, another system or a new use | Route to the commercial owner as a change request; no data change | Signed change in scope before any work starts |
What goes into an issue record so a repair can be traced end to end?
One issue record per feedback item, opened when the feedback arrives and closed only when the repair is verified and recipients are notified. The fields below let anyone answer three questions later: what exactly was wrong, what changed, and who has the old version.
At a glance
What goes into an issue record so a repair can be traced end to end
- issue_id, received_at, received_from (buyer role, not a person's name) and dataset_version the buyer was using.
- class and subtype from the taxonomy, with the name of whoever classified it.
- testable_claim: the complaint rewritten as a statement that can be checked, for example "ticket IDs repeat across the March and April files".
- affected_records: the record IDs or the query that selects them, plus a count.
- source_links and derived_links: the source system records behind each affected item, and every file, split or label set derived from them.
- root_cause_stage: extraction, transformation, de-identification, labeling, packaging or source system.
- repair: what was changed, at which stage, and by whom.
- regression_checks: the checks added or rerun, each with its result before and after the repair.
- recipients: every buyer and internal user who received an affected version, with the version and delivery date.
- change_notice: reference to the release note or notice sent, and the new version number.
- status: open, classified, repaired, verified, notified, closed, or declined with a reason.
How severe is the issue, and who has to hear about it first?
Rank by consequence, not by how loudly it was reported. Rights and privacy issues come first because every further copy or training run makes them harder to undo; defects that touch evaluation labels come next because they distort every measurement taken with them; utility and cosmetic items follow.
Where personal data is involved, the law may set the notification duty. Under Article 19 of the EU General Data Protection Regulation, a controller must tell each recipient of personal data about any rectification, erasure or restriction carried out under the related articles, unless that proves impossible or involves disproportionate effort, and must tell the data subject who those recipients are on request. Whether and how that applies to a given delivery is a question for counsel; the recipient log in the issue record is what makes compliance possible either way.
| Severity | Typical classes | First action | Who is told first |
|---|---|---|---|
| 1: stop | Rights, privacy, records that should never have been shared | Pause further delivery of affected records | Rights owner, privacy lead and every recipient |
| 2: measurement at risk | Defects in evaluation labels or test splits | Flag affected splits as unreliable | Recipients using those splits |
| 3: training quality | Defects in training records; consistency and completeness | Schedule repair in the next version | Recipients of the affected version |
| 4: packaging | Format, encoding, documentation errors | Repair in a patch release | Recorded in the release notes |
| Not a repair | Utility gaps and scope requests | Reply with the classification and options | The buyer who raised it |
Which released examples or recipients are affected by this source change?
Answer it from lineage, not memory. Every delivered record should carry a stable ID that maps back to its source record, and every derived artifact (splits, label files, aggregates) should list the record IDs it was built from. When a source record is corrected or withdrawn, a query over those links returns the affected files, and the recipient log returns who holds them.
Version numbers should say what kind of change happened. The Croissant 1.1 metadata specification recommends semantic versioning for datasets: a patch increment when the data is the same but serialized or packaged differently, a minor increment when data is added while old data stays retrievable, and a major increment when existing data is edited, removed or moved between splits. It also recommends recording file checksums so a recipient can confirm which files they hold.
Under that scheme most defect repairs are major releases, because they edit existing records. Treat this as useful information for the buyer rather than an embarrassment: results computed on the old version are not directly comparable with results on the new one.
How do you know the repair actually worked?
Write the check before the fix: a query, test or sample review that fails on the current version and should pass on the repaired one. Run it on the repaired version, then run the full existing test suite to make sure nothing else moved. Keep the new check permanently, so the same defect cannot return unnoticed.
Corrections themselves can be wrong. The label-errors project, which published corrected test labels for ten common machine learning benchmarks using confident learning and human validation, states plainly that its corrections are better than the original labels but not perfect, and invites further reports. Expect the same of your own repairs: verify a fresh sample, record the residual error estimate, and keep the issue reopenable.
Close the loop with the buyer. Ask whoever raised the issue to confirm on the new version, with the same test they used to find it. Their confirmation goes in the issue record.
Worked example (illustrative): three complaints about a support ticket dataset
The case is invented to show the method; it does not describe a real company, buyer or dataset. A software company delivered version 2.0.0 of a de-identified support ticket dataset. The buyer sent three comments in one email after a first training run.
- Comment 1, "some tickets appear twice": classified as defect, consistency. Testable claim: ticket IDs repeat across monthly files. Query finds 1.8 percent of tickets duplicated (illustrative figure), root cause a date filter with an inclusive boundary at month ends. Repair at extraction; regression check for duplicate IDs added; released as 3.0.0 because existing records were removed.
- Comment 2, "resolution codes are too coarse": classified as utility, granularity. The help desk system holds only six resolution codes, and finer reasons live in free text. Reply with the classification and an option to scope a labeling project; no change to 3.0.0.
- Comment 3, "some messages include signatures with names of a partner's staff": classified as rights and privacy, severity 1. Delivery of affected tickets paused; de-identification rule extended to signature blocks; affected records replaced in 3.0.0; recipient notice sent the same week and logged.
- A follow-up request for chat transcripts from a second system is logged as scope and passed to the commercial owner.
When should feedback be declined rather than repaired?
Decline, with a reason, when the change would make the data less true to its source. A buyer may ask for labels to be changed to match their own taxonomy, for outliers to be removed, or for a field to be filled in where the source system left it blank. Those are transformations the buyer can apply on their side; doing them in the supplier's release blurs what the records actually show.
Also stop and narrow the response when the feedback raises a question the data team cannot answer alone. Rights and consent questions go to whoever owns them, usually counsel and the business owner of the source system. Disputes about whether a delivery met an agreed acceptance threshold belong to the people who signed the agreement, not the issue queue.
Datasheets for Datasets asks creators, before distribution, whether there will be an erratum and how updates and corrections will be communicated to consumers. Answering that question up front, and pointing to the issue process, sets expectations before the first complaint arrives.
Agreeing the feedback loop before a dataset is licensed
A business licensing its records for AI benefits from settling the feedback process before signing: which classes it will repair, how versions are numbered, how recipients are notified and what counts as a scope change. SourceX runs that rights review, contracting, preparation and delivery when a business licenses selected operational records to AI developers.
Each deal's signed Dataset Deal Confirmation fixes the delivery date, exclusivity and term, and de-identification is done by SourceX through approved processing routes. A short company assessment, with no data uploads, is the first step.
Frequently asked questions
What is a dataset buyer feedback taxonomy?
It is a fixed set of classes for sorting what dataset buyers report, so each item reaches the right owner and the right kind of fix. The version here uses four classes: defect, utility, rights and scope. Each class has subtypes, a typical response and a check that proves the response worked.
Is a utility complaint ever a defect?
Only when the specification promised what the buyer says is missing. If the agreement specified a minimum count of a case type and the delivery fell short, that is a completeness defect. If it did not, the gap is a utility issue and the answer is a new collection or a different slice.
Should every repair produce a new dataset version?
Yes, and the version number should say what changed. Under the semantic versioning approach recommended in the Croissant specification, packaging-only fixes are patch releases, additions are minor releases, and edits or removals of existing records are major releases.
How long should the recipient log be kept?
For as long as any recipient may still hold or use an affected version, which in practice means the life of the license and any retention period after it. Without the log, a later correction or withdrawal cannot be passed on to the people who need it.
Who should classify incoming feedback?
One named owner per dataset, usually the data operations lead, with a fast route to the rights owner for anything that might be a rights or privacy issue. Classification should be recorded with the classifier's name so disagreements can be revisited.
Sources
- Croissant 1.1 recommends semantic versioning for datasets (patch for repackaging, minor for additions, major for edits or removals) and recording file checksums. Source
- The label-errors project published human-validated corrections to ten benchmark test sets and notes the corrections are better than the originals but not perfect. Source
- GDPR Article 19 requires notifying each recipient of rectification, erasure or restriction unless impossible or disproportionate, and naming recipients to the data subject on request. Source
- Datasheets for Datasets asks whether there is an erratum and how dataset updates and corrections will be communicated to consumers. Source
- Sambasivan et al. (CHI 2021) interviewed 53 AI practitioners and described data cascades as pervasive, invisible and delayed but often avoidable. Source
- ISO/IEC 25012 defines a data quality model of fifteen characteristics viewed as inherent or system dependent. Source
See if your company qualifies
A short company assessment. No data uploads are needed.