Dirty Tables with Ground-Truth Corrections for Data-Cleaning AI

Why synthetic error injection is not enough for data-cleaning models

Synthetic errors teach a model the noise generator, not the business. Research benchmarks show the gap: BIRD was built because earlier text-to-SQL sets held few real rows and ignored messy cell values [1], and Spider 2.0 treats enterprise work as wrangling and transformation on real application databases, not just querying [2]. Many academic cleaning benchmarks start from a clean table and inject errors at controlled rates, so ground truth is known by construction rather than observed.

Controlled injection makes measurement easy but distorts the distribution. Real errors cluster by source system, data-entry screen, migration batch and time period. A typo generator never reproduces a vendor master where "ACME Corp", "Acme Corporation" and "ACME CORP." were created by three different AP clerks over six years, or a ZIP code column silently truncated from ZIP+4 during a CSV export.

A few academic collections now use real errors, but public real-error sets remain small and narrow by domain. For a data-prep copilot that must work on ERP, CRM, billing and ticketing tables, you need licensed operational data where the corrections were made by people doing the job.

Where real before-and-after pairs exist inside businesses

The best ground truth is a correction someone already made and recorded for operational reasons. Look for these sources when you write a request:

• Master-data change history. SAP change documents (CDHDR/CDPOS), Oracle EBS audit tables, NetSuite system notes and MDM hub audit trails record old value, new value, user and timestamp per field.
• CRM deduplication and merge logs. Salesforce and HubSpot merge events record surviving and merged record IDs; these are also strong labels for entity matching, covered in entity resolution training data from master data.
• Data-migration and cleansing projects. ERP cutovers usually produce a legacy extract, a cleansed load file and a mapping or exception workbook. Those three files are a ready-made before/after/log triple.
• Data-quality tool outputs. Rule results and exception queues from dbt tests, Great Expectations, Informatica Data Quality or Talend record which rows failed which rule and how stewards resolved them.
• Finance reconciliations and journal corrections. Reversing entries, bank reconciliation adjustments and invoice-matching exceptions encode wrong amounts, wrong GL codes and wrong vendor IDs plus the fix.

Vendor guidance on ERP data for AI describes the same error families: duplicates, missing values and inconsistent records that must be resolved before use [6]. For broader context on transactional sources see ERP transaction and master data for AI training.

Which error types to request and how to label them

Ask for a fixed error taxonomy at cell level so detection metrics are comparable across suppliers. Detection systems that ensemble many detector configurations, each tuned to a different error family, can only be trained and diagnosed when your labels distinguish those families.

Illustrative example: invented to show structure; it does not describe an available dataset.

Keep two fields per labeled cell: error_type and correction_source (change log, steward decision, source document, external reference). Corrections derived from an external reference, such as geocoding or a postal reference file, should be flagged separately from human corrections because their error profile differs.

The record schema for a before/after correction dataset

Deliver dirty and clean versions as separate tables with stable row keys, plus a long-format correction log. Never accept only the cleaned table: once the original is overwritten, you cannot measure detection.

Illustrative example: invented to show structure; it does not describe an available dataset.

{
  "table_id": "vendor_master_v3",
  "row_key": "V-004817",
  "column": "postal_code",
  "dirty_value": "0213",
  "clean_value": "02139",
  "error_type": "TRUNC",
  "correction_source": "steward_decision",
  "rule_id": "ZIP_LEN_5_OR_9",
  "detected_by": "dq_rule",
  "corrected_at": "2024-03-18T14:02:11Z",
  "corrector_role": "AP data steward",
  "confidence": "verified_against_source_document"
}

Also ask for the data dictionary, allowed-value lists and any business rules active at the time of correction. Rules are both features and labels, and they are what a data-prep agent must learn to infer. The page on data dictionaries and schema documentation as AI grounding data covers what to request.

How to evaluate detection, repair and data-prep agents

Score detection and repair as separate tasks, because a model can flag the right cell and still propose the wrong value. Use these measures:

• Detection: cell-level precision, recall and F1 against the correction log, reported per error type. Aggregate F1 hides a model that catches every FMT error and misses every DUP.
• Repair: exact-match repair accuracy on detected cells, plus a normalized match for formats (dates parsed equal) and a tolerance for numerics.
• Over-correction rate: share of clean cells the model changed. In production this matters more than recall, since a copilot that rewrites valid data loses user trust.
• Downstream effect: train your own downstream model on the dirty and repaired versions and compare; a repair that does not move the end task is not worth automating.
• Agent tasks: for wrangling agents, score the final output table and the transformation script, in the spirit of Spider 2.0's workflow tasks [2].

ISO/IEC 5259-2 defines data quality measures and 5259-4 a process framework covering labeling for training and evaluation [4][5]; aligning your error types to those terms makes results easier to defend with internal reviewers. Treat the correction log itself as imperfect: label audits of widely used ML test sets found an average error rate of at least 3.3% [3]. Have a second annotator review a stratified sample of corrections and record agreement per error type before freezing an eval set. For more on that discipline, see building a golden evaluation dataset from real business records.

Diligence questions before you license dirty data

Dirty tables raise privacy and confidentiality questions that cleaned extracts hide. Errors often sit in free-text and contact fields, and the dirty version is where a name pasted into a "notes" column survives. Ask suppliers:

1. Was the uncorrected snapshot retained as captured, or reconstructed from the change log? Reconstruction can drop intermediate states.
2. What share of corrections came from human stewards versus automated rules? Rule-generated "corrections" teach your model the rule, including its bugs.
3. How was personal data handled in both versions, and was de-identification applied consistently so row keys still join? PII tools such as Presidio state they cannot guarantee finding all sensitive values [7], so ask how residual risk was sampled.
4. Are rejected corrections included? Proposed fixes that a steward declined are valuable negatives.
5. Which time span, source systems and schema versions are covered, and were any columns dropped?
6. Does the license allow training, evaluation and publication of aggregate metrics?

The SourceX answer on whether AI labs want raw or cleaned data explains why keeping the raw version matters, and what happens when supplied data contains errors covers how errors are treated in a supply relationship. The data quality scorecard helps you score candidate datasets on acceptance.

Request template for correction-paired tables

Describe the data, not the companies you think hold it, and state the error families you need.

Illustrative example: invented to show structure; it does not describe an available dataset.

Use: train and evaluate a cell-level error detector and repair model for a data-prep copilot. Tables: vendor master, customer master and AP invoice headers from US mid-market companies on any ERP. Versions: original snapshot, corrected snapshot, cell-level change log with old/new value, timestamp, role and reason code. Error families: DUP, MISS, FMT, CODE, RANGE, XFIELD; at least two source systems. Extras: data dictionary, allowed-value lists, DQ rule definitions, declined corrections. Privacy: personal fields replaced consistently across both versions; method documented. Allowed uses: training, internal evaluation, publication of aggregate scores.

Related reading in this cluster: the structured data buyer's guide, synthetic tabular data evaluation if you plan to augment real errors, and spreadsheet task benchmarks from real workbooks. When you are ready to scope a request, describe your correction-paired data needs to SourceX.

Source real dirty tables with recorded corrections

SourceX sources operational datasets such as finance workflows and support and sales histories from US companies on request, so a match is not guaranteed, and every release is approved by the supplying company. Each dataset is rights-reviewed, personal details are removed or replaced before delivery with the method recorded, and terms are agreed in a license defining records, uses, term and delivery. Start a buyer request for data-cleaning training data.