Systems and records
Dataset quality remediation: fix the missing fields or narrow the scope?
By SourceX Editorial · Updated
9 sections
Short answer
To choose between repairing missing dataset fields and narrowing the scope, model both against the same acceptance test: accepted records gained per hour of effort, plus a bias check on what you drop or recover. Repair wins when values still exist in a reachable source and the task needs them. Narrowing wins when gaps cluster in a definable slice whose removal does not skew outcomes. Often a documented hybrid is best.
Checklist
Dataset quality remediation
Key takeaways
- Classify why each field is missing first; the cause decides whether repair is possible at all.
- Score repair and narrowing on accepted records, effort, recovery rate, bias, buyer fit and evidence trail.
- Missing or untrustworthy outcomes, broken links and silent schema changes sink acceptance even when every file opens.
- Dropping records whose gaps track the outcome biases the dataset, so compare kept and dropped slices before cutting.
- Intentionally removed fields, such as de-identification blanks, are not defects and should never be repaired.
Fix the missing fields or narrow the scope: which costs less to get accepted?
Model both options against the same acceptance test and pick the one that delivers more accepted records per hour of effort without biasing the dataset. Repairing missing fields wins when the values still exist somewhere you can reach, such as another system, an attachment or an audit log, and when the gap sits in fields the buyer's task depends on. Narrowing the scope wins when the gaps cluster in a slice you can define cleanly and drop without distorting what remains.
The mistake is to choose by instinct. Teams that like engineering work repair everything, including fields nobody will use. Teams in a hurry cut the scope by date range, and sometimes cut away exactly the records that carry the outcomes a buyer needs. A short comparison, built from validators and a reconciliation worksheet, turns that into a measured decision.
This guide gives that comparison, the checks that feed it and a worked example. It assumes the package is meant for a declared task, such as training a model to predict job outcomes from work orders, because quality only means something relative to the use. A file that opens cleanly can still fail that test.
Why is the field missing? Classify before you model anything
Start by sorting each gap by cause, because the cause decides whether repair is even possible. A field can be missing because it was never collected, because the export left it out, because a migration dropped it, because someone removed it on purpose or because it does not apply to that record type.
Record the cause in the data itself, not only in a memo. The Frictionless Table Schema standard lets a schema declare which strings count as missing values and attach labels to them, with examples such as OMITTED and REFUSED, so a consumer can tell kinds of missingness apart. Its required constraint states whether a field may be null at all. A schema that marks not applicable separately from not recorded prevents a buyer from reading a structural blank as a quality defect.
Datasheets for Datasets draws a similar line: it asks whether information is missing from individual instances and why, and separates that from information intentionally removed, such as redacted text. De-identification blanks belong in the second group and should never be repaired back.
The cause also tells you whether dropping records is safe. Kang's review of missing-data handling notes that missing data can reduce statistical power and produce biased estimates, and defines data as missing completely at random only when missingness is unrelated to both the missing value and the observed responses. If outcome codes are missing mostly on jobs that went badly, cutting those records does not just shrink the dataset; it changes what the dataset teaches.
The remediation and scope reduction comparison
Score both options on the same rows. Every cell should be filled from a measurement or a named estimate, and the bias row should never be left as not applicable.
| Dimension | Remediate missing fields | Narrow the scope | How to measure it |
|---|---|---|---|
| Accepted records after the change | In-scope records plus those whose fields are recovered | Records in the retained slice that already pass | Run the acceptance validators on a sample under each option |
| Effort | Hours to locate, extract, join and verify recovered values | Hours to redefine scope, re-export and update documentation | Estimate from a timed pilot on 50 to 100 records |
| Recovery rate | Share of gaps actually filled from a source of record | Not applicable to recovery; record the share of gaps removed | Pilot recovery on a random sample of gap records |
| Bias introduced | Risk that recovered values differ systematically from originals | Risk that dropped records differ on outcomes or key variables | Compare distributions of outcome and key fields between kept and dropped or recovered and native records |
| Buyer fit | Keeps the declared task and coverage intact | May shrink coverage below what the task needs | Check retained scope against the task definition and volume needs |
| Evidence trail | Field-level provenance for each recovered value | Written scope rule and exclusion counts | Exception log and manifest entries |
| Ongoing risk | Repair logic may break on the next export | Excluded slice may be needed later | Note what reruns each option needs |
Validators and a reconciliation worksheet to size the gap
Measure the gap with automated checks and a hand reconciliation before you estimate either option. The checks tell you where the holes are; the reconciliation tells you whether the export even contains what the source system holds.
At a glance
Validators and a reconciliation worksheet to size the gap
For field completeness, use a validator that reports a pass rate rather than a binary result. Great Expectations, for example, provides an expectation that column values are not null with a mostly parameter, which succeeds if at least that fraction of values match. Its documentation also warns that empty strings do not count as null unless coerced, which is exactly the trap that makes an export look complete when it is not. Run each check by slice, such as by year, region or record type, so clustering shows up.
ISO/IEC 5259-2, published in 2024, sets out a data quality model and measures for analytics and machine learning data, building on ISO/IEC 25012. A team does not need the standard to run these checks, but naming each check after a quality characteristic, such as completeness, consistency or currentness, makes the report easier for a buyer's reviewer to read.
- Scope reconciliation: record and case totals in the export against counts run in the source system for the same filters.
- Row and case totals: parent records against child records, such as work orders against line items and notes.
- Field completeness by slice: null and empty-string rates per field, per year and per record type.
- Timestamp validity: impossible dates, bulk-entered times and closing dates before opening dates.
- Schema history: fields added, renamed or retyped during the date range, and when.
- Attachment integrity: referenced files present, readable and matched to the right record.
- Outcome labels: share of records with a final status a buyer can treat as the outcome.
- Exception log: every record excluded or altered, with the reason and the rule that triggered it.
What defect could sink acceptance even when every file opens?
The defect that changes acceptance is usually a missing or untrustworthy outcome, not a missing descriptive field. A work order without a resolution code, a ticket without a final status or a claim without a decision cannot teach a model what happened, however complete the rest of the record is.
Close behind are broken links and silent schema changes. If notes cannot be joined to their work orders after a migration, or a status field changed meaning halfway through the range, every file opens and the validators may pass, yet the package fails its declared task. Sambasivan and colleagues at Google describe data cascades as compounding events causing negative downstream effects from data issues, often invisible and delayed; a schema change found by the buyer after training is a textbook example.
Put these three defects at the top of the comparison. If repair cannot restore outcomes or links for a slice, that slice is a candidate for removal regardless of how complete its other fields are.
Running the yield math for both options
Use one formula for both options so they can be compared: accepted records equals in-scope records times the share that passes every acceptance check. Then compute what each option changes and what it costs.
For remediation, accepted records rise by the gap records multiplied by the recovery rate from your pilot, and effort is the per-record repair time multiplied by the number of gap records you attempt. For narrowing, accepted records equal the passing records in the retained slice, and effort is the re-export and documentation work, usually smaller and more predictable.
Divide the gain in accepted records by effort for each option, then apply the bias check. Compare the distribution of the outcome and two or three key variables, such as job type and region, between records you keep and records you drop or repair. If the dropped slice has a visibly different outcome mix, narrowing has a hidden cost the per-hour figure does not show. If recovered values come from a different source with different conventions, repair has one.
A hybrid is often best: repair the fields whose values are recoverable and matter to the task, drop the slice whose outcomes cannot be restored, and document both.
Illustrative: an HVAC service company's work order export
All figures in this example are illustrative counts, durations and percentages. A regional HVAC service company prepares six years of work orders from one service management system for a dataset meant to train a model that predicts first-visit resolution. The export holds 84,000 work orders.
The validators find three gaps. Equipment model numbers are blank on 22% of records, mostly from the first two years. Resolution codes are missing on 9%, concentrated in a 7-month period after a system migration. Technician notes are present but empty strings on 4%, which the first null check missed because it counted empty strings as values.
A timed pilot on 80 records shows that model numbers can be recovered from photographed nameplates attached to the jobs for about two thirds of gap records, at roughly 4 minutes each. Resolution codes cannot be recovered for the migration period; the old system's audit log was not migrated. The bias check shows that jobs in that period are more often multi-visit jobs, so their missing outcomes are not random.
The team compares options. Dropping the first two years would remove the model-number gap in one step but also remove a third of the history buyers value. Repairing model numbers adds about 12,000 accepted records for an estimated 1,200 hours of work spread over several weeks. For the migration period, the exception, repair is impossible and dropping it biases the multi-visit share, so the team excludes it, records the exclusion and its reason in the exception log, and states the resulting under-representation of multi-visit jobs in the dataset documentation.
The result is a hybrid: repair one field, narrow one window, and disclose the known skew. A buyer reviewing the package sees the decision and its trade-off rather than discovering it later.
Signals that neither option will rescue the package
Stop and reconsider the package when outcomes are missing across most of the range, when links between parent and child records are broken everywhere, or when the remaining scope after narrowing no longer matches the declared task. No amount of field repair fixes a dataset that cannot show what happened.
Request specialist review when recovered values would come from a source with different access or privacy permissions than the original records, when missing fields might hide personal information that was removed on purpose, or when the outcome definition itself is disputed inside the company. Those are rights, privacy and domain questions, not data engineering ones.
Narrow and wait is a legitimate result. If the newest two years are clean and the older history is not, a smaller package now and a repaired extension later may serve both the supplier and the buyer better than a delayed, larger one.
Taking the chosen scope into a licensing review
Whichever option you choose, the comparison table, the validator results and the exception log become part of what a licensing review examines. SourceX takes companies that license selected records directly to SourceX through rights review, preparation and delivery, and a scope decision that is already documented leaves that review less to reconstruct.
Watch volume while narrowing: buyers generally want two or more years of consistent history and tens of thousands of records from one system, so a cut that takes the retained scope well below that needs a reason, such as rare work. Before preparing anything, a brief assessment of the company, which needs no uploaded data, shows whether the records are worth the effort.
Frequently asked questions
Is it better to impute missing values or drop incomplete records in a dataset for licensing?
For a licensed training dataset, recover real values from a source of record where possible rather than imputing estimates, and label any imputed field clearly. Drop records only after checking that the dropped slice does not differ on outcomes or key variables, because removing non-random gaps biases what the dataset teaches.
How do I find out whether missing fields are random?
Compare the records with gaps against complete records on the outcome and a few key variables, such as year, region and job type. If the gap rate changes sharply across those slices or tracks the outcome, treat it as non-random and avoid simply deleting those records.
Do empty strings count as missing data?
They should, but many tools treat them as values. Great Expectations, for example, notes that empty strings are not counted as null unless coerced. Normalize empty strings and placeholder values to a declared missing marker before measuring completeness.
How much remediation effort is reasonable before narrowing scope?
Time a pilot on a small random sample of gap records and multiply out. If the cost per accepted record from repair is well above the cost of re-scoping, and the excluded slice passes the bias check, narrowing is usually the better trade.
Should redacted or de-identified fields be repaired?
No. Fields removed on purpose for privacy are not quality defects and should be labelled as intentionally removed in the schema and dataset documentation. Repairing them would undo the preparation the license depends on.
Sources
- The Frictionless Table Schema standard lets a schema declare labelled missing values, such as OMITTED and REFUSED, and a required constraint for fields that cannot be null. Source
- Great Expectations' not-null expectation accepts a mostly threshold and does not count empty strings as null unless they are coerced. Source
- Missing data can reduce statistical power and bias estimates; data are missing completely at random only when missingness is unrelated to the missing value and the observed responses. Source
- Data cascades are compounding events that cause negative downstream effects from data issues and build technical debt. Source
- ISO/IEC 5259-2:2024 defines a data quality model and measures for analytics and machine learning, building on ISO/IEC 25012. Source
- Datasheets for Datasets asks whether information is missing from individual instances and why, separating intentionally removed information such as redacted text. Source
See if your company qualifies
A short company assessment. No data uploads are needed.