Skip to content

AI uses for records

Dataset outcome logging bias: a protocol to measure documentation gaps

By SourceX Editorial · Updated

9 sections

Short answer

Measure it before training. Take a record source that logs every case whatever the outcome, such as invoices or dispatch, sample within each outcome class, and score the same completeness checks on every case. Compare completeness by outcome with intervals, then test how the verdict moves under different assumptions about cases the archive dropped. This page sets out that protocol and reports no results.

Provenance trail

Dataset outcome logging bias

This page covers dataset outcome logging bias: a protocol to measure documentation gaps.Illustrative

Key takeaways

  • Operational archives often record failures, disputes and escalations in more detail than routine successes, because forms ask for evidence only when something goes wrong.
  • The archive cannot be its own sampling frame; reconcile it against a source that logs every case whatever its outcome.
  • Score presence and sufficiency of documentation separately, with the outcome hidden from coders where possible.
  • Size the sample per outcome class, report intervals, and account for clustering by technician or site.
  • Show the result under several assumptions about unmatched cases, and lock the plan in a dated preregistration before reading outcomes.

How do you account for successful work leaving less documentation than failures?

Treat thin documentation as a measurement problem that depends on the outcome, and measure it before you train on the archive. Pick a record source that logs every case whatever happened, such as a billing ledger or a dispatch log, draw a sample stratified by outcome, and score each case on the same completeness checks. If successful cases carry fewer notes, attachments or reason codes than failed ones, the archive teaches a model that trouble is what work looks like.

What follows is a study design for your own archive, with no measured gap from anyone else's. It sets out how to design and run an outcome dependent documentation completeness study: the question, population, sampling frame, variables, comparison, metrics, uncertainty, limits and a preregistration checklist. No results are reported here, and every count in the examples is labeled illustrative.

The method matters most for buyers who plan to train classifiers, agents or forecasting models on operational records. A support ticket that closed on first contact may hold one status change and nothing else, while an escalated ticket holds a thread of notes, a root cause field and a refund approval. A model that learns from text volume, field presence or attachment counts will read the second kind of case as typical.

  • Name the decision the model will support and the outcome classes it must distinguish.
  • Choose a frame that records every case regardless of outcome, and reconcile it against the archive.
  • Sample within each outcome class, and score the same completeness measures on every sampled case.
  • Report completeness per outcome class with intervals, then test how the verdict moves under different assumptions about the missing cases.

Why would clean outcomes leave thinner records than failed ones?

Because most business systems ask for evidence only when something goes wrong. A refund needs a reason, a warranty claim needs photos, a dispute needs the full thread and a failed inspection needs a corrective action. A job that went to plan often closes with a default status and the minimum mandatory fields, so the archive is rich exactly where the work was unusual.

Missing data methods give this a name. Stef van Buuren's Flexible Imputation of Missing Data, following Rubin's 1976 classification, separates data missing completely at random, where the chance of being missing is the same for every case; missing at random, where it is the same only within groups defined by observed data; and missing not at random, where it varies for reasons you cannot see. Outcome dependent documentation sits in the second class when the outcome is recorded for every case, and drifts into the third when the outcome itself is missing or recorded only for the cases that went wrong.

NIST Special Publication 1270 makes a related point about AI bias: statistical biases come from systematic rather than random error and can arise with no intent to discriminate. Nobody decided to under-document success. The form simply asked for less.

  • Mandatory fields that trigger on exception codes, such as a reason for a credit note or a return.
  • Review queues that sample flagged work, so reviewer comments exist mostly on problem cases.
  • Retention and legal holds that keep disputed files longer than routine ones.
  • Workarounds where staff close routine work in bulk at the end of a shift.
  • Archive migrations that carried open or disputed records forward and left quiet closures behind.

Which record source can count the cases the archive left thin?

Use a source that creates a row for every unit of work at the moment it starts, independent of how it ends. The archive you plan to license is usually the wrong frame, because it is the thing being measured. A good frame is boring: an invoice ledger, a call detail record, a dispatch board export, a sequential ticket number range or a production order list.

Reconcile the frame against the archive before sampling. Every frame case should match zero or one archive record; cases with zero matches are the extreme form of thin documentation, and their outcome mix is the first number the study reports. Record why each frame case cannot be matched rather than dropping it.

Settle outcome maturity at the same time. A case that is still open, still inside a return window or still awaiting payment has no final outcome yet, and its documentation will grow. Set a maturity rule in advance, such as excluding cases opened within the last period that covers most closures in your business, and report how many cases it removed.

Which record source can count the cases the archive left thin?
Candidate frameWhy it sees every caseCommon gapReconciliation key
Billing or invoice ledgerRevenue work is invoiced whatever the quality of the jobWarranty and no-charge work never invoicedJob or order number on the invoice line
Ticket number sequenceSystems issue numbers in order, so gaps are visibleDeleted and merged tickets leave holes with no reasonTicket ID and creation timestamp
Telephony or contact center logsEvery inbound contact creates a call recordContacts by email or chat are outside the logCaller reference or case ID captured by the agent
Dispatch or scheduling boardEvery visit is planned before it happensSame-day add-ons booked by phoneVisit ID and technician-day
Production or work order listEvery order is released before work startsRework orders created under a new numberOrder number and parent order

The outcome dependent documentation completeness study

The table below is the study design. Fill every row before anyone looks at the archive, keep the filled table with the dataset, and treat any change after data access as a documented deviation. The example column shows one illustrative choice for a field service archive; replace it with your own.

Write the research question so it can come out either way. The study asks whether completeness differs by outcome and by how much, not whether success is under-documented.

Keep the study small enough to finish. One archive, one deployment population and three or four outcome classes are enough for a first pass, and the design can be repeated for other systems later. A study that tries to cover every branch, product line and year at once tends to end with thin strata everywhere and no defensible comparison anywhere.

The outcome dependent documentation completeness study
ElementWhat to specifyIllustrative choice (field service)What invalidates it
Research questionOne sentence naming outcome classes and completeness measuresDoes note and photo presence differ between first-time fixes and repeat visits?Question rewritten after seeing the data
PopulationUnits of work the model will see in deploymentResidential service visits at all branches over the licensed periodPopulation defined by what the archive happens to hold
Sampling frameSource that lists every unit regardless of outcomeDispatch board export reconciled to the job archiveFrame drawn from the archive being tested
Sampling designStrata, allocation and selection methodRandom sample within each outcome class and branchConvenience sample of easy-to-read jobs
Outcome classesMutually exclusive outcome codes with a maturity ruleFirst-time fix, repeat visit within the window, canceled, disputedOutcome inferred from note length
Completeness variablesFixed list of presence and sufficiency checksNote present, note above a word floor, photo present, parts line presentChecks added once differences appear
ComparisonWhich classes are compared and in which directionEach failure class against first-time fixComparison chosen by largest gap
Primary metricOne headline measure stated in advanceDifference in share of cases passing all checksSeveral metrics with the best one reported
UncertaintyInterval method and clustering adjustmentWilson intervals per class, cluster bootstrap by technicianPoint estimates without intervals
LimitsKnown blind spots of the frame and measuresNo-charge warranty visits absent from dispatchLimits written after results

What counts as documented, measure by measure?

Score presence and sufficiency separately. Presence asks whether a field, note or attachment exists at all; sufficiency asks whether it contains enough to reconstruct what happened. A note that says ok is present and insufficient, and a model trained on note presence will miss that.

Define each measure so two coders reach the same answer without reading the outcome. Code completeness with the outcome field hidden where you can, and have a second coder repeat a share of the sample so you can report agreement. Automated checks such as word counts or attachment counts can run on every case; judgment checks belong on the sample.

Check the export before you score anything. Many systems hold attachments, email threads or call recordings in a separate store, and a flat export can drop them for every case alike or only for older ones. If the export path differs by record age, and outcome mix differs by age, the study will measure the export rather than the business. Pull a few cases of each outcome directly from the live system and confirm the export shows the same notes and files.

What counts as documented, measure by measure?
MeasureDefinitionHow to computeTrap to avoid
Narrative presentAt least one free-text note written by a personCount notes excluding system-generated textTemplates and auto-replies counted as notes
Narrative sufficientNote states what was done and whyCoder rubric with outcome hiddenLonger notes scored higher by default
Reason or cause codeStructured cause filled with a non-default valueShare of cases with a code other than the defaultDefault value treated as an answer
Evidence attachmentPhoto, document or reading linked to the caseAttachment count after removing signatures and logosExports that drop attachments, read as absence
Linked downstream recordInvoice, part order or follow-up linked by IDJoin on the documented keyFuzzy joins that favor messy cases
Closure detailClosed time, closer identity and resolution codeField presence at final statusBulk closures stamped in one minute

How large a sample, and how precise must each class be?

Size the sample per outcome class, not overall. The quiet successful cases are usually the majority of the frame and the minority of rich records, while rare failure classes may hold only a handful of cases. Set a minimum number of cases per class that gives an interval narrow enough for the decision, and oversample small classes rather than letting the total drive allocation.

Report each class with an interval for the completeness share and an interval for the difference between classes. The statsmodels proportion_confint function offers Wilson, Agresti-Coull, Jeffreys and Clopper-Pearson intervals among others; its documentation notes that the Clopper-Pearson interval has coverage of at least the nominal level but is in general conservative. Pick one method in the preregistration and use it everywhere.

Cases are not independent when one technician, branch or customer contributes many of them. Documentation habits cluster by person, so an analysis that ignores clustering will report intervals that are too narrow. Resample whole clusters in a bootstrap, or report the number of distinct clusters per class beside every estimate so a reader can judge concentration.

  • Illustrative allocation: equal numbers per outcome class within each branch, with small classes taken in full.
  • State the minimum per class and the action if a class cannot reach it, such as merging classes or reporting it as descriptive only.
  • Record the number of technicians or sites per class and the largest single contributor's share.

Which assumption about the missing cases flips the verdict?

Run the analysis under more than one assumption and show where the conclusion changes. The completeness gap you measure on matched cases says nothing about frame cases with no archive record at all, and those are often the quietest successes. Van Buuren's book recommends sensitivity analysis when missingness may be nonignorable, because adjustments rest on assumptions the data cannot check.

A simple version uses bounds. Recompute the primary metric assuming every unmatched case was a success with no documentation, then assuming each unmatched case resembles matched cases of the same outcome. A second version shifts the assumed completeness of unmatched cases by a stated amount and reports the shift at which the conclusion reverses, following the delta adjustment idea described in the book.

Write the result as a statement about decisions, not about truth. The useful output is that the gap remains under all assumptions tried, or that it depends on what unmatched cases looked like, which tells the buyer what to request next.

Which assumption about the missing cases flips the verdict?
ScenarioAssumption about unmatched casesWhat it testsReport
Complete caseIgnore unmatched casesGap among cases the archive holdsPrimary metric with interval
Worst case for successAll unmatched cases were undocumented successesUpper bound on under-documentation of successMetric and change from complete case
Like matchedUnmatched cases resemble matched cases in the same classMissing at random given outcomeReweighted metric
Delta shiftUnmatched completeness lower by a stated amountTipping point for the conclusionShift at which the verdict reverses

Preregistration checklist for the completeness study

Lock the plan before anyone with access to outcomes reads the archive. The Center for Open Science describes preregistration as posting a time-stamped, read-only version of the study plan before data collection or analysis begins, and changes are still allowed when the final report states them and gives the reason. An internal, dated copy filed with the data request serves the same purpose for a private study.

Checklist · 11 items

  1. Research question in one sentence, with outcome classes and completeness measures named.
  2. Deployment population, its period and its sites or channels.
  3. Sampling frame, reconciliation key and rule for unmatched cases.
  4. Outcome maturity rule and the number of cases it removes.
  5. Strata, allocation and the minimum per class.
  6. Completeness measures with definitions, coder rubric and blinding plan.
  7. Primary metric, comparison direction and interval method.
  8. Clustering unit and how intervals account for it.
  9. Sensitivity scenarios and the tipping point to report.
  10. Known limits of the frame and measures, written before results.
  11. Who may see outcomes before coding is complete, and where deviations are logged.

0 of 11 ticked. Tick each item as you go.

What to ask a data supplier before outcome-skewed records reach training

Ask how records are created and kept by outcome, not only how many there are. Datasheets for Datasets asks dataset creators whether a dataset is a sample of a larger set and what sampling strategy was used, and whether information is missing from individual instances. Put the same questions to the supplier for each outcome class, and ask for the frame counts that let you run the reconciliation yourself.

If the study shows a gap, the fix depends on its form. A presence gap can be handled by dropping field presence and note length as model features. A sufficiency gap may need supplementary records, such as invoices or parts lines, that exist for every case. A gap among unmatched cases needs a different frame or a narrower scope that says which outcomes the dataset represents.

A buyer requesting operational archives through SourceX can make this study part of the ask: SourceX sources records from U.S. businesses only after a request arrives, so the outcome classes, the reconciliation frame and the per-class minimums can go straight into the request's scope, volume and format fields. Suppliers that qualify usually hold two or more years of consistent history and tens of thousands of records in one system, though rare work can qualify with less, and every dataset is judged on its own.

Frequently asked questions

What is outcome logging bias in a dataset?

It is a difference in how completely cases are recorded that depends on how they turned out. In many operational archives, failures, disputes and escalations collect notes, codes and attachments while routine successes close with little detail. A model trained on such records can confuse documentation volume with the outcome itself.

Is this the same as survivorship bias?

They are related but not identical. Survivorship bias drops failed cases from view entirely, while outcome logging bias usually keeps all cases but records some outcomes in far more detail. The same study design catches both, because reconciling the archive against a complete frame reveals dropped cases as well as thin ones.

Can I fix the gap by imputing the missing notes?

Imputation can fill structured fields under stated assumptions, but it cannot recreate what a technician saw or did. For text and attachments, the safer moves are removing presence-based features, adding records that exist for every case, or narrowing the scope of the dataset. Any imputation should be shown under several assumptions.

Does this page report how big the gap usually is?

No. This is a protocol for measuring the gap in a specific archive, and no results are reported. Gaps vary by business, system and process, so a figure from one archive would not tell you about another.

Who should review a completeness study before it is used?

A research statistician should check the sampling, intervals and sensitivity analysis, and someone who runs the operation should confirm the outcome classes and what counts as sufficient documentation. Coders should not see outcomes while scoring completeness where that can be arranged.

Sources

  • Flexible Imputation of Missing Data (van Buuren), chapter 1: Rubin's MCAR, MAR and MNAR categories and what each assumes about the chance of a value being missing. Source
  • Flexible Imputation of Missing Data, chapter 3: when ignorability is suspect, study the impact of different assumptions by sensitivity analysis, including a delta shift. Source
  • NIST SP 1270: statistical and computational biases stem from systematic rather than random error and can occur without prejudice or intent. Source
  • statsmodels proportion_confint: interval methods for a proportion, with Clopper-Pearson described as conservative. Source
  • Center for Open Science: preregistration posts a time-stamped, read-only study plan before data collection or analysis, and documented deviations remain acceptable. Source
  • Datasheets for Datasets: questions on whether a dataset is a sample, its sampling strategy and missing information in instances. Source

Tell us what data you need

Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.

See if your company qualifies