Skip to content

Systems and records

Dataset coverage for manual workarounds outside the system of record

By SourceX Editorial · Reviewed by Noah Loul ·

10 sections

Short answer

To account for important business work outside the system of record, define the population the dataset should represent, interview operators with their consent about what they handle by phone, email, paper or spreadsheet, check each workaround against independent traces, and log every gap with its size, concentration and likely missingness type. Then recover what evidence supports, disclose the rest and narrow claims you cannot defend.

System export

Dataset manual workaround

This page covers dataset coverage for manual workarounds outside the system of record.Illustrative

Key takeaways

  • Manual workarounds cluster in urgent and unusual cases, so their absence from an export is usually biased rather than random.
  • Define the deployment population first; a gap only exists relative to the cases the model will face.
  • Interviews work only when participation is voluntary and operators know the notes describe the data, not their performance.
  • Size each workaround against an independent trace such as invoices or phone logs, and type it as MCAR, MAR or MNAR.
  • Give every gap one response, recover, disclose or narrow, and hand the register to buyers with the data.

How do you account for important work outside the system of record?

Treat off-system work as a coverage question about the dataset, and answer it with evidence rather than assumption. Define the population of work the dataset is supposed to represent, interview the operators who do that work, with their consent, about what they handle outside the system, check each claimed workaround against whatever traces exist, and record every gap with an estimate, a cause and a decision to recover, disclose or narrow the scope.

The reason to bother is that manual workarounds concentrate in the cases that matter most to a model: the rush job, the exception, the customer who only calls. An archive that looks complete can be systematically missing them, and a model trained or evaluated on it will look better than it is on exactly those cases.

This page gives the method as a protocol a data team can run with operators: an interview guide, a gap register, a worked example labeled illustrative and the decision rules for each gap. It is general information, not legal advice; consent and employment questions depend on jurisdiction and should be reviewed by counsel.

Why off-system work is rarely missing at random

Work that bypasses the system usually does so for a reason connected to the work itself, which makes the gap biased rather than random. Missing data methods distinguish three cases, and the difference decides what you can repair.

Stef van Buuren's Flexible Imputation of Missing Data describes data as missing completely at random when the probability of being missing is the same for all cases, missing at random when it is the same only within groups defined by the observed data, and missing not at random when it varies for reasons unknown to the analyst. The same text notes that MCAR is often unrealistic and that it is not possible to test MAR against MNAR from the data alone, because the information needed is missing.

Off-system work tends to fall in the last two groups. If phone-dispatched jobs are missing more often on weekends and the weekend flag is recorded, the gap is closer to MAR and can be modeled. If they are missing because they were urgent and urgency is never recorded, the gap is MNAR, and only outside evidence such as interviews, phone logs or invoices can show its size.

Define the deployment population before counting gaps

A gap only exists relative to a population, so write down the population first. The population is the set of real cases the model will face once deployed, not the set of rows the export happens to contain.

At a glance

Define the deployment population before counting gaps

Datasheets for Datasets, by Gebru and coauthors, asks dataset creators whether the dataset contains all possible instances or a sample of a larger set, what that larger set is, and whether the sample is representative of it and how that was validated. It also asks whether information is missing from individual instances and why. Those two questions, answered honestly, are the frame for everything in this protocol.

Google's Data Cards Playbook describes Data Cards as structured summaries of the facts stakeholders need about a dataset, including upstream sources and collection methods. A gap register produced by this protocol belongs in that kind of documentation, not in a private note.

  • Unit: one job, order, ticket, claim or case, as the model will see it.
  • Scope: which sites, teams, channels, customer types and products are in and out.
  • Period: the dates the dataset covers, and any changes in process or system during them.
  • Sampling frame: the list from which records were drawn, such as all closed work orders in the field service system.
  • Known exclusions: anything deliberately left out, such as warranty jobs handled by a manufacturer portal.

Running operator interviews people can freely agree to

Operators should take part because they choose to, understand what the interview is for and know what will happen to what they say. Workaround questions can feel like an audit of rule-breaking, so voluntariness is a quality requirement as well as an ethical one: people who feel exposed describe the official process.

The Belmont Report, the U.S. framework for research involving people, breaks informed consent into information, comprehension and voluntariness, and says agreement is valid only when given free of coercion and undue influence. A business interview is not necessarily human subjects research, but the three elements are a sound checklist for any interview whose answers end up in a dataset description.

  • Information: say the purpose is to describe gaps in the records, not to evaluate the person, and say who will see the notes.
  • Comprehension: give the explanation in plain language and invite questions before starting.
  • Voluntariness: make declining easy, keep managers out of the room, and never tie participation to performance reviews.
  • Minimization: record workarounds and volumes, not customer names or personal details, and do not record audio unless the operator agrees.
  • Use limits: state that interview notes describe the dataset and are not themselves part of any licensed data unless separately agreed.

Interview guide: questions that surface workarounds

The guide asks about concrete recent cases rather than general habits, because people describe the official process when asked in general and the real one when asked about last Tuesday. Use the questions in order, and ask for an example for every yes.

  • Walk me through the last job you handled that did not start in the system. How did it reach you?
  • Which kinds of work do you handle by phone, text, personal email or paper, even if they are entered later?
  • When work is entered later, what gets left out or changed compared with what happened?
  • Which spreadsheets, notebooks or shared folders do you keep because the system does not fit the task?
  • Which customers, sites or job types almost never appear in the system the way they really happened?
  • What happens on nights, weekends, outages or peak days that does not happen the rest of the time?
  • If a new colleague learned only from the system records, what would they get wrong in the first month?
  • Is there any trace of these cases somewhere else, such as a phone log, an invoice or a delivery note?

The evidence gap register

Every workaround the interviews surface becomes one row in a gap register, with its evidence and its likely missingness type. The register is the asset that turns anecdotes into a disclosure a buyer or reviewer can check.

The evidence gap register
Register fieldWhat to recordIllustrative entry
Gap ID and descriptionA short name for the off-system workG-03: emergency callouts dispatched by phone
Source of evidenceInterviews, plus any independent traceThree dispatcher interviews; phone system call log; invoices
Population share estimateSize of the gap relative to the population, with methodIllustrative: about 8 percent of jobs, from invoice match
ConcentrationWhere the gap clusters: customer, site, time, channelNights and weekends; two large customers
Likely missingness typeMCAR, MAR given a recorded variable, or MNARMNAR: urgency is not recorded
Fields affectedWhat is absent or altered when the case is entered lateArrival time, diagnosis notes, parts used
Consent and confidentiality notesWhat operators agreed to and any limitsNotes anonymized; no audio; operators reviewed summary
DecisionRecover, disclose or narrow, with owner and dateDisclose and recover from invoices; owner: data lead

Worked example: urgent jobs dispatched by phone

The example is an illustrative composite of a regional field service company, not a real business, and every figure in it is invented and labeled illustrative. The company plans to license several years of work orders from its field service system for an AI scheduling model.

In interviews, dispatchers describe emergency callouts that arrive by phone after hours and go straight to a technician by text. Some are entered the next morning, some at month end and some never, when the technician also handled billing on paper. The official process says every job starts as a work order.

Checking traces settles the size. Matching an illustrative quarter of invoices against work orders shows that roughly 8 percent of invoiced jobs have no work order, and that the unmatched jobs cluster on nights and weekends. Among jobs entered late, the arrival time is often the time of entry rather than the time on site. The export, taken alone, would teach a model that after-hours emergencies are rare and quick.

The decision is to recover what the invoices support, disclose the rest with the estimate and its method, and flag arrival times on late-entered jobs as unreliable. One exception changes the plan: for one large customer the operators say jobs are logged in that customer's own portal, not the company's system. Those records may not be the company's to license at all, so the team narrows the scope to exclude them pending a rights review rather than trying to recover them.

Recover, disclose or narrow: choosing a response for each gap

Each gap gets exactly one primary response, chosen by whether the missing cases can be recovered with evidence and whether the company has the right to use that evidence. The table is the decision rule.

Recover, disclose or narrow: choosing a response for each gap
ConditionResponseWhat to document
An independent trace exists and the company controls itRecover: rebuild the cases from the trace and mark them as reconstructedTrace source, match method, fields that could not be rebuilt
The gap is sized but cannot be rebuiltDisclose: describe size, concentration and causeEstimate, method and the population it is relative to
The gap depends on a recorded variableDisclose and supply the variable so users can reweightWhich variable and how it relates to missingness
The work happens in another party's systemNarrow: exclude from scope until rights are reviewedWho holds the records and what review is pending
The gap cannot be sized at allNarrow the claimed population, or stopWhy no estimate is possible and what was tried

Where this protocol stops and a statistician starts

The protocol finds and sizes gaps; it does not prove what a model will do with them. Bring in a research statistician when a gap is large, MNAR and in a slice the model is meant to serve, because the choice between reweighting, imputation, targeted collection or a narrower claim depends on the model's use.

Regulated uses raise the bar. For high-risk AI systems, the EU AI Act's Article 10 requires data governance practices that include examination of possible biases and identification of relevant data gaps or shortcomings that prevent compliance with the Regulation, and how those can be addressed. A gap register is evidence for that work, but whether a given system is in scope and what suffices is a question for counsel and the system's provider.

Stop or narrow when the operators' account and the traces disagree and nobody can explain why, when interviews reveal work done in systems the company does not control, or when operators decline to take part and no independent trace exists. A dataset described more narrowly and honestly is worth more than one whose coverage claim cannot be defended.

Telling a dataset buyer what the archive leaves out

Buyers need the gap register, or a summary of it, alongside the data: the population definition, each material gap with its estimate and method, and the fields that are unreliable on reconstructed or late-entered records. That is what lets an evaluation team decide whether the data fits their deployment.

On the supply side, SourceX runs rights review and data preparation for companies that license selected records as AI datasets, and nothing ships until the company has approved scope and terms and signed. Datasets for AI buyers are sourced on request from U.S. businesses, with the buyer describing scope, volume, format and licensing needs, which is the point at which coverage questions like these belong in the request.

Frequently asked questions

What is off-system work in a dataset?

It is real work the dataset is meant to represent that was handled outside the system the data was exported from, such as jobs taken by phone, approvals in personal email or tracking in a private spreadsheet. It is missing or distorted in the export even when the export itself is complete.

Why are manual workarounds a coverage problem for AI training data?

Workarounds cluster in urgent, unusual or high-value cases, so their absence is usually not random. A model trained or evaluated on the export learns a world where those cases are rare or simpler than they are.

Do you need consent to interview employees about workarounds?

Treat participation as voluntary and explain the purpose, who sees the notes and how they will be used, because people who feel exposed describe the official process. Whether formal consent is legally required depends on the jurisdiction and the setting, so have counsel review the plan.

Can missing off-system cases be imputed?

Sometimes, if the missingness depends on recorded variables. When cases are missing for reasons that were never recorded, such as urgency, the data alone cannot tell you how biased the gap is, and you need outside evidence like invoices or phone logs.

What should a dataset buyer ask about off-system work?

Ask for the population definition, the known gaps with their estimated size and method, where they cluster and which fields are unreliable on late-entered records. Ask also whether any excluded records sit in another party's system.

Sources

  • Flexible Imputation of Missing Data defines missing completely at random, missing at random and missing not at random, calls MCAR often unrealistic, and notes MAR cannot be tested against MNAR because the needed information is missing. Source
  • Datasheets for Datasets asks whether a dataset contains all instances or a sample of a larger set, whether the sample is representative and how that was validated, and whether information is missing from individual instances. Source
  • Google's Data Cards Playbook describes Data Cards as structured summaries of essential facts about ML datasets, including upstream sources and data collection methods. Source
  • The Belmont Report treats informed consent as information, comprehension and voluntariness, and requires agreement free of coercion and undue influence. Source
  • Article 10 of the EU AI Act requires data governance for high-risk AI systems that covers examination of possible biases and identification of relevant data gaps or shortcomings and how they can be addressed. Source

Tell us what data you need

Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.

See if your company qualifies