AI uses for records
Business dataset negative outcome coverage: where did the failures go?
By SourceX Editorial · Updated
9 sections
Short answer
Negative outcome coverage is the share of a business's failed deals, reopened cases and cancelled jobs that actually reached the dataset. To find what is missing, count each outcome in the source system, compare that with the dataset by period, channel and customer segment, trace every gap to a cause such as an export filter, deletion or unfinished records, and report the shortfall with an uncertainty range before anyone trains or tests on it.
System export
Business dataset negative
Key takeaways
- Exports built around finished, billable work quietly drop failed deals, reopened cases and cancelled jobs.
- Count each outcome in the source system first; coverage means nothing without a population denominator.
- Compare negative-outcome coverage with positive-outcome coverage per segment, not just in total.
- Recover records lost to filters and joins; reweight only what is gone for good, and disclose every remaining gap.
- Report estimated shares with an interval and state the maturity window, because recent records undercount negatives.
Why do business datasets lose their failed deals, reopened cases and cancelled jobs?
Because the path from a source system to an export is built around finished, billable work. Extracts are often filtered on an invoice, a won stage or a completed status; migrations move active customers and leave old pipelines behind; and cleanup projects delete stale opportunities and duplicate tickets. Each step quietly removes more negative outcomes than positive ones.
The systems themselves make some negatives harder to keep. Dynamics 365 Customer Service requires all case activities to be closed before a case can be cancelled, and once a case is resolved or cancelled its details cannot be updated in the interface unless an administrator allows updates through a flow or the API. Salesforce keeps standard field history for 18 months, and up to 24 months through the API, unless an add-on extends it, so the stage changes that show a deal slipping before it was lost can age out of the record.
There is also a structural reason that no export fix can solve. Outcomes are observed only on the branch a decision allowed: a declined quote never shows whether the job would have gone well. Lakkaraju and colleagues called this the selective labels problem, and it means some negatives are not missing from the dataset but missing from the world. A coverage worksheet has to tell those two apart.
How do CRM, helpdesk and field service systems record a negative outcome?
Each system records failure as a state plus, if configured, a reason, and the reason is usually optional. Before measuring coverage, write down exactly which field and value defines each negative outcome in each source, because a dataset can contain the record and still lose the outcome.
| Negative outcome | How it is recorded | System example | How it usually goes missing |
|---|---|---|---|
| Lost deal | Closed as lost with competitor, close date and description | Dynamics 365 Sales Close as Lost, with details kept on an Opportunity Close activity | Export reads the opportunity row and drops the close activity holding the reason |
| Lost deal reason | A default reason property on the deal | HubSpot Closed lost reason deal property | Field left blank or filled with a generic value; not exported by default views |
| Reopened deal | A closed opportunity reopened later | Dynamics 365 Reopen Opportunity command | Only the latest state is exported, so the earlier loss disappears |
| Cancelled case | Case status set to cancelled after activities are closed | Dynamics 365 Cancel Case | Filtered out as noise, or merged into duplicates during cleanup |
| Reopened case | Resolved case reopened, with resolution history | Dynamics 365 case resolution history | Export keeps the final resolution and loses the reopen |
| Cancelled job | Job or work order status cancelled, sometimes with a reason code | Field service and dispatch systems, where configured | Extract joins jobs to invoices, and cancelled jobs rarely have one |
| Slipped stage before loss | Field history on stage and close date | Salesforce field history, 18 to 24 months by default | History purged before the extract was taken |
What should the negative outcome coverage and sampling worksheet include?
It should hold enough inputs that someone else can recompute every coverage figure, and enough notes that they can disagree with an assumption. Fill one row per input for each source system. The illustrative values belong to the worked scenario later on this page and are invented.
At a glance
What should the negative outcome coverage and sampling worksheet include
| Input | Definition | Where it comes from | Illustrative value | Decision it feeds |
|---|---|---|---|---|
| Population window | Dates and systems that define the business population | Agreed with the business before any extract | 24 months, dispatch system | Which records count as missing |
| Population count by outcome | Records per outcome in the source system for the window | System report run directly on the source | 12,000 jobs, 1,450 cancelled | Denominator for coverage |
| Dataset count by outcome | Records per outcome in the delivered dataset | Dataset profiling | 10,900 jobs, 380 cancelled | Numerator for coverage |
| Strata | Segments the deployment cares about: channel, region, customer type, product, period | Deployment requirements | Channel and quarter | Where gaps are hidden by totals |
| Exclusions | Records removed on purpose, with reason and count | Extract specification | Test jobs and duplicates, 120 | Separates design from loss |
| Maturity window | Time after which an outcome is considered final | Business process and reopen history | 60 days after scheduled date | Which recent records to set aside |
| Missingness cause | Why each gap exists: filter, deletion, retention, migration, unrecorded | Source system logs and interviews | Invoice join in extract | Fix, reweight or disclose |
| Audit sample | Random records pulled from the population to check fields | Simple random sample per stratum | 200 cancelled jobs | Estimates field completeness |
| Uncertainty | Interval around each estimated share | Wilson interval for proportions | 95 percent interval reported | Whether the gap is real or noise |
Which formulas and decision rules turn the worksheet into a decision?
Three ratios do most of the work, each computed per stratum as well as in total. The thresholds are decisions the buyer and supplier agree in writing; the numbers below are illustrative starting points, not standards.
- Outcome coverage: dataset records with the outcome divided by population records with the outcome. Compute it for every negative and for the positive outcome, then compare them.
- Coverage gap ratio: negative-outcome coverage divided by positive-outcome coverage. A value near 1 means negatives survived the pipeline as well as positives; a value well below 1 means the dataset is tilted toward success.
- Reason completeness: share of negative records with a usable reason value, estimated from the audit sample with an interval.
- Rule 1: if the gap ratio in any stratum the deployment depends on falls below an agreed floor (illustratively 0.8), trace the cause before using the data.
- Rule 2: if the cause is a filter or join, re-extract; do not reweight records that can still be recovered.
- Rule 3: if records are gone for good, disclose the gap, and reweight only when the missing records are plausibly similar to the ones kept.
- Rule 4: exclude records still inside their maturity window from outcome counts, and report how many were set aside.
Which users and cases does the dataset underrepresent?
Define the people and cases the model or agent will actually serve, then measure coverage against that population, not against the archive. If an agent will triage every inbound request, its population includes the requests that were abandoned, cancelled or escalated, not only those that ended in a sale or a completed job.
Datasheets for Datasets asks whether a dataset contains all possible instances or a sample, what the larger set is, whether the sample is representative, and whether any instances are missing information. For negative outcomes, answer those questions per stratum. Losses are often concentrated: one sales channel with a short cycle, one region with seasonal cancellations, one product line with frequent callbacks. A dataset can match the overall loss rate while missing nearly all of the losses from the channel the agent will handle.
Check the people as well as the records. When one team or a few staff recorded most negative outcomes carefully, the dataset reflects their habits. Compare who closed each negative record with who handled the population, and flag strata where a small group accounts for most reasons.
Then sample to the strata, not to the total. A simple random sample of the whole population will contain few records from small but important segments, so draw a separate random sample within each stratum the deployment depends on and record the sampling fraction for each. When the strata are later combined for a single estimate, weight each one by its share of the population, and say so in the worksheet. Disproportionate sampling is normal and useful here; undocumented sampling is not.
Illustrative worked scenario: cancelled jobs lost in an invoice join
This scenario is illustrative and invented; the counts show how the worksheet reads, not results from any company. A home services business exports 24 months of jobs from its dispatch system for an AI developer building a scheduling and triage agent.
The source report shows 12,000 jobs in the window, of which 1,450 were cancelled. The delivered dataset holds 10,900 jobs, of which 380 are cancelled. Coverage of completed jobs is about 99.7 percent (10,520 of 10,550), while coverage of cancelled jobs is about 26 percent (380 of 1,450). The gap ratio is roughly 0.26, far below any reasonable floor.
Tracing the cause takes one query: the extract joined jobs to invoices, and most cancelled jobs never generated an invoice. The 380 that survived had a cancellation charge or a partial visit. Rule 2 applies, so the fix is a re-extract without the join, not a reweighting of 380 unusual records to stand in for 1,450 ordinary ones.
An audit sample of 200 cancelled jobs pulled straight from the dispatch system finds 37 with no cancellation reason, or 18.5 percent. A Wilson 95 percent interval for that share runs from roughly 13.7 to 24.5 percent, which is what the worksheet reports instead of a single number. Reopened work appears as separate callback jobs linked by customer and address, so the worksheet adds a linking step before callbacks can be counted as negative outcomes of the original job.
After the re-extract, the worksheet is rerun by channel and quarter. Phone bookings and online bookings now show similar gap ratios, but the most recent quarter still shows fewer cancellations than earlier ones because many of its jobs are inside the 60-day maturity window. Those jobs are set aside and counted, and the delivery note states the window, the corrected counts and the reason-completeness interval.
Which assumption could flip the coverage verdict, and how wide is the uncertainty?
The assumption that most often flips the decision is that the population count is complete. If lost deals or cancelled jobs were deleted from the source system before the report was run, the denominator is too small and coverage looks better than it is. Ask when cleanup jobs ran, what they deleted and whether audit or backup records can recover the counts; if not, show coverage as a lower and upper bound rather than a point.
The second is the maturity window. A case resolved last week may still reopen, and a scheduled job may still be cancelled, so recent records undercount negatives. Show results with and without the most recent period and state the window chosen. The third is that missing records resemble the kept ones; if the gap came from a filter, they almost certainly do not.
For estimated shares, report an interval. The NIST Engineering Statistics Handbook gives the Wilson method for a confidence interval on a proportion, and the worksheet uses it rather than the simple normal approximation. Borrowing from survey practice helps set expectations: NCES statistical standards require a nonresponse bias analysis when a unit or item response rate falls below 85 percent. A dataset that holds well under that share of a negative outcome deserves at least the same scrutiny.
Should each gap be re-extracted, reweighted or disclosed?
Match the fix to the cause. Recoverable records should be recovered; reweighting is for gaps that cannot be closed, and disclosure is required in every case where a gap remains.
| Gap cause | Preferred fix | When it is not enough | What to disclose |
|---|---|---|---|
| Extract filter or join | Re-extract without the filter and re-profile | Filter also shaped which fields were populated | Original and corrected counts by outcome |
| Deleted or purged records | Recover counts from audit logs or backups | Records unrecoverable, only counts known | Bounds on coverage and the deletion dates |
| History retention limits | Export field history before it ages out | Older stage changes already gone | Period for which history is complete |
| System migration | Pull the legacy system's archive | Legacy system decommissioned | Date range covered by each system |
| Outcome never observed | Label as unobserved, never as success or failure | Deployment depends on those cases | Share of cases with no observable outcome |
| Recent records not mature | Set aside until the maturity window passes | Buyer needs recent data for drift | Count set aside and window used |
What should a data request say about negative outcomes?
A request for business records should name the negative outcomes explicitly and ask for the population counts alongside the data, so coverage can be checked on delivery rather than discovered in training. Buyers generally look for consistent history, typically two or more years, and tens of thousands of records or more in one system, though smaller sets can qualify when the work is rare and each deal is assessed on its own; whatever the size, the share of failed and cancelled work is part of what makes the history useful.
Through SourceX, histories like these are sourced on request from U.S. businesses, with SourceX handling the rights review, preparation and delivery for the company that licenses them. An AI team can name the outcomes, strata, date range, volume and format it needs on the request form, and ask for the worksheet inputs to arrive with the data.
Frequently asked questions
How do you find failed deals, reopened cases and cancelled jobs missing from a dataset?
Run a count of each outcome directly in the source system for the same date range, compare it with the dataset per segment and period, and compute coverage for negative and positive outcomes separately. Where negative coverage falls behind, trace the cause in the extract logic, deletion history, retention settings or migrations.
What is a good negative outcome coverage ratio?
There is no universal standard. Compare negative-outcome coverage with positive-outcome coverage in the same dataset; a ratio close to 1 means the pipeline treated them alike. Agree a floor with the supplier for the segments the model depends on, and investigate any segment below it.
Can reweighting fix missing negative outcomes?
Only when the missing records are gone for good and plausibly resemble the ones kept. If records were lost through a filter or join, the survivors are usually unusual, so recover them with a re-extract instead. Always disclose the gap and the method used.
Why do reopened cases disappear from exports?
Many exports take the current state of each record. A case that was resolved, reopened and resolved again looks like a single resolution unless the resolution history or status change log is exported too. Ask for the history alongside the case table.
How large should an audit sample be for checking negative outcome fields?
Large enough that the interval around the estimated share is narrow enough to change the decision. A few hundred records per important stratum is a common starting point; report a Wilson interval rather than a single percentage so readers can see the uncertainty.
Do negative outcomes raise extra privacy or rights questions?
Often they do. Lost deals can name competitors and contain pricing discussions, and cancelled jobs may include complaints or personal circumstances. The business must hold the right to license those records, and personal details need de-identification; neither step substitutes for the other.
Sources
- Dynamics 365 Sales lets users close an opportunity as lost with competitor, close date and description, keeps those details on an Opportunity Close activity, and allows a closed opportunity to be reopened. Source
- Dynamics 365 Customer Service requires all case activities to be closed before a case is cancelled and keeps case resolution history for reopened cases. Source
- Dynamics 365 does not allow updating details of resolved or cancelled cases in the interface, but allows updates through Power Automate flows or the API when the setting is enabled. Source
- HubSpot's default deal properties include Closed lost reason, which records why a deal was lost. Source
- Salesforce retains standard field history for up to 18 months, and up to 24 months through the API, unless Field Audit Trail extends retention. Source
- Lakkaraju et al. (KDD 2017) describe the selective labels problem, where the decision process determines which outcomes are observed. Source
- Datasheets for Datasets asks whether a dataset is a sample of a larger set, whether it is representative, and whether instances are missing information. Source
- NCES statistical standards require any survey stage with a unit or item response rate below 85 percent to be evaluated for nonresponse bias. Source
- The NIST Engineering Statistics Handbook describes the Wilson method for a confidence interval on a proportion. Source
Tell us what data you need
Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.