Systems and records
Missing failures in an archive dataset: deleted, overwritten or never logged?
By SourceX Editorial · Reviewed by Noah Loul ·
9 sections
Short answer
Failures drop out of a business archive in three ways: someone deleted them, a later success overwrote them, or nobody logged them. Audit each separately before training or testing on the archive. Read field history and audit logs for overwrites, deletion views and schedules for purges, and for unlogged failures, match two independent witness sources and estimate the remainder neither one caught.
System export
Missing failures archive
Key takeaways
- Deleted, overwritten and never-logged failures leave different traces, need different fixes and support different disclosures, so run three audits, not one.
- The evidence runs on a clock: Zendesk keeps deleted tickets for 30 days, Microsoft Purview Audit (Standard) keeps records for 180 days and Salesforce field history lasts about 18 months in the org.
- Count attempts, not final records: a ticket reopened twice and a job that needed a second visit both close looking like clean successes.
- Two independent witness sources and their overlap give a capture-recapture estimate of failures neither source logged, as long as the sources are independent and matched accurately.
- When a mechanism cannot be measured, narrow the claim: an archive that lost its failures can still teach correct procedure, but not failure rates or recovery.
Where do failures go when an archive looks clean?
They are deleted, overwritten by a later success or never written down, and each route leaves different evidence behind. An archive with almost no failed jobs, lost deals or abandoned tickets is more often a record of what survived than a record of a business that rarely fails.
The decision at stake is whether an archive can support claims about failure: how often it happens, why, and how the business recovered. Agent builders need those cases most. Anthropic's guidance on agent evaluations says capability evals should start at a low pass rate so a team has something to climb; tasks cut from a success-only archive tend to start high and teach little about recovery.
Useful evidence is a count of failed attempts from somewhere other than the archive itself, split by the mechanism that removed them. The method below classifies the mechanism, makes the attempt the sampling unit, estimates unlogged failures with two witnesses, records everything in a worksheet and ends with what the archive cannot claim.
Deleted, overwritten or never logged: which mechanism removed them?
Classify before counting, because the mechanism decides where to look and whether the gap can be repaired. Deletion removes a record that existed. Overwriting keeps the record but replaces the failed state with a later one. Never-logged failures happened outside the system, or before anyone opened a record.
Overwriting is the mechanism most audits miss. A Jira issue reopened twice still exports with the status Done. A field service job that needed a second visit closes as completed under one job number. A pull request merged with squash leaves a single commit on the base branch; GitHub's documentation notes that the intermediate commits are not kept there as separate entries, so attempts that failed their tests disappear from that branch's history.
Never-logged failures are the hardest. A technician who fixes a fault by phone, a quote never entered because the customer walked away, a refund issued straight from the payment processor: none creates a record in the system being licensed, though many leave a trace in some other system. Even administrative traces expire; Microsoft Purview Audit (Standard) keeps audit records for 180 days by default.
The six ways failures leave an archive, with the evidence each leaves and how long that evidence lasts:
| Mechanism | Business system example | Evidence that survives | How long the evidence lasts |
|---|---|---|---|
| Deleted by a user or admin | A help desk agent deletes the ticket for a failed order | Deleted-items view, ID gaps, exports taken earlier | Zendesk: 30 days in the deleted tickets view, then permanent |
| Deleted by a retention schedule | Closed-lost deals purged after a set period | The schedule itself, old reports, counts in finance systems | Until those reports are purged as well |
| Overwritten in place | Opportunity stage reverted, ticket reopened then solved | Field history and audit trails | Salesforce field history: about 18 months in the org, 24 through the API |
| Compacted on merge | Pull request squashed into one commit | The pull request page and CI logs | As long as the repository host keeps them |
| Never logged in the system | Fault fixed by phone, quote never entered | Witness systems: billing credits, call logs, warranty claims | Each witness system's own retention |
| Logged only in an audit trail | Failed sign-in or permission change before a task | Admin audit log | Microsoft Purview Audit (Standard): 180 days |
Why the sampling unit must be the attempt, not the record
Sample attempts, because a final-state record hides every try before the last one. A dataset of closed records answers what the final outcome was; a failure audit needs what happened on each attempt. The difference is largest in the workflows buyers care about most, such as repairs, collections and claims.
Rebuild attempts from history before drawing any sample. Field history tables, ticket audit events, status-change logs and job visit records each turn one final record into an ordered list of states. The attempt-to-record ratio, by month and by workflow, is the first number the audit produces; a ratio that drops sharply after a migration or a retention change usually marks lost history, not better performance.
Define the attempt in operational terms for each workflow: for a help desk, every transition from solved back to open starts a new attempt; for field service, every visit; for sales, every stage regression; for code, every CI run on a pull request. Then stratify by what followed each attempt, because a simple random sample of attempts is still dominated by first-time successes. Oversample attempts that preceded a reopen, a revisit or a credit, and keep the sampling weights so overall rates can be restored.
The clocks make this urgent. Salesforce's help documentation says field history is kept for about 18 months, with data up to 24 months old reachable through the API, unless the org extends retention with Field Audit Trail. History that has aged out cannot be rebuilt from the record itself, so the audit has to run before the export is cut, not after delivery.
How do you count failures nobody logged?
Use two independent witness sources and their overlap. If one source records n1 failures, a second records n2, and m appear in both, the two-source capture-recapture estimate of all failures is n1 × n2 ÷ m. The idea comes from counting animal populations and is used in public health to estimate cases that every surveillance list missed.
A CDC-published study describes the approach as examining the overlap in how sources identify cases and using that overlap to estimate the cases no source identified. A CDC-published sensitivity analysis of workplace injury undercounts found that positive dependence between sources had the greatest impact on its estimates, which is the assumption to test first in a business setting.
Choose witnesses by how they learn about a failure, not by convenience. A billing credit witnesses failures customers pushed back on; a warranty claim witnesses failures that broke something; a technician's parts return witnesses failures that needed hardware. Witnesses that learn from the same event, such as a complaint that triggers both a ticket and a credit, are dependent by construction, so prefer pairs that observe failure through different routes.
Illustrative: an installer's help desk tags 420 failed installations in a year, and its billing system shows 300 installations credited for failure, 180 of them matching help desk tags. That leaves 240 seen only by the help desk and 120 only in billing. The estimate is 420 × 300 ÷ 180 = 700 failed installations. The two sources together saw 540, so about 160 failures appear in neither, and the help desk captured 60 percent of the estimated total.
The estimate holds only if the sources are independent, the population is closed over the period and matching is accurate. Positive dependence, such as large customers being both more likely to complain and more likely to receive credits, pushes the estimate down, so read it as a floor. Match on a de-identified key such as a hashed job number generated inside the supplier's environment, stratify by segment or region where catch rates plausibly differ, and add a third witness, such as warranty claims, to check dependence between the first two.
The missing-failure sampling worksheet
The worksheet fixes definitions and evidence before any rate is computed, so two reviewers reach the same count. Its entries feed the parts of a standard dataset card: the Hugging Face template asks for data selection criteria and filtering under data collection and processing, and for technical and sociotechnical limitations in a section of their own.
Complete one row per line for each workflow in scope; the entries shown are illustrative:
| Worksheet line | What to record | Illustrative entry | Evidence source |
|---|---|---|---|
| Failure definition | What counts as a failed attempt, in operational terms | Installation needing a second visit or a credit within 30 days | Process owner sign-off |
| Attempt population | All attempts in scope, rebuilt from history, with the period covered | All installation visits, January 2024 to December 2025 | Job visit table and field history |
| Export selection rules | Filters applied: statuses, dates, record types, completeness | Closed jobs only; drafts excluded | Export query and its version |
| Mechanism split | Count or estimate for each mechanism: deleted, overwritten, never logged | Overwritten 210; deleted unknown; unlogged about 160 | History tables, deletion views, two-witness estimate |
| Witness sources | Systems outside the archive that record failures, with their retention | Billing credits; warranty claims as a third check | Finance and warranty exports |
| Matching key | De-identified key that links witnesses, and how matching errors were checked | Hashed job number generated by the supplier | Supplier matching log plus a manual sample |
| Evidence clocks | Retention window of each trace and the date the audit ran | Field history 18 months; audit log 180 days | System documentation and admin settings |
| Correlation | Clustering of failures by site, technician, product or customer | 40 percent of failures at 3 of 25 sites | Grouped counts from the attempt table |
Test the strongest alternative: does the business simply fail less?
Sometimes the archive is right and failures really are rare. A mature process, strict intake screening or a simple product can all produce few failures, and an audit that assumes missing data will invent a problem. The witnesses settle it: if billing credits, warranty claims and call logs also show few failures, the low rate is probably real.
Three patterns point the other way. Many reopen events in history tables alongside very few failures in final states indicate overwriting. A failure share that drops at a migration date or a retention boundary indicates deletion or lost history. Failures concentrated in one channel, such as email but not phone, indicate a logging gap rather than a difference in performance.
Repeated cases distort both directions. One failure that generated five tickets inflates the logged count, and five failures filed under one parent ticket deflate it. Count failures at the attempt level after linking duplicates, and report how many raw records collapsed into each counted failure, because a large record count can hide both repetition and absence.
One limit applies to every method here: failures nobody noticed at the time, such as a wrong answer a customer accepted, appear in no witness and no estimate recovers them. The honest disclosure is that the audit measures noticed failures only.
What can the archive claim once failures are missing?
Write the limitations as claims that are supported, claims that are not and the evidence that would change the verdict. An archive that lost its failures can still be valuable for teaching correct procedure; it cannot carry failure rates, recovery behavior or error-handling evaluations without witness evidence.
The coverage limitations table for the illustrative installer archive:
| Claim a buyer might make | Supported when | Not supported when | Disclose as |
|---|---|---|---|
| Failure rate estimate | A witness-based estimate exists with dependence checked | Only the archive's own failure flags exist | Archive rate plus the two-witness floor |
| Training on recovery steps | Attempt history shows the failed and the corrected states | Overwrites erased the failed state | Share of failures with full attempt history |
| Evaluating agent error handling | Failed attempts sampled across sites and segments | Failures cluster in a few sites | Per-site counts and excluded sites |
| Root-cause modeling | Failure reason recorded at the time | Reason reconstructed later from free text | Reason source and missing-reason share |
| Time-to-resolution analysis | Open cases kept and treated as censored | The export kept closed jobs only | Selection rule and open-case count |
Re-extract, recover, supplement or narrow: deciding per mechanism
Overwritten failures are the cheapest to repair: re-extract with history tables and audit events before their retention windows close. Deleted failures can sometimes be recovered from earlier exports or backups, but only where the company may lawfully keep and use them; records removed under a deletion request or a legal obligation stay deleted, whatever their value to a dataset.
Never-logged failures can be supplemented only from witness systems inside the licensed scope, such as billing credits or warranty claims linked by a de-identified key. Where no witness is licensed, narrow the claim and record the two-witness estimate in the dataset documentation, so a buyer can judge how much of the failure space the archive covers.
Timing decides more than method. Zendesk's documentation says deleted tickets stay in a deleted tickets view for 30 days before automatic permanent deletion, and that tickets archived more than 120 days after being solved do not appear in that view at all. Run the audit before the export is cut, and record the date it ran next to every count.
Whatever the decision, write the mechanism split and the witness estimate into the dataset card's collection and limitations sections, with the audit date, so a buyer reading it a year later knows which clocks had already run out when the counts were taken.
Asking for failures in a dataset request
State the failure definition and the evidence you will accept in the request itself. SourceX holds nothing in stock and sources from U.S. businesses only when asked; its request form already collects licensing needs, timing, format, language, volume and scope, and it is also where a failure requirement belongs.
Buyers who plan to evaluate agents on error handling should also ask for the attempt-to-record ratio by month. It shows at a glance whether history is intact across the whole period or thins out before a migration or a retention boundary.
A useful request names the workflow, the failure definition, attempt-level history as a required field, any witness sources the buyer wants linked and an acceptance test, such as a minimum share of failures carrying full attempt history. Whether a particular company's history tables or witness systems can be licensed is settled in that dataset's rights and privacy review; a request is a specification, not a promise that such records exist.
Frequently asked questions
How can you tell whether failures were deleted or never recorded?
Look for traces inside the system first: gaps in record IDs, deleted-items views, retention schedules and history tables point to deletion or overwriting. If none shows a gap but witness systems such as billing credits or warranty claims record failures the archive lacks, the failures were probably never logged there.
What is a two-witness estimate of missing failures?
It is the two-source capture-recapture method: count failures in two independent sources, count the overlap, and estimate the total as the product of the two counts divided by the overlap. The difference between that total and the failures either source saw estimates the failures nobody logged. Dependence between sources biases it, usually downward.
Do audit logs and field history keep failed attempts indefinitely?
Usually not by default. Microsoft Purview Audit (Standard) keeps audit records for 180 days, and Salesforce keeps field history for about 18 months in the org with up to 24 months through the API, unless extended retention is added. Check each system's settings before relying on its history.
Can deleted failure records be restored for an AI dataset?
Sometimes, from earlier exports or backups, but only where the company may lawfully keep and use them. Records deleted under a privacy request or a legal obligation must stay deleted. Counsel should review any recovery plan before such records join a licensed dataset.
Why do AI evaluations need failed attempts from business archives?
Capability evaluations are meant to start with a low pass rate, and real failures supply the hard cases. An archive stripped of failed attempts shows what a finished job looks like but not how work went wrong or how people recovered, which is what agents are most often tested on.
Sources
- Anthropic says capability evals should start at a low pass rate so teams have something to improve against. Source
- GitHub's squash merging combines a pull request's commits into one commit on the base branch, without keeping the intermediate commits as separate entries. Source
- Zendesk keeps deleted tickets in a deleted tickets view for 30 days before permanent deletion; tickets archived more than 120 days after being solved do not appear there. Source
- Salesforce keeps field history for about 18 months in the org, with up to 24 months reachable through the API, unless Field Audit Trail extends retention. Source
- Microsoft Purview Audit (Standard) retains audit records for 180 days by default. Source
- A CDC-published study describes capture-recapture as using the overlap between data sources to estimate cases no source identified. Source
- A CDC-published sensitivity analysis of workplace injury undercounts found that positive dependence between sources had the greatest impact on capture-recapture estimates. Source
- The Hugging Face dataset card template asks for data selection criteria and filtering methods, and for technical and sociotechnical limitations. Source
Tell us what data you need
Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.