Systems and records
Post-outcome edits and dataset leakage: a field-by-field rollback test
By SourceX Editorial · Updated
9 sections
Short answer
Post-outcome edits cause dataset leakage when a record's fields are rewritten after the result is known, so a task built from today's export hands the model hindsight. Prevent it field by field: read each field's change history, roll inputs back to the moment of decision, move fields first written after the outcome to the label side, and withhold any field whose history has expired instead of trusting its current value.
Evaluation split
Post outcome edits dataset
Key takeaways
- An export shows each record as it stands today, not as it stood when the decision was made.
- Classification, urgency, ownership and narrative fields are the usual suspects; a 2018 study of issue-tracker research found summary, component and assignee mattered most.
- Event-time joins are not enough: Feast's documentation notes that a value corrected after the event can still be returned unless write time is filtered too.
- History expires: Salesforce's standard field history keeps up to 18 months in the app, and Delta Lake time travel defaults to 30 days.
- Sort every input field as frozen, rolled back, label-only or unrecoverable, then measure the score gap between as-of and current versions of the task.
What is post-outcome edit leakage in a dataset?
Post-outcome edit leakage happens when a field in a business record is created or rewritten after the outcome, and a dataset then uses that later value as an input to a task that is supposed to come before the outcome. The model sees part of the answer. Kaufman, Rosset, Perlich and Stitelman, in their 2012 ACM TKDD paper on leakage in data mining, frame leakage as information about the target that should not legitimately be available to learn from, and observe that it usually slips in through collection, aggregation and preparation rather than by intent.
Edited records are a quiet route for it. A support ticket's category is corrected once engineering finds the real cause. A deal's amount is updated to the signed figure. A claim's severity is reclassified after the payout. Each edit is sound housekeeping for the business, and each one rewrites the past for anyone who exports the record later.
The fix is not to distrust every field. It is to know, for each field a task uses, which value it held at the decision moment, and to prove that from the system's own change history. What follows is a four-status field audit, the history sources that make a rollback possible and where they run out, a way to measure how much the edits matter, and an annotated packet for one decision.
Which fields get rewritten after the result is known?
Fields that classify, rank or assign work are rewritten most, because people tidy them once they know how the case ended. Free text comes next, since agents edit summaries and descriptions to match the final diagnosis.
At a glance
Which fields get rewritten after the result is known
Software research has measured this. Tu, Zhu, Zheng and Zhou (ESEC/FSE 2018) examined how studies used issue-tracker data and found that issue attributes change over time while many studies read them as of data collection rather than as of the moment being predicted. Of the 58 studies they reviewed, 11 had a leakage problem and 44 more were suspected. When they reproduced three of them, the summary, component and assignee attributes had the largest effect, and the bias they measured was real but not striking.
That last finding matters as much as the first. Hindsight edits do not always wreck a dataset; the damage depends on the field and the task. Start with the field families listed below, then measure rather than assume.
- Classification: category, component, product area, issue type and reason codes, often corrected once the cause is known.
- Urgency: priority, severity and escalation flags, raised or lowered after the fact.
- Ownership: assignee, team and queue, which can show where the work ended rather than where it began.
- Narrative: summary, description and subject lines edited to match the final diagnosis.
- Commercial: amount, close date, discount and loss reason, updated to the signed or final figure.
- Backfills: fields created later and filled across old records by a script or a migration.
Event time or write time: which clock decides what the model may see?
Write time decides. A value belongs in a task input only if it had been written by the decision moment, not merely if it describes something that happened before then. A join keyed on event time alone lets later corrections straight through.
Feature-store documentation states the trap plainly. Feast's guide to point-in-time joins says the join reproduces feature values as of each row's timestamp, scanning backward from it, but by default constrains only the event timestamp, so a value backfilled or corrected after that moment can still be returned. Its remedy is an extra filter on the created timestamp, which the guide describes as keeping backfilled values out of training data and reproducing what the online store would have served at the time.
Business records carry the same two clocks. A ticket's category can describe the problem as it stood on day 0 yet have been written on day 4. A CRM loss reason can describe a March meeting yet have been typed in May. For every input field, the audit needs the write time of the value used, not only the business date that value refers to.
Where does the past survive, and when does it expire?
The past survives in change logs and snapshots, each with a limit that sets how far back a rollback can reach. Check those limits before promising anyone as-of fields, because once a history expires only the current value is left.
Application history is selective. Salesforce's standard field history covers up to 20 tracked fields per object and, without the paid Field Audit Trail add-on, keeps that data for up to 18 months in the app and up to 24 months through the API. Jira Cloud's changelog endpoint returns a paginated list of every change to an issue, sorted from the oldest. HubSpot stores property value updates in each record's property history, which the API returns when a request names properties in the propertiesWithHistory parameter.
Warehouse history depends on what the data team actually ran. dbt snapshots record changes to a mutable table as type-2 slowly changing dimensions, with dbt_valid_from and dbt_valid_to columns, and dbt's own guide says snapshots are only useful if run frequently; two changes between runs leave a single trace. Delta Lake time travel can query an older version of a table, by default up to 30 days back, and running VACUUM deletes the data files that older versions depend on. The table below sets the five sources side by side.
| History source | What it keeps | Limit to check | Use in a rollback |
|---|---|---|---|
| Salesforce field history | Old and new values of tracked fields | Up to 20 tracked fields per object; 18 months in the app and 24 via the API without Field Audit Trail | Roll tracked fields back; untracked fields hold current values only |
| Jira Cloud changelog | Every change to an issue, oldest first | Paginated, so each issue must be read to the end | Replay changes up to the decision time |
| HubSpot property history | Past values of record properties | Returned only for properties named in the request | Take the last value written before the decision |
| dbt snapshots | Type-2 rows with valid-from and valid-to times | Catches only the state at each scheduled run | Join on the row valid at the decision time |
| Delta Lake time travel | Older versions of a whole table | 30 days by default; VACUUM removes the files old versions need | Read the table version current at the decision time |
Frozen, rolled back, label-only or unrecoverable: sorting every input field
Give every field a task might use exactly one of four statuses, decided from its history rather than its name. The status fixes what enters the task packet: the current value, a reconstructed as-of value, nothing on the input side, or an explicit gap.
Two rules keep the audit honest. A field with no tracked history cannot be called frozen just because nobody remembers editing it; it stays unrecoverable until a sample shows it never changes. And a field first written after the outcome is label-only even when its value looks harmless, because the mere fact that it was filled in can tell a model the case is closed.
| Status | Test from the history | Illustrative field | What enters the task packet |
|---|---|---|---|
| Frozen | No write after the decision time in the change log | Channel the ticket arrived through | The current value |
| Rolled back | Written after the decision time, and the history covers the change | Category corrected on day 4 | The last value written before the decision |
| Label-only | First written at or after the outcome | Root cause, resolution code, satisfaction score | Nothing on the input side; may serve as a label |
| Unrecoverable | Changed, or may have changed, after the decision with no history to show it | Customer tier on an object without tracking | A gap marker with the reason, never the current value |
How much do the edits matter? Measure the hindsight inflation
Build the task twice, once from as-of values and once from the current export, and score one model on both versions; the gap between the two scores is the hindsight inflation for that task. Tu and colleagues found the bias real but modest in the studies they rebuilt, so measure it task by task rather than assuming it is large or small.
Three numbers per field make the audit reviewable. The post-decision write rate is the share of records with any write to the field after the decision time. Value drift is the share where the as-of value differs from today's value. The late-fill share is the share of records where the field was first filled at or after the outcome. Report each with its denominator and the date range of the records, since a migration year can look nothing like an ordinary one.
Fix the decision rules before running the comparison:
- Value drift below a preset threshold, such as 1 percent of records (an illustrative setting): use the as-of value anyway and publish the rate.
- Drift above the threshold with history for every record: roll the field back and record the drift figure in the data card.
- Drift above the threshold with history missing for some records: roll back where possible, mark the rest as gaps, and test whether gapped records differ in outcome from the others.
- As-of score well below the current-export score: treat any earlier result built on the export as inflated and rerun it on the as-of version.
One escalation decision, packed with only what existed at hour 2 (illustrative)
The decision: two hours after a billing complaint arrives, should a tier 1 agent escalate it? The packet holds only what existed at that moment, with each element's source and status, and the figure traces how the same ticket's fields changed afterward.
Three edits would have leaked the answer. The category moved from Billing to Product defect on day 4, after engineering found a sync bug. The description gained the agent's diagnosis the same day. The priority rose to High at hour 5, after the very escalation it would help predict. A model trained on the export would learn that product-defect tickets marked High get escalated, which is true only in hindsight.
The packet records one gap instead of hiding it. This illustrative system does not track customer tier, and the account was upgraded on day 20, so the current value is withheld and the gap is listed with its reason. The assignee change at hour 2 is the decision itself, so it becomes the label, not an input.
| Packet element | As-of value | Where the value comes from | Status |
|---|---|---|---|
| Customer message | The text as first submitted | First version in the ticket's audit trail | Rolled back |
| Category | Billing | Change log entry before day 4 | Rolled back |
| Priority | Normal | Change log entry before hour 5 | Rolled back |
| Prior contacts in 90 days | 2 | Tickets created before the decision time | Frozen |
| Refund policy in force | The version effective that week | The policy document's revision history | Rolled back |
| Customer tier | Withheld | No field history on this object | Unrecoverable, gap listed |
| Escalated to tier 2 | Yes | Assignee change at hour 2 | Label |
| Root cause and resolution code | Excluded | Written on day 3 and later | Label-only |
Would joining the full change history make the task clearer?
No. A change history is a tool for reconstruction, not an input. Joining every changelog entry puts post-decision writes straight back into the packet, and even the entries from before the decision add noise and linkage risk that rarely sharpen the task.
Pre-decision history can earn a place in narrow cases. How often a priority changed before the decision, or how long a ticket sat unassigned, can be genuine signals for an escalation task. Include them as derived counts computed only from entries written before the decision time, not as raw log rows.
Raw logs also carry people. Change entries usually record who made each change, which is personal data and a key that links records across systems. Replace authors with roles, such as tier 1 agent or automation, and drop entries whose only content is a person's identity. Every extra join widens what could be re-identified without telling the model anything more about the decision.
Putting as-of requirements into a data request
Ask for history, not just records, and say how you will test it. A supplier can preserve as-of values only if it extracts them before retention limits or cleanup jobs remove them, so the requirement has to sit in the request, ahead of extraction.
SourceX sources each dataset from U.S. businesses after a buyer describes the need, rather than selling from stock, so an as-of requirement can shape the extraction itself. Add it to the scope, volume, language, format, timing and licensing needs the request form already collects. The supplying company's systems still decide what history exists; no request can bring back a log that has already expired.
A request that covers these six points gives a supplier something it can check before any extraction starts:
Checklist · 6 items
- Name the decision moment for each task and the timestamp that marks it.
- List the input fields and ask for each one's change history, or a statement that none exists.
- Ask for write times as well as business dates for every value supplied.
- Require label-only fields to arrive in a separate file from the inputs.
- Ask for a gap list: fields whose history expired or was never tracked, with record counts.
- State the acceptance test: on a sample, rebuilt as-of values must match the change log.
0 of 6 ticked. Tick each item as you go.
Frequently asked questions
What is hindsight leakage in a training dataset?
It is information created after a task's outcome that ends up in the task's inputs, so a model learns from facts nobody had at decision time. In business records it usually comes from fields edited once the result was known, such as a corrected category or an updated deal amount. Scores built on that data look better than the model will perform.
How can I tell whether a field was edited after the outcome?
Read the field's change history and compare each write time with that record's decision and outcome times. Salesforce field history, the Jira changelog and HubSpot property history all return past values with timestamps. If a field has no history at all, treat it as unrecoverable until a sample shows it never changes.
Can a record be rebuilt if field history was never turned on?
Only in part. Warehouse snapshots or table time travel may hold older copies, but dbt snapshots catch only the state at each run and Delta Lake keeps 30 days of versions by default. Where no older copy exists, withhold the field from the inputs and list it as a gap rather than using today's value.
Is a point-in-time join enough to stop leakage?
Not by itself. A join keyed on event time picks values that describe the right moment but may have been written later; Feast's documentation notes that corrected or backfilled values can still be returned unless created time is filtered as well. Join on the time each value was written.
Should resolution notes ever be part of the input?
Only when the task begins after resolution, such as summarizing a closed case for a handover. For any task that predicts or makes the decision, resolution notes, root causes and satisfaction scores belong on the label side, kept in a separate file.
Sources
- Kaufman, Rosset, Perlich and Stitelman (ACM TKDD, 2012) describe leakage as information about the target that should not legitimately be available to learn from, usually introduced unintentionally during collection, aggregation and preparation. Source
- Tu, Zhu, Zheng and Zhou (ESEC/FSE 2018) found 11 of 58 issue-tracker studies had a time-related leakage problem and 44 more were suspected; summary, component and assignee had the largest impact, and the measured bias was real but not striking. Source
- Feast's point-in-time join guide says joins constrain only the event timestamp by default, so values backfilled or corrected later can still be returned, and a created-timestamp filter restricts retrieval to values available at the time. Source
- Salesforce Help: standard field history tracks up to 20 fields per object and, without Field Audit Trail, retains history for up to 18 months, and up to 24 months via the API. Source
- Jira Cloud's REST API returns a paginated list of all changelogs for an issue, sorted from the oldest. Source
- HubSpot stores property value updates in a record's property history, retrieved through the propertiesWithHistory parameter. Source
- dbt snapshots record changes to mutable tables as type-2 slowly changing dimensions with dbt_valid_from and dbt_valid_to, and are only useful if run frequently. Source
- Delta Lake time travel queries older table versions, by default up to 30 days back, unless VACUUM has run or retention settings changed. Source
Tell us what data you need
Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.