Skip to content

Systems and records

Post-outcome edits and dataset leakage: a field-by-field rollback test

By SourceX Editorial · Updated

9 sections

Short answer

Post-outcome edits cause dataset leakage when a record's fields are rewritten after the result is known, so a task built from today's export hands the model hindsight. Prevent it field by field: read each field's change history, roll inputs back to the moment of decision, move fields first written after the outcome to the label side, and withhold any field whose history has expired instead of trusting its current value.

Evaluation split

Post outcome edits dataset

This page covers post-outcome edits and dataset leakage: a field-by-field rollback test.Illustrative

Key takeaways

  • An export shows each record as it stands today, not as it stood when the decision was made.
  • Classification, urgency, ownership and narrative fields are the usual suspects; a 2018 study of issue-tracker research found summary, component and assignee mattered most.
  • Event-time joins are not enough: Feast's documentation notes that a value corrected after the event can still be returned unless write time is filtered too.
  • History expires: Salesforce's standard field history keeps up to 18 months in the app, and Delta Lake time travel defaults to 30 days.
  • Sort every input field as frozen, rolled back, label-only or unrecoverable, then measure the score gap between as-of and current versions of the task.

What is post-outcome edit leakage in a dataset?

Post-outcome edit leakage happens when a field in a business record is created or rewritten after the outcome, and a dataset then uses that later value as an input to a task that is supposed to come before the outcome. The model sees part of the answer. Kaufman, Rosset, Perlich and Stitelman, in their 2012 ACM TKDD paper on leakage in data mining, frame leakage as information about the target that should not legitimately be available to learn from, and observe that it usually slips in through collection, aggregation and preparation rather than by intent.

Edited records are a quiet route for it. A support ticket's category is corrected once engineering finds the real cause. A deal's amount is updated to the signed figure. A claim's severity is reclassified after the payout. Each edit is sound housekeeping for the business, and each one rewrites the past for anyone who exports the record later.

The fix is not to distrust every field. It is to know, for each field a task uses, which value it held at the decision moment, and to prove that from the system's own change history. What follows is a four-status field audit, the history sources that make a rollback possible and where they run out, a way to measure how much the edits matter, and an annotated packet for one decision.

Which fields get rewritten after the result is known?

Fields that classify, rank or assign work are rewritten most, because people tidy them once they know how the case ended. Free text comes next, since agents edit summaries and descriptions to match the final diagnosis.

At a glance

Which fields get rewritten after the result is known

Software research has measured this. Tu, Zhu, Zheng and Zhou (ESEC/FSE 2018) examined how studies used issue-tracker data and found that issue attributes change over time while many studies read them as of data collection rather than as of the moment being predicted. Of the 58 studies they reviewed, 11 had a leakage problem and 44 more were suspected. When they reproduced three of them, the summary, component and assignee attributes had the largest effect, and the bias they measured was real but not striking.

That last finding matters as much as the first. Hindsight edits do not always wreck a dataset; the damage depends on the field and the task. Start with the field families listed below, then measure rather than assume.

  • Classification: category, component, product area, issue type and reason codes, often corrected once the cause is known.
  • Urgency: priority, severity and escalation flags, raised or lowered after the fact.
  • Ownership: assignee, team and queue, which can show where the work ended rather than where it began.
  • Narrative: summary, description and subject lines edited to match the final diagnosis.
  • Commercial: amount, close date, discount and loss reason, updated to the signed or final figure.
  • Backfills: fields created later and filled across old records by a script or a migration.

Event time or write time: which clock decides what the model may see?

Write time decides. A value belongs in a task input only if it had been written by the decision moment, not merely if it describes something that happened before then. A join keyed on event time alone lets later corrections straight through.

Feature-store documentation states the trap plainly. Feast's guide to point-in-time joins says the join reproduces feature values as of each row's timestamp, scanning backward from it, but by default constrains only the event timestamp, so a value backfilled or corrected after that moment can still be returned. Its remedy is an extra filter on the created timestamp, which the guide describes as keeping backfilled values out of training data and reproducing what the online store would have served at the time.

Business records carry the same two clocks. A ticket's category can describe the problem as it stood on day 0 yet have been written on day 4. A CRM loss reason can describe a March meeting yet have been typed in May. For every input field, the audit needs the write time of the value used, not only the business date that value refers to.

Where does the past survive, and when does it expire?

The past survives in change logs and snapshots, each with a limit that sets how far back a rollback can reach. Check those limits before promising anyone as-of fields, because once a history expires only the current value is left.

Application history is selective. Salesforce's standard field history covers up to 20 tracked fields per object and, without the paid Field Audit Trail add-on, keeps that data for up to 18 months in the app and up to 24 months through the API. Jira Cloud's changelog endpoint returns a paginated list of every change to an issue, sorted from the oldest. HubSpot stores property value updates in each record's property history, which the API returns when a request names properties in the propertiesWithHistory parameter.

Warehouse history depends on what the data team actually ran. dbt snapshots record changes to a mutable table as type-2 slowly changing dimensions, with dbt_valid_from and dbt_valid_to columns, and dbt's own guide says snapshots are only useful if run frequently; two changes between runs leave a single trace. Delta Lake time travel can query an older version of a table, by default up to 30 days back, and running VACUUM deletes the data files that older versions depend on. The table below sets the five sources side by side.

Where does the past survive, and when does it expire?
History sourceWhat it keepsLimit to checkUse in a rollback
Salesforce field historyOld and new values of tracked fieldsUp to 20 tracked fields per object; 18 months in the app and 24 via the API without Field Audit TrailRoll tracked fields back; untracked fields hold current values only
Jira Cloud changelogEvery change to an issue, oldest firstPaginated, so each issue must be read to the endReplay changes up to the decision time
HubSpot property historyPast values of record propertiesReturned only for properties named in the requestTake the last value written before the decision
dbt snapshotsType-2 rows with valid-from and valid-to timesCatches only the state at each scheduled runJoin on the row valid at the decision time
Delta Lake time travelOlder versions of a whole table30 days by default; VACUUM removes the files old versions needRead the table version current at the decision time

Frozen, rolled back, label-only or unrecoverable: sorting every input field

Give every field a task might use exactly one of four statuses, decided from its history rather than its name. The status fixes what enters the task packet: the current value, a reconstructed as-of value, nothing on the input side, or an explicit gap.

Two rules keep the audit honest. A field with no tracked history cannot be called frozen just because nobody remembers editing it; it stays unrecoverable until a sample shows it never changes. And a field first written after the outcome is label-only even when its value looks harmless, because the mere fact that it was filled in can tell a model the case is closed.

Frozen, rolled back, label-only or unrecoverable: sorting every input field
StatusTest from the historyIllustrative fieldWhat enters the task packet
FrozenNo write after the decision time in the change logChannel the ticket arrived throughThe current value
Rolled backWritten after the decision time, and the history covers the changeCategory corrected on day 4The last value written before the decision
Label-onlyFirst written at or after the outcomeRoot cause, resolution code, satisfaction scoreNothing on the input side; may serve as a label
UnrecoverableChanged, or may have changed, after the decision with no history to show itCustomer tier on an object without trackingA gap marker with the reason, never the current value

How much do the edits matter? Measure the hindsight inflation

Build the task twice, once from as-of values and once from the current export, and score one model on both versions; the gap between the two scores is the hindsight inflation for that task. Tu and colleagues found the bias real but modest in the studies they rebuilt, so measure it task by task rather than assuming it is large or small.

Three numbers per field make the audit reviewable. The post-decision write rate is the share of records with any write to the field after the decision time. Value drift is the share where the as-of value differs from today's value. The late-fill share is the share of records where the field was first filled at or after the outcome. Report each with its denominator and the date range of the records, since a migration year can look nothing like an ordinary one.

Fix the decision rules before running the comparison:

  • Value drift below a preset threshold, such as 1 percent of records (an illustrative setting): use the as-of value anyway and publish the rate.
  • Drift above the threshold with history for every record: roll the field back and record the drift figure in the data card.
  • Drift above the threshold with history missing for some records: roll back where possible, mark the rest as gaps, and test whether gapped records differ in outcome from the others.
  • As-of score well below the current-export score: treat any earlier result built on the export as inflated and rerun it on the as-of version.

One escalation decision, packed with only what existed at hour 2 (illustrative)

The decision: two hours after a billing complaint arrives, should a tier 1 agent escalate it? The packet holds only what existed at that moment, with each element's source and status, and the figure traces how the same ticket's fields changed afterward.

Three edits would have leaked the answer. The category moved from Billing to Product defect on day 4, after engineering found a sync bug. The description gained the agent's diagnosis the same day. The priority rose to High at hour 5, after the very escalation it would help predict. A model trained on the export would learn that product-defect tickets marked High get escalated, which is true only in hindsight.

The packet records one gap instead of hiding it. This illustrative system does not track customer tier, and the account was upgraded on day 20, so the current value is withheld and the gap is listed with its reason. The assignee change at hour 2 is the decision itself, so it becomes the label, not an input.

One escalation decision, packed with only what existed at hour 2 (illustrative)
Packet elementAs-of valueWhere the value comes fromStatus
Customer messageThe text as first submittedFirst version in the ticket's audit trailRolled back
CategoryBillingChange log entry before day 4Rolled back
PriorityNormalChange log entry before hour 5Rolled back
Prior contacts in 90 days2Tickets created before the decision timeFrozen
Refund policy in forceThe version effective that weekThe policy document's revision historyRolled back
Customer tierWithheldNo field history on this objectUnrecoverable, gap listed
Escalated to tier 2YesAssignee change at hour 2Label
Root cause and resolution codeExcludedWritten on day 3 and laterLabel-only
An edit trace for one illustrative support ticket: six field lanes on a time axis from opening to export, with category, priority and description rewritten after the escalation decision, root cause written only after the outcome, and customer tier changed with no history, each marked with its use in the task packet.
One ticket, traced field by field. Illustrative timings and values.

Would joining the full change history make the task clearer?

No. A change history is a tool for reconstruction, not an input. Joining every changelog entry puts post-decision writes straight back into the packet, and even the entries from before the decision add noise and linkage risk that rarely sharpen the task.

Pre-decision history can earn a place in narrow cases. How often a priority changed before the decision, or how long a ticket sat unassigned, can be genuine signals for an escalation task. Include them as derived counts computed only from entries written before the decision time, not as raw log rows.

Raw logs also carry people. Change entries usually record who made each change, which is personal data and a key that links records across systems. Replace authors with roles, such as tier 1 agent or automation, and drop entries whose only content is a person's identity. Every extra join widens what could be re-identified without telling the model anything more about the decision.

Putting as-of requirements into a data request

Ask for history, not just records, and say how you will test it. A supplier can preserve as-of values only if it extracts them before retention limits or cleanup jobs remove them, so the requirement has to sit in the request, ahead of extraction.

SourceX sources each dataset from U.S. businesses after a buyer describes the need, rather than selling from stock, so an as-of requirement can shape the extraction itself. Add it to the scope, volume, language, format, timing and licensing needs the request form already collects. The supplying company's systems still decide what history exists; no request can bring back a log that has already expired.

A request that covers these six points gives a supplier something it can check before any extraction starts:

Checklist · 6 items

  1. Name the decision moment for each task and the timestamp that marks it.
  2. List the input fields and ask for each one's change history, or a statement that none exists.
  3. Ask for write times as well as business dates for every value supplied.
  4. Require label-only fields to arrive in a separate file from the inputs.
  5. Ask for a gap list: fields whose history expired or was never tracked, with record counts.
  6. State the acceptance test: on a sample, rebuilt as-of values must match the change log.

0 of 6 ticked. Tick each item as you go.

Frequently asked questions

What is hindsight leakage in a training dataset?

It is information created after a task's outcome that ends up in the task's inputs, so a model learns from facts nobody had at decision time. In business records it usually comes from fields edited once the result was known, such as a corrected category or an updated deal amount. Scores built on that data look better than the model will perform.

How can I tell whether a field was edited after the outcome?

Read the field's change history and compare each write time with that record's decision and outcome times. Salesforce field history, the Jira changelog and HubSpot property history all return past values with timestamps. If a field has no history at all, treat it as unrecoverable until a sample shows it never changes.

Can a record be rebuilt if field history was never turned on?

Only in part. Warehouse snapshots or table time travel may hold older copies, but dbt snapshots catch only the state at each run and Delta Lake keeps 30 days of versions by default. Where no older copy exists, withhold the field from the inputs and list it as a gap rather than using today's value.

Is a point-in-time join enough to stop leakage?

Not by itself. A join keyed on event time picks values that describe the right moment but may have been written later; Feast's documentation notes that corrected or backfilled values can still be returned unless created time is filtered as well. Join on the time each value was written.

Should resolution notes ever be part of the input?

Only when the task begins after resolution, such as summarizing a closed case for a handover. For any task that predicts or makes the decision, resolution notes, root causes and satisfaction scores belong on the label side, kept in a separate file.

Sources

  • Kaufman, Rosset, Perlich and Stitelman (ACM TKDD, 2012) describe leakage as information about the target that should not legitimately be available to learn from, usually introduced unintentionally during collection, aggregation and preparation. Source
  • Tu, Zhu, Zheng and Zhou (ESEC/FSE 2018) found 11 of 58 issue-tracker studies had a time-related leakage problem and 44 more were suspected; summary, component and assignee had the largest impact, and the measured bias was real but not striking. Source
  • Feast's point-in-time join guide says joins constrain only the event timestamp by default, so values backfilled or corrected later can still be returned, and a created-timestamp filter restricts retrieval to values available at the time. Source
  • Salesforce Help: standard field history tracks up to 20 fields per object and, without Field Audit Trail, retains history for up to 18 months, and up to 24 months via the API. Source
  • Jira Cloud's REST API returns a paginated list of all changelogs for an issue, sorted from the oldest. Source
  • HubSpot stores property value updates in a record's property history, retrieved through the propertiesWithHistory parameter. Source
  • dbt snapshots record changes to mutable tables as type-2 slowly changing dimensions with dbt_valid_from and dbt_valid_to, and are only useful if run frequently. Source
  • Delta Lake time travel queries older table versions, by default up to 30 days back, unless VACUUM has run or retention settings changed. Source

Tell us what data you need

Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.

See if your company qualifies