AI uses for records
Document version decision dataset: keep later edits out of the past
By SourceX Editorial · Updated
10 sections
Short answer
A document version decision dataset pairs each recorded decision with the document text that existed when it was made, not the text an export returns today. Join on the decision's own timestamp, choose the version by when it could be read as well as when it took effect, test whether later edits touch the decision, and mark every element you cannot reconstruct as a gap rather than filling it with the current version.
Document stack
Document version decision
Key takeaways
- Exports return current text, so a naive join hands a past decision information that arrived later.
- Keep two document clocks: effective-from for which rule governed, visible-from for what the decider could read.
- Revision history is perishable: Google Drive can purge binary revisions after about 30 days, and Salesforce keeps field history up to 18 months, or 24 through the API.
- An edit-after test decides whether a missing history matters: if later edits touch the relevant passage, the current text stays out.
- A five-status completeness map turns unrecoverable versions into labeled gaps a buyer can count.
How do you build a document version decision dataset without hindsight?
Pair each decision with the version of every document it relied on as that version stood at the decision's timestamp, recovered from revision history, and label anything you cannot recover as a gap instead of substituting today's text. Success is easy to state: nothing in the task packet should be something the decision maker could not have read at that moment.
Exports make this harder than it sounds, because most systems hand over the current state by default. A returns policy article, a pricing sheet, a support macro or a project brief edited in place carries every later change, including changes made because of the very case being modeled. Joined naively, the dataset teaches a model to decide with information that arrived afterward.
This guide covers the join for documents edited in place: which clock to join on, where earlier versions survive, the as-of rule with an edit-after test, an annotated minimal task packet and a completeness map. It is a method; worked entries are Illustrative.
How a later edit leaks into an earlier decision
Through four routes that are common in ordinary business systems and invisible in any single export.
At a glance
How a later edit leaks into an earlier decision
Agent evaluation has met the same failure in another setting. Anthropic's engineering guide to agent evals says each trial should be isolated by starting from a clean environment, warning that state left over between runs, such as leftover files and cached data, can cause correlated failures and can also inflate measured performance. A task packet that carries later document text is leftover state from the future.
Leakage of this sort is not a niche mistake. Kapoor and Narayanan's survey of leakage in machine-learning-based science (arXiv 2207.07048) found errors reported across 17 research fields.
The four routes into a decision dataset:
- Silent overwrite: the document keeps one ID and one URL while its text changes, so a join on document ID returns today's text for a decision made last year.
- Feedback edits: a playbook or policy is revised because a case exposed a gap, and the revised text now contains the answer the original decider lacked.
- Backdated effect: an amendment is published after the decision but given an earlier effective date, so it governed the case on paper while nobody could have read it.
- Drift in linked records: an account tier, a price or an owner field changes later, and the export shows the new value beside the old decision.
Which clock decides the join: when a rule applied, or when it could be read?
Both, kept in separate fields, because they answer different questions. Effective-from says which rule governed the case; visible-from says what the decider could have read. A task about information available at decision time joins on visible-from, and a task about compliance with the governing rule joins on effective-from.
The decision has a clock problem of its own. Use the moment the decision was made, such as an approval or status change in the record's audit trail, not the time the record was closed, exported or last modified; those later stamps drag later versions into the join.
Illustrative: a returns policy article is published as version 1 on January 6. A refund is decided on March 5. Version 2, published on March 10, carries an effective date of March 1. Version 3 is an in-place edit on March 20 that adds guidance drawn from this very case. Joined on visible-from, the decision gets version 1; joined on effective-from, it gets version 2, flagged as not readable at the time; a plain export returns version 3, which is wrong under either rule. The figure lays out this timeline.
For a fuller model of business clocks, including when each person could have seen a fact, see the guide to temporal reasoning in context graphs; this page stays with the join.
Where earlier versions survive, and where they are trimmed away
Earlier versions survive only where a system was set to keep them, and many systems trim history on a schedule. Check retention before promising any as-of join, because a hole in history becomes a hole in the dataset.
At a glance
Where earlier versions survive, and where they are trimmed away
Five common sources, with the limit that matters for the join:
- Google Drive: for binary files such as PDFs, revisions other than the head that are not marked Keep Forever are purgeable, typically kept for 30 days but removable sooner once a file has 100 such revisions and a new one is uploaded; up to 200 revisions per file can be marked Keep Forever.
- SharePoint and OneDrive: when versions exceed a library's limits, the excess versions are marked for permanent deletion that bypasses the recycle bin, and the automatic setting thins out older versions judged least likely to be used.
- Salesforce: with Field Audit Trail turned off, field history is retained for up to 18 months, and up to 24 months through the API, and it exists only for the fields selected for tracking.
- Zendesk: ticket audits are a read-only history of all updates to a ticket, so fields and comments can be replayed up to the decision time.
- SQL Server temporal tables: an AS OF query reconstructs data as it was at a past moment by combining the current table with its history table. That moment is when the database held the row, which can differ from a document's effective date.
The as-of join rule and the edit-after test
For each document a decision relied on, select the latest version whose visible-from time falls at or before the decision time, and require exactly one match. Then compare that version with the current one and record whether any later edit touches the passage the decision depends on.
The edit-after test decides what a missing history costs. If no later edit touches the relevant passage, current and as-of text agree where it matters and the element can be marked reconstructed. If a later edit does touch it, the current text stays out of the packet, however convenient it would be.
Six steps, in order:
- Fix the decision time from the audit-trail event that records the decision, in UTC, and keep the name of the field it came from.
- List the documents the decision could have relied on, by rule: those linked from the record, cited in its notes or in scope for its case type, not every document in the system.
- For each document, select the latest version visible at or before the decision time. Zero matches is a gap; two matches is an ambiguity to resolve, never one to settle by taking the newer.
- Store the effective-from date beside it, and flag any version that took effect before the decision but became visible after it.
- Diff the selected version against the current one, limited to the passages the decision depends on, and record whether later edits touch them.
- Give each packet element a status from the completeness map, and write down the reason for every gap.
An annotated minimal task packet
A minimal packet holds only the evidence the decision could have used, each element tagged with how its version was established. Seven elements were considered for the example below: two are kept out, one because it postdates the decision and one because it informs nothing, and one more is withheld as a gap.
Naming the exclusions in the label pays off at grading time. A model that cites the March 20 playbook edit can then be scored as leaking rather than rewarded as clever, and a model that ignores the missing macro is not penalized for it.
The example is an Illustrative refund exception decided by a support agent on March 5, built on the timeline above:
| Element | As-of content (Illustrative) | Version status | Why it is in or out |
|---|---|---|---|
| Ticket thread to the decision | Customer messages and agent notes through the approval event, replayed from ticket audits | Captured at the time | The decision's own evidence; later replies and the closing survey are cut off |
| Returns policy article | Version 1, visible from January 6 | Reconstructed from revision history | The text the agent could read; version 2's effective date flagged, version 3 kept out |
| Customer plan terms | The contract version signed before the purchase | Captured at the time | Sets the refund window the agent had to apply |
| Account tier field | Tier as of March 5, recovered from field history | Reconstructed from history | The tier changed in April, so the exported value would mislead |
| Reply macro | Only the current macro text survives, and it was edited in March | Gap: current only | Withheld from the input and flagged, so the label does not credit the macro |
| Playbook edit of March 20 | Guidance added because of this case | Excluded: after the decision | Contains the answer; kept in an audit file, never in the input |
| Unrelated order history | Earlier orders with no bearing on this refund | Excluded: not relevant | Adds linkage risk and noise without informing the decision |
The context completeness map
Every element gets one of five statuses, and each status carries a rule for model input and for the label. The map makes gaps countable, so a buyer can see how much of a dataset rests on reconstruction and how much is simply missing.
Five statuses and what each one permits:
| Status | Definition | Allowed in model input? | What the label must say |
|---|---|---|---|
| Captured at the time | Stored as it stood when the decision was made, such as a signed contract or an audit-trail replay | Yes | Nothing extra |
| Reconstructed from history | Recovered from revision or field history, with the edit-after test recorded | Yes, with the history source noted | The history source and the test result |
| Gap: current only | Only today's version survives, and later edits touch the relevant passage or cannot be ruled out | No | That the element existed but is withheld, so graders do not penalize its absence |
| Gap: absent | The decision relied on it, but no version survives | No | That the decision used evidence the packet lacks |
| Excluded | Created after the decision, outside the approved scope or irrelevant to the decision | No | Nothing in the input; post-decision material may inform the label separately |
Would joining more records make the task clearer?
Usually not beyond the evidence the decision could have used. Every extra join adds rows the model must learn to ignore, raises the odds of a wrong match between systems and widens what could re-identify a person or a customer, without adding anything the decider knew.
Add an element only when it passes three tests: the decider could have seen it before deciding, it plausibly changed the decision, and its version can be established. An element that fails the first is leakage; one that fails the second is noise; one that fails the third goes on the map as a gap, not into the input.
A wrong match does more harm than a missing one because it is silent. When two systems disagree on a customer or document ID, a fuzzy join can attach another customer's contract and nothing in the packet reveals it; an exact-key join with a gap code is safer than a near match with no flag.
There is a fair objection: richer context sometimes shows why a decision was made. When that matters, the answer is an explanation field taken from the decider's own notes at the time, not more records joined from afterward.
Ship, reconstruct or withhold: the call for each element
Make the call element by element, then judge the dataset by its share of gaps among the elements that carry the decision.
At a glance
Ship, reconstruct or withhold: the call for each element
Four outcomes per element:
- Ship as captured: elements stored as they stood, with the decision time and its source field recorded.
- Ship as reconstructed: elements whose as-of version came from a revision store, or whose edit-after test shows no later change to the relevant passage.
- Withhold and flag: current-only elements whose relevant passage changed after the decision, or where the test cannot rule that out.
- Drop the task: decisions whose governing document has no surviving version, when the task asks which rule applied.
Requesting decision data with version history intact
Ask about revision history before asking about volume. A request that names the decision events, the documents they relied on and the retention settings of the systems holding them tells a supplier whether as-of joins are possible before a single record is exported.
SourceX sources each dataset against a written buyer request rather than from stock, and nothing leaves the company holding the records until it has approved the scope and terms and signed. The request is therefore the place to say that each packet must be built as of its decision, and to include the acceptance test. SourceX has not published results from datasets built this way; this page sets out the method.
Frequently asked questions
What is an as-of join for documents?
An as-of join attaches to each record the version of a document that was current at that record's timestamp, instead of requiring an exact key match. In a decision dataset, the timestamp is the decision time and the version is the latest one readable at or before it.
Why not use the latest version of each document?
The latest version carries every later edit, including edits made because of the case itself. A model trained or graded on it learns to decide with information the original decider never had, which inflates measured performance.
What if the revision history has already been purged?
Mark the element as a gap rather than filling it with current text, and record why. Then check retention settings in the systems that still hold history, since services such as Google Drive and SharePoint can trim older versions on a schedule.
Should the effective date or the publication date pick the version?
It depends on the question the task asks. Use the date a version became readable for tasks about what the decider knew, and the effective date for tasks about which rule governed; keep both fields and flag any version that took effect before it could be read.
Does reconstructing document versions add privacy risk?
Every join widens what could identify a person or customer, which is one reason to keep the packet minimal. Reconstruction does not change what the company approved for licensing, and de-identification and rights review still apply to every element.
Sources
- Anthropic's engineering guide to agent evals says each trial should be isolated by starting from a clean environment, because state shared between runs can cause correlated failures and inflate performance. Source
- Kapoor and Narayanan's survey found leakage-related errors reported across 17 fields that adopted machine-learning methods. Source
- Google Drive purgeable revisions are typically kept 30 days but can be purged earlier once a file has 100 non-Keep-Forever revisions; up to 200 per file can be kept forever. Source
- When SharePoint versions exceed library limits they are marked for permanent deletion that bypasses the recycle bin; automatic limits thin out older versions least likely to be used. Source
- With Field Audit Trail off, Salesforce retains field history data for up to 18 months, and up to 24 months via the API. Source
- Zendesk ticket audits are a read-only history of all updates to a ticket. Source
- SQL Server's AS OF subclause reconstructs data as it was at a past time by combining a temporal table with its history table. Source
Tell us what data you need
Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.