Skip to content

Privacy and preparation

Expired policy training dataset: tag validity, then test what changed

By SourceX Editorial · Updated

10 sections

Short answer

An expired policy training dataset holds accurate history with stale lessons: each record was right under the rule in force when it happened, but a model will learn that rule as current. Tag every record with its governing policy version and that version's validity dates, label historical correctness and present applicability separately, then decide per record whether to train as current, train with its date, keep for evaluation or drop.

Redaction

Expired policy training

This page covers expired policy training dataset: tag validity, then test what changed.Illustrative

Key takeaways

  • Historical correctness and present applicability are two labels; a dataset that stores only one hides the other.
  • The IRS raised the 2026 business mileage rate from 72.5 to 76 cents per mile for expenses on or after July 1, so two correct reimbursements weeks apart teach different rules.
  • Applicability is decided by the anchor date the policy names, such as when an expense was incurred, not by when a record was processed or stored.
  • Database system time records when a row changed, not when a rule took effect, so validity dates must be captured as business fields.
  • No one has measured how common expired-policy records are in licensed business data; this page publishes the plan, and results are not yet published.

What goes wrong when a dataset outlives its policies?

The records stay true while the rules behind them expire, and a model trained on them learns the old rule as though it still applies. Every record can pass a quality review, with complete fields and labels that were right on the day, and the dataset can still teach the wrong behavior for today.

A public rule shows how quickly this happens. The IRS set the 2026 business standard mileage rate at 72.5 cents per mile, up 2.5 cents from 2025. Announcement 2026-11 then raised it to 76 cents for expenses paid or incurred on or after July 1, 2026, citing recent fuel price increases, while the earlier notice's rates still govern expenses before that date. An expense system's June approvals at 72.5 cents are correct, and so are its August approvals at 76 cents. A model trained mostly on first-half records will reimburse a September trip at the old rate, and nothing in the data flags it.

The same pattern runs through refund windows, discount approval limits, credit policies, warranty terms and escalation rules. The answer is not to throw history away. It is to make time explicit, so each record states which rule made it right and whether that rule still holds.

Two labels, not one: correct then, applicable now

Give every policy-governed record two separate labels. Historical correctness asks whether the decision was right under the rule in force on its anchor date; present applicability asks whether that rule would govern the same case if it happened on the deployment date. Together they place each record in one of four quadrants, each with a default use.

The fourth quadrant is the one teams miss. A decision that broke the old rule but happens to match the new one looks correct to anyone reading today's policy, yet it records a mistake, not a lesson. Counting it as a good example teaches the model that ignoring the rule in force is acceptable.

The examples in the table use an employee mileage reimbursement process on either side of the July 1, 2026 rate change.

Two labels, not one: correct then, applicable now
QuadrantWhat it meansIllustrative recordDefault use
Correct then, applicable nowRight under its rule, and that rule still governsAn August 2026 trip reimbursed at 76 centsTrain as current behavior
Correct then, expired nowRight under its rule, but the rule has been replacedA June 2026 trip reimbursed at 72.5 centsTrain only with date and rule in context, or keep for evaluation
Wrong then, still wrongBroke the rule in force and breaks the current oneA trip reimbursed at a rate neither period usedFix the label or drop the record
Wrong then, matches nowBroke its own rule but agrees with the newer oneA June 2026 trip reimbursed at 76 centsNever count as a correct example
A 2026 timeline with two policy versions, 72.5 cents per mile until July 1 and 76 cents per mile after it, and a row of records below: those before July 1 turn amber as correct then but expired now, those after stay green as applicable now, with a today marker.
Two correct records, two different rules. Rates from the IRS (2026); record dots are illustrative.

Which date decides whether a policy applies?

The anchor date the policy itself names decides it. For the 2026 mileage change, the IRS ties the rate to when expenses were paid or incurred, so a June trip submitted in August still takes the June rate. Processing, approval and storage dates are evidence about a record, not the test of whether its rule applies.

System timestamps are the usual trap. Microsoft's documentation for SQL Server temporal tables says the database sets a row's ValidFrom value to the begin time of the transaction, in UTC, from the system clock. That records when a row changed in the database, which can fall days after a policy took effect, or before it. A pipeline that reads system time as business time will file records under the wrong policy version.

The same documentation gives a clean rule for intervals: a row counts as of a given time when its start is at or before that time and its end is after it. Store each policy version with a start date and an exclusive end date, resolve every record against its anchor date with that test, and the boundary day stops being ambiguous.

Question answering research uses the same idea. SituatedQA pairs questions with timestamps because their answers depend on temporal context, and samples dates between an answer's start and end dates. A policy-governed business record is a question-plus-timestamp pair in exactly that sense.

The time-aware data card: fields to add before training

Add policy and time fields to the dataset's documentation and to each record, so anyone can tell which rule governed a record and whether it still holds. Existing documentation frameworks ask part of this; none of it fills itself in.

Datasheets for Datasets asks over what timeframe data was collected, whether that matches when the data was created, and whether and how the dataset will be updated, and its authors note that the questions can strain for dynamic datasets. The Data Provenance Initiative (arXiv 2310.16787), auditing more than 1,800 text datasets, found that hosting sites left out license information for more than 70 percent of popular datasets. If licenses go unrecorded that often, policy and validity fields will be absent unless a buyer asks for them.

The fields below extend a datasheet for policy-governed records. Values are illustrative and continue the mileage example.

The time-aware data card: fields to add before training
FieldWhat it recordsIllustrative valueCheck before training
Policy familyWhich rule set governs the decisionEmployee vehicle reimbursementEvery policy-governed record maps to one family
Policy versionVersion identifier and its source documentMileage rate effective July 1, 2026The version exists in the version table
Valid from and valid toStart date and exclusive end date of the version2026-07-01 to openNo gaps or overlaps between versions
Anchor fieldWhich record date the policy uses to decide applicabilityDate the expense was incurredAnchor present and plausible on each record
Label dateWhen the correctness label was set, and against which versionLabeled 2026-09-15 against the July versionNo new rule back-applied to older records
Historical correctnessRight or wrong under the version in force on the anchor dateCorrectA second reviewer agrees on a sample
Present applicabilityWhether that version governs as of a stated dateApplicable as of 2026-10-26Recomputed whenever a version changes
Change logEach policy change and the records it touchedJuly 1, 2026 rate change; first-half records marked expiredCount of affected records stored with the entry

The change-impact checklist for every policy change

Run the same short checklist each time a governing policy changes, before the next training or evaluation run. It turns a policy memo into a list of affected records and a decision for each one.

The checklist assumes the data card fields above are in place:

Checklist · 9 items

  1. Identify the changed clause and its exact effective date from the policy source, not from a summary of it.
  2. Confirm the anchor date the new version uses; it can differ from the old version's anchor.
  3. Close the old version's interval at the effective date and open the new one, leaving no gap or overlap.
  4. Select records whose anchor dates fall inside the closed interval and mark them expired for present applicability.
  5. Check labels set after the change for back-application of the new rule to older records.
  6. Flag records that broke the old rule but match the new one, and remove them from the correct examples.
  7. Recompute the share of training records under each version and compare it with the version mix expected at deployment.
  8. Decide per quadrant: train as current, train with date context, move to evaluation or history, or drop.
  9. Move temporal test cases to after the change, and record the change and its affected-record count in the card's log.

0 of 9 ticked. Tick each item as you go.

How to measure version coverage against the task you will deploy

Count training records by governing version and compare that mix with the version the deployed system will face, which is usually the current one. The headline measure is the expired-version share: records whose governing version has ended, divided by records whose anchor date could be resolved.

Report records with unresolved anchors as a separate share and never fold them into either side, because a dataset with a large unresolved share has not shown that it is current. Report the measure per task family as well, since a refund policy can change while an escalation policy stays put.

Validate both labels the way evaluation tasks are validated. Anthropic's engineering guidance on agent evaluations treats agreement as the mark of a good task: two domain experts judging it independently would hand down the same pass or fail result. For time-labeled records, that agreement is only possible when the anchor date and governing version travel with the record, so measure reviewer agreement on historical correctness and present applicability separately.

Original evidence plan: a small, permissioned record set

The next evidence needed is a measured count of the four quadrants in real business datasets, and SourceX has not collected it. This page publishes the plan; results are not yet published.

The design: the sampling frame is policy-governed decision logs from a small number of companies that approve the use, across at least three policy families such as expense reimbursement, refunds and credit limits, each with a documented policy change inside the record window. The unit is one decision record. The variables are anchor date, governing version, historical correctness, present applicability and label date. The metrics are the expired-version share, quadrant counts, the unresolved-anchor share and reviewer agreement. Records whose policy text cannot be recovered are excluded and counted, not dropped silently.

Shares would be reported with intervals that allow for clustering within each company, since records from one company share policies and reviewers. A few companies cannot stand for an industry, and the report would say so in its first paragraph.

One example would challenge the headline: late and retroactive work. Expense reports for June trips filed in August, claims judged under the policy in force on the date of loss, and audits of past decisions all need the old rule, so for those tasks the expired records are exactly right and deleting them would make a model worse. The plan therefore reports quadrant counts by task rather than one verdict for the whole dataset.

Could newer records make the dataset worse?

They can, which is why the remedy is a test rather than a reflex. Newer records may carry immature outcomes and unsettled labels, may reflect a changed task rather than a changed rule, and can make scores from before and after a change impossible to compare.

Research on temporal misalignment points in both directions. Kelvin Luu and colleagues (NAACL 2022) found that models do worse when their training data comes from an earlier period than their test data, and that the size of the effect varies considerably across domains and tasks. That favors weighting recent records, but the variation means the benefit has to be measured for the task at hand.

A preset comparison settles it. Train three arms on the same budget: current-version records only, all records with date and version in context, and all records with no time fields. Score each on held-out cases from after the change, and choose by a rule written before the run.

Requesting records that carry their policy history

Ask suppliers for policy versions and their dates along with the records, because a dataset without them cannot be relabeled the next time the rules move.

With SourceX, the buyer's request starts the search: matching records are located at U.S. businesses once a request arrives rather than drawn from stock, and the form gathers the buyer's licensing needs and timing along with scope, volume, language and format. A requirement for this use should name the policy families, the anchor field for each, the version table with effective dates and an acceptance test covering both labels. Records move only after the supplying company has approved the scope and terms and executed the agreement; a request implies no current stock and no particular model improvement.

Methods note: sources, date and limits

Checked on 2026-10-26: the IRS release on the 2026 business mileage rate and Internal Revenue Bulletin 2026-29 carrying Announcement 2026-11; Microsoft's SQL Server temporal tables documentation, read in its source repository; the SituatedQA repository; Datasheets for Datasets; the Data Provenance Initiative audit; Luu et al. (NAACL 2022); and Anthropic's evaluation guidance. The quadrant labels, data card fields, checklist and measurement plan are this article's own framework.

Not measured: how common expired-policy records are in any dataset, how the three training arms affect any model, and policies that were never written down. The mileage example is a public rule chosen because its dates are documented; it is not drawn from any company's records.

Frequently asked questions

Should I delete training records governed by an expired policy?

Usually not. Records that were right under their own rule are accurate history, and late, retroactive or audit tasks need exactly that. Keep them with the governing version and date attached, use them as current behavior only if the rule still applies, and drop only records that were wrong under their own rule.

What is the difference between system time and business validity time?

System time is when a database recorded a change; SQL Server temporal tables, for example, set a row's validity start from the transaction's begin time on the system clock. Business validity time is when a rule or fact was actually in force. A training dataset needs the second as an explicit field, because the two can differ by days or more.

How do I find the anchor date for a policy?

Read the policy itself: it usually names the event that decides applicability, such as the date an expense was incurred, an order was placed or a loss occurred. The IRS's mid-2026 mileage announcement, for instance, applies the new rate to expenses paid or incurred on or after July 1, 2026. If the text is silent, record the assumption and have the policy owner confirm it.

Do dataset documentation frameworks include policy validity fields?

Not as a set field. Datasheets for Datasets asks when data was collected, whether that matches when it was created and how the dataset will be updated, which is a good starting point. Policy family, version, validity dates and anchor field are additions a buyer has to specify.

Has SourceX measured how many licensed records follow expired policies?

No. This page publishes the measurement plan and the data card fields; no records have been counted and results are not yet published. If the study runs, it will report quadrant counts by task with intervals and list its exclusions.

Sources

  • The IRS set the 2026 business standard mileage rate at 72.5 cents per mile, up 2.5 cents from 2025. Source
  • Announcement 2026-11 (Internal Revenue Bulletin 2026-29) set a 76 cents per mile business rate for expenses paid or incurred on or after July 1, 2026, citing fuel price increases; Notice 2026-10 rates govern earlier expenses. Source
  • SQL Server temporal tables set ValidFrom to the transaction's begin time from the system clock, and AS OF returns rows whose start is at or before the time and whose end is after it. Source
  • SituatedQA pairs questions with timestamps because answers depend on temporal context, sampling dates between an answer's start and end dates. Source
  • Datasheets for Datasets asks about collection timeframe versus creation timeframe and dataset updates, and notes limits for dynamic datasets. Source
  • The Data Provenance Initiative audit of 1,800+ text datasets found license omission above 70 percent on popular hosting sites. Source
  • Temporal misalignment between training and test periods degrades performance, and the effect varies considerably across domains and tasks. Source
  • Anthropic's evaluation guidance: a good task is one where two domain experts would independently reach the same pass/fail verdict. Source

Tell us what data you need

Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.

See if your company qualifies