Systems and records
Event log policy context enrichment: attach the rule in force
By SourceX Editorial · Updated
10 sections
Short answer
To attach the rule that applied when a business event occurred, keep every rule as dated versions with a half-open validity interval, pick the event clock the rule actually depends on, normalize both sides to UTC, and run an as-of join by rule scope that must return exactly one version per event. Then validate coverage, overlaps, boundary events and backdated changes, and reconcile event counts to the source.
Record timeline
Event log policy context
Key takeaways
- Policy context enrichment is a point-in-time join: each event gets the one rule version whose validity interval contains the event's decisive timestamp.
- Choose the clock per rule: occurrence, recording or decision time can each be the right one, and the choice belongs in the specification.
- System history tables record when a change was saved, not when a rule took effect, so they are a weak substitute for business effective dates.
- A backward as-of join alone can attach a retired rule; the specification also checks the version's end date and records unmatched events instead of dropping them.
- Every enriched log should pass coverage, overlap, gap, boundary, time zone, backdating and reconciliation checks before anyone trains or evaluates on it.
How do you attach the rule that applied when an event occurred?
Store each rule as a series of versions, each valid over a half-open interval from its effective start to the next version's start, and join every event to the single version whose interval contains the event's decisive timestamp within the same rule scope. Record the version ID on the event, flag any event that matches zero or two versions, and reconcile the enriched log back to the source before it is used.
That is event log policy context enrichment in one paragraph. The rest of this page turns it into a specification a data engineering team can adopt as is: the contract for the rule version table, a five-step join procedure, ten validation checks with pass conditions, confidence labels for uncertain matches, and an illustrative worked example.
Why it matters: an event log records what people and systems did, but not the rules they were working under. A refund approved without a manager looks like a control failure if today's approval threshold is applied to last year's events, and looks normal once the threshold in force that week is attached. Models trained or evaluated on such logs learn the wrong lesson unless the rule context travels with each event.
Which clock decides the rule?
The rule in force is decided by the timestamp the rule itself refers to, which is not always when the row was written. Pick it per rule type and write it into the specification, because the same event can fall on different sides of a rule change depending on the clock.
Separate business effective time from system time. A dbt snapshot implements a type-2 slowly changing dimension with dbt_valid_from and dbt_valid_to columns, filled from an updated_at column or from the time the snapshot ran, and dbt leaves dbt_valid_to null for the current row by default. SQL Server system-versioned temporal tables set the period start from the transaction time in UTC. Both tell you when a database learned about a change. Neither tells you when the business decided the rule took effect, which may be earlier (a backdated policy) or later (a rule approved now, effective next month).
| Clock | Typical field | Use it when the rule refers to | Common trap |
|---|---|---|---|
| Occurrence time | occurred_at, service date, transaction date | When the business act happened, such as a sale or a shipment | Back-entered events carry a later recording time |
| Decision time | approved_at, decided_at | Who could approve and under which limit at the moment of approval | Several approvals on one case; use the decisive one |
| Request or submission time | submitted_at, created_at of a request | Deadlines and eligibility tied to when a request was made | Resubmissions reset the clock only if the rule says so |
| Recording time | recorded_at, inserted_at | System behavior, such as which validation code ran | Rarely the right clock for a business policy |
| Contract or period date | contract_date, billing period | Terms fixed at signing or per period | Periods that straddle a change need splitting |
Where do rule versions live, and what is usually missing?
Rule versions usually live in four places: configuration tables in the operating system (approval limits, discount ceilings, routing rules), controlled policy documents with revision histories, change tickets that approved each change, and audit or history tables. Most teams have the content of the rules but not clean effective dates.
The usual gaps are predictable. Configuration tables keep only the current value. Policy documents have a revision date but not an effective date. Change tickets record approval, not deployment. History tables carry system time only. Reconstructing the version table therefore means reading at least two of these sources and recording which one supplied each date.
If the log is exported as object-centric event data, the OCEL 2.0 specification can record changes to object attribute values over time and qualified relationships between objects and events. A rule can be modeled as an object whose attributes change on dated events, which keeps the policy history inside the exchange format rather than in a side file.
The rule version table contract
Every rule version is one row, and the table enforces that versions of the same rule in the same scope never overlap. Half-open intervals avoid double matches at the boundary: PostgreSQL's range types use lower-inclusive, upper-exclusive bounds as their standard form, and an exclusion constraint on a range column can reject overlapping versions at write time. Use the same convention in any warehouse, with an explicit open end for the current version.
At a glance
The rule version table contract
| Field | Content | Constraint | Why it is there |
|---|---|---|---|
| rule_id | Stable identifier of the rule, such as refund_approval_limit | Not null | Groups versions of one rule |
| scope_key | Where the rule applies: region, product line, channel, customer tier | Not null; same vocabulary as the event log | The join matches scope before time |
| version_id | Identifier of this version | Unique | Written onto every matched event |
| valid_from | Business effective start, UTC | Not null; inclusive | Lower bound of the interval |
| valid_to | Business effective end, UTC, or open for the current version | Exclusive; greater than valid_from | Upper bound; prevents attaching retired rules |
| clock | Which event timestamp this rule is evaluated against | One of the clocks in the specification | Makes the clock choice explicit per rule |
| rule_value | The parameters: limit, rate, threshold, routing target | Typed, not free text | What a model or analyst needs from the context |
| date_source and recorded_at | Which source supplied valid_from, and when the row was written | Not null | Separates effective time from system time and supports audits of backdating |
| supersedes | version_id this row replaces, if a correction | Optional | Keeps corrections traceable instead of overwriting history |
The five-step join procedure
Run the join in a fixed order so its results are reproducible. Most engines offer an as-of join for step four: pandas merge_asof matches on the nearest key rather than an equal key, requires both inputs sorted on that key, can match equal keys with by first, and in the default backward direction picks the last right-side row whose key is less than or equal to the left key. DuckDB's ASOF join returns at most one right-side match, and its ASOF LEFT JOIN keeps unmatched left rows with nulls instead of dropping them.
Checklist · 5 items
- 1. Normalize time. Convert every event timestamp and every valid_from and valid_to to UTC. RFC 3339 timestamps carry an explicit offset, and the offset is local time minus UTC, so subtracting it gives UTC; reject naive timestamps with no zone rather than guessing.
- 2. Choose the decisive timestamp. For each event, copy the field named in the rule's clock into a join_time column, and record which field it came from.
- 3. Map scope. Translate the event's attributes into the rule's scope_key vocabulary with a reviewed mapping table; unmapped events go to an exceptions file, not to a default scope.
- 4. Join as of join_time. Within each rule_id and scope_key, select the version with the latest valid_from at or before join_time, keeping events with no match. A version starting exactly at join_time applies, which matches the inclusive lower bound.
- 5. Confirm the end bound. Keep the match only if join_time is earlier than valid_to; otherwise mark the event no_rule_in_force. A plain backward as-of join skips this check and will attach a rule that had already been retired.
0 of 5 ticked. Tick each item as you go.
The validation specification
A join that runs is not a join that is right. Each check below has a pass condition and a fixed action on failure, so a reviewer can rerun the specification and get the same answer.
At a glance
The validation specification
| Check | Pass condition | On failure |
|---|---|---|
| Row conservation | Enriched event count equals source event count for the window | Stop; a join multiplied or dropped rows |
| Single match | Each event has exactly one version per applicable rule, or an explicit no_rule_in_force flag | Send double matches to exceptions; fix overlaps in the version table |
| No overlap | No two versions of a rule in one scope share any instant | Reject the version table load |
| Gap report | Every gap between consecutive versions is listed with a reason | Unexplained gaps block release of affected events |
| Time zone integrity | No naive timestamps; offsets present on all inputs | Reject rows; do not infer the zone |
| Boundary review | Events within a set window of any valid_from are listed | Owner confirms the side of the change for a sample |
| Backdating audit | Versions whose recorded_at is later than valid_from are flagged | Re-run affected events and keep both readings |
| Scope coverage | Share of events mapped to a scope is reported per source system | Unmapped events stay out of the training set |
| Outcome consistency | Events whose recorded outcome contradicts the attached rule are counted | Sample them; they reveal wrong dates or real exceptions |
| Source reconciliation | Totals by month and by activity match the source system reports, and a rerun on pinned inputs gives identical version IDs | Investigate before release; pin input snapshots and sort orders |
Could clock drift, identifier mismatch or an unrecorded transition change the attached rule?
Yes, and the risk concentrates near rule changes. An event a few minutes either side of a valid_from can switch versions if a server clock drifted, if one system stores local time, or if the event was entered after the fact. Identifier mismatches do the same with scope: a store renamed in one system but not the other maps to the wrong region's rule.
Unrecorded transitions are the hardest case. If a rule was suspended for a week without a configuration change, no version row exists for the suspension, and every event that week gets the normal rule. Only the outcome consistency check tends to reveal it, as a cluster of contradictions in one period.
Label confidence on each event instead of hiding the uncertainty. A simple scheme works: exact when the event is well clear of any change and the scope map is direct; boundary when it falls inside the review window; inferred when the scope or the date came from a secondary source; and unresolved when two readings remain. Buyers and analysts can then filter, and the label explains itself.
Worked example (illustrative): a refund limit that changed twice
An illustrative retailer requires manager approval for refunds above a limit. The limit was raised on 1 March, then lowered again on 1 June, but the June change was decided and saved on 10 June with an effective date of 1 June. All values here are illustrative, and the limit is recorded only as a version ID, not as an amount.
| Event | Decisive time (UTC) | Attached version | Label and note |
|---|---|---|---|
| Refund approved by clerk | 28 February, 23:58 | V1 (before raise) | Boundary: two minutes before the change; clerk approval breaches V1, sample for review |
| Refund approved by clerk | 1 March, 00:03 | V2 (raised limit) | Boundary: inside window; consistent with V2 |
| Refund approved by clerk | 4 June, 14:10 | V3 (lowered, backdated) | Inferred: V3 recorded 10 June; outcome contradicts V3 because staff were still working to V2 |
| Refund approved by manager | 12 June, 09:30 | V3 | Exact: consistent |
| Refund in a store missing from the scope map | 20 June, 16:45 | None | Unresolved: held in exceptions until the store is mapped |
Which events should ship without a rule attached?
Ship an event without policy context when the decisive timestamp is missing, the scope cannot be mapped, or two credible effective dates remain after the owner's review. A blank, labeled field is honest; a guessed version silently teaches a model a rule that never applied.
The third event in the example shows the exception that matters most. The backdated V3 is legally or contractually correct from 1 June, but the people doing the work followed V2 until 10 June. Keep both readings for that window, the effective-date reading and the as-practiced reading, and let the use case choose. Narrow the scope, or ask the workflow owner, when a rule's history depends on undocumented decisions, when a rule is a regulatory requirement whose interpretation is disputed, or when more than a small share of events land in the boundary or unresolved labels.
Policy context in a log you plan to license
Policy context makes an event log far more useful for training and evaluating agents, but the rule history is often more confidential than the events themselves, since it can reveal approval limits, pricing logic or internal controls. Treat the rule version table as a separate item in any rights review: having access to it, privacy permission for the people in the log, ownership of the records and contractual rights to license them are four different questions, and de-identifying the events does not answer the rights question for internal policy.
SourceX can carry an enriched log from sourcing and rights review through contracting, preparation, delivery and payment when a company licenses it for AI. The company decides in the signed scope whether rule context is included, approving scope and terms before any data is delivered, and eligibility is checked with a short assessment requiring no upload.
Frequently asked questions
What is event log policy context enrichment?
It is adding to each event in a business event log the version of the rule or policy that was in force when the event happened. The usual method is a point-in-time join between the log and a table of dated rule versions, followed by checks that every event matched exactly one version.
What should the policy context join and validation specification include?
A rule version table with scope, half-open validity intervals, a declared clock and the source of each date; a fixed join procedure covering time zones, scope mapping and the end bound; and validation checks for row conservation, single matches, overlaps, gaps, boundaries, backdating, outcome consistency and reconciliation to source totals.
Can a database history table replace a policy version table?
Usually not. System-versioned tables and snapshots record when a row changed in the database, which is system time. A rule's business effective date can be earlier or later than that, so the history table is a useful source of evidence but not the effective dates themselves.
Why use half-open intervals for rule versions?
With an inclusive start and exclusive end, an event exactly at a change time belongs to the new version only, so boundary events never match two versions. It also lets consecutive versions share a timestamp without overlapping.
How do you handle a rule that was backdated?
Record both when the version was saved and when it took effect, flag events that fall between the two, and keep both readings for them. Whether the effective-date or the as-practiced reading is right depends on what the dataset is used for.
Sources
- pandas merge_asof matches on the nearest key, requires sorted inputs, matches by-columns first and, searching backward, picks the last key less than or equal to the left key. Source
- DuckDB ASOF joins return at most one match, and ASOF LEFT JOIN keeps unmatched rows with nulls. Source
- PostgreSQL ranges default to an inclusive lower and exclusive upper bound, and exclusion constraints can forbid overlapping ranges. Source
- dbt snapshots implement type-2 slowly changing dimensions with dbt_valid_from and dbt_valid_to, null for current rows by default. Source
- SQL Server temporal tables set period start from transaction time in UTC and filter AS OF with start at or before and end after the given time. Source
- RFC 3339 timestamps carry an offset from UTC, calculated as local time minus UTC. Source
- The OCEL 2.0 specification records changes to object attribute values and qualified relationships between objects and events. Source
See if your company qualifies
A short company assessment. No data uploads are needed.