Systems and records
Dataset time encoding for business tasks: event, observation and export time kept apart
By SourceX Editorial · Updated
10 sections
Short answer
Represent the three clocks as separate fields: event time for when the business occurrence happened, observation time for when the source system recorded each version of it, and export time for when the extract read the data. Give each a declared source field, precision and format, store export time once per run in a manifest, and run consistency checks such as observation never later than export.
System export
Dataset time encoding
Key takeaways
- Event time, observation time and export time answer different questions, and merging them leaks future knowledge into features and distorts durations.
- Write instants in RFC 3339 with an offset or Z, keep date-only fields as dates, and name the local zone separately when clock time matters.
- System stamps such as a platform-maintained modified time are good observation clocks for the latest version only; versions need a history source.
- Record export time and the run identifier in a manifest and on every row, with the scheduled window for incremental extracts.
- Nine checks, from observation-before-export to version order, can be run by a buyer without the supplier's systems.
- When a source cannot supply a clock, mark it unknown rather than copying another clock into it.
What are event time, observation time and export time?
Event time is when the business occurrence happened, observation time is when the source system recorded that version of the fact, and export time is when the extract read the data. Keep each in its own field, each with a declared meaning, source field and precision, and never let one fill in for another.
The three clocks are not a new idea. Stream processing already separates them: the Apache Flink documentation defines event time as the time each event occurred on its producing device and processing time as the system time of the machine running the operation. The CloudEvents specification defines its time attribute as the timestamp of when the occurrence happened, and lets a producer fall back to another time only if all producers for the same source do so consistently.
Business records add a twist that streams rarely have. A technician finishes a repair on Tuesday afternoon but closes the work order on Wednesday morning. A sales rep moves an opportunity's close date backwards after the quarter ends. A finance clerk posts an invoice dated last month. Each of those records carries an event time that the system learned later, and sometimes learned twice. The export that ships to a buyer then adds a third moment: the instant the extract saw the data, which decides what the dataset could possibly know.
Which business tasks go wrong when one timestamp stands in for all three?
Any task that asks what was known at a moment, how long something took, or what happened in what order breaks when the clocks merge. The damage is quiet, because the merged column still looks like a valid timestamp.
The table below lists the failures that show up most often when a delivered dataset has a single updated_at or a single export date doing the work of all three.
| Business task | Clock it needs | What breaks if clocks are merged | Symptom in the data |
|---|---|---|---|
| Forecasting or next-step prediction | Observation time for features, event time for the label | Features include facts the business only learned after the prediction point | Validation scores far above production performance |
| Cycle time and SLA measurement | Event time at both ends | Entry lag is counted as work time | Durations cluster around office opening hours |
| Agent trajectory replay | Observation time to order actions as the user saw them | Steps replay in occurrence order, not in the order the screen showed them | An agent appears to act on information it had not seen yet |
| Data freshness and coverage claims | Export time | A snapshot taken weeks later is described as current | Records modified after the stated cut-off appear in the file |
| Correction and restatement analysis | Observation time per version | Only the latest version survives, so restatements vanish | Every record has exactly one version |
| Point-in-time joins across systems | Observation time in each system plus each export time | Two tables exported days apart are joined as if simultaneous | Child rows reference parents that did not exist yet |
The three-clock timestamp schema, field by field
The schema below is the asset this page exists to provide. It adds ten fields to every record, grouped by clock, and each field states what a validator must check. Field names are suggestions; the meanings and checks are the part to keep.
Event fields describe the occurrence. Observation fields describe one version of what the system knew. Export fields describe the run that produced the file and are identical for every row from that run.
| Field | Clock | Meaning | Format | Check |
|---|---|---|---|---|
| event_time | Event | When the business occurrence happened, per the source field named in event_time_basis | RFC 3339 instant with Z or a numeric offset | Present unless event_time_status is unknown; never equal to export_time by default |
| event_time_basis | Event | Source field or rule the value came from, such as a completion field or a user-entered date | Text naming a data dictionary entry | Every distinct value appears in the data dictionary |
| event_time_precision | Event | Finest unit the source actually captured | One of day, minute, second, millisecond | Day-precision values carry no time of day and are not shifted across dates |
| event_zone | Event | IANA time zone where the occurrence happened, when local clock time matters | IANA zone name, optionally as an RFC 9557 suffix | Offset in event_time agrees with the zone at that instant |
| observed_at | Observation | When the source system recorded this version of the fact | RFC 3339 instant | Never later than export_time |
| observed_at_basis | Observation | System field used, such as a created stamp, a user-modified stamp or a history row | Text naming a data dictionary entry | Same basis for every row of one table, or a documented reason |
| record_version | Observation | Ordinal of this version within record_id | Integer starting at 1 | Strictly increasing with observed_at inside each record_id |
| export_time | Export | Instant the extract read the source, the as-of moment of the file | RFC 3339 instant | Constant within export_run_id and equal to the manifest value |
| export_window_start and export_window_end | Export | Scheduled interval an incremental extract was meant to cover, start included, end excluded | Two RFC 3339 instants | Every observed_at in an incremental file falls inside the window |
| export_run_id | Export | Identifier of the extract run, used as the join key to the manifest | UUID or other unique text | Unique per run and listed in the package manifest |
How each clock should be written on disk
Write every instant as an RFC 3339 timestamp with an explicit offset or Z, keep date-only fields as dates, and declare in the file format whether a stored value is an instant or a local wall-clock reading.
RFC 3339 originally used an offset of -00:00 to say that the UTC time is known but the local offset is not. RFC 9557, which updates RFC 3339, revises that convention so that Z is used for the unknown-offset case, and adds an optional bracketed suffix that names a time zone, as in 2022-07-08T00:14:07+02:00[Europe/Paris] for a summer instant. RFC 9557 also treats an offset that disagrees with the named zone as an inconsistency, which is exactly the check the event_zone field needs.
Columnar files need the same declaration at the type level. The Apache Parquet specification gives the TIMESTAMP type an isAdjustedToUTC parameter. When it is true the value identifies a single instant and the original offset is lost on write; when it is false the value is a local reading that does not identify an instant without extra information. A package should state which of the three clocks are stored as instants (normally all three) and keep the local zone in its own field when local clock time matters, for example for shift work or business hours.
Date-only values deserve special care. A due date or a close date entered as a calendar day is a day, not midnight UTC. Casting it to a timestamp and converting to UTC moves it to the previous day for every zone west of Greenwich, which is why event_time_precision exists.
Where observation time comes from in real source systems
Observation time comes from system-maintained stamps and from history or audit tables, and the choice of stamp changes what the clock means. Name the stamp in observed_at_basis rather than leaving the buyer to guess.
Salesforce is a useful example because it exposes several stamps on most objects. Its GraphQL documentation describes LastModifiedDate as the date and time when a user last modified the record, and SystemModstamp as the date and time when a user or an automated process, such as a trigger, last modified it. The API documentation for update calls says the platform sets fields such as LastModifiedDate and SystemModstamp itself; a caller cannot supply them. That makes them trustworthy as observation clocks, but they describe the latest change only, not every version.
A last-modified stamp therefore gives one observation per record, taken at export. To deliver several versions of a fact, the extract has to read a history source: a field history table, an audit log, a change data capture stream or a series of snapshots. Each history row becomes one record_version with its own observed_at. If the history source has a retention limit, the earliest versions are gone, and the package should say from which date version coverage is complete.
User-entered dates are event-time candidates, not observation clocks. A close date, a service date or an invoice date is what someone said happened. Keep it in event_time with its basis, and keep the system stamp beside it; the gap between the two is often the most informative feature in the record.
Recording the export clock so the as-of view can be rebuilt
Store export_time once per run in the manifest and copy it onto every row, together with the run identifier and, for incremental files, the window the run was meant to cover. A buyer can then answer what the file could have known without access to the source.
Table formats already model this. The Apache Iceberg specification gives every snapshot a unique snapshot-id and a timestamp-ms recording when the snapshot was created, and from format version 4 requires that timestamp to be strictly greater than its parent's on the same branch. Delivering the snapshot identifier with the export time lets a buyer pin a reproducible as-of view rather than a moving table.
Lineage standards separate the scheduled time from the actual run time. The OpenLineage specification attaches an eventTime to each run event and defines a nominalTime run facet that captures the time a run is scheduled for, noting that it will differ from the time the job actually runs. In the schema above, export_window_start and export_window_end play the nominal role and export_time records the actual read.
Two practical rules keep this clock honest. First, export_time is the moment the extract read the data, not the moment the file was compressed, uploaded or delivered; record those separately in the manifest if they matter. Second, when an extract takes hours and reads tables at different moments, record an export_time per table, because one value for the whole package would overstate how simultaneous the tables are.
Consistency checks a buyer can run without the supplier's systems
A buyer should be able to verify the three clocks from the package alone. Each check below needs only the delivered rows and the manifest, and each produces a count of failing rows that can go into an acceptance report.
At a glance
Consistency checks a buyer can run without the supplier's systems
- C1 observation before export: observed_at is never later than export_time. A failure usually means a local wall-clock time was written as UTC, or rows from a later run were mixed in.
- C2 one export clock per run: every row with the same export_run_id carries the same export_time, and that value matches the manifest.
- C3 window coverage: in an incremental file, every observed_at falls in the half-open interval from export_window_start to export_window_end, and consecutive windows neither overlap nor leave gaps.
- C4 version order: within each record_id, record_version starts at 1, has no gaps or duplicates, and increases with observed_at.
- C5 future events are declared: rows where event_time is later than observed_at are allowed only for event_time_basis values the dictionary marks as planned, such as a due date or a scheduled visit.
- C6 precision is respected: values with day precision carry no time of day, and a sample shows no date moved by a time zone conversion.
- C7 zone and offset agree: when event_zone is present, the offset in event_time matches that zone's rules at that instant.
- C8 no silent fill: report the share of rows where event_time equals observed_at or export_time to the second. The acceptable share depends on the source; a high share deserves a question to the supplier, because it often means a missing field was back-filled from a system stamp.
- C9 types match meaning: in Parquet deliveries, columns declared as instants use isAdjustedToUTC true, and local wall-clock columns, if any, are named as such in the data dictionary.
Illustrative worked record: a repair finished late in the day, entered next morning and corrected
The example below is constructed to show the schema and checks; it is not a real customer record. A field technician finishes a repair at 4:40 pm in Chicago, closes the work order in the mobile app the next morning, and a supervisor corrects the completion time two days later. A monthly incremental extract runs on April 1.
The failing row shows two common defects. Its observed_at is a UTC value written without an offset, which a consumer then reads as Chicago time, pushing it five hours later and outside the export window. Its event time has been silently filled from the export clock.
| Field | Version 1 (valid) | Version 2 (valid) | Failing row | Check result for failing row |
|---|---|---|---|---|
| event_time | 2026-03-03T16:40:00-06:00 | 2026-03-03T16:10:00-06:00 | 2026-04-01T02:00:00Z | C8 flags: equals export_time |
| event_time_basis | Work order completed-at field | Work order completed-at field | Work order completed-at field | Passes, but value contradicts the basis |
| event_zone | America/Chicago | America/Chicago | America/Chicago | C7 not applicable to a filled value |
| observed_at | 2026-03-04T14:12:00Z | 2026-03-06T15:30:00Z | 2026-03-31T22:50:00 with no offset, read as Chicago time | Becomes 2026-04-01T03:50:00Z; C1 still passes |
| record_version | 1 | 2 | 1 | C4 fails: duplicate version 1 for the record |
| export_time | 2026-04-01T05:00:00Z | 2026-04-01T05:00:00Z | 2026-04-01T05:00:00Z | C2 passes |
| export window | 2026-03-01T00:00:00Z to 2026-04-01T00:00:00Z | Same window | Same window | C3 fails: 03:50Z on April 1 is after the window end |
| export_run_id | run-2026-04 (illustrative) | run-2026-04 (illustrative) | run-2026-04 (illustrative) | Passes |
When a dataset can honestly carry fewer than three clocks
Some sources legitimately collapse two clocks into one, and the right response is to declare it rather than to invent the missing clock. Three cases come up often.
Append-only event logs, such as telephony call records or machine logs written at the moment of the event, record the occurrence and the observation together. Set observed_at equal to event_time, set observed_at_basis to say so, and expect C8 to report a high share, which is then explained rather than suspicious.
Snapshot-only exports from systems without field history carry one version per record, observed at an unknown moment before the export. Leave observed_at empty with a status of unknown, keep the system's last-modified stamp if there is one, and tell buyers that restatement analysis is not possible from this file.
Systems that never captured when something happened, only when someone typed it, have no trustworthy event time. Mark event_time_status as unknown instead of copying observed_at into it. A buyer can decide to use observation time as a proxy; a supplier should not make that decision silently. The schema is also not a substitute for a data dictionary or a lineage record: it carries the clocks, and the dictionary explains the source fields behind them.
Requesting business records with all three clocks
When a dataset is sourced from operating businesses rather than pulled from a public set, the clock fields can be specified before extraction instead of repaired afterwards. Nothing is held in stock at SourceX: records are gathered from U.S. businesses after a buyer asks, and the buyer's request form spells out scope, volume, language, format, timing and licensing terms.
A request that names the three clocks, the checks in this guide and the history depth needed for versions gives the extract a definition of done. Because SourceX handles sourcing, rights review, contracting, preparation, delivery and payment between the business and the AI developer, those requirements travel with the deal from the first scoping conversation.
Frequently asked questions
What is the difference between event time and observation time in a business dataset?
Event time is when the occurrence happened in the business, such as a repair being finished. Observation time is when the source system recorded that version of the fact, such as the moment the work order was closed or edited. The two differ whenever entry is late, backdated or corrected.
Is export time the same as the delivery date?
No. Export time is the instant the extract read the source, which fixes what the file could know. Compression, upload and delivery happen later and can be recorded separately in the manifest, but they say nothing about the content.
Should all timestamps in a delivered dataset be in UTC?
Instants should be stored unambiguously, which usually means UTC or an explicit offset. When local clock time matters, for example for business hours, keep the IANA time zone in its own field, because normalizing to UTC loses the original offset.
How do I get observation times for older versions of a record?
Read a history source rather than the current record: a field history table, an audit log, change data capture or a series of snapshots. Each history row becomes one version with its own observation time, and any retention limit on that history should be stated in the package.
What if the source system only has a last-modified date?
Use it as the observation time of the single version you have, name the field in the basis column, and state that earlier versions are not available. Do not copy it into event time unless a buyer has agreed to that proxy.
Can a buyer check the timestamps without access to the source system?
Yes, if the package carries the three clocks, the run identifier and a manifest. Checks such as observation before export, one export time per run, window coverage and version order need only the delivered rows.
Sources
- Apache Flink defines event time as the time each event occurred on its producing device and processing time as the system time of the machine executing the operation. Source
- The CloudEvents time attribute is the timestamp of when the occurrence happened; producers may use another time only if all producers for the same source do so consistently, and the value must follow RFC 3339. Source
- RFC 3339 defines an offset of -00:00 for a time whose UTC value is known but whose local offset is unknown. Source
- RFC 9557 updates RFC 3339 so that Z is used for an unknown local offset, adds a bracketed time zone suffix, and treats an offset that disagrees with the named zone as an inconsistency. Source
- The Parquet TIMESTAMP type has an isAdjustedToUTC parameter: true values identify a single instant and lose the original offset, false values are local readings that do not identify an instant on their own. Source
- Salesforce describes LastModifiedDate as when a user last modified a record and SystemModstamp as when a user or an automated process such as a trigger last modified it. Source
- The Salesforce API updates fields such as LastModifiedDate and SystemModstamp automatically, and a caller cannot specify those values. Source
- Every Apache Iceberg snapshot has a unique snapshot-id and a timestamp-ms for when it was created; from format version 4 that timestamp must be strictly greater than its parent's on the same branch. Source
- OpenLineage run events carry an eventTime, and the nominalTime run facet captures the time a run is scheduled for, which differs from the time it actually runs. Source
Tell us what data you need
Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.