Systems and records
Dataset, event and task interoperability: keep logs and tasks linked
By SourceX Editorial · Updated
10 sections
Short answer
Keep event logs and derived task datasets linked by giving every source event a deterministic ID, every task its own ID, and every task a derivation record listing the exact source event IDs, the rule and its version, and the version and hash of the log it read. Package all three with declared keys, relative paths and file hashes so a buyer can resolve every reference and rebuild tasks without the supplier's systems.
Record timeline
Dataset
Key takeaways
- Event IDs must be deterministic, so a re-export of the same records produces the same IDs.
- Every task needs a derivation record naming its source events, rule version and the hash of the log it read.
- Separate the input window from the label events, and validate that no input event falls after the window end.
- Package logs, tasks and derivations together with declared keys, relative paths and file hashes.
- A re-export is a new log version; rebuild or re-validate tasks rather than patching IDs by hand.
How do you keep an event log and a derived task dataset linked?
Give every source event a stable ID that never changes after export, give every derived task its own ID, and connect the two with a derivation record that lists the exact source event IDs, the rule and rule version that produced the task, and the version and hash of the log it was read from. Ship the log, the tasks and the derivation records in one package whose manifest declares the keys, so any reference can be checked.
That is the whole idea; the work is in the details. Dataset, event and task interoperability fails quietly: the task file still loads, the log still loads, and nobody notices that a third of the tasks now point at events that were renumbered in a later export. The schema below is built so that failure is caught by a script, not by a model that trains on mismatched context.
A task here means any derived unit a buyer trains or evaluates on: an agent episode built from a case, a next-step prediction example, a classification label attached to a window of events, or an evaluation item with a known outcome. An event log means the time-stamped activity record it came from, whether a flat case log or an object-centric one.
Where do the links usually break between export and delivery?
Links break at the moments when a supplier regenerates identifiers or drops rows without telling the task builder. Each of the patterns below has a field in the schema that exists to stop it.
At a glance
Where do the links usually break between export and delivery
- Row numbers used as event IDs: a re-export with one extra filter shifts every number after the first removed row.
- Re-pseudonymization: a fresh pseudonym key run maps the same customer or order to a new token, so object IDs in old tasks no longer match.
- Deduplication after task building: duplicate events are removed from the log but not from tasks that cited them.
- Window drift: a task built from events up to a decision point later includes events after it, leaking the outcome into the input.
- Silent rule changes: the derivation script is edited and tasks built before and after the change are mixed in one release.
- Absolute paths in manifests: references point at a supplier's internal storage and cannot be resolved by anyone else.
Five identifier layers and who mints each one
Five identifier layers are enough for most packages. Keep each layer's minting rule written down, because the rule is what lets a later export reproduce the same IDs.
- Deterministic event IDs can follow the name-based pattern of RFC 9562: a version 5 UUID computed from a namespace and a name is always the same for the same inputs, so hashing a canonical string such as system, table, primary key and event type gives a reproducible ID.
- Derivation IDs suit a version 7 UUID, which places a millisecond Unix timestamp in its leading bits so records sort by creation time.
- RFC 9562 cautions against name-based UUIDs as database primary keys; use them as stable external identifiers and let storage use its own keys.
| Identifier | What it names | Who mints it | Stability rule |
|---|---|---|---|
| event_id | One event in the source log | Supplier, at first export | Deterministic from source system, record key and event type, so a re-export yields the same ID |
| object_id | A business object such as an order, ticket or invoice | Supplier, after pseudonymization | One pseudonym key per delivery program; never re-keyed between versions |
| task_id | One derived training or evaluation unit | Task builder | Unique within the package and never reused for a different task |
| derivation_id | One run of one rule that produced one task | Task builder | Random or time-ordered UUID per derivation record |
| log_version and package_version | A frozen export and a frozen release | Supplier and task builder | Semantic version plus content hash of each file |
The derivation record, field by field
The derivation record is the bridge. One record per task, stored as its own table, so the task file stays clean and the lineage can be audited without opening every task.
At a glance
The derivation record, field by field
| Field | Type | Purpose | Required |
|---|---|---|---|
| derivation_id | UUID | Identify this derivation run for this task | Yes |
| task_id | string, foreign key to tasks | Name the task this record explains | Yes |
| source_event_ids | list of event_id, foreign keys to events | Every event the task's input was built from | Yes |
| source_object_ids with qualifier | list of object_id and role pairs | Objects the task is about, with the role each plays | Yes for object-centric logs |
| rule_id and rule_version | string and version | Which derivation rule produced the task, at which revision | Yes |
| source_log_version and source_log_sha256 | version and hash | The exact log export the events were read from | Yes |
| input_window_end | timestamp with offset | Last moment of evidence the task input may contain | Yes for prediction and agent tasks |
| field_transforms | list of field, operation pairs | Which fields were copied unchanged, masked or aggregated | Yes |
| label_source_event_ids | list of event_id | Events that establish the task's outcome or label | Yes when the task has a label |
| notes | text | Exceptions a reviewer should know | No |
How do OCEL, OpenLineage, PROV and Data Package express the same links?
None of the standards does the whole job alone, but each covers one link and gives the schema familiar names. Using them keeps the package readable by tools a buyer already runs.
| Standard or tool | What it links | Fields to borrow |
|---|---|---|
| OCEL 2.0 | Events to objects, each relation qualified by the object's role | Event-to-object relations with a qualifier; the relational form keys them on event ID, object ID and qualifier |
| XES as used by pm4py | Events to one case in a flat log | case:concept:name for the case, concept:name for the activity, time:timestamp |
| OpenLineage | Jobs and runs to input and output datasets | Run IDs that must be UUIDs, inputs and outputs per run, column lineage marked IDENTITY or MASKED |
| W3C PROV-O | A derived entity to the entity it came from | wasDerivedFrom, with a qualified derivation naming the activity |
| Frictionless Data Package | Tables to each other inside one package | primaryKey, foreignKeys with a reference resource, relative paths, resource hash |
| Croissant | Record sets and files in an ML dataset description | key, references between fields, sha256 on each file object |
Illustrative record: a blocked-delivery task traced to its order events
The record below is illustrative and invented for this page; it is not drawn from any company's data. It shows one task derived from an order-to-cash log in which a delivery was blocked by a credit hold and later released.
The task asks an agent, given the events up to the hold, which action should come next. The label is the action a person actually took, taken from a later event that the input window excludes.
- task_id: T-00731. Task type: next action after a credit hold. Input: four events and two objects, rendered as text for the agent.
- derivation_id: a version 7 UUID minted when the rule ran.
- source_event_ids: E-ord-create, E-dlv-create, E-credit-check, E-dlv-block (four deterministic event IDs from the order log).
- source_object_ids with qualifier: order O-5521 as subject, delivery D-8810 as blocked item.
- rule_id and rule_version: next-action-after-block, version 1.3.0.
- source_log_version: 2.1.0, with the SHA-256 of the events file as exported.
- input_window_end: the timestamp of E-dlv-block, with its UTC offset.
- label_source_event_ids: E-credit-release, which falls after the window end and therefore never appears in the input.
- field_transforms: customer name masked, order amount bucketed into a range, activity names copied unchanged.
- Validator result: every event and object reference resolves, the log hash matches the manifest, and no input event is later than the window end. The record passes.
Can a buyer validate the package without the supplier's systems?
Yes, if the package carries its own keys, hashes and relative paths. A buyer should be able to run every check below from the delivered files alone, with no call to the supplier's systems and no internal URLs.
At a glance
Can a buyer validate the package without the supplier's systems
The Data Package standard makes the mechanics concrete: a table schema can declare a primary key and foreign keys that reference another resource in the same package, a resource can carry a hash of its file, and a multi-file path must be either all relative paths or all URLs. Croissant expresses the same idea for ML datasets with keys, field references that act like foreign keys, and a sha256 checksum on each file, which its specification strongly recommends recording in versioned datasets.
- Manifest integrity: every file listed exists, every file present is listed, and each file's hash matches the manifest.
- Key uniqueness: event_id, object_id, task_id and derivation_id are unique in their tables.
- Reference resolution: every source_event_id, source_object_id and label_source_event_id resolves to a row in the delivered log.
- Coverage: every task has exactly one derivation record and every derivation record names an existing task.
- Version match: each derivation's source_log_version and hash equal the delivered log's version and hash.
- Window rule: no source event is later than input_window_end, and every label event is later than it.
- Rule census: report task counts per rule_id and rule_version so mixed releases are visible.
- Rebuild test: re-run one published rule version on the delivered log for a sample of tasks and compare the output with the delivered tasks.
A failing example: what the validator should catch
A failing case shows what each check is for. In this invented example, the supplier re-ran pseudonymization before a second export, and the task builder reused tasks from the first release.
The fix is not to patch IDs by hand. Rebuild the affected tasks from the new export with the same rule version, or ship the old log version alongside them, and record which path was taken in the release notes.
| Check | What the validator finds | Cause | Correct response |
|---|---|---|---|
| Reference resolution | Some source_object_ids do not exist in the delivered log | New pseudonym key produced new object tokens | Rebuild those tasks from the new log or deliver the old log version |
| Version match | Derivations cite log 2.0.0 but the package holds 2.1.0 | Tasks were not rebuilt after the re-export | Fail the release until versions agree |
| Window rule | One task's input includes an event after its window end | Late-arriving event was merged into the case after task building | Rebuild the task and add the late-arrival case to the rule's tests |
| Coverage | Several tasks have no derivation record | A manual task batch skipped the derivation step | Remove the tasks or derive them with a named rule |
What happens to task links when the source log is re-exported?
Treat a re-export as a new log version, never as an update in place. If the deterministic event IDs and the pseudonym key are unchanged, most tasks will still resolve and only the changed events need attention; the validator tells you which.
OpenLineage offers a useful pattern for recording this. Each run of a job carries a run ID that must be a UUID, lists the datasets it read and wrote, and can carry column-level lineage that marks an output field as an identical copy of its input or as masked. Logging every export and every task build as a run gives an operational history that sits next to the per-task derivation records rather than replacing them.
At the semantic level, PROV-O describes the relation itself: a derivation is a transformation of an entity into another, an update that results in a new one, or the construction of a new entity from an existing one, recorded with wasDerivedFrom. A task built from events is a derivation in exactly that sense, which is why the derivation record names both the entity it came from and the activity, the rule run, that produced it.
Requesting a linked log and task package
When a buyer licenses business records for agent or process work, asking for the log, the tasks and the derivation records as one versioned package is the single request that prevents most of the failures above. SourceX sources linked packages like this from U.S. businesses only when a buyer asks, and its request form collects scope, volume, language, format, timing and licensing needs. The format field is the place to name this schema, the derivation table and the validator as delivery requirements.
Behind that request, SourceX handles sourcing, rights review, contracting, preparation and delivery for the businesses licensing the records. Naming the identifier and pseudonymization rules in the request lets them be settled before the first export instead of repaired after it.
Frequently asked questions
Why not just use row numbers as event IDs?
Row numbers change whenever an export adds a filter, removes a duplicate or sorts differently, so tasks built on them break silently. A deterministic ID computed from the source system, record key and event type survives re-export.
Should the derivation record live inside each task file?
Keep it in its own table with a foreign key to the task. That keeps task files clean for training and lets a reviewer audit lineage for the whole release without parsing every task.
Does pseudonymization break the link between logs and tasks?
Only if the pseudonym key changes between exports. Use one key per delivery program, apply it before any task is built, and treat a new key as a new log version that requires rebuilding tasks.
Is OpenLineage enough on its own?
It records which jobs read and wrote which datasets and when, which is valuable operational lineage. It does not by default say which source events a single task came from, so per-task derivation records are still needed.
What should a buyer reject at delivery?
Reject a package where references do not resolve, hashes do not match the manifest, tasks lack derivation records, or input events fall after the window end. Each is a defect that would put wrong or leaked context into training.
Sources
- OCEL 2.0 relates events to objects through event-to-object relations qualified by the object's role. Source
- pm4py's constants define case:concept:name as the case identifier column for event logs. Source
- OpenLineage run IDs must be UUIDs, and run events describe a job's input and output datasets. Source
- The OpenLineage column lineage facet maps output fields to input fields and marks transformations such as IDENTITY or MASKED. Source
- PROV-O defines derivation as transforming, updating or constructing a new entity from a pre-existing one, using wasDerivedFrom. Source
- Data Package table schemas can declare a primary key and foreign keys that reference fields in another resource of the same package. Source
- A Data Package resource can carry a file hash, and a multi-file path must be all relative paths or all URLs. Source
- Croissant record sets declare keys and field references, and file objects can carry sha256 checksums recommended for versioned datasets. Source
- RFC 9562 makes UUIDv5 deterministic for the same namespace and name, puts a Unix millisecond timestamp in UUIDv7, and cautions against name-based UUIDs as primary keys. Source
Tell us what data you need
Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.