Privacy and preparation
Event log reconstructed task validation: replay it with an operator
By SourceX Editorial · Updated
9 sections
Short answer
Validate a reconstructed task by having an operator who does that work narrate each sampled case from the source system, then compare their account step by step with the episode your rules built. First reconcile counts to source totals and fix clock and identifier mapping. Log every mismatch as a missing, extra, misordered, wrong-case or wrong-boundary step, fix the rule, and replay again.
Record timeline
Event log reconstructed task
Key takeaways
- Operator replay compares a reconstructed episode with a person's account of the real work, catching errors a model mined from the same log would share.
- Reconcile counts to source totals and fix clock, timestamp format, identifier and role mapping before any operator session.
- Have the operator narrate from the source system before seeing the reconstruction, so the session measures accuracy rather than agreement.
- Classify mismatches as extra, missing, misordered, wrong-case, wrong-boundary or wrong-state steps, and tie each to a rule change and a retest.
- Keep recorded events when an operator's memory disagrees, add fields for what the log cannot show, and mark unobservable steps instead of inventing events.
How do you validate a reconstructed task by replaying it with an operator?
Pick a sample of reconstructed cases, ask a person who does the work to walk through each one from the source system's own records, and compare what they say happened with the episode your rules produced. Every difference goes into a discrepancy report with a type, the evidence the operator pointed to and the rule change it triggers.
A reconstructed task is an episode your pipeline assembled from raw events: it chose a case key, mapped system codes to activity names, ordered events by timestamp and decided where the task starts and ends. Each of those choices can be wrong in a way that no schema check notices. Conformance checking compares a log with a process model, but if the model was mined from the same log, both can share the same mistake.
The operator brings what the log cannot: knowledge of which steps were real work, which were system noise, and which happened off-system on the phone or on paper. That is why the replay is a validation step and not a demo.
Checks that must pass before an operator sees a case
Do not spend an operator's time on errors a query can find. Reconcile the reconstruction to the source and fix time and identity first, so the session surfaces the problems only a person can see.
Time needs the most care. RFC 3339 defines a numeric offset as local time minus UTC, so 18:50:00-04:00 is the same instant as 22:50:00Z, and mixing offset and non-offset timestamps reorders events silently. PM4Py's documentation warns that automatic timestamp parsing looks at each value in isolation and can read 2020-02-01 as January 2 instead of February 1, and it advises specifying the timestamp format.
| Check | How to run it | Pass condition | If it fails |
|---|---|---|---|
| Source totals | Compare case and event counts per day and per status with the source system's own reports | Counts match, or every gap has a documented cause | Find the dropped or duplicated extract before any replay |
| Clock mapping | List every source clock with its time zone and offset handling; convert to one reference | No event moves across a step boundary when converted | Fix the conversion; flag systems that store local time without an offset |
| Timestamp format | Parse with an explicit format, never inference | No day-month swaps in a hand sample | Re-parse with the declared format |
| Identifier mapping | Trace each case key to the source record IDs it joins | One reconstructed case per real case in a sample | Split or merge rules need revision before replay |
| Role meanings | Map user and queue codes to roles with the workflow owner | Every actor code in the sample has a role | Hold cases with unmapped actors out of the sample |
Running the replay session without leading the operator
Ask the operator to narrate the case from the source records before they see your reconstruction. If they read your episode first, they tend to confirm it, and the session measures agreement with you rather than accuracy.
At a glance
Running the replay session without leading the operator
Choose operators who do the work today or did it in the period covered, not the engineer who wrote the rules. Sample deliberately rather than at random alone: include the most common variants, rare variants, the longest cases, cases that touch several objects such as an order with split shipments, and cases near your task boundaries.
- Blind narration: the operator opens the case in the source system and says what happened, step by step, and why.
- Side-by-side comparison: show the reconstructed episode and mark each step as confirmed or as a discrepancy.
- Evidence capture: for each discrepancy, note which screen, document or message the operator relied on.
- Boundary question: ask where this piece of work started and finished for them, and compare it with your boundary.
- Notes, not recordings: keep written notes keyed to case IDs; screen recordings capture personal data you then have to manage.
A discrepancy taxonomy borrowed from alignments
Classify each mismatch with a small fixed set of types, so findings from different sessions can be counted and traced to rules. Process mining alignments are a useful model: PM4Py describes synchronous moves where log and model agree, moves on log that appear in the trace but not the model, and moves on model that appear in the model but not the trace.
In a replay, the operator's account plays the role of the model. A step the operator confirms is a synchronous move. An event in your episode they say was not a step resembles a move on log. A step they did that left no event resembles a move on model. Ordering, case and boundary errors need their own types because a sequence alignment does not express them.
Object-centric logs add one more type. OCEL 2.0 lets one event relate to several objects, with a qualifier for each object's role, and it keeps object attribute values as their own timed changes rather than tying them to events. An operator will often spot a step assigned to the wrong order, or a status that was read from the wrong moment.
| Discrepancy type | What the operator says | Likely cause | Typical fix |
|---|---|---|---|
| Extra step | That event was not me doing anything | Automated job, notification or sync written to the same audit table | Filter by actor type or event source |
| Missing step | I called the customer before releasing it | Work done off-system or in an unlogged tool | Record as an unobservable step; do not invent an event |
| Wrong order | I approved before I edited, not after | Clock drift, batch posting or completion-only timestamps | Correct clock mapping or use start times where they exist |
| Wrong case | That email was about the other order | Identifier mismatch or one event relating to several objects | Revise the join or use object qualifiers |
| Wrong boundary | The task really started at the credit check | Boundary rule anchored on the wrong event | Move the start or end anchor and rebuild |
| Wrong state | The order was already on hold then | Object attribute read at the wrong time | Read state as of the event timestamp |
The human replay validation and discrepancy report
This is the reusable asset. It has a header that pins exactly what was validated and a row per discrepancy, so a buyer or reviewer can trace each correction from finding to fixed rule to retest.
At a glance
The human replay validation and discrepancy report
Version everything the result depends on. A report that does not name the log version and the rule version cannot be checked again after either changes.
- Header: log extract version and checksum, reconstruction rule version, source systems and date range, report date.
- Sample design: how many cases, how they were chosen, which strata, and which cases were excluded and why.
- Operators: role, team and period of experience with the process; no names in the shared copy.
- Pre-checks: results of source totals, clock mapping, timestamp format, identifier and role checks.
- Per-case result: case ID, reconstructed step count, steps confirmed, discrepancy count, operator confidence.
- Per-discrepancy row: case ID, step reference, discrepancy type, operator statement, evidence pointed to, cause hypothesis.
- Resolution: rule changed (with new rule version), marked unobservable, or rejected with the evidence that overruled the operator.
- Retest: rerun result on the same cases and on a fresh sample, with remaining discrepancies by type.
- Limits: process areas the replay did not cover and why, and the overall confidence statement.
Do clock drift, identifier mismatches and unrecorded transitions distort a replayed case?
Yes, and each changes it in a recognizable way. Clock drift between two systems reorders steps near each other in time; an identifier mismatch moves steps between cases or merges two cases into one; an unrecorded transition leaves a gap that the reconstruction either skips or bridges with a wrong boundary.
Completion-only timestamps are a related trap. The XES lifecycle extension defines a transactional model in which an activity has transitions such as start and complete; when a source stores only completion, durations and overlaps between concurrent work are inferred rather than observed, and an operator may describe two steps as parallel that the log shows as sequential. XES is maintained as IEEE 1849, whose 2023 revision superseded the 2016 version.
Retries are the other case worth asking about directly. A failed submission followed by a successful one may be two events for one step or two real attempts. Only the person who did it, or the system's error record, can tell you which.
Illustrative case: replaying credit-hold releases
Treat this case as illustrative: the distributor is invented and so are the counts. A distributor reconstructs credit-hold release tasks from its ERP audit trail and order history. Pre-checks find one system writing local time without an offset, which is fixed before sampling. The team samples 30 cases across five strata and runs replay sessions with two credit analysts.
Of 214 reconstructed steps, analysts confirm 181. The 33 discrepancies cluster: 14 extra steps from a nightly job that rewrote the hold status, 9 missing steps where the analyst phoned the sales rep, 6 order swaps from the offset problem in a second system, and 4 boundary errors where the task actually began at a credit limit review. The rules change, the job's events are filtered, phone calls are recorded as unobservable steps, and a rerun on the same 30 cases plus 20 new ones leaves 5 discrepancies.
The exception: in two cases an analyst insisted they released the hold, but the audit trail showed a manager's user ID. The manager confirmed acting on the analyst's request by chat. The record was right about who pressed the button and the analyst was right about who decided, so the report keeps the event as recorded and adds the decision-maker as a separate field, rather than overwriting the log with a recollection.
Cases an operator cannot vouch for
Some steps have no human witness. Fully automated stages, integrations between systems and work done by people who have left cannot be validated by replay, so the report should mark those parts as validated by reconciliation only.
Narrow the scope when discrepancies stop falling. If two rounds of rule changes leave the same discrepancy types in the same parts of the process, the reconstruction probably cannot represent that sub-process from the available events, and it is better to deliver the parts that validate than to keep tuning rules.
Bring in specialists when the replay touches records with personal or regulated content on screen, when operators work under agreements that limit what they may discuss, or when the workflow owner disagrees with an operator about how the process is meant to run. That last case is a process question, not a data one.
Validated task episodes as licensable data
Episodes that have survived an operator replay, with a report showing what was checked and corrected, are more useful to AI developers building agents than raw event exports, because the buyer can see how the reconstruction was tested.
Validated episodes of this kind are what SourceX packages when it licenses operational records directly from a company, a process that runs from sourcing and rights review through contracting, preparation, delivery and payment. Buyers usually look for two or more years of consistent history held in one system, at a volume in the tens of thousands of records, with rarer work considered case by case even at smaller volumes.
Before any event logs or reports change hands, the company approves the scope and terms and signs the agreement, and whether its records qualify can be checked first through a short assessment that involves no uploads.
Frequently asked questions
How many cases should an operator replay?
Enough to cover each variant and stratum you care about, including rare and long cases, rather than a fixed number. Start with a small stratified sample, fix the rules, then replay the same cases plus a fresh sample to check that the fixes hold.
Should the operator see the reconstructed episode before narrating?
No. Ask them to narrate from the source system first, then compare. Showing the reconstruction first tends to produce agreement with it rather than an independent account.
What if the operator's memory and the event log disagree?
Ask for evidence. When the log records who acted and the operator recalls who decided, both can be right, so keep the event and add a separate field. Do not overwrite recorded events on recollection alone.
Is operator replay the same as conformance checking?
No. Conformance checking compares a log with a process model, for example through token-based replay or alignments. Operator replay compares a reconstructed episode with a person's account of real work, which catches errors a model mined from the same log would share.
What should happen to steps that left no event?
Record them as unobservable steps with the operator's evidence, and leave them out of the event sequence. Inventing events to fill the gap makes the episode look complete while teaching a model behavior the systems never captured.
Sources
- PM4Py's documentation warns that automatic timestamp parsing can read 2020-02-01 as January 2 rather than February 1 and advises specifying the timestamp format. Source
- PM4Py documents token-based replay for conformance checking and alignments made of synchronous moves, moves on log and moves on model. Source
- The OCEL 2.0 specification relates each event to the objects involved with qualifiers describing their roles, and records object attribute values as timed values not connected to events. Source
- RFC 3339 defines numeric offsets as local time minus UTC, so 18:50:00-04:00 is the same time as 22:50:00Z. Source
- The XES lifecycle extension describes which transition of an activity's transactional model an event represents, with a standard model including start and complete. Source
- The XES site says IEEE 1849-2023 superseded the 2016 version of the XES standard. Source
See if your company qualifies
A short company assessment. No data uploads are needed.