Systems and records
Process data extraction completeness: did the export keep the history?
By SourceX Editorial · Updated
9 sections
Short answer
Check what one row means. A current-state snapshot has one row per record with latest values and a last-modified date. A historical event export has many rows per record, each with a case ID, what changed, old and new values and a timestamp. Confirm it by profiling rows per case, replaying events to rebuild current values, and reconciling counts to source totals.
System export
Process data extraction
Key takeaways
- One row per record means a snapshot; many rows per record with changed fields and timestamps means an event history.
- Most systems keep history apart from records, so a standard export reads current state even when history exists.
- History exists only for what was configured to be tracked and only as long as retention keeps it; read both from settings.
- Replaying events to rebuild current values is the strongest single test that a history is complete.
- Grade every reconstructed case and keep the grade with the data, rather than treating every case as equally complete.
How do you tell a historical event export from a current-state snapshot?
Look at what one row means. In a current-state snapshot, one row is one record, holding its latest values and usually a last-modified date. In a historical event export, one row is one change or activity, carrying a case identifier, what changed, the old and new value where relevant, and when it happened, so a single record appears many times.
The file name and the vendor's label are not evidence. A report called 'activity' can be a snapshot of the latest activity per record, and a 'full export' can contain every record but none of their history. The reliable test is behavioral: count rows per case, replay the events and check that they rebuild the current state, and reconcile both against totals taken from the source system at the same cutoff.
This matters because the two files answer different questions. A snapshot can say how many orders are open today. Only an event history can say how long each order waited in each step, how often a status went backwards, or what an agent would have known at the moment it had to decide. Process mining tools make the requirement concrete: pm4py's dataframe formatting function checks for a case identifier, an activity and a timestamp column and raises an error when one is missing.
What each kind of file looks like, side by side
Most files can be classified in a few minutes from their shape. The table lists the properties that separate the two, and the quick check that confirms each.
| Property | Current-state snapshot | Event or change history | Quick check |
|---|---|---|---|
| Rows per record | Exactly one | Many, varying by how busy the record was | Group by record ID and look at the distribution of row counts |
| Meaning of the timestamp | Last modified or created date of the record | When each change or activity happened | Check whether the same record has several distinct timestamps |
| Previous values | Absent | Present as old and new value pairs, or derivable from consecutive rows | Look for columns such as old value, from, previous or field name |
| Status coverage | Only statuses that records sit in today | Every status a record passed through, including ones now rare | Compare the status vocabulary with the workflow's documented states |
| Deleted or merged records | Usually missing without trace | Often present as delete or merge events | Search for delete, merge or archive event types |
| Growth between two pulls | Row count tracks the number of records | Row count grows with activity even if records do not | Compare two exports of the same scope taken days apart |
Where history lives in common systems, and what a standard export leaves out
In most business systems the history sits in a separate place from the records, behind a separate setting, endpoint or retention rule. A standard export or report reads the record tables and therefore produces a snapshot, even when the system holds the history.
Two limits recur. History exists only for what was configured to be tracked, and only for as long as the retention rule keeps it. Both have to be read from the system's settings, not inferred from the file.
| System | Record export gives | Where the history is | Limit to check |
|---|---|---|---|
| Salesforce | Current field values per record | Field history tracking, for fields an admin selected | Up to 20 tracked fields per object, formula and roll-up fields cannot be tracked, and without Field Audit Trail history is kept up to 18 months in the UI and 24 months via the API |
| Zendesk | Tickets with current fields | Ticket Audits: each audit is one update holding one or more events, such as a field change with its previous value | Audits must be pulled per ticket or through an export path that includes them |
| Jira Cloud | Issues with current fields | Issue changelogs, available from a paginated endpoint that returns all changelogs for an issue oldest first | Confirm the extract used the changelog endpoint rather than whatever an issue search returned inline |
| HubSpot | Current property values | Property history, returned when records are read with the propertiesWithHistory parameter | History comes only for the properties named in the request |
| Microsoft Dataverse | Current row values | Audit logs for tables and columns where auditing is turned on | Update logs are written only when a value changes, retrieve and export operations are not audited, and logs past the retention period are deleted |
| A database behind a custom app | Current rows | Change data capture, such as Debezium, which emits an event for each committed row-level change | Captures changes from when capture was set up, not the history before it |
The extraction scope verification checklist
The checklist is the asset. Run it on every extract before anyone builds cases, features or task episodes from it, and keep the results with the data. Each check has a pass condition that can be computed, and a defined response when it fails.
| Check | How to run it | Pass condition | If it fails |
|---|---|---|---|
| 1. Case identity | Confirm every row carries a stable case ID and that IDs match the record export | No blank IDs; every event ID joins to a record | Find the mapping table or stop; events without cases cannot be reconstructed |
| 2. Rows per case | Profile event counts per case by record type and period | Multi-row cases are common and busy records have more events | A file with one row per case is a snapshot; request the history source |
| 3. Event vocabulary | List distinct event types and changed fields | The workflow's documented states and key fields all appear | Compare with the tracked-field and audit settings; record what is untracked |
| 4. Replay to current state | Apply events in order to rebuild each record's latest values | Rebuilt values match a snapshot taken at the same cutoff for a sampled set of records | Investigate gaps: untracked fields, retention, deleted events or import edits |
| 5. History window | Compare the first event date per case with the record's created date | Events start at creation for records inside the retention window | Mark cases created before the window as partial and record the window |
| 6. Clock mapping | Document each timestamp's source, time zone and precision | One documented clock per field, converted consistently | Hold sequence-sensitive analysis until the mapping is confirmed |
| 7. Source totals | Reconcile case counts and event counts by period to totals from the source system | Differences explained by written rules such as filters or deletions | Unexplained gaps block acceptance of that period |
| 8. Deleted, merged and archived records | Search for delete and merge events and for archived records | Their treatment is stated and consistent | Request the archive or note the exclusion in the data card |
| 9. Concurrency and retries | Find identical events within seconds on the same case and parallel work items | Duplicates and parallel branches are flagged, not silently dropped | Add a dedupe rule with the key it uses |
| 10. Reconstruction confidence | Grade each case: complete, partial window, inferred steps or unreliable | Every case carries a grade and the reason | Exclude or label low-grade cases before use |
Can clocks, identifiers or missing transitions change a rebuilt case without anyone noticing?
Yes. Each of the three can change the case without making the file look broken. Clock drift reorders steps when events come from systems with different clocks or time zones, so a reply can appear before the message it answered. Identifier mismatch splits one case into two or merges two into one when systems key records differently, for example a ticket number in the helpdesk and an order number in the ERP.
An unrecorded transition is the hardest, because the gap is invisible. If a status was changed by an integration that bypassed the audit path, or a field was not tracked when the change happened, the replay jumps from one state to another with nothing in between. Check 4 catches some of these as a replay mismatch, and comparing consecutive states against the allowed transitions in the workflow catches more.
History can also change after the fact. Microsoft documents that Dataverse audit records show the current label of a choice column by default, so renaming a label later changes what old audit entries display unless a setting stores the label as recorded. Its retention notes add that each audit record keeps the retention period in force when it was written. Both are reasons to record the system settings alongside the extract.
Worked example (illustrative): a ticket export that looked like history
The following example is illustrative. A support team hands over a CSV of 12,000 tickets with columns for status, priority, assignee, created date, updated date and a free-text 'history' column. At first glance it looks like history, because the text column contains lines such as 'status changed to pending'.
The checklist shows otherwise. Check 2 finds exactly one row per ticket. The history column, parsed, holds only the last three changes per ticket, so check 5 finds that for most long-running tickets the events start weeks after creation. Check 4 cannot rebuild priority changes at all, because priority never appears in the text. The file is a snapshot with a truncated change summary attached.
The team then pulls the ticket audits for the same tickets and cutoff. Rows per ticket now vary from 2 to over 60, replay rebuilds current status and priority for every ticket in a 200-ticket sample, and counts reconcile by month to the helpdesk's own reports. Roughly one in twenty tickets is graded 'partial' because it was imported from an earlier tool with no audit trail, and the grade travels with the data.
The in-between case: a series of scheduled snapshots
A folder of weekly or nightly snapshots is neither a single snapshot nor an event log. Differencing consecutive files shows which records changed between pulls, which is real history, but every change that happened and was undone between two pulls is lost, and the time of each change is known only to within the interval.
Label such data as sampled state history and state the interval. It can support questions about trends and backlogs, but not questions about the order of steps within a day or the exact time to respond. If the source system still holds an audit trail for the same period, prefer it, and use the snapshots to check that the replay lands on the same values at each snapshot date.
When the reconstruction needs the workflow owner, not more data
More rows will not fix a definitional problem. Bring in the workflow owner when the replay shows transitions the process documentation says cannot happen, when the same status label meant different things before and after a process change, or when the team cannot say which of two clocks records the business event.
Bring in the same owner when the extract and the process disagree about what a case is. A support team may treat a merged duplicate as one case while the helpdesk keeps two tickets, and a field service team may count a return visit as part of the original job while the system opens a new work order. Those are business decisions about case boundaries, and they change every count and duration that follows, so the owner's ruling should be written down with the extract.
Also stop when reconstruction depends on inference for a large share of cases. A case where most steps were inferred from free text or from a later snapshot is a model of the process, not a record of it, and should be labeled as such or left out.
How do event histories fare in a licensing review?
For AI work, verified event histories are usually more useful than snapshots, because they show how work moved and how it ended. Buyer interest usually starts at about two years of consistent history and tens of thousands of records in a single system, while rarer kinds of work can qualify with smaller volumes; SourceX assesses each dataset deal by deal.
Where a company licenses event histories like these through SourceX, rights review, contracting and preparation are part of the service, and a large delivery can stay in the company's own storage or travel on encrypted drives rather than being hosted. Running the checklist above first tells you what you hold before the short company assessment, which asks for no data upload.
Frequently asked questions
Is a full data export the same as a complete history?
Not necessarily. A full export usually means every record, which is completeness across records, not across time. Many systems keep change history in separate audit or history tables that a standard export does not read.
How can I tell if my CRM export includes field history?
Look for multiple rows per record with field names, old values, new values and change dates. In Salesforce, history exists only for fields an admin chose to track, and without Field Audit Trail it is kept up to 18 months in the interface and 24 months through the API.
Can I rebuild history from old backups or snapshots?
Partly. Differencing snapshots shows which values changed between backups, but not changes made and reversed in between, and not the exact time of each change. Label the result as sampled state history with its interval.
What minimum columns does an event log need for process analysis?
A case identifier, an activity or event type, and a timestamp. Process mining libraries such as pm4py check for exactly these three columns, and useful logs add the actor, the changed field and old and new values.
Why does replaying events not match the current record?
Common causes are untracked fields, history deleted by a retention rule, changes written by integrations that bypass the audit path, and records imported from an earlier system. Each mismatch should be traced to one of these causes and recorded with the data.
Sources
- pm4py's format_dataframe function expects a case identifier, an activity and a timestamp column and raises an error when one is missing. Source
- Salesforce lets admins track up to 20 standard and custom fields per object, and formula, roll-up summary and auto-number fields cannot be tracked. Source
- Without Field Audit Trail, Salesforce retains field history for up to 18 months, and up to 24 months through the API. Source
- Zendesk ticket audits are a read-only history of updates to a ticket; each audit is one update containing one or more events. Source
- Jira Cloud's issue changelog endpoint returns a paginated list of all changelogs for an issue sorted by date, starting from the oldest. Source
- HubSpot stores property value updates in a record's property history, retrievable through the API with the propertiesWithHistory parameter. Source
- Dataverse auditing logs changes only where it is turned on, writes update logs only when a value changes, does not audit retrieve or export operations, deletes logs past the retention period and by default shows a choice column's current label. Source
- Debezium is a change data capture platform whose applications consume an event for each committed row-level change to a monitored database. Source
See if your company qualifies
A short company assessment. No data uploads are needed.