Systems and records
Event log retry deduplication: telling a retry from a second action
By SourceX Editorial · Updated
9 sections
Short answer
A retry repeats the same intended action, so it carries that action's identity: the same idempotency key, delivery ID or event ID, against the same object and the same prior state, with no new state change. A second business action has its own intent, moves the object to a new state and matches its own entry in the system of record. Classify on identity, state and reconciliation before deleting any row.
Record timeline
Event log retry
Key takeaways
- A retry shares the original action's identity and causes no new state change; a second business action has its own intent, transition and record.
- At-least-once delivery in queues and webhooks makes repeated events normal, so deduplication is a classification task, not a cleanup chore.
- Event IDs, delivery IDs, idempotency keys and state transitions decide most cases; timestamps only break ties.
- Reconcile the classified log to totals in the system of record and record the difference as a confidence label.
- Never delete raw events; keep ambiguous pairs, label them and resolve them with the workflow owner.
How do you tell a retry from a second business action?
Ask three questions of every repeated-looking event, in order. Does it carry the same identity as an earlier event, such as the same event ID, delivery ID or idempotency key? Did it change the state of the business object beyond what the earlier event already did? Does it reconcile to a separate entry in the system of record, such as a second invoice, payment or shipment?
Same identity, no new state change and no separate record means a retry, and the episode should count one business action. A new state change that reconciles to its own record means a second business action, even if the payload looks identical. Anything in between is ambiguous and should be labeled as such, not silently merged.
The distinction matters for AI work because an agent trained or graded on episodes learns from counts. If a retried payment is counted twice, the episode teaches that customers were charged twice; if a genuine second order is collapsed into the first, the episode hides a real decision.
Why business event logs contain retries in the first place
Retries are built into the delivery mechanisms that carry business events, so duplicates are expected, not exceptional. Most messaging and webhook systems prefer delivering an event more than once to losing it.
Apache Kafka's design documentation says Kafka provides at-least-once delivery by default, meaning messages are never lost but may be redelivered. It explains that a producer that did not receive an acknowledgment used to resend, which could write the message to the log again, and that the idempotent producer option deduplicates using a producer ID and a sequence number. It also notes that a consumer that crashes after processing but before saving its position will process the first few messages again.
Amazon's SQS documentation says standard queues provide at-least-once delivery and that more than one copy of a message might be delivered, so applications should be designed to handle the same message more than once. Stripe's webhook documentation says endpoints might occasionally receive the same event more than once and recommends logging processed event IDs. GitHub's documentation notes that a redelivered webhook keeps the same X-GitHub-Delivery header as the original, and that GitHub does not redeliver failed deliveries automatically, so redeliveries come from the receiver's own scripts or manual action.
Even when the effect is idempotent, the log may not be. RFC 9110 defines an idempotent request method as one where multiple identical requests have the same intended effect as a single request, and it explicitly allows a server to log each request separately. A clean business outcome can therefore sit next to several log rows.
The retry and duplicate event classification guide
The guide sorts repeated events into classes, each with the signals that identify it and the treatment it gets when events become business task episodes. Examples are illustrative; adapt the field names to your systems.
| Class | Identifying signals | Illustrative example | Treatment in the episode |
|---|---|---|---|
| Transport redelivery | Same event ID or delivery ID, identical payload, later receipt time | A webhook redelivered after the receiver timed out | Keep one business event; record the extra receipt as a delivery attempt |
| Client retry with idempotency key | Same key and parameters, replayed response, no new object | A payment request resent after a network error | One action; attach the attempts to it |
| Retry without a key | Same payload and target within a short window, no new state change in the system of record | An integration resubmits an order update after a timeout | Merge provisionally; confirm against source totals |
| Consumer reprocessing | Same message processed twice after a crash or rebalance | Two status-change rows written by a restarted worker | Keep the first; flag the second as reprocessing |
| Replay or backfill | Original event ID and source time, much later ingest time | A month of events reloaded into the warehouse | Drop the copy; keep ingest lineage |
| Two notifications for one change | Different event IDs, same object ID and event type, same transition | A platform emits two events for one update | One business transition; keep both IDs as aliases |
| Failed attempt then success | Error response then success for the same intent | A shipment label call fails, then succeeds | One action with a recorded failure, which is useful evidence of recovery |
| Second business action | New intent, new state transition, its own record in the system of record | A customer places a second identical order | Two actions; never merge |
| Ambiguous double submission | Different keys, same content, seconds apart, both created records | A user clicks submit twice and two tickets open | Keep both; label ambiguous; resolve with the workflow owner |
Which fields decide the class of a repeated event?
Identity fields decide most cases, state fields decide the rest, and timestamps only break ties. Capture these fields for every raw event before any cleaning, because a deduplication step that drops them cannot be audited later.
- Event ID and, where the source has one, delivery ID, kept separately because a redelivery can share one and differ on the other.
- Idempotency key and the request parameters it covered.
- Business object ID and event type, which together catch two notifications for one change.
- State before and after, or the status transition, from the source system's history table.
- Source timestamp and ingest timestamp, kept as two columns.
- Actor or integration that produced the event, and the response status the caller received.
- A hash of the normalized payload, used only as a supporting signal.
| Signal | Strength | Known limit | Source note |
|---|---|---|---|
| Event ID | Strong for redelivery | Some platforms emit two IDs for one change | Stripe recommends tracking event IDs and pairing object ID with event type for split events |
| Delivery ID | Strong for webhook redelivery | Only identifies the delivery, not the business change | GitHub keeps the same delivery header on redelivery |
| Idempotency key | Strong while the key is retained | Keys can expire, after which a reuse may run as new | Stripe retains v1 API keys for at least 24 hours and may remove them after; its v2 API uses a longer window |
| State transition | Decisive for second actions | Missing when the source keeps only current state | Read from history or audit tables |
| Timestamp gap | Weak | Clock drift and batching distort gaps | Stripe warns that distinct events can share a created time |
Could clock drift, identifier mismatch or an unrecorded transition change the case?
Yes, each of the three can flip a classification, which is why the guide ranks identity and state above time. The table shows how each threat misleads and the check that catches it.
| Threat | How it misleads | Check before trusting the class |
|---|---|---|
| Clock drift between systems | A retry appears to happen before the original, or two actions appear simultaneous | Order by source sequence or version where available; measure offset on events known to be paired |
| Ingest time used as event time | A backfill looks like a burst of new actions | Keep source and ingest time separate; compare against the source's own history |
| Identifier mismatch across systems | The same order has a different ID in the CRM and the ERP, so repeats look distinct | Build a crosswalk table and test it on a sample before matching |
| Unrecorded transition | A second action left no history row, so it looks like a retry | Reconcile to the system of record; unexplained totals reopen the case |
| Expired idempotency key | A late retry with the old key creates a new object | Treat it as a second object and flag it, since the effect was real |
Reconcile the cleaned log back to source totals
After classification, count business actions by type and compare them with the totals in the system of record for the same period: payments with the ledger, orders with the order table, tickets with the help desk's own counts. The episodes are reproducible only if the counts match or every difference is explained.
Record the reconciliation as part of the dataset, not in a separate notebook. A short table per action type, with the raw event count, the count after classification, the source total and the unexplained difference, lets a reviewer see how much of the cleaned log rests on judgment.
Use the difference to set a confidence label per episode. Episodes that reconcile exactly are high confidence. Episodes that depend on a timestamp-only merge, or that sit in a period with an unexplained gap, are lower confidence and should be kept apart for evaluation.
Worked example: one payment, three rows (illustrative)
This example is illustrative; the identifiers and times are invented. An accounts receivable integration posts a customer payment. The raw log shows three payment-created rows for the same invoice within two minutes.
Rows one and two share an idempotency key and parameters; the second carries a replayed response and no new ledger entry, so it is a client retry. Row three has a different event ID but the same object ID and event type and the same transition, so it is a second notification for the same change. The ledger shows one payment. The episode keeps one business action with two attempts and two event aliases.
The exception: a week later the same customer pays the same invoice amount again, with a new key, a new ledger entry and a later refund. That is a second business action, an overpayment followed by a refund, and collapsing it into the first payment would erase the most instructive part of the case.
- Raw rows: 3 for the first payment, 1 for the second, 1 refund.
- After classification: 2 payments and 1 refund as business actions.
- Ledger total for the period: 2 payments, 1 refund, so the episode reconciles.
- Confidence: high, because identity and state decided every row.
When the log cannot settle it, keep both rows and label them
When identity and state cannot decide, keep both rows, label the pair ambiguous and record why. Deleting one of them turns a known uncertainty into a hidden error.
Never drop raw events. Store the classification as a column next to the raw data, with the rule that assigned it, so the decision can be reversed when a workflow owner explains what happened. Event log standards help here: OCEL 2.0 treats each event as an atomic, typed occurrence with its own identifier and timestamp, which suits a design where retries become attributes or attempts of one event rather than extra events.
Bring in the workflow owner when ambiguous pairs cluster in one integration or period, because that usually means a missing key or a known incident. Bring in a data engineer who knows the source when totals will not reconcile, since the gap may be a missing history table rather than a classification problem.
Deduplicated event logs in a licensing review
A log whose repeats are classified and reconciled is far easier to review for licensing, because each episode shows one business action per real decision and states its own confidence. Raw logs full of redeliveries are harder to scope and harder to price.
A company that wants to license cleaned event logs to AI developers can use SourceX for the whole path, from sourcing and rights work through contract, preparation, delivery and payment. The logs stay put until the company signs off on scope and terms in a signed agreement, and the first contact is a short assessment that asks for no data.
Being able to export a log is a question of system access. Whether the people in it may be shared is a privacy question, and whether the company may license it is a question of ownership and contracts. Cleaning the log answers none of them, so the review covers each.
Frequently asked questions
What is event log retry deduplication?
It is the step that identifies events which repeat an earlier attempt at the same action, such as a redelivered webhook or a retried API call, and counts them as one business action. Done well, it keeps the extra attempts as evidence instead of deleting them, and it reconciles the result to the system of record.
Can I deduplicate on timestamp and payload alone?
Only as a weak, provisional signal. Distinct events can share a timestamp, clocks drift between systems, and a customer can legitimately repeat an identical action. Use event IDs, idempotency keys and state transitions first, and confirm any timestamp-based merge against source totals.
Are retries noise to throw away?
No. A failed attempt followed by a success shows how a workflow recovered, which is useful for training and testing agents. Keep the attempts attached to the single business action they belong to.
Why do my webhook logs show the same event twice?
Most webhook and queue systems deliver at least once, so a timeout or crash on the receiving side leads to a repeat. Check whether the event ID or delivery ID matches an earlier row; if it does, it is a redelivery, not a new action.
What if the idempotency key expired before the retry?
Then the platform may have treated the late retry as a new request and created a second object. Count it as a second object because the effect was real, flag it as a late retry, and check whether a reversal or refund followed.
Sources
- Kafka provides at-least-once delivery by default; the idempotent producer deduplicates with a producer ID and sequence number; a consumer that crashes before saving its position reprocesses messages. Source
- Amazon SQS standard queues provide at-least-once delivery, and more than one copy of a message might be delivered. Source
- Stripe webhook endpoints might occasionally receive the same event more than once; Stripe recommends logging processed event IDs, notes that distinct events can share a created timestamp, and suggests pairing data.object ID with event type when two events are generated. Source
- Stripe saves the first result for an idempotency key and replays it on retries, and retains keys for at least 24 hours, after which a reused key can execute a new request. Source
- A redelivered GitHub webhook carries the same X-GitHub-Delivery header as the original delivery. Source
- GitHub does not automatically redeliver failed webhook deliveries. Source
- RFC 9110 defines idempotent methods by intended effect and allows a server to log each request separately. Source
- OCEL 2.0 defines events as atomic, typed occurrences with an identifier and a timestamp. Source
See if your company qualifies
A short company assessment. No data uploads are needed.