Systems and records
Business archive deduplication timing: when can you remove duplicates?
By SourceX Editorial · Reviewed by Noah Loul ·
10 sections
Short answer
Deduplicate a business archive late and reversibly. First preserve the full export with a checksum for every file, reconcile totals and record every location each item lived in: custodian, folder, thread and attachment family. Then decide what each duplicate group proves before suppressing copies in a working set, keeping a map back to every original location. Never deduplicate items under a hold or financial duplicates that are themselves evidence.
System export
Business archive
Key takeaways
- Separate storage deduplication, which can happen early, from record deduplication, which should happen after review.
- Before access ends, capture the full export, a checksum manifest, reconciled totals and a location row for every item.
- Extra copies can prove who received a record, when, and in which conversation; record that before suppressing them.
- Near-duplicates are usually versions and financial duplicates are often the evidence itself; keep both.
- Custodian-level grouping, or global grouping with a full location map, keeps later scope changes reversible.
When is it safe to deduplicate a business archive?
It is safe once three things are true: the original export is preserved with checksums, every copy's location has been recorded, and someone has decided what each group of duplicates proves. Until then, remove nothing. Deduplicate a working copy, keep the preserved export untouched, and keep a map from each kept item back to every place a copy lived.
The pressure to clean up early is real. A shutdown, a system migration or an expiring vendor subscription produces a large export, and duplicates look like waste: the same invoice PDF in five mailboxes, the same contract attached to twenty emails, the same ticket exported twice. Removing them before review saves storage and review hours, and it also discards facts about who held a record, when they received it and what it was attached to.
The fix is to separate two jobs that people often merge. Storage deduplication stores identical content once to save space. Record deduplication decides which copies a reviewer or buyer sees. The first can happen early because it loses nothing if done properly. The second should happen late, after the evidence each copy carries has been captured.
Can you save storage without losing where each copy lived?
Yes, if the storage layer separates content from the paths that point to it. Preservation formats already work this way, and the same pattern is easy to copy in a plain export folder with a manifest.
The Oxford Common File Layout (OCFL) specification is a useful model. Its implementation notes say OCFL separates the content path of a stored file from its logical path in each version, which is what permits deduplication: identical content is stored once, and later references to it appear only in the inventory's state block. The specification defines that state block as digest values, each mapped to the list of logical paths that hold that content. In other words, one stored file can carry several names and locations, and none of them is lost.
Translate that into an archive project and the rule is simple. Store one copy of identical bytes if you must, but write down every original path, mailbox folder, channel, custodian and parent item that pointed to it. If that location list cannot be produced, you have not deduplicated; you have deleted.
What does a duplicate copy prove that the kept copy does not?
A duplicate often proves possession, timing or context that the surviving copy cannot show. Before suppressing a group, ask what a reviewer, an auditor or a model trainer would lose if only one copy remained.
| Duplicate pattern | What the extra copies can prove | Lost if removed early | Keep it how |
|---|---|---|---|
| Same email in several mailboxes | Who received it, and whether it was filed, forwarded or left unread | Distribution and awareness by person or team | Keep one body, plus a list of every custodian and folder that held it |
| Same attachment on many emails | Which conversations the document was sent into, and when | The link between a document and each decision thread | Keep one file, plus parent-item references for every email family |
| Same record exported twice from one system | Usually nothing, if both exports have identical fields and timestamps | Little, after totals reconcile | Keep one, and log which export run it came from |
| Near-identical contract or quote versions | What changed between drafts, and who changed it | Negotiation history and the approved version | Treat as versions, not duplicates; keep all with order and dates |
| Duplicate invoice or payment entries in an ERP | That a duplicate was created and later caught or not caught | The exception itself, which may be the most useful record | Never deduplicate; flag and keep both with their resolution trail |
| Repeated chat message or ticket comment | A retry, a reminder or an escalation | How long an issue waited and how often it was chased | Keep timestamps for each occurrence even if text is stored once |
What has to be captured before access to the source ends?
Capture the full export, a checksum for every file, the export totals and the software notes needed to read it, in that order. Deduplication decisions can wait months; a closed subscription or a wiped server cannot be undone.
A checksum manifest is what lets you prove later that cleanup did not alter the preserved copy. BagIt (RFC 8493) is a widely used packaging format for this: each payload file is listed in a payload manifest with a checksum, a complete bag lists every payload file, and a valid bag is complete and every checksum verifies. You do not need to adopt BagIt itself, but the same two tests, everything listed and everything verifying, should pass before anyone touches duplicates.
Reconcile the export against the source system's own counts too. If the helpdesk reports a ticket total for a date range and the export holds fewer, a missing page of results looks exactly like a successful deduplication afterward. Record the totals on both sides with the date and the person who checked them.
The preservation-first duplicate review guide
Use these ten steps in order. Each has a record to keep, a pass test and a stop condition. Steps one to four happen before access ends; steps five to ten can happen on the preserved copy at any pace.
| Step | What to record | Pass test | Stop if |
|---|---|---|---|
| 1. Freeze and export | Source system, export method, date range, run date, operator | Export completes without errors or skipped items | The tool reports throttling, partial pages or size caps |
| 2. Fixity manifest | A checksum per file and the algorithm used | Every exported file appears once and every checksum verifies | Files exist that the manifest does not list |
| 3. Reconcile totals | Source counts and export counts per object type and period | Counts match, or each gap has a written reason | A gap has no explanation |
| 4. Location inventory | Custodian, mailbox or channel, folder path, parent item and family ID for every item | Each item has at least one location row | Items arrive with no source path |
| 5. Exact grouping | Hash of normalized content and the grouping scope chosen | Group members share the same hash and item type | Hashes were computed on files that were later modified |
| 6. Role classification | For each group, the pattern from the table above and what the copies prove | A reviewer has named a role for every group above a size you set | A group mixes distinct records, such as two invoices with one PDF |
| 7. Near-duplicate and thread review | Version order, inclusive messages and unique attachments | No unique text or attachment is suppressed | The threshold or ignore rules were never reviewed |
| 8. Action per group | Keep all, keep one with a location map, or keep one | Every suppressed copy maps to a kept item | Any copy would be dropped with no map entry |
| 9. Hold and rights check | Holds, disputes, excluded custodians and third-party content | Counsel or the records owner has signed off on the scope | A hold or open dispute covers the items |
| 10. Decision log | Who decided, when, and how to reverse it | The working set can be rebuilt from the preserved copy and the log | The preserved copy is scheduled for deletion |
Exact, near and thread duplicates need different rules
Only byte-identical content is a true exact duplicate, and even then the copies may sit in different contexts. Near-duplicates and email threads need human-reviewed rules, because similarity is a setting, not a fact.
The EDRM Processing Glossary defines deduplication as identifying and suppressing identical copies of messages or documents based on hash values or other criteria. Note the word suppressing: in review practice, duplicates are usually withheld from view, not destroyed. The same email in two mailboxes can also hash differently if headers or export formats differ, so decide and document what you hash, for example the message body and attachment hashes rather than the full file.
Microsoft Purview eDiscovery shows how much judgment sits inside near-duplicate and thread analysis. Its documentation describes a configurable similarity threshold and word-count limits, groups documents above the threshold with one marked as the pivot, flags inclusive emails that contain all unique content in a thread, and flags emails that have attachments not found elsewhere in the thread. It also lets a case ignore boilerplate text such as disclaimers. Every one of those settings changes what gets suppressed, so record the values used, and never treat a non-inclusive email as safe to delete when it carries a unique attachment.
Contract drafts, revised quotes and edited policies are versions, not duplicates. A version chain is often exactly what a buyer of negotiation or approval records wants, so keep all versions with dates and authors and let the review decide which one is final.
Custodian-level or global: which scope keeps the evidence?
Choose custodian-level grouping, or global grouping with a full location map. Plain global deduplication with no map is the one choice that cannot be undone later.
The EDRM guidelines for e-discovery processing describe custodian deduplication as keeping one copy per custodian, while global deduplication runs across all custodians and leaves a single copy. They also recommend fields such as all custodians and all source paths, so that the places a suppressed copy lived stay visible after deduplication.
This matters for licensing in a specific way. Suppose the only kept copy of a widely circulated pricing email came from an HR manager's mailbox, and HR mailboxes are later excluded from the licensed scope. Without a location map, excluding that custodian silently removes a record that twenty sales mailboxes also held. With the map, the reviewer promotes another custodian's copy and the record stays in scope.
- Use custodian-level grouping when who-knew-what matters, for example in dispute, compliance or account-management archives.
- Use global grouping with all-custodian and all-path fields when the volume is large and only the content matters, for example product documentation or published price lists.
- Decide the scope before hashing and write it into the decision log, because switching scope later changes every group.
Worked example (illustrative): a closing distributor's quote archive
This example is illustrative; the counts are invented to show the method, not observed results. A regional distributor is closing its sales office. Its email tenant ends in six weeks, and the owner wants to keep the quoting history: emails between reps and customers, the attached quote PDFs and the ERP sales orders that followed.
Weeks one and two cover steps one to four. The team exports 14 mailboxes, writes a checksum manifest and reconciles the export against the tenant's item counts; one mailbox comes up short by 3 percent, and a re-export with a narrower date range closes the gap. Every item gets a location row: mailbox, folder, thread ID and attachment family.
After the tenant closes, the team groups exact duplicates on message body and attachment hashes. About a third of items fall into groups, mostly internal forwards of the same customer request. The reviewer classifies the groups: forwards become keep-one-with-map, the quote PDFs become keep-one-with-parent-references, and quote revisions are reclassified as version chains because near-duplicate review shows changed quantities and dates.
The exception: the ERP export contains pairs of sales orders with the same customer, items and quantities on the same day. A hash-based cleanup would collapse them. The reviewer checks the order history and finds that some pairs were genuine repeat orders and some were accidental double entries that were later canceled. Both kinds stay, flagged, with their cancellation records, because the double-entry cases are the most useful records for anyone studying order exceptions.
The result is a working set roughly two thirds the size of the export, a location map that can rebuild every original view, and a preserved copy that still verifies against its manifest.
When does a hold or dispute stop the cleanup?
Stop deduplication of any items that are under a legal hold, tied to a dispute or likely to be requested in one, and ask counsel how to proceed. In those cases the form and completeness of what you kept can matter as much as the content.
Under U.S. Federal Rule of Civil Procedure 37(e), courts can act when electronically stored information that should have been preserved in anticipation or conduct of litigation is lost because a party failed to take reasonable steps to preserve it, and it cannot be restored or replaced through additional discovery. Rule 34(b)(2)(E) also says that, unless a request specifies a form, information is produced in the form in which it is ordinarily maintained or in a reasonably usable form. A preserved original plus a reversible decision log keeps both options open; an irreversible cleanup does not.
Other signals that call for a pause and a specialist review:
- A government inquiry, audit or insurance claim touches the period covered by the archive.
- The duplicates are financial entries, such as invoices, payments or credits, where the duplicate itself may be the evidence.
- Custodians have left and their mailboxes are the only source for some threads.
- The archive includes customer or third-party content whose contracts limit retention or reuse.
- Nobody can explain how the export tool treats duplicates by default.
Handing a reviewed archive to a licensing assessment
An archive that went through this review is easier to assess for any later use, including licensing, because a reviewer can see what was suppressed, why, and where each copy came from. Send the decision log and the location map along with the working set, and keep the preserved export until every use has been decided.
Archive owners who decide to explore licensing can have SourceX run the rights review, contracting, preparation and delivery for selected records. The starting point is a brief company assessment with no uploads, and no record leaves the archive before the owner approves the scope and terms and signs.
Frequently asked questions
Should we deduplicate before exporting to save time?
No. Export everything first, write a checksum manifest and reconcile totals while you still have access to the source. Deduplication can then run on a copy at any pace, and it stays reversible because the original is untouched.
Is a matching hash enough to call two emails duplicates?
Only for the content that was hashed. Two copies of the same message can sit in different mailboxes, folders and threads, and that context is lost if only the content hash is kept. Record every location before suppressing any copy.
Should duplicates be removed before or after de-identification?
Group duplicates on the original content first, then de-identify consistently across the group. Masking can make different records look identical, or make copies of one record look different, which corrupts the grouping if it runs afterward.
Can we delete the suppressed copies once the working set is built?
Keep them, or at least keep the preserved export and the location map, until holds, rights reviews and every planned use are settled. Storage for identical content can be shared, so keeping the evidence rarely costs much.
Are near-duplicates the same as duplicates?
No. Near-duplicates are usually versions, such as revised quotes or contract drafts, and the differences between them are often the useful part. Treat them as version chains with dates and authors rather than removing them.
Sources
- The OCFL 1.1 implementation notes say OCFL separates a stored file's content path from its logical path, which permits deduplication, and that later references to already-stored content should occur in the inventory's state block only. Source
- The OCFL 1.1 specification defines a version's state block as digest values, each mapped to an array of the logical paths that hold content with that digest. Source
- RFC 8493 (BagIt 1.0) has payload manifests that pair each payload file with a checksum; a complete bag lists every payload file in every payload manifest, and a valid bag is complete with every checksum verifying. Source
- The EDRM guidelines for e-discovery processing describe custodian-level and global deduplication and recommend fields listing all custodians and all source paths for a deduplicated item. Source
- The EDRM Processing Glossary defines deduplication as the identification and suppression of identical copies of messages or documents in a data set based on hash values or other criteria. Source
- Microsoft Purview eDiscovery documentation describes near-duplicate grouping with a configurable similarity threshold and a pivot document, inclusive emails, a flag for unique attachments in a thread, and text to ignore such as disclaimers. Source
- Microsoft Purview review set export documentation describes expanding an export to family items that share the same FamilyId. Source
- Federal Rule of Civil Procedure 37(e) addresses electronically stored information that should have been preserved in anticipation or conduct of litigation, is lost because a party failed to take reasonable steps, and cannot be restored or replaced. Source
- Federal Rule of Civil Procedure 34(b)(2)(E) says that if a request does not specify a form, electronically stored information is produced in the form in which it is ordinarily maintained or in a reasonably usable form. Source
See if your company qualifies
A short company assessment. No data uploads are needed.