Systems and records
How do you prove an archive export was not silently truncated?
By SourceX Editorial · Updated
10 sections
Short answer
Prove an archive export was not truncated with three independent checks taken before access ends: control totals from the live system reconciled by object and month, a log showing every extraction ended on the system's documented end-of-results flag rather than a cap, and a random replay sample checked in both directions. Then freeze the evidence in a checksum manifest; a checksum alone cannot show completeness.
System export
Archive preservation proof
Key takeaways
- A checksum proves a file is unchanged since hashing, not that it holds every record.
- Search caps, early pagination stops and spreadsheet row limits truncate exports without raising an error.
- Control totals must come from the live system through a different route than the export, by object and month.
- Replay in both directions: forward finds omissions, reverse finds altered or misattached records.
- Sign off on hashed reconciliation and replay results, never on an export status of complete.
- Assemble the packet before shutdown, because two of its three checks need the source to answer.
How do you prove an archive export was not silently truncated?
You prove it with three independent kinds of evidence that a truncated export cannot fake at the same time: control totals captured from the live system and reconciled against the archive, a cap and pagination check showing no limit stopped the extraction early, and a sampled replay that looks up randomly chosen records in both directions. A checksum manifest then freezes the result so nobody can alter it later.
Each piece covers a gap the others leave. Counts catch missing volume but not a swapped or duplicated record. Replay catches missing or wrong records that counts hide, but only for the records sampled. The cap check explains why a gap happened, which is what tells you whether the fix is a re-run or a different export route.
This is the archive preservation proof of completeness that a later reviewer, an acquirer or a data buyer can inspect without access to the old system. It has to be assembled before access ends, because two of its three parts need the live source to answer.
Why a valid checksum says nothing about missing records
A checksum proves the file you hold is the file that was hashed, not that the file holds everything. NIST's Secure Hash Standard, FIPS 180-4, describes hash digests as a way to detect whether a message has changed since the digest was generated. If an export stopped at record 10,000, the hash of that short file is perfectly valid and will verify forever.
Export tools that ship checksums inherit the same limit. Google Vault, for example, includes a checksum file listing MD5 hash values for the files in an export, which lets you confirm the download arrived intact. It does not tell you whether the search behind the export matched every message you meant to keep.
The BagIt packaging format, RFC 8493, makes the distinction explicit. A complete bag has every file its manifests list, and a valid bag is a complete bag whose checksums all verify. Its Payload-Oxum field, a byte and file count, is described as strictly an optimization for spotting incomplete bags quickly; the RFC says implementations must still run full checksum validation before calling a bag valid. Both definitions are about the package against its own manifest. Neither says anything about the package against the source system, which is the question truncation raises.
Which limits cut exports short without an error?
Most truncation comes from documented limits that return a normal-looking response, not from crashes. The table lists the ones that recur in business systems, with the documented behavior, the symptom you will see in the archive and the test that exposes it.
Treat any record count that lands on a round number, or on a value printed in a vendor's limits page, as suspect until the reconciliation explains it. A count of exactly 10,000 or 1,000 from a search-based export is the commonest sign that a cap, not the data, decided the size.
| Cause | Documented behavior | Symptom in the archive | Test that exposes it |
|---|---|---|---|
| Search result cap | HubSpot's CRM search endpoints return at most 10,000 total results per query; Zendesk's Search API returns up to 1,000 results per query while its count field still reports the true total | Count equals the cap exactly; newest or oldest records missing | Compare against an unfiltered total; split the query by date range until every slice is under the cap |
| Pagination stopped early | Salesforce query results arrive in batches with done set to false and a nextRecordsUrl until the last batch; Zendesk incremental exports continue until end_of_stream is true | Count is a multiple of the batch or page size | Log the final page flag; fail the job unless done or end_of_stream was true |
| Last page guessed from page size | Zendesk advises against using the count field to detect the last page because pages can hold more than the usual count | Job ended on a short page that was not the last | Stop only on the documented end flag, never on a short page |
| Spreadsheet row ceiling | Excel worksheets hold 1,048,576 rows by 16,384 columns | CSV opened and resaved in Excel ends at that row | Count lines in the raw file with a tool, not a spreadsheet |
| Deleted or archived records excluded | Salesforce's queryAll includes deleted records that match the query; the ordinary query does not | Totals match the active list but not the historical report | State which statuses the export includes and reconcile per status |
| Download window lapsed | Google Vault export files are available for 15 days after the export starts | Some files in a multi-part export never downloaded | Reconcile the downloaded file list against the export's own file list |
What should the count, checksum and sampled replay evidence packet include?
It should hold enough that someone who never saw the source system can repeat each check and reach the same verdict. That means the reference numbers, how they were captured, the reconciliation, the replay sample and its results, the manifest, and a signed decision, each stored as a plain file with a date.
Use the packet below as is. Columns describe what each item records, how to produce it and the condition under which it passes. If an item cannot pass, the packet records the exception rather than leaving it out; an explained gap is evidence, a missing item is not.
| Packet item | What it records | How to produce it | Passes when |
|---|---|---|---|
| 1. Scope statement | Systems, objects, statuses and date range the archive is meant to cover | Written by the archive owner before extraction | Every later count refers to this scope |
| 2. Control totals | Record counts per object per month, plus one summed field such as invoice amount or hours where it exists | Captured from the live system through a different path than the export, such as a native report | Captured with timestamp, path and screenshot before extraction starts |
| 3. Extraction log | Each request, page or batch with its count and final end-of-results flag | Written by the export job | Every object ends on a documented end flag; no page equals a known cap |
| 4. Reconciliation table | Control total, archive count and difference for every object and month | Join items 2 and 3 | Difference is zero or each nonzero cell has a written reason |
| 5. Boundary check | Lowest and highest IDs and earliest and latest dates per object | Query source and archive for minimum and maximum | Source and archive boundaries match |
| 6. Forward replay sample | Randomly chosen source records looked up in the archive | Draw IDs from the live system; search the archive | Every sampled record found with matching key fields |
| 7. Reverse replay sample | Randomly chosen archive records looked up in the source | Draw from the archive; open each in the live system | Every sampled record exists in the source with matching fields |
| 8. Manifest | SHA-256 for every archive file, plus a hash of items 1 to 7 | Hash at the end of extraction; package as a BagIt bag if possible | Manifest validates after every copy |
| 9. Exceptions register | Each known gap: deleted, out of scope, unrecoverable | Filled from items 4, 6 and 7 | No unexplained gap remains |
| 10. Sign-off | Who reviewed the packet, the verdict and the date | Archive owner and one independent reviewer | Signed before source access ends |
Capturing control totals while the source still answers
Take the reference numbers first, from the live system, through a route the export does not use. If the export uses an API, take totals from the product's own reporting screens or an administrator count; if it uses a bulk export file, take them through the API. Two routes that agree are much stronger than one route checked against itself.
Count by object and by month rather than one grand total. A grand total can match by accident when one month is missing and another is duplicated, and a monthly grid shows exactly where a cap or a time filter bit. Where the system holds a numeric field that sums naturally, such as invoice amounts, hours or quantities, record its monthly sum too. Counts catch missing rows; sums also catch rows that were cut mid-field or loaded twice.
Record the boundaries as well: the lowest and highest record ID and the earliest and latest created date for each object. An export that ends at a suspiciously early date for the newest records is the classic footprint of a sorted search that hit its cap.
Write down the time each total was taken. Live systems keep changing, so the reconciliation compares like with like only if the export's own cut-off time is recorded next to the control total.
State the time zone of every boundary as well. A report that groups by month in the account's local time and an API that filters in UTC will disagree on the records created in the hours around midnight at each month end. That disagreement looks like a small, regular gap at every month boundary; once the time zones are aligned it disappears, and any gap that remains is real.
Sampled replay: what to check in each direction
Replay means taking specific records and checking that each exists, intact, on the other side. Run it in both directions, because each direction catches a different failure. Forward replay, from the source to the archive, catches omissions. Reverse replay, from the archive to the source, catches records that were altered, duplicated or attached to the wrong parent during extraction.
Draw the sample at random within strata rather than from the top of a list. Stratify by object and by period, and add deliberate edge picks: the first and last days in scope, the largest records, records with long text or many child rows, and records changed close to the extraction time. Edges are where caps, time filters and encoding faults show up.
For each sampled record, compare a fixed set of fields chosen in advance: the key, created and closed dates, status, owner, the count of child records such as comments or line items, and one long text field compared in full.
A clean sample supports a decision; it is not proof that nothing is missing. NIST's Engineering Statistics Handbook describes acceptance sampling as drawing a random sample from a lot to decide whether to accept or reject it, and stresses that the purpose is that decision rather than an estimate of the lot's quality. Here the archive is the lot. Write the conclusion that way: what was sampled, how, the acceptance rule and the result, never a claim that no record anywhere is missing. The counts and end-of-results flags carry completeness at full scale; the replay shows the records behind those counts are the right ones. For a regulatory hold or a disputed transaction, have a statistician or auditor size the sample against the risk.
Checklist · 4 items
- Decide the sample size and the acceptance rule before drawing, and write both into the packet.
- Use a zero-defect rule for record existence: one missing sampled record fails the slice and triggers a full re-check of that slice.
- Allow documented differences only where the source changed after the extraction time, and record each one.
- Keep the sampled IDs, the comparison output and who ran it, so the replay can be repeated.
0 of 4 ticked. Tick each item as you go.
Worked case (illustrative): a CRM export before the subscription ends
This case is hypothetical and every figure in it is an illustrative count or duration. A distributor is ending its CRM subscription in 21 days and wants a complete archive of contacts, companies, deals and activity history for the last five years.
Starting state: an analyst has already pulled contacts through the CRM's search API into a CSV and reports 10,000 contacts. The native contact report on the live system shows 26,400. Evidence: the extraction log shows one query sorted by last modified date, ending on a normal response. Applicable rule: the vendor documents a 10,000 result ceiling per search query. Decision: the first file is rejected as truncated, not merely incomplete, and the job is rebuilt to query one creation month at a time, with each slice checked against its monthly control total and required to stay under the cap.
Outcome check: the rebuilt export reconciles to the monthly grid in every month but one, where it holds 31 fewer contacts. Forward replay finds two sampled contacts from that month absent. The exceptions register traces all 31 to a bulk deletion an administrator ran during the extraction window, confirmed in the system's audit log, and records them as deleted after the control total was taken. Reverse replay of 200 archive records finds every one in the source with matching key fields.
Counterexample: the deal object reconciled perfectly on counts, yet reverse replay showed 14 sampled deals carrying the wrong company because the join used company names, which repeat. Counts alone would have passed it. The fix was to re-export with internal IDs and repeat the replay before sign-off on day 12.
What disappears if the proof waits until after shutdown?
Almost everything that makes the proof independent. After access ends, the archive can only be checked against itself, which is the circular test this packet exists to avoid.
- Live control totals and native reports, the second route that makes counts meaningful.
- The ability to look up a sampled ID in the source, so forward and reverse replay become impossible.
- Audit logs that explain deletions, merges and permission changes during the extraction window.
- Export files with limited download windows, such as Vault exports available for 15 days.
- Administrators who know which statuses, archived objects and custom fields were in use and why a filter was set.
Which record proves the decision, not just the status?
The signed reconciliation and replay results prove the decision; a status field does not. An export screen showing complete, a green check or a job that exited without errors only says the process finished. It says nothing about whether the process asked for everything, which is the failure every row of the limits table produces.
Link the sign-off to evidence by hash. Hash the reconciliation table, the extraction log and the replay results, list those hashes in the manifest, and record the manifest's own hash in the sign-off. Anyone checking later can then confirm that the verdict refers to these exact files. Keep one copy of the manifest apart from the archive, so damage to one location cannot erase both the data and the proof.
The sign-off note itself can be short: the scope it covers, the reconciliation verdict per object, the replay rule and result, the open exceptions with their reasons, the manifest hash, and two names with dates. A reviewer who disagrees later can then point to the exact line and file they dispute, which is the point of keeping the verdict separate from the evidence.
Where a completeness packet helps in a data licensing review
A licensing review asks first whether the company may license the records, and only then whether the records are what they claim to be. The packet answers the second question; it does not create rights, and de-identification alone does not either.
When a company licenses operational records directly to SourceX, the qualification, rights review, contracting, preparation, delivery and payment are handled by SourceX. Its first step is a short company assessment that does not require uploading any data. Buyers typically want two or more years of consistent history from one system, which is why an archive that can prove its years are whole is easier to scope.
Frequently asked questions
Does a matching checksum prove an export is complete?
No. A checksum shows the file has not changed since it was hashed, so a truncated file hashed at export time still verifies. Completeness needs a comparison with the source, such as control totals and sampled lookups taken while the system is live.
What is the most common cause of a quietly truncated export?
A documented result limit that returns a normal response. Search endpoints such as HubSpot's CRM search or Zendesk's Search API cap results per query, so an export built on one query stops at the cap without an error. Splitting the query by date range fixes it.
How many records should a replay sample check?
Decide the size and acceptance rule before drawing, based on what the archive will be used for. A zero-defect rule per slice is a practical default for record existence. For high-stakes uses, have a statistician or auditor size the sample against the risk.
Why replay in both directions?
Forward replay from source to archive finds records the export missed. Reverse replay from archive to source finds records that were altered, duplicated or attached to the wrong parent. Counts can match while either failure is present.
Can completeness be proven after the old system is shut down?
Only weakly. Without the source you can check the archive against itself and its own manifest, but not against what the system held. Capture control totals, logs and replay results before access ends.
Sources
- NIST FIPS 180-4 says hash digests are used to detect whether messages have been changed since the digests were generated. Source
- RFC 8493 defines complete and valid bags and calls Payload-Oxum strictly an optimization that does not replace checksum validation. Source
- Google Vault exports include a checksum file listing MD5 hashes for the export's files, and export files are available for 15 days. Source
- HubSpot's CRM search endpoints return at most 10,000 total results per query. Source
- Zendesk's Search API returns up to 1,000 results per query, and the count field reports the true total. Source
- Zendesk incremental exports continue with the after cursor until end_of_stream is true, and the count should not be used to detect the last page. Source
- Salesforce returns query results in batches with done set to false and a nextRecordsUrl until the final batch. Source
- Salesforce's queryAll resource includes deleted records that match the query. Source
- Excel worksheets hold at most 1,048,576 rows by 16,384 columns. Source
- NIST's Engineering Statistics Handbook describes acceptance sampling as using a random sample to decide whether to accept a lot, not to estimate its quality. Source
See if your company qualifies
A short company assessment. No data uploads are needed.