Skip to content

Systems and records

Data preservation: attachment completeness before export links expire

By SourceX Editorial · Updated

10 sections

Short answer

To preserve attachments before their links expire, download the files themselves, not the links, while the source account is still live; then prove completeness. Reconcile every attachment reference in the export against a downloaded file, reject zero-byte and error-page downloads, record a SHA-256 checksum for each file in a manifest, and validate the package again after every copy. A link-only export is not an archive.

System export

Data preservation

This page covers data preservation: attachment completeness before export links expire.Illustrative

Key takeaways

  • A link-only export is not an archive; download the files while the account and signatures are still valid.
  • Your deadline is the earliest clock: signed link expiry, backup download window, deletion of the parent object or account closure.
  • Reject false successes such as zero-byte files and error pages saved under document names before reconciling.
  • Prove completeness per parent record and validate SHA-256 checksums after every copy, not just file counts.
  • Capture IDs, visibility, versions and human context with the bytes, and restrict files that are not yours to reuse.

Because many exports carry pointers to files rather than the files, and those pointers depend on an account, a session or a signature that ends. Once the subscription lapses, the workspace is deleted or the signature expires, the export still lists the attachment but nothing answers at the other end.

Slack is a clear case. Its help center says workspace data exports include links to files but not the files themselves, and that if a file, or the channel or workspace it was shared in, is deleted, the file will no longer be accessible through the link in the export. Zendesk documents a similar dependency for ticket attachments: an authenticated request to an attachment's content URL returns a redirect to a temporary download URL, so the address you can see in a ticket export is not a permanent file location.

Cloud storage links expire by design. Amazon S3 presigned URLs signed with Signature Version 4 can be valid for at most seven days, and for 12 hours at most when created in the console; a URL signed with temporary credentials stops working when those credentials expire. Google Cloud Storage V4 signed URLs accept an expiration of up to 604,800 seconds, which is seven days. Any export that hands you signed links has a deadline built into every row.

Which clock sets your real preservation deadline?

The earliest of several clocks, and it is rarely the contract end date. Write each one down with its source before planning the download, because the one that ends first decides the schedule.

Treat the deadline as a working day, not a date on a contract. Large attachment sets take longer to pull than most teams expect, because APIs throttle requests and some files fail and need retries. Plan to finish the first full pass with at least a third of the time left, so that the second pass, the reconciliation and the copy to a second location all happen while the source can still answer.

Which clock sets your real preservation deadline?
ClockDocumented exampleWhat it means for the plan
Signed link expiryS3 SigV4 presigned URLs last at most seven days; GCS V4 signed URLs at most 604,800 secondsDownload within the window of each export run, or regenerate links before they lapse
Backup download windowConfluence Cloud says its backup file download link is available for 14 daysStart the transfer the day the backup completes, not at the end of the window
Deletion of the parent objectSlack: a deleted file, channel or workspace makes the export's file link inaccessibleFreeze deletions and retention purges until downloads are verified
Account or subscription endVaries by vendor; confirm the date access ends in the contract and the admin consoleFinish downloads and verification before the cancellation date, with slack for retries
Credential lifetimeS3 URLs signed with temporary credentials stop working when those credentials expireUse credentials that outlive the transfer, issued to a named owner
People who can explain the filesThe admin or analyst who knows what the attachments mean leaves at shutdownCapture their context notes while they are still available

The attachment download and integrity verification runbook

This runbook turns a deadline into evidence. Each step produces something a later reviewer can check, and the pass condition says when to move on. A team can run it as is for a help desk, a project tracker, a chat workspace or a CRM.

Run steps one and two on the first day, even if the download itself must wait for credentials. A reference inventory taken while the system is live is the yardstick for everything after it; without it, nobody can say later whether the archive is missing anything.

Keep the outputs in plain formats, such as CSV for the reference and reconciliation tables and plain text for the manifest and README. They will need to be read long after the tools that produced them have been retired.

The attachment download and integrity verification runbook
StepActionEvidence producedPass condition
1. Inventory referencesParse the export and list every attachment reference with its parent record ID, file name, stated size and linkreferences.csv with one row per attachmentRow count matches the system's own attachment count or report, or the gap is explained
2. Set the deadlineRecord every clock from the deadline table and pick the earliestDated deadline note signed by the archive ownerDownload plan finishes before the earliest clock with time for retries
3. Download filesFetch each file with authenticated, rate-limited requests, following redirects to the temporary URL immediatelyFiles stored under a path that includes the parent record IDEvery reference has a download attempt logged with status code
4. Reject false successesFlag zero-byte files, HTML or JSON bodies saved under document names, and size mismatchesrejects.csv with reason per fileEvery reject is retried or explained
5. ReconcileJoin references to downloaded files by record ID and attachment IDReconciliation table: referenced, downloaded, rejected, missingMissing count is zero or each missing file has a documented reason
6. HashCompute SHA-256 for every file at download timePayload manifest with checksum and relative pathOne checksum line per downloaded file
7. PackagePlace files, manifest and metadata into a BagIt bagBag with bag-info and payload manifestBag validates as complete and valid
8. Copy and re-verifyCopy the bag to a second storage location and validate it thereValidation log from the destinationDestination checksums match the source manifest
9. Capture contextWrite a README: system, export date, field meanings, known gaps, who to askREADME in the bag's tag filesA newcomer can map any file to its record and purpose
10. Sign offArchive owner confirms the reconciliation and validation resultsDated sign-off noteSigned before the source system is closed

How should the download job behave so failures are visible?

It should fail loudly and record everything, because silent partial success is the most common way an attachment archive goes wrong. Every request needs a log line with the attachment ID, the parent record ID, the HTTP status, the bytes received and the time.

Fetch the temporary download URL as soon as it is issued rather than collecting links first and downloading later. Where a system redirects to a short-lived address, a batch of links gathered at the start of a long run can expire before the job reaches them, and the job then saves error responses instead of files.

Respect the source system's rate limits and back off when it signals throttling. A job that hammers the API may get its token revoked mid-run, which turns the remaining downloads into error pages. Save files under a path built from the parent record ID and the attachment ID, never from the file name alone, because file names repeat across records and a later download can silently overwrite an earlier one.

Finally, make the job restartable. It should skip files that already exist with a recorded checksum and retry only the failures, so a second pass on day six does not start from zero.

How do you prove the archive is complete and not just large?

By reconciling, record by record, what the source said existed against what you hold, and by validating checksums rather than file counts. A folder with the right number of files can still be wrong: a script that saved a login page or a 403 error under each file name produces a perfectly sized set of useless files.

BagIt, the packaging format published as RFC 8493, gives the two words you need. A complete bag contains every file listed in its payload manifests; a valid bag is a complete bag in which every checksum in every manifest has been verified against the corresponding file. The RFC's Payload-Oxum field, a byte and file count, is described as strictly an optimization: implementations must still run full checksum validation before calling a bag valid.

The Library of Congress bagit-python tool implements this. It generates SHA-256 and SHA-512 manifests by default, validates a bag with its validate option, and offers a fast mode that checks only the Payload-Oxum, which is useful as a quick look but not as proof.

Reconcile at the level of the parent record as well as the whole archive. If the export says ticket 48213 had three attachments and the archive holds two, the archive is incomplete even if the overall totals look close, because that ticket's history no longer makes sense without the missing file.

Which checksum should you record, and when should you check it again?

Record SHA-256 for each file at the moment of download, then check it after every transfer and on a schedule. SHA-256 is widely supported by preservation tools and command-line utilities. Avoid SHA-1 for new manifests: NIST announced in December 2022 that SHA-1 should be phased out by December 31, 2030 and replaced with SHA-2 or SHA-3.

The National Digital Stewardship Alliance's fixity guidance, Checking Your Digital Content, advises creating fixity information once materials are received, treats moving data between storage systems as a point where content can be harmed, and says fixity information has to be used through scheduled audits. For a shutdown archive, that means three checks at minimum: at download, after each copy to a new location, and at a fixed interval while the archive is retained.

Keep the manifest with the files and a second copy apart from them. A checksum stored only next to the file it protects can be damaged or overwritten along with it.

What context disappears with the system if you wait?

The files survive more often than the meaning around them. When the source system closes, so do the screens that showed which record each file belonged to, who uploaded it, when, whether it was internal or customer-visible, and which version replaced which.

Capture these alongside the bytes:

  • Parent record ID, comment or message ID, and the attachment's own ID, so the file can be rejoined to its history.
  • Original file name, uploader role, upload timestamp and visibility setting (internal note, customer-visible, private).
  • Version chains for documents that were replaced, and inline images embedded in rich-text comments, which some exports store separately from attachments.
  • The software needed to open unusual formats, such as CAD drawings, proprietary diagram files or email message files, recorded in the README.
  • Notes from the person who knows what the attachments are for, captured before that person leaves.

Worked example: a help desk closing in ten days (illustrative)

This example is hypothetical; its figures are illustrative counts and durations. A company is closing its help desk account in ten days. The ticket export lists 41,300 attachment references across 18,900 tickets. The team sets the deadline at day seven to leave room for retries, downloads through the API with a service account owned by the archive lead, and follows each redirect to the temporary download URL as soon as it is returned.

The first run logs 41,300 attempts. The reject check flags 1,140 files: 960 are small HTML bodies saved under PDF and PNG names because a token expired mid-run, and 180 are zero-byte. A second run with a fresh token recovers 1,090. Reconciliation leaves 50 references with no file; 44 point to attachments that were deleted inside the help desk before the export, and six belong to tickets that were redacted under a deletion request. Each is listed with its reason.

The exception is a group of 300 attachments uploaded by customers that contain scanned identity documents and third-party contracts. They are downloaded and preserved under the company's retention obligations, but tagged as excluded from any secondary use until privacy and contract review decide otherwise. The bag is then copied to a second storage location, validated there, and signed off on day eight.

Which attachment sets need counsel, privacy or security review before copying?

Stop and get advice when preservation itself is legally constrained or when what you are copying is not yours to keep. Speed matters, but copying the wrong material quickly creates a different problem.

  • A litigation hold, regulator request or dispute covers the records: counsel decides scope and chain of custody before anyone filters or deletes.
  • Attachments include health, financial account, identity or children's information: narrow to what the retention obligation requires and restrict access.
  • Files belong to customers or third parties under contracts that limit retention or reuse: preserve if required, but tag them as restricted.
  • The only available credentials belong to a departing employee or a former vendor: get a properly owned account first.
  • Files are flagged as malware by the source system: quarantine them rather than downloading them into the main archive.

If the archive may later support an AI data license

A verified archive is only the starting point; possession of the files is not the same as the right to license them, and de-identification does not create that right on its own. Rights, privacy and contract review come next.

If an archive like this later becomes a licensing candidate, SourceX covers qualification, rights review, contracting, data preparation, delivery and payment for the business licensing it to AI developers. The start is a short assessment with no data upload, and large deliveries can remain in the company's storage or move on encrypted drives, since SourceX does not host multi-terabyte datasets.

Frequently asked questions

Do Slack exports include file attachments?

Slack's help center says standard workspace exports include links to files, not the files themselves. If a file, or its channel or workspace, is deleted, the link in the export stops giving access. Download the files while the workspace is live.

How long do presigned or signed download links last?

Amazon S3 presigned URLs signed with Signature Version 4 last at most seven days, and at most 12 hours when created in the console. Google Cloud Storage V4 signed URLs allow up to 604,800 seconds, or seven days. Many applications issue much shorter links.

What is the difference between a complete and a valid archive?

In BagIt terms, a complete bag holds every file listed in its payload manifests. A valid bag is complete and every checksum has been verified against its file. Counts and sizes alone only suggest completeness.

Which checksum algorithm should I use for preserved attachments?

SHA-256 is a practical default and is generated by common preservation tools such as bagit-python. Avoid SHA-1 for new work, since NIST has set December 31, 2030 as the date to complete its phase-out.

How do I detect attachments that downloaded as error pages?

Check each file's size and its first bytes against the expected type. HTML or JSON content saved under a PDF or image name, or zero-byte files, usually mean an expired token or a permission error. Log and retry them before reconciling.

Sources

  • Slack says workspace data exports include links to files but not the files, and a deleted file, channel or workspace makes the export's file link inaccessible. Source
  • Zendesk documents that an authenticated request to an attachment's content URL returns a 302 redirect whose Location header points to a temporary download URL. Source
  • Amazon S3 presigned URLs created with Signature Version 4 can be valid for up to seven days, up to 12 hours from the console, and expire with temporary credentials. Source
  • Google Cloud Storage V4 signed URLs accept expiration values up to 604,800 seconds (seven days). Source
  • Atlassian says the Confluence Cloud backup download link is available for 14 days and attachments are included only when selected. Source
  • RFC 8493 defines a complete bag and a valid bag, and says Payload-Oxum is an optimization that does not replace checksum validation. Source
  • The Library of Congress bagit-python tool generates SHA-256 and SHA-512 manifests by default and provides full and fast validation modes. Source
  • NIST announced in December 2022 that SHA-1 should be phased out by December 31, 2030 in favor of SHA-2 or SHA-3. Source
  • The NDSA's Checking Your Digital Content advises creating fixity information on receipt, flags transfers as a risk point and calls for scheduled audits. Source

See if your company qualifies

A short company assessment. No data uploads are needed.

See if your company qualifies