Privacy and preparation
Dataset acceptance evidence packet: what to ship with a delivery
By SourceX Editorial · Updated
9 sections
Short answer
A buyer accepts a dataset faster when the supplier ships evidence the buyer can rerun, not assurances. The packet holds a checksummed file manifest, a reconciliation of delivered counts against the agreed scope, validator output with the rules that produced it, sampled review records, the schema and its change history, and an exception log, all tied to the acceptance criteria by a one-page review evidence index.
Checklist
Dataset acceptance evidence
Key takeaways
- Files that open prove little; acceptance turns on comparisons with the source system, the scope and the schema history.
- Reconcile counts by period and record type, and check parent and child links, before running any other check.
- Ship validator results as data with their rule configuration so the buyer can rerun them and get the same answer.
- A review evidence index maps every acceptance criterion to the file that answers it and the person who produced it.
- Keep corrected failures in the output; a visible fix is more convincing than a clean result with no history.
- Rights, privacy and scope findings hold the delivery; small measured quality defects ship with a disclosed exception.
What evidence packet supports a buyer's dataset acceptance decision?
The packet that gets a dataset accepted quickly is one that lets the buyer reproduce every acceptance check without asking the supplier a question. It holds a file manifest with sizes and checksums, a reconciliation of what was delivered against the agreed scope, machine-readable validator output for each acceptance criterion, records of any sampled human review, the schema and its change history, and an exception log. An index ties each acceptance criterion to the file that answers it.
Readable files are where acceptance starts, not where it ends. A buyer's reviewer is asking whether the package is complete, internally coherent and usable for the task in the agreement. Without a packet, the reviewer rebuilds that answer from scratch, finds questions only the supplier can answer, and the acceptance window runs out in email.
This guide is written for the company preparing records for license. The same structure works for a buyer who wants to specify what a supplier must deliver, and it complements, rather than replaces, the acceptance criteria and procedure in the contract.
What defect could change the acceptance decision even if every file opens?
The most damaging defects are the ones a file viewer never shows: records missing from a period, the same case exported twice, timestamps that silently shift by hours, outcome labels left empty for open cases, attachments listed but not delivered, and a field whose meaning changed partway through the history. Each one can turn a usable dataset into a misleading one for training or evaluation.
At a glance
What defect could change the acceptance decision even if every file opens
These defects share a pattern. They are only visible when the delivery is compared with something outside itself: the source system's counts, the agreed scope, the schema history, the attachment index. That is why the packet is mostly comparisons, not descriptions.
- Truncated period: an export job hit a row limit or a date filter, so one month holds a fraction of its usual cases.
- Duplicate cases: a re-run export appended rather than replaced, so a set of cases appears twice with different row IDs.
- Timestamp shift: local times exported without an offset, so durations and orderings are wrong across time zones or daylight saving changes.
- Unlabeled outcomes: open or abandoned cases mixed with closed ones, with no field that tells them apart.
- Orphan references: records point to attachments, notes or line items that are not in the delivery.
- Meaning drift: a status or category field was redefined mid-history, so the same value means different things before and after a date.
Reconciling the delivery to the scope both sides signed
Reconcile counts by period and by record type against the source system before anything else, because every later check assumes the right records are present. The reconciliation table shows, for each month or week in scope, the count in the source system, the count exported, the count removed by agreed filters, the count removed by de-identification or privacy rules, and the count delivered. Each row should balance, and any row that does not gets an entry in the exception log.
Use a count the buyer could, in principle, recompute: a saved report or query in the source system, with its filter written out and the date it was run. A screenshot of a dashboard total is weaker evidence, because nobody can tell which filters it applied.
Reconcile relationships as well as rows. If a case has messages, line items or attachments, report how many parent records have zero children where children are expected, and how many child records point to a parent that is not in the delivery. Those two figures catch most broken joins.
Validator output a buyer can rerun instead of trust
Ship the validator results as data, with the rules that produced them, so the buyer can rerun the same rules on the same files and get the same answer. Open-source tools already work this way. Deequ, from AWS Labs, describes itself as unit tests for data: checks define constraints, the verification run computes metrics, and the result reports each constraint that failed with its observed value. Great Expectations turns its Expectations and Validation Results into Data Docs, static web pages a reviewer can read without installing the tool.
At a glance
Validator output a buyer can rerun instead of trust
Whatever tool you use, each check in the output should carry the same fields, so a reviewer can sort and filter them. Where the tabular files have a declared schema, include it. The Frictionless Table Schema standard, for example, lets a schema declare a primary key, which it defines as one or more fields that uniquely identify each row and cannot be null, and list the strings that should be treated as missing values. A declared primary key turns the duplicate-case defect into a single, rerunnable check.
- Check ID and the acceptance criterion it implements, quoted or referenced by clause number.
- The rule as executable configuration, not prose: the constraint, the column and the threshold.
- Scope: which files, record types and periods the check ran on.
- Result: pass, fail or warning, the observed value and the threshold it was compared with.
- Run metadata: run ID, tool and version, the manifest checksum of the files checked, and the time of the run with its UTC offset.
- Disposition for any failure: fixed and rerun, accepted under a documented waiver, or open.
The review evidence index, line by line
The index is the first file a reviewer opens: one row per piece of evidence, pointing to where it lives in the packet and which acceptance criterion it answers. Adapt the file names to your delivery; the structure is what matters.
| Evidence item | File in the packet | What it proves | Criterion it answers | Produced by |
|---|---|---|---|---|
| File manifest | manifest file listing path, size in bytes and SHA-256 per file | Every delivered file arrived complete and unaltered | Delivery completeness and integrity | Supplier export job, checked by the buyer on receipt |
| Scope reconciliation | reconciliation table by period and record type | Delivered counts balance against source counts and agreed filters | Scope and volume | Supplier data owner |
| Validator results | machine-readable results file plus the rule configuration | Each automated criterion passed, or failed with a disposition | Field completeness, validity, uniqueness, ranges | Supplier validation run, rerunnable by the buyer |
| Rendered validation report | static report pages generated from the same run | The same results in a form a non-engineer can read | Reviewer sign-off | Validation tool |
| Schema and change history | schema file and a dated list of field changes | Every field's type, meaning and the dates its meaning changed | Field definitions and label consistency | Supplier system administrator |
| Sampled review record | sample frame, sample IDs, reviewer findings | Human checks on labels, text quality and residual personal data | Sampled accuracy and privacy criteria | Supplier reviewer or processing partner |
| Attachment integrity report | counts of referenced, delivered and missing attachments | No orphan references, or a list of each one | Attachment completeness | Supplier export job |
| Exception log | dated list of every known defect, waiver and open item | What is wrong, how much, and what was agreed about it | Disclosure of known defects | Supplier data owner |
| Packet index and version | this index with dataset version and packet date | Which delivery the evidence describes | Traceability | Supplier data owner |
Timestamps, schema history and attachments: where quiet defects hide
Three checks catch defects that pass most schema validators. Timestamps come first: require every timestamp to carry an explicit offset or be in UTC. RFC 3339, the internet profile of ISO 8601 timestamps, defines a numeric offset as local time minus UTC, so 1996-12-19T16:39:57-08:00 names the same instant as 1996-12-20T00:39:57Z. Report the share of timestamps without an offset, events that occur before the record was created, and durations that come out negative.
Schema history comes second. Export the field definitions as they stand, then list every change to fields in scope with its date: added values, renamed values, fields that became required. A buyer training on several years of records needs to know that a status value meant something else before a certain date, and the export alone will not say so.
Attachments and file integrity come third. The BagIt packaging format in RFC 8493 pairs a payload manifest of checksums and file paths with an optional Payload-Oxum value giving the total payload size in octets and the file count, which tools can use to detect an incomplete bag quickly before running full checksum validation. The Frictionless Data Resource standard has similar bytes and hash properties, and the Croissant metadata format strongly recommends SHA-256 checksums for every file in a versioned dataset. Any of these lets the buyer confirm on receipt that the files match the manifest.
Worked example: an invoice-exception delivery from an accounts payable system (illustrative)
Suppose a distributor licenses three years of invoice exceptions from its accounts payable system: invoices that failed a match against the purchase order or receipt, with the resolution notes and the final action. Every figure here is illustrative.
The reconciliation shows that one month delivers about 40% of the cases the source report counts. The exception log explains it: an export filter excluded invoices still open at month end, which the agreement says should be included with an open-status flag. The supplier fixes the filter, reruns the export and the validators, and the month now balances. The original failure and the rerun both stay in the validator output, so the buyer sees the correction rather than a clean result with no history.
The schema history notes that a new exception reason, price variance above tolerance, was split out of a general mismatch reason in the second year. The packet says so, and the buyer decides to map both periods to a common label for training and keep the finer label for evaluation on recent data.
The exception: the attachment report shows about 3% of invoice image references with no delivered file, all from a storage migration period. The agreement allows a documented shortfall below an agreed limit, so the supplier lists every missing reference in the exception log and the buyer accepts with that waiver recorded. If the missing files had concentrated in one exception reason, the same shortfall would have skewed the data for that reason, and the right call would have been to fix or exclude that slice before acceptance.
Findings that should hold the packet back rather than ship with a note
Hold the delivery when a finding affects rights, privacy or scope rather than quality. Residual personal information found in sampled review, records from a system or period the agreement does not cover, or a source the company has no right to license are not defects to disclose in a note. They change what can lawfully be delivered, and they go to the company's counsel and privacy owner before anything ships.
Narrow rather than delay when a defect is confined to one slice. If one period or record type fails reconciliation and cannot be fixed in time, deliver the slices that pass with the failed slice excluded and documented, provided the agreement allows partial delivery.
Ship with a disclosed exception when the defect is small, measured, understood and within what the agreement tolerates. The exception log is the right place for it, with counts, affected IDs and the reason. A disclosed, bounded defect is easier for a buyer to accept than a clean-looking packet that turns out to have hidden one.
Assembling the packet when licensing records through SourceX
When a company licenses selected records directly to SourceX, the records pass through rights review, contracting and preparation before delivery and payment. Because the company approves the scope and terms and signs before any data is delivered, the packet's reconciliation has a signed scope to balance against.
De-identification is performed by SourceX through approved processing routes, so the de-identification row of the reconciliation comes from that step rather than from the company's own export. Deliveries too large to move stay in the seller's storage or ship on encrypted drives, which is when a checksummed manifest earns its place: the files may never pass through a shared bucket. Whether a company qualifies is answered by a short company assessment that needs no data uploads.
Frequently asked questions
What is a dataset acceptance evidence packet?
It is the set of files a supplier ships alongside a dataset so the buyer can decide on acceptance without rebuilding the checks. It typically holds a checksummed manifest, a scope reconciliation, validator results with their rules, sampled review records, the schema history, an exception log and an index. It supports the contract's acceptance criteria rather than replacing them.
Is a checksum manifest enough for acceptance?
No. A manifest shows the files arrived complete and unaltered, but it says nothing about whether the right records are in them. Missing periods, duplicate cases, shifted timestamps and broken references all pass a checksum check. Reconciliation and validator output are what show the contents are usable.
Should the supplier or the buyer run the validators?
Both, ideally with the same rules. The supplier runs them before shipping and includes the results and the rule configuration; the buyer reruns the same rules on receipt. Matching results confirm the packet; differences point straight to an environment or version problem.
Should failed checks be removed from the validator output after they are fixed?
Keep them. Show the original failure, the fix and the rerun, each with its run ID and time. A history of corrected failures is more convincing to a reviewer than a clean result with no history, and it documents what changed between runs.
Does the packet need to include personal data findings?
It should include the results of sampled checks for residual personal information, described in counts and categories, without copying the personal information itself into the packet. Any confirmed finding should go to the company's privacy owner and counsel before the delivery ships.
Sources
- Deequ is a library for defining unit tests for data: checks define constraints, a verification run computes metrics, and the result lists constraints that failed with their observed values. Source
- Great Expectations Data Docs translate Expectations, Validation Results and other metadata into human-readable documentation saved as static web pages. Source
- The Frictionless Table Schema standard defines a primary key as one or more fields that uniquely identify each row and cannot be null, and a missingValues list of strings to treat as missing. Source
- The Frictionless Data Resource standard defines a bytes property for file size and a hash property that defaults to MD5 and can name other algorithms with a prefix. Source
- The Croissant 1.1 specification lets each FileObject carry a SHA-256 checksum and strongly recommends recording checksums for all files in versioned datasets. Source
- RFC 8493 defines BagIt payload manifests of checksums and file paths and an optional Payload-Oxum giving payload octet count and file count, used to detect incomplete bags before checksum validation. Source
- RFC 3339 defines internet timestamps as a profile of ISO 8601, with numeric offsets calculated as local time minus UTC. Source
See if your company qualifies
A short company assessment. No data uploads are needed.