Privacy and preparation
Dataset clean room evaluation: can a buyer test utility without a copy?
By SourceX Editorial · Reviewed by Noah Loul ·
10 sections
Short answer
A buyer can test most of a dataset's utility without a copy by running approved analyses in a clean room and receiving only agreed outputs: coverage counts, field completeness, summary statistics and model scores on a held-out slice. The data owner should fix the questions, dataset version, output limits, access dates, logging and deletion evidence in a written brief before granting access.
Evaluation split
Dataset clean room
Key takeaways
- Most pre-purchase utility questions can be answered with counts, rates and scores that never expose a record.
- Clean room platforms enforce what runs and what leaves differently, so write your own limits into a brief.
- A manifest with digests, a configuration export and query logs let both sides prove what was evaluated.
- Aggregate outputs need a minimum group size, a fixed dimension list and a cap on runs.
- Exporting trained weights or keeping records is a licensing decision, not part of an evaluation.
- Close every evaluation with a written record of revocation, recomputed digests and deletion.
Can a buyer judge your dataset's utility without taking a copy?
Often, yes. A buyer can answer most pre-purchase utility questions inside a clean room: whether the records cover the cases it cares about, whether labels and outcomes are filled in, how fields are distributed, and how a baseline model scores on a held-out slice. What the buyer receives is a set of approved outputs, not the records.
The approach fails when the buyer's real question needs the records themselves to leave, for example a full fine-tuning run whose weights the buyer wants to keep, or a manual read of thousands of documents. Those are training or licensing decisions, not evaluation, and they belong in a signed agreement.
This guide is written for the data owner. It turns the evaluation into a requirements brief you issue before any access is granted, so approval, integrity, access, revocation and retention are each tied to a record both parties can check later.
What clean room platforms enforce, and what they leave to you
Each major platform restricts what can run and what can leave, but in different places. Read the controls as building blocks; none of them decides on its own whether a particular buyer should see a particular output.
Two details matter most for an evaluation. Google's BigQuery documentation says analysis rules can be set only on views, so a table shared directly into a clean room without rules gives subscribers raw access, and it says analysis rules might not fully prevent sophisticated extraction, recommending query templates for approved queries. Snowflake notes that join and column policies are respected only if the template uses policy filters.
| Platform | What controls what runs | What controls what leaves | Record you can keep |
|---|---|---|---|
| AWS Clean Rooms | Analysis rules on each configured table: aggregation, list, or custom analyses limited to reviewed analysis templates | Only members allowed to receive results get query output | Analysis logs in CloudWatch with query text, rule and template, when logging is enabled |
| BigQuery data clean rooms | Analysis rules on views and query templates for approved queries | Restricted egress by default; egress settings cannot change after the listing is created | The listing configuration and your project audit logs |
| Snowflake Data Clean Rooms | Templates the data owner makes available; free-form SQL only if granted | Collaborators get aggregated results, not direct queries on raw data | Template code the consumer can usually read before running |
| Databricks Clean Rooms | Notebooks every other collaborator must approve before the runner runs them | Output tables are temporary and read-only, readable by the runner unless shared output is enabled | Notebook versions, approvals and output tables |
Which utility questions fit inside the room, and which do not
Sort the buyer's questions before you design anything. A question fits when its answer can be expressed as counts, rates, distributions or a score, and does not need any single record to be readable outside the room.
Ask the buyer to rank its questions. A buyer with two decisive questions rarely needs a broad template library, and a short list keeps the output review, the run cap and the log review small enough for one person on your side to handle. Questions marked partial need an extra control, usually a supervised session or a rule that per-record outputs stay inside.
| Buyer question | Method inside the room | What leaves | Fit |
|---|---|---|---|
| Do the records cover our target cases? | Aggregation template grouped by category, period and region | Counts above a minimum group size | Good |
| How complete are labels and outcome fields? | Null and value-range profile per field | Per-field completeness rates | Good |
| Is the text long and varied enough? | Length and vocabulary statistics | Summary statistics, no excerpts | Good |
| How does a baseline model score on a held-out slice? | Approved notebook or job runs the buyer's model against a fixed slice | Metrics per agreed slice | Good, if model outputs per record stay inside |
| Can our team read examples to judge quality? | Supervised view-only session on a small reviewed sample | Reviewer notes without copied text | Partial |
| Will fine-tuning on this data improve our model? | Short training run inside the room | Before and after scores only; weights stay inside | Partial; exporting weights is a license question |
| Can we keep a copy to compare against other suppliers? | None | Records | No: needs a signed agreement |
The clean room evaluation requirements brief
Issue this brief to the prospective buyer before provisioning. Each line names the requirement, the evidence that shows it was met and who holds that evidence. Fill the values for your own platform and dataset; the structure works for any of the platforms above.
Checklist · 3 items
- Attach the brief to the evaluation agreement so the question list and output limits are terms, not courtesy.
- Version the brief. If the buyer adds a question mid-evaluation, issue version two and record who approved it.
- Keep a short exceptions log: any run that was refused, any output withheld, and why.
0 of 3 ticked. Tick each item as you go.
| Brief section | Requirement | Evidence produced | Owner |
|---|---|---|---|
| 1. Purpose and questions | List the utility questions in writing; anything not listed is out of scope | Signed question list with version number | Buyer lead, data owner |
| 2. Approved dataset version | Name the exact snapshot, row count and field list provisioned | Manifest with SHA-256 digest of each file or table export | Data owner |
| 3. Preparation applied | State what was removed, masked or pseudonymized before provisioning | Preparation log and reviewer sign-off | Data owner, privacy reviewer |
| 4. Permitted analyses | Only listed templates or approved notebooks run; no free-form queries unless agreed | Template or notebook IDs and approval records | Data owner, security engineer |
| 5. Output limits | Minimum group size, allowed metrics, no record-level output, cap on runs | Analysis rule configuration export | Security engineer |
| 6. Access and accounts | Named accounts, least access needed, start and expiry dates | Membership list with dates | Security engineer |
| 7. Logging | Query or job logging on before first access; both sides keep a copy | Log destination and first log entry timestamp | Both parties |
| 8. Results handover | Outputs reviewed by the owner before release to the buyer | Released output list with digests | Data owner |
| 9. Revocation and expiry | Access ends on the expiry date or on written notice, whichever is first | Membership removal record | Security engineer |
| 10. Deletion and retention | Provisioned copies and buyer-side artifacts removed; logs kept for the agreed period | Deletion confirmation and retention schedule | Both parties |
How both parties verify the package and the access limits
Agree on the evidence before access starts, so neither side reconstructs it from memory afterward. Three records do most of the work: a manifest, a configuration export and a log.
The manifest names each table or file provisioned and its digest. NIST's Secure Hash Standard describes digests as a way to detect whether a message has changed since the digest was generated, so the same digest recomputed at the end shows the buyer evaluated exactly the version you approved.
The configuration export captures the analysis rules, templates and result receivers as they were on the first day. On BigQuery, egress controls are fixed once a listing exists, which makes the listing itself part of the record. On Databricks, notebook approvals show which code was allowed to run.
The log shows what actually ran. AWS describes its analysis logs as letting members determine whether queries complied with the analysis rules and the collaboration agreement, and lists fields such as the query text, the analysis rule and the template used. Turn logging on before the buyer's first query; AWS notes it can take a few minutes for log delivery to begin.
Why aggregate results still need a ceiling
Aggregate outputs are not automatically safe. NIST defines a reconstruction attack as a privacy attack that uses published statistics to reconstruct individual data points from the original private dataset, and many narrow counts over a small population can do exactly that.
Set three ceilings in the brief: a minimum group size below which a count is suppressed, a fixed list of dimensions the buyer may group by, and a cap on the number of runs. Platform features such as aggregation thresholds help, but the cap on runs and the fixed dimension list are yours to write.
Model scores raise a quieter version of the same issue. If the buyer's model produces a prediction per record and those predictions leave the room, the buyer can learn about individual records. Release metrics per agreed slice and keep per-record outputs inside.
Illustrative: a 30-day evaluation of resolved support tickets
Everything in this example is illustrative. A software company holds resolved customer support tickets with categories, resolution notes and satisfaction scores. A prospective buyer wants to know whether the tickets would help train an assistant for billing disputes.
Starting state: the buyer asks for a copy of 10,000 tickets. The owner instead issues the brief with four questions: share of tickets tagged billing, completeness of the resolution field, length distribution of resolution notes, and how the buyer's current model scores at predicting the resolution category on a held-out slice.
Provisioning: the owner pseudonymizes customer and agent names, removes payment details, exports the snapshot, records digests in the manifest and provisions a view with an aggregation rule and two templates. Access opens on day 1 and expires on day 30, with logging on before the first query.
Outcome check: the logs show 14 template runs, all within the cap of 20. The owner releases four result sets, each with a digest. At the end, the snapshot digest still matches the manifest, the buyer's account is removed and both sides sign the deletion confirmation.
The exception: on day 22 the buyer asks to fine-tune a small adapter inside the room and export it. The owner refuses under section 4 of the brief, because exported weights would carry information learned from the tickets. The request is logged, and the parties move it into the license negotiation instead.
Closing out: revocation, deletion and the record you keep
End every evaluation with a written close-out, even when the buyer walks away. It should show the date access ended, the account removals, the digests recomputed against the manifest and the list of released outputs.
For copies made to provision the room, follow your own sanitization policy. NIST's media sanitization FAQ for SP 800-88 Revision 2 says an organization's policy should cover documentation and evidence requirements, meaning certificates of sanitization. For cloud tables, the practical evidence is usually a deletion record from the platform plus the removal of the share.
Keep the logs for the period the brief names. They are your answer if the buyer later makes a claim about the data, and your evidence if a model trained elsewhere appears to reflect records it should never have seen.
When the evaluation should become a license negotiation
Stop the evaluation and move to a contract when the buyer needs a copy, wants to keep anything trained on the data, asks for record-level outputs or asks to add questions that need fields you excluded. An evaluation agreement should state that it grants no right to train, keep or redistribute the data.
Clean room access answers who can see what. It does not settle whether you may license the records at all: customer contracts, privacy notices and ownership of the underlying content still decide that, and counsel should review them before a sensitive dataset is offered, even for evaluation.
- The buyer asks to keep released outputs that are fine-grained enough to stand in for records: tighten the output limits first.
- The buyer asks for a second dataset version or a broader date range: treat it as a new brief, not an extension.
- Results suggest the buyer wants exclusive use: exclusivity and term belong in the signed license agreement, not in the evaluation terms.
- Your own review finds a field that should not have been provisioned: pause access, record the finding and re-provision.
If the evaluation leads to licensing through SourceX
When an evaluation turns into a license, SourceX can carry the data owner through the rest: sourcing and qualification, rights review, contracting, preparation, delivery and payment. The owner approves scope and terms and signs before SourceX receives anything, and a dataset too large to move can stay in the owner's storage or go out on encrypted drives.
Whether a company is eligible is settled by a brief assessment that involves no data transfer.
Frequently asked questions
What is a dataset clean room evaluation?
It is a pre-purchase test in which the buyer runs approved analyses on the owner's data inside a controlled environment and receives only agreed outputs. The buyer learns whether the data is useful without receiving the records. The owner sets the questions, output limits, access dates and logging in advance.
Can a buyer fine-tune a model inside a clean room during evaluation?
Some platforms can run training jobs, and a short run with before and after scores can be a fair test. The risk is the trained weights: if they leave the room, they carry information learned from the data. Treat exporting weights as a licensing question, not an evaluation step.
Is a clean room the same as a sandbox or a sample?
No. A sample hands over a subset of records, and a sandbox usually lets users query freely. A clean room restricts which analyses run and what leaves, so the buyer sees results rather than records. Many owners combine a clean room with a small, reviewed view-only sample.
Do aggregate results protect individuals?
Not by themselves. Many narrow counts over a small group can be combined to reconstruct individual records, which NIST describes as a reconstruction attack. Minimum group sizes, a fixed list of dimensions and a cap on runs reduce that risk.
What should an owner keep after the evaluation ends?
Keep the brief, the manifest with digests, the configuration export, the logs, the list of released outputs and the deletion confirmation. Together they show what the buyer could do, what it did and what it took away. Keep them for the period the brief names.
Sources
- AWS Clean Rooms analysis rules include aggregation, list and custom types, with custom analyses limited to reviewed and allowed analysis templates. Source
- AWS Clean Rooms analysis logs record query text, rule and template so members can check queries against the analysis rules and collaboration agreement. Source
- BigQuery data clean rooms restrict egress by default, fix egress controls once a listing exists, apply analysis rules only to views and recommend query templates. Source
- Snowflake Data Clean Rooms give collaborators aggregated results through templates the data owner makes available, without direct queries on raw data. Source
- Databricks clean room notebooks need approval from every collaborator other than the uploader before the runner can run them. Source
- Databricks clean room output tables are temporary, read-only tables readable by the runner unless shared output is enabled. Source
- NIST defines a reconstruction attack as using published statistics to reconstruct individual data points from the original private dataset. Source
- NIST's Secure Hash Standard specifies hash algorithms whose digests are used to detect whether messages have changed since the digests were generated. Source
- NIST's SP 800-88 Rev. 2 FAQ says a sanitization policy should cover documentation and evidence requirements such as certificates of sanitization. Source
See if your company qualifies
A short company assessment. No data uploads are needed.