Skip to content

Leadership and readiness

Dataset rejection postmortem: eight parts, four sharing tiers

By SourceX Editorial · Reviewed by Noah Loul ·

10 sections

Short answer

A dataset rejection postmortem should leave three things behind: a dated account of what was expected and what was delivered, contributing causes tied to evidence, and decisions that change because of them. Write expectations from the documents in force, check whether the test or the data failed, and ask whether any technical fix could have helped. Then share each part only as widely as its contents allow.

Checklist

Dataset rejection postmortem

This page covers dataset rejection postmortem: eight parts, four sharing tiers.Illustrative

Key takeaways

  • Treat the postmortem as a decision document: if no request, test, checklist or price changes, it was a status report.
  • Quote the acceptance criteria and request versions that were in force, because memory rewrites expectations once the outcome is known.
  • Classify each cause as a data defect, test defect, scope gap, rights, terms or task mismatch; only the first two respond to technical work.
  • Give every part a sharing tier and abstract details as they travel: names become roles, counts become bands, records become identifiers.
  • A changed decision needs an owner, a date and an observable check, or it will not survive the next deal.

What should a dataset rejection postmortem leave behind?

A dataset rejection postmortem should leave behind a dated account of what each side expected and what was delivered, the contributing causes with the evidence for each, and a short list of decisions that change as a result. Anything else, including who was at fault, is noise that makes the next team less willing to write one.

The format borrows from incident practice in software operations. Google's site reliability engineering book defines a postmortem as a written record of an incident, its impact, the actions taken to mitigate or resolve it, the root causes and the follow-up actions to prevent recurrence, and argues for writing it blamelessly, on the assumption that people acted on the information they had at the time. A rejected dataset is not an outage, but the discipline carries over: the record exists to change what happens next.

Two features make dataset postmortems harder than incident reviews. Two organizations hold the evidence, and much of it is confidential to one of them: the buyer's model results, the supplier's customer contracts, both sides' prices. The template in this guide is built around that problem. Each part has a default sharing tier, and its details change form as the document moves outward.

When is a rejection worth a postmortem?

Write one whenever a delivery or sample fails a written acceptance test, is accepted only in part, ends in a no-deal after a sample, or starts a dispute. Agree the triggers before the first delivery, so that asking for a postmortem reads as routine rather than as an accusation.

Google's SRE book takes the same approach to incidents. It lists objective triggers such as data loss or user-visible downtime beyond a threshold, says the criteria should be set before an incident happens, and lets any stakeholder request a postmortem. Agreed triggers protect the person who asks. They also produce a set of postmortems written to the same standard, which can be compared later in a way that ad hoc reviews cannot.

Reasonable triggers for a dataset deal:

  • A sample or delivery fails any criterion in the written acceptance test.
  • A buyer accepts part of a delivery and rejects the rest.
  • A deal ends after a sample was prepared or tested.
  • Either side disputes a test result or the meaning of a criterion.
  • The same defect forces a second re-delivery.

The eight-part postmortem template

Eight parts take a rejected dataset from expectation to changed decision, and each carries a default sharing tier, so the document can be written once and released in layers. The tiers are this guide's own, modeled on the idea behind the Traffic Light Protocol, which FIRST maintains as a set of four labels telling recipients how far information may travel (version 2.0, authoritative since August 2022). FIRST notes that TLP was not designed to handle licensing terms or information-handling rules, so the confidentiality agreement between the parties still governs.

The four tiers run from narrowest to widest. Named reviewers see everything, including people, prices and record identifiers. The author's own organization sees the same parts with people and prices removed. Both deal teams see a joint version they have agreed. A public lesson keeps only the scope line, the cause class and the changed practice. The illustrative entries follow one invented rejection of a support-ticket sample, worked through in full later in the guide.

Use the parts in this order:

The eight-part postmortem template
PartWhat to recordEvidence it must citeWidest default tierIllustrative entry
1. Scope lineDeal code, dataset version, dates of request, delivery and decision, and the tier of each partRequest and delivery manifestsPublic lesson, in generic formSupport-ticket sample version 2, rejected on day 38
2. Expectation, written and understoodThe criteria in force, quoted with their version, then what each side took them to meanSigned request and acceptance test versionsBoth deal teamsCriterion 4 asked for resolution codes; the supplier read it as status codes
3. What was deliveredScope, banded counts, fields, filters and de-identification steps as documented before the testDelivery manifest and datasheetBoth deal teamsAbout 2,000 tickets, 14 fields, agent names replaced
4. Evidence of the rejectionResults by criterion, test and grader versions, failing sample identifiersTest logs and result filesBoth deal teamsCriterion 4 failed on 61 percent of tickets; the other criteria passed
5. Dated timelineEach event with its date and source record, from request to decisionEmail, tickets and file timestampsOwn organizationDay 0 request, day 21 sample, day 30 test, day 38 notice
6. Contributing causesEvery cause with its class and the strength of its evidenceParts 2 to 5Both deal teams; cause class only in publicScope gap, strong evidence; no test defect found
7. The fix testFor each cause, whether a technical fix could have changed the outcomeRepair estimate or a re-test of a repaired subsetBoth deal teamsResolution codes sit in a second system; a linked re-export could cure it
8. Changed decisionsEach change with an owner, a date and how anyone will see it happenedThe updated documents, linkedOwn organization; generic form widerRequest template defines resolution versus status; owner and date named
A grid of the eight postmortem parts against four sharing tiers: named reviewers and the author's organization see every part, both deal teams see six parts plus generic changed decisions, and a public lesson keeps only the scope line, cause class and changed practice in generic form.
Default sharing tiers for the eight parts. Illustrative defaults; the parties' agreement governs.

Why write the expectation from documents rather than memory?

Because once the outcome is known, both sides remember having expected it. Baruch Fischhoff's 1975 study introduced tasks for measuring hindsight bias, and his 2025 retrospective on fifty years of that research describes it as the tendency to revise how likely outcomes seemed once the actual result is known. After a rejection, the buyer tends to recall a clear requirement and the supplier an ambiguous one, and neither memory is evidence.

So part two quotes the request and acceptance test that were in force on the delivery date, word for word and with their version numbers, before recording what each side understood them to mean. The gap between the written criterion and the understood one is often the real finding: a term such as resolution, active customer or complete record that each side defined differently and nobody wrote down.

Part three applies the same rule to the delivery. Datasheets for Datasets, the documentation framework proposed by Timnit Gebru and colleagues, organizes its questions around a dataset's motivation, composition, collection process, preprocessing, uses, distribution and maintenance. A delivery manifest that answered the composition and preprocessing questions before the test ran is far more reliable than a description assembled after the rejection.

Did the data fail, or did the test?

Check the instrument before blaming the data: run the acceptance test on examples both sides agree are good, and read the failing cases rather than only the score. A postmortem that skips this step can record a test defect as a data defect and teach the supplier the wrong lesson.

At a glance

Did the data fail, or did the test

Public benchmarks show how often the instrument is at fault. OpenAI's announcement of SWE-bench Verified describes professional software developers screening the original SWE-bench samples because some unit tests were overly specific or unrelated to the issue and could reject correct solutions; 500 samples survived the screening. Anthropic's engineering guide to agent evaluations separates the transcript, the full record of a trial, from the outcome, and notes that with frontier models a 0 percent pass rate across many trials most often signals a broken task rather than an incapable agent. Reading the failing records is the dataset equivalent of reading the transcript.

Part six then assigns each contributing cause to one of six classes and records how strong the evidence for it is:

  • Data defect: the delivered records lack something the criteria clearly required, such as a field, a link or an outcome.
  • Test defect: the test rejects examples both sides agree are good, or measures something the criteria never asked for.
  • Scope gap: the written criteria were ambiguous or silent, and each side read them differently.
  • Rights: the holder could not license the use, or could license only part of the scope.
  • Terms: price basis, exclusivity, term or permitted use could not be agreed.
  • Task mismatch: the source records never captured what the buyer's task needs.

Would any technical fix have changed the outcome?

Part seven answers that question cause by cause, because the most common postmortem error is proposing a data fix for a problem data cannot solve. Data defects and test defects respond to technical work. Rights, terms and task mismatch do not, and a scope gap needs a conversation and a definition before it needs any engineering.

Rights show the trap most clearly. When the Data Provenance Initiative traced the licenses of more than 1,800 text datasets (Nature Machine Intelligence, 2024; MIT Media Lab), it reported that over 70 percent of licenses were omitted and over half were wrong on popular hosting sites, with the true terms frequently stricter than the label. Cleaning cannot change what a license allows. If part seven says better redaction would have fixed a rights cause, the postmortem has confused privacy treatment with permission.

Where a fix is plausible, record the evidence for it: a timed repair estimate or, better, a re-test of a small repaired subset against the same test version. An untested fix is a hypothesis, and any changed decision that rests on it should say so.

Sharing the postmortem without exposing either side

Write the full version once at the narrowest tier, then derive each wider version by abstracting details rather than deleting whole parts. Names become roles, counts become bands, records become identifiers and model results become pass or fail by criterion. A reader at every tier should still be able to follow the causal chain from expectation to changed decision.

Personal data needs particular care, because postmortems tend to collect quoted examples. Article 5 of the GDPR requires personal data to be adequate, relevant and limited to what is necessary for the purpose. A postmortem rarely needs record content to make its point, so cite failing records by identifier and leave the records themselves in the controlled evaluation environment. Counsel should confirm what the specific records and agreement require.

The abstraction ladder shows how common details change form as a postmortem moves outward:

Sharing the postmortem without exposing either side
DetailNamed reviewersOwn organizationBoth deal teamsPublic lesson
OrganizationsLegal namesLegal namesBuyer and supplierAn AI developer and a services company
PeopleNames and rolesRoles onlyRoles onlyOmitted
TaskProduct and model namesProduct and model namesTask family and intended useA broad task type, such as ticket triage
CountsExactExactBanded, such as about 2,000Omitted, or small versus large
Failing recordsIdentifiers, with content kept in the evaluation environmentIdentifiers onlyIdentifiers onlyNone
Model resultsScores and runsScores and runsPass or fail by criterionOmitted
Prices and termsAs negotiatedTerms only, prices removedOnly terms that bear on a causeNever

Turning causes into changed decisions

Each cause should change at least one document the next deal will use: the request template, the acceptance test, the sampling plan, the rights checklist or the pricing basis. A change that touches no document, such as a promise to communicate earlier, is an intention, and intentions rarely survive a staff change.

Give each changed decision an owner, a date and a check someone else can observe: the revised request template sits in the shared folder, the test now runs on known-good cases before every delivery, the rights checklist names the platform terms that blocked this deal. Google's SRE book treats review as part of the process, saying an unreviewed postmortem is of little use, and stores finished postmortems where others can find them. A dataset team can do the same with a folder both deal owners can read and a periodic look across postmortems for causes that repeat.

Repeated causes are where the effort pays off. Three rejections that trace to the same undefined term are a template problem rather than three supplier problems, and the fix belongs in the request every future supplier receives.

Illustrative: one rejected support-ticket sample, start to finish

This case is invented to show the template; every count and date in it is illustrative. A buyer requested support tickets with resolution codes to evaluate a triage model. The supplier delivered about 2,000 tickets on day 21. On day 30 the buyer's test failed criterion 4, resolution code present, on 61 percent of tickets, and the buyer rejected the sample on day 38.

Part two found the gap: the request said resolution code, the supplier exported the status field, and nobody had defined the term. The instrument check came next: on 20 tickets both sides agreed were complete, the test passed all 20, so it was sound. Part six recorded one cause, a scope gap, with strong evidence. Part seven located the resolution codes in a second system, and a re-test on 100 linked tickets passed criterion 4, so a linked re-export could cure the defect.

Part eight changed two documents. The buyer's request template now defines resolution versus status, and the supplier's export checklist now asks which system holds outcomes. The public lesson ran to two sentences: an undefined outcome field caused a rejection, and the fix was a definition in the request.

Writing the next data request with the postmortem in hand

The most useful output of a dataset rejection postmortem is a better request. A buyer who has learned which terms were ambiguous, which criteria rejected good data and which rights questions surfaced late can write all three into the next requirement before any supplier prepares a record.

Buyers tell SourceX the scope, volume, language, format, timing and licensing terms they need through its request form, and the data is sourced to order from U.S. businesses rather than drawn from stock. A request that states the intended use, defines its terms and attaches the acceptance test lets candidate data be screened against the same criteria the postmortem sharpened. It is not a promise of a match or of acceptance; a precise request simply makes either outcome quicker to reach and easier to explain.

Frequently asked questions

What is a dataset rejection postmortem?

A written review after a dataset or sample fails acceptance. It records what was expected and delivered, the contributing causes with their evidence, and the decisions that change as a result. Unlike a rejection notice, it is written for the next deal rather than for the counterparty.

Should the buyer and supplier write one postmortem or two?

Usually two internal versions and one agreed joint summary. Each side holds evidence the other may not see, such as model results or customer contracts, so each writes its full version at its own narrowest tier, and then both agree what the joint version says.

How do you keep confidential details out of a shared postmortem?

Abstract details instead of deleting whole parts: replace names with roles, exact counts with bands, records with identifiers and scores with pass or fail by criterion. Cite failing records by identifier and keep the records themselves inside the controlled evaluation environment.

Who should own the actions from a dataset postmortem?

Each changed decision needs one named owner, a date and a check another person can observe, such as a revised request template in a shared folder. Actions that change no document, such as communicating better, rarely outlast the next staff change.

What if the acceptance test caused the rejection?

Record it as a test defect, fix the test and re-run it on cases both sides agree are good before judging any data again. Public benchmark work, such as the screening behind SWE-bench Verified, shows tests can reject correct work, so this check belongs in every postmortem.

Sources

  • Google's SRE book defines a postmortem as a written record of an incident, its impact, mitigation actions, root causes and follow-up actions, advocates blameless postmortems, sets triggers in advance, lets any stakeholder request one and treats review as essential. Source
  • FIRST's Traffic Light Protocol is a set of four labels indicating sharing boundaries; version 2.0 is authoritative from August 2022, and TLP was not designed to handle licensing terms or information-handling rules. Source
  • Fischhoff's 2025 retrospective on his 1975 study describes hindsight bias as the tendency to revise likelihood estimates once the actual outcome is known, and notes the 1975 paper introduced tasks for studying the bias. Source
  • Datasheets for Datasets proposes documentation questions organized by motivation, composition, collection process, preprocessing, uses, distribution and maintenance. Source
  • OpenAI's SWE-bench Verified announcement describes professional developers screening SWE-bench samples because some unit tests were overly specific or unrelated and could reject correct solutions; 500 samples were verified. Source
  • Anthropic's agent evaluation guide distinguishes the transcript, the full record of a trial, from the outcome, and says a 0 percent pass rate across many trials most often signals a broken task with frontier models. Source
  • The Data Provenance Initiative's audit of more than 1,800 text datasets (Nature Machine Intelligence, 2024) reported license omission above 70 percent and error rates above 50 percent on popular hosting sites, with correct licenses often more restrictive. Source
  • GDPR Article 5(1)(c) requires personal data to be adequate, relevant and limited to what is necessary for the purposes of processing. Source

Tell us what data you need

Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.

See if your company qualifies