Definitions and comparisons
SourceX dataset acceptance stages: what each status really confirms
By SourceX Editorial · Reviewed by Noah Loul ·
10 sections
Short answer
Each SourceX dataset acceptance stage records that one check happened, not that the dataset is accepted. Qualification needs no data uploads, rights review examines documents, signing fixes scope before any data moves, and only review of the delivered, de-identified data against written quality thresholds and the buyer's task test can support acceptance. The signed agreement, not a status label, defines each stage, so read the terms before the status.
Checklist
SourceX dataset acceptance
Key takeaways
- A status is a summary of one completed check; the signed agreement for the deal defines what each stage means and what follows from it.
- The early stages examine paperwork: a supplier qualifies without uploading data, and rights review works from contracts and notices, not the records.
- De-identification changes the data, so test results from before preparation do not carry over to the delivered dataset.
- Measured quality and buyer fit are separate tests; a dataset can meet every written threshold and still fail the task it was bought for.
- Before fixing a dataset that was not accepted, find the failing gate in order: scope, rights and privacy, measurement, fit, then commercial terms.
What does each SourceX dataset acceptance stage actually mean?
Each stage records that one specific check was completed; none of them, short of a decision recorded under the signed terms, means the dataset has been accepted. Reading a stage correctly means knowing what it confirms, what it leaves open and which document defines it.
SourceX's public description lists its work in this order: sourcing and qualification, rights review, contracting, data preparation, delivery and payment. It also states that every dataset goes through rights, privacy, quality and buyer-fit review. SourceX has not published a glossary of status labels, so the stage names in this guide are descriptive rather than system labels, and the signed agreement for each deal is the authority on what each stage means.
The practical decision for a delivery or acceptance owner is simple to state: which evidence to ask for at each stage, and which question to answer first when a delivery is not accepted. The sections below give a reading guide, a five-gate decision tree and a rejection log that traces each failure to the stage that should have caught it.
Which document decides what a status means?
The signed agreement decides; a public explanation, a portal label or an email only summarizes it. When a summary and the signed terms disagree, the signed terms win and the mismatch should be logged so the summary gets corrected.
SourceX's public facts point the same way. The order runs approval, signature, then delivery: SourceX receives no data until the supplying company has accepted the scope and terms and signed, and from then on SourceX's right to use the de-identified dataset, sublicense it and distribute it reaches only as far as the agreement allows. Each deal's signed agreement also fixes exclusivity and term. Scope, permitted use and duration all live in the signed document, so a status cannot widen them.
Log each mismatch with four fields: the summary's wording and where it appeared, the clause of the signed agreement it conflicts with, who noticed it, and the date the summary was corrected. A short log like this is how public explanations stay aligned with signed terms as templates and pages change.
Federal contracting gives a precise reference definition. Under FAR 46.501, acceptance acknowledges that delivered supplies or services conform with the contract's quality and quantity requirements, it can happen before, at or after delivery depending on the contract, and it is ordinarily documented on an inspection or receiving report. A data license is not a federal supply contract, but the three ideas transfer: acceptance is defined by the agreement, tied to stated requirements and recorded in writing.
The stage reading guide: what each status confirms and leaves open
Read every stage as a narrow statement. The guide below covers the eight stages a buyer-side or supplier-side owner will meet, from a buyer's request through a recorded decision; payment, the last item in SourceX's public description, is governed by the signed terms.
Because SourceX finds datasets for buyers on request, from U.S. businesses, instead of keeping stock, the first stage is a written request, not a catalog pick. Use the guide to decide what to ask for before treating a stage as settled:
| Stage | What it confirms | What it leaves open | Evidence to ask for |
|---|---|---|---|
| Request received | A buyer stated scope, volume, language, format, timing and licensing needs | Whether matching records exist anywhere | The written request, intended use and acceptance test |
| Supplier qualified | A company passed a short assessment without uploading data | Anything about the records' quality or fit | Assessment result and system inventory |
| Rights reviewed | Authority to license the selected records was examined | Privacy treatment, quality and buyer fit | Findings listing the contracts and notices checked |
| Scope and terms signed | The supplier approved exact scope and terms before any data moved | Whether the delivered data matches that scope | Signed agreement and its scope definition |
| Prepared and de-identified | Records went through de-identification by SourceX or its processing partner | Whether the data still passes the buyer's test | Method description and re-run test results |
| Delivered | Data reached the agreed route, such as the seller's storage or encrypted drives | Completeness, usability and acceptance | Manifest with counts, checksums and validator output |
| Reviewed for quality and fit | Data was measured against written thresholds and the task test | Payment, or any change to the signed scope | Measurement report with sample sizes and results |
| Decision recorded | The party with authority under the signed terms decided | Whether a model trained on it will improve | Dated written decision citing the agreement |
Which stages examine the data, and which only examine paperwork?
Only the stages after preparation examine the dataset a buyer will actually receive. Everything earlier examines descriptions, documents or a different version of the data, so it cannot by itself support acceptance.
Qualification is paperwork by design: SourceX's first step for a supplying company is a short assessment, and no data uploads are needed to see whether the company qualifies. Rights review works from contracts, notices and ownership, and it cannot lean on license labels. When MIT Media Lab researchers audited more than 1,800 text datasets, popular hosting sites had left licenses off at rates above 68 percent and labeled them incorrectly at rates above 50 percent.
Preparation changes the data. De-identification, performed by SourceX directly or through its processing partner, removes or transforms fields, so a sample that passed before preparation says little about the prepared version. Delivery is a logistics stage: a large dataset either remains in storage the seller controls or travels on encrypted drives, and arrival proves only that the data moved.
Format validation is the first real data check, and it is narrow. OpenAI's fine-tuning data-preparation notebook counts structural errors such as missing message lists, unrecognized roles, missing content and conversations without an assistant turn, and warns that examples over 16,385 tokens are truncated. Passing those checks shows the file is usable, not that it fits the task.
Measured quality versus buyer-specific acceptance
Quality and fit are separate tests and deserve separate statuses. Quality asks whether defects stay under written thresholds; fit asks whether the data does the job the buyer bought it for, and a dataset can pass the first while failing the second.
Quality measurement borrows from acceptance sampling. The NIST/SEMATECH handbook defines the acceptable quality level as the producer's baseline defect rate and the lot tolerance percent defective as a defect level the consumer would find unacceptable, and it frames the plan around producer's risk and consumer's risk. Writing both numbers down, with the sample size, turns a quality review into a check either side can rerun.
Fit needs a task test. Anthropic's January 2026 evaluation guide sets the bar for a good task at agreement: two domain experts, working separately, should arrive at the same pass-or-fail call. It also recommends a reference solution that passes every grader to prove the task is solvable. A buyer-specific acceptance test built that way stops an acceptance decision from resting on one reviewer's taste.
Documentation keeps both tests honest. Datasheets for Datasets asks creators to record motivation, composition, collection process, preprocessing, uses, distribution and maintenance, and MLCommons' Croissant RAI vocabulary gives machine-readable fields for collection, limitations, intended uses and personal or sensitive information. A delivery that records these lets a reviewer tell a quality miss from a fit miss.
The not-accepted decision tree: find the failing gate first
When a delivery is not accepted, identify the gate that failed before changing any data. The gates run in a fixed order because an earlier failure makes later work wasted: there is no point repairing records the signed scope excludes or the rights do not cover.
At a glance
The not-accepted decision tree: find the failing gate first
Work through the gates in order and stop at the first one that fails:
- Gate 1, scope: are the failing records inside the signed scope? If not, the cure is a new signed approval or removal, not cleanup.
- Gate 2, rights and privacy: would the fix need records, fields or uses that the signed terms or the privacy treatment do not permit? If so, stop and return to rights review.
- Gate 3, measurement: does the failure reproduce when the agreed measurement is rerun on the agreed sample? If not, correct the test or the sample before touching the data.
- Gate 4, fit: does data that meets every written threshold still fail the buyer's task test? If so, the problem is task mismatch, and the options are to re-scope or walk away.
- Gate 5, terms: is the disagreement really about price, exclusivity, term or volume? If so, it is a commercial question, and no technical fix will settle it.
When a technical fix cannot change the outcome
A technical fix only cures a technical defect. Three kinds of rejection survive any amount of cleaning, and spotting them early saves a repair cycle that cannot succeed.
A rights gap is the first. Cleaning, relabeling or de-identifying records does not create authority to license them; if the supplier lacked authority for some records, the only cures are removing them or obtaining the authority, and either may change the signed scope.
Commercial terms are the second. Exclusivity and term are set in each deal's signed agreement, so a disagreement about them is a negotiation, not a data defect, and it belongs with the people who can change the agreement.
Genuine task mismatch is the third. If the records lack the decision, outcome or context the task needs, no format change adds it. De-identification can sharpen this problem: removing identifiers can also remove signal, and adding fields back reopens the privacy review rather than closing the acceptance question.
The stage-of-origin rejection log
Log every rejection with the stage that should have caught it, not only the stage where it surfaced. A rejection found later than its stage of origin is a stage leak, and counting leaks by stage shows which earlier check needs tightening.
Each row below pairs a common rejection with its stage of origin, the kind of cure it allows and the evidence that closes it:
| Rejection reason | Stage that should have caught it | Cure class | Evidence that closes it |
|---|---|---|---|
| Records outside the signed scope | Scope and terms signed | New signed approval or removal, not cleanup | Diff of delivered records against the signed scope |
| Supplier lacks authority for some records | Rights reviewed | Remove records or obtain authority | Updated rights findings |
| Residual identifiers in prepared data | Prepared and de-identified | Rerun de-identification, then retest | Privacy check rerun on a stated sample |
| Wrong format or truncated examples | Delivered | Technical repair | Validator output before and after the fix |
| Defect rate above the written threshold | Reviewed for quality and fit | Repair, then re-measure on a fresh sample | Measurement report with sample size and limits |
| Meets thresholds, fails the task test | Request received, where the task was underspecified | Re-scope or walk away | Revised acceptance test agreed in writing |
| Dispute over price, exclusivity or term | Scope and terms signed | Commercial negotiation | Signed change to the agreement, or a recorded no-deal |
What SourceX has not published, and how to measure stage leaks
SourceX has not published a status glossary, counts of deals at each stage, rejection rates or review timing, and this article reports none. The current supplier process is described on SourceX's how-it-works page, and the signed agreement for each deal governs its stages and any timing.
The stage-leak measure can be run by any team that keeps a rejection log. Its design is set out below; results are not yet published.
Checklist · 4 items
- Unit: one rejection finding, tagged with the stage where it surfaced and the stage of origin from the log above.
- Leak rate per stage: findings that originated at that stage but surfaced later, divided by all findings that originated there.
- Cohorts and denominators: deliveries grouped by the quarter the scope was signed, with the number of deliveries in each cohort reported beside every rate.
- Limits: small cohorts reported as counts, not percentages; findings re-coded by a second reviewer on a sample to check agreement; no comparison across buyers whose acceptance tests differ.
0 of 4 ticked. Tick each item as you go.
Writing the acceptance test into a SourceX data request
The cheapest stage leak to prevent is an underspecified request. Buyers who use the SourceX request form already share scope, volume, language, format, timing and licensing needs; adding the intended use and an acceptance test at that point lets every later stage check against the same target. Datasets are sourced to that request, so no stage before review of the delivered data can promise acceptance, and a precise test is what lets the final review reach a clear answer.
A useful acceptance test answers five questions before any data moves:
- Task: what the model or evaluation must do with the data, written as one sentence a supplier can test against.
- Reference: the outcome field, grader or expert rubric that decides a pass, with one worked example that passes it.
- Thresholds: the defect types that count, the acceptable and unacceptable defect levels, and the sample size used to measure them.
- Privacy constraints: the fields the de-identified data must still carry for the task, so preparation and fit are checked against one list.
- Decision owner: who records the decision under the signed terms, and which evidence that person needs to see.
Frequently asked questions
Does a signed agreement mean the dataset has been accepted?
No. At SourceX the agreement is signed, with its scope and terms approved by the supplying company, before any data is delivered, so the signature precedes the data in delivered form. Acceptance is a later decision about the delivered, prepared data, made under the signed terms.
Why can a dataset pass quality checks and still not be accepted?
Because quality and fit are different tests. Quality checks count defects against written thresholds, while fit asks whether the data does the buyer's task, for example whether records carry the outcome a model must learn. A dataset can be clean and complete and still lack what the task needs.
Can de-identification cause a dataset to fail acceptance?
Yes. De-identification removes or transforms fields, and some of them may carry signal the task needs, so results from before preparation do not carry over. Rerun the acceptance test on the prepared data, and treat any request to restore fields as a new privacy review.
What should a buyer put in a request so acceptance can be decided cleanly?
State the intended use, the task test or grader, the written quality thresholds with a sample size, the privacy constraints and who records the decision. A test that two experts would grade the same way keeps the decision from turning on one reviewer's judgment.
Where are SourceX's acceptance terms actually defined?
In the signed agreement for each deal. Public pages, including SourceX's how-it-works page, summarize the current process, and this guide states no review timing; where a summary and the signed terms differ, the signed terms govern.
Sources
- Under FAR 46.501, acceptance acknowledges conformance with contract quality and quantity requirements, can occur before, at or after delivery, and is ordinarily documented on an inspection or receiving report. Source
- An audit of more than 1,800 text datasets found license omission above 68 percent and error rates above 50 percent on popular hosting sites. Source
- OpenAI's fine-tuning data-preparation notebook checks structural errors and warns that examples over 16,385 tokens are truncated. Source
- The NIST/SEMATECH handbook defines AQL and LTPD and the producer's and consumer's risk in acceptance sampling. Source
- Anthropic's January 2026 guide calls a task good when two experts would reach the same verdict and recommends a reference solution that passes all graders. Source
- Datasheets for Datasets proposes documenting each dataset's motivation, composition, collection process, preprocessing, uses, distribution and maintenance. Source
- The Croissant RAI vocabulary provides fields for data collection, limitations, intended uses and personal or sensitive information. Source
Tell us what data you need
Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.