Leadership and readiness
Dataset RFP rejection transparency: 8 questions and an attrition ledger
By SourceX Editorial · Reviewed by Noah Loul ·
9 sections
Short answer
Dataset RFP rejection transparency means the vendor discloses what it filtered out before the records reached you: each rule, its version, how many records it removed and which segments it thinned. Ask for that as an attrition ledger with stratified counts, score each answer from 0 to 3, and read the result against your intended use, because a filter that helps one use can strip out the cases another needs.
Provenance trail
Dataset RFP rejection
Key takeaways
- Every delivered dataset is what survived a chain of filters, and an RFP that asks only about the survivors cannot see the selection effects.
- Published corpora show the effect: the C4 blocklist filter disproportionately removed text from and about minority individuals (Dodge et al., EMNLP 2021).
- Ask for an attrition ledger: stage, rule version, records in and out, reason codes and the segments each filter thinned.
- A high rejection rate is not proof of rigor; without criteria and strata it cannot be interpreted.
- Score answers from 0 (no ledger) to 3 (stratified, versioned and rerunnable), then choose proceed, rerun, pilot or stop.
Why should a dataset RFP ask about rejected records?
Because the records a vendor delivers are the survivors of every filter it ran, and those filters decide what the dataset can represent. Deduplication, language detection, quality scoring, privacy removal, license screening and completeness checks each drop records for a reason, and the reasons are rarely random. An RFP that asks only about the delivered records sees the result of those choices without the choices themselves.
Dataset RFP rejection transparency is the vendor's written disclosure of that selection: which rules ran, in what order and version, how many records each removed and which segments each one thinned. It does not mean handing over the removed content, which privacy law, contracts or legal holds may forbid. Criteria, counts and segment profiles are enough to see selection effects.
The RFP is the cheapest moment to ask. Before price is on the table, every vendor answers the same questions in writing, and a vendor that does not track its own attrition shows that before you have paid for a pilot to find out. The AI training data RFP template covers the full structure of a data RFP; this guide supplies the section on removed records, with an attrition ledger, eight questions, a scoring rubric and a decision memo.
How filtering changes what a dataset represents
A filter changes a dataset's composition whenever the records it removes differ from the records it keeps, and published corpora show this happening at scale. Documenting the C4 web corpus, Dodge and colleagues (EMNLP 2021) evaluated the text removed by its banned-word blocklist and found that the filter disproportionately removed text from and about minority individuals. Nobody set out to remove those voices; the effect came from what the word list happened to match.
Rights screens reshape data too. The Stack, a code dataset, records in its Croissant RAI example that a license detector found no license for about 81 percent of collected repositories, which were then excluded, and that roughly 40 percent of permissively licensed files were near-duplicates. The same record notes that languages such as SQL, Batchfile and TypeScript were less likely to be permissively licensed, 4 percent against an average of 10 percent, which may bias their representation. A legally necessary filter still shifted the language mix.
Filters also trade one use against another. After pretraining 28 models of 1.5 billion parameters on differently filtered data, Longpre and colleagues (A Pretrainer's Guide to Training Data, NAACL 2024) reported that quality filtering raised both downstream performance and toxic generation, that toxicity filtering cost the models some ability to identify toxic content, and that no single filtering approach suited every use. Business records can behave the same way: a completeness filter that drops tickets without a resolution code can also drop the escalations and abandoned cases an agent most needs to learn from.
The attrition ledger every vendor response should include
Ask each vendor for one ledger row per filter, from the source population to the records offered, with counts broken down by the strata in your specification. The structure borrows from clinical trial reporting: CONSORT guidance asks trials to report losses and exclusions in each group together with the reasons, and to keep participants later found ineligible in the flow diagram. Dataset attrition deserves the same discipline, where every removal is counted and nothing leaves the denominator unexplained.
A ledger like this is achievable even for very large datasets. The Stack's Croissant RAI record walks from 220.92 million repository names to 137.36 million public, accessible repositories before its license and deduplication steps. MLCommons' Croissant RAI vocabulary gives the description a machine-readable home in its data preprocessing protocol field, defined as the steps required to bring collected data to a state a model can process, such as filtering out incomplete entries.
The table lists the stages a business-records dataset usually passes through and what each row must report. The figure uses illustrative bars to show how a completeness filter can strip out hard cases while routine cases barely move.
| Stage | What the row records | Reason codes to require | Coverage effect to report |
|---|---|---|---|
| Source population | System, date range and record unit before any filter, with the total count | None; this row is the denominator | Starting mix by period, channel, product and outcome |
| Eligibility window | Date, account or region rules that define the scope | Out of window, out of region, test or internal account | Which periods or regions the scope cuts off |
| Rights screen | Rule used to decide permission, its data source and version | No permission found, contract restriction, opt-out, unknown status | Segments where permission was rarer, such as one product line or customer tier |
| Privacy and legal removal | Rule and approver for personal-data, legal-hold and sensitive-source removals | Erasure request, legal hold, sensitive source | Counts only; the content stays out |
| Deduplication | Exact and near-duplicate method, threshold and which copy was kept | Exact duplicate, near duplicate, superseded version | Whether revisions or repeat cases were collapsed |
| Quality and completeness | Each check with its threshold and version | Missing field, too short, failed parse, low score | Share of hard, long or unresolved cases removed |
| Language and format | Detector, threshold and handling of mixed records | Non-target language, mixed language, unreadable attachment | Code-switched or attachment-heavy records lost |
| Sample and offer selection | How the offered set and any sample were drawn from the survivors | Not selected, reserved for another use, manual removal | Differences between the sample and the full offer |
Eight RFP questions about the records a vendor removed
Paste the eight questions into the data specification section and require answers in the ledger format, so responses can be compared line by line. Two deserve emphasis. Question six follows Datasheets for Datasets, which asks dataset creators whether the raw data was saved alongside the cleaned version; without it, no filter can ever be revisited. Question seven matters because permission metadata is often wrong at the source: the Data Provenance Initiative's 2024 audit of more than 1,800 text datasets found license omission above 70 percent and error rates above 50 percent on popular hosting sites, so a rights screen is only as good as the data it reads.
The questions, ready to paste:
- Q1. List every filter between the source population and the records offered, in order, with the rule version and the date it ran.
- Q2. For each filter, give records in and records out, broken down by the strata in our specification: period, channel, product line, language and outcome.
- Q3. For each filter, give the reason codes used and the share of removals under each code.
- Q4. Which thresholds were tuned, on what data, and did any tuning use our evaluation material or a public benchmark?
- Q5. Were any records removed by hand? Name the written rule, the role that applied it and the count removed.
- Q6. Is the pre-filter data retained, and could a quality or completeness filter be rerun at thresholds we set, within the same rights?
- Q7. What data source did the rights screen use to decide permission, and how were records with unknown status treated?
- Q8. Will the same filters, at the same versions, run on the full delivery and every refresh, and how will a change be notified?
Scoring the answers, from no ledger to a rerunnable one
Score each response from 0 to 3 on how complete its ledger is, then read the content. Completeness tells you whether the vendor can see its own selection; the content tells you whether that selection fits your use. A vendor that scores 0 has not necessarily delivered biased data, but neither of you can tell.
Apply the rubric question by question rather than on overall impression, and keep the scored responses in the RFP file. The request is not unusual: NIST's Generative AI Profile (NIST AI 600-1, 2024) includes documenting training data curation policies among its suggested actions, so a vendor running a disciplined pipeline should already hold most of this record.
| Score | What the response contains | What you can conclude | Next step |
|---|---|---|---|
| 0: No ledger | General statements about rigorous quality control, with no rules or counts | Selection effects cannot be assessed | Make the ledger a condition of shortlisting |
| 1: Criteria only | Filter names and rules, without counts | You know what was removed in principle, not how much | Ask for counts per stage before any sample |
| 2: Counts without strata | Records in and out per stage, as totals | The volume of attrition is visible; who was removed is not | Ask for counts by your strata on the two largest filters |
| 3: Stratified and versioned | Counts by your strata, reason codes, rule versions and a rerun option | Selection effects can be read against your intended use | Compare the segments removed with the cases your use depends on |
Could a high rejection rate or a security badge tell you the same thing?
No. A high rejection rate sounds like rigor, but without criteria and strata it cannot be interpreted. An illustrative vendor that discards nine records in ten might be applying strict quality rules, or a completeness filter that removes every hard case, and the rate alone cannot tell the two apart.
Security attestations describe how a vendor protects its systems, not which records its filters removed. Catalog size and case studies are the other usual stand-ins: a large catalog says nothing about attrition inside any one dataset, and case studies report the buyers for whom the selection happened to fit. All of these are fair to weigh after the ledger, not instead of it.
There are legitimate limits on disclosure, and a fair RFP allows for them. Removed content may be personal data, privileged, under a legal hold or restricted by a customer contract, and it should stay removed. The request is never to send what was deleted; it is to state the rules and how many records each one took, by segment. A vendor can answer that without exposing anything it removed.
Which filters matter for your intended use?
Read the ledger against the use the data must serve, because the same filter can be harmless for one purpose and damaging for another. The pairings below are worth checking first; they follow from how each use consumes data, not from measured results.
- Evaluation sets: any filter that removes ambiguous, long or unresolved cases makes the test easier than production and inflates scores.
- Safety and abuse classifiers: toxicity and sensitive-content filters remove the very examples the classifier must recognize, the trade-off the Pretrainer's Guide measured.
- Workflow and agent training: completeness filters that require a closed status or final code drop interrupted, escalated and abandoned cases.
- Retrieval over document collections: deduplication that keeps one copy can drop the superseded versions a system must learn to tell apart.
- Multilingual or regional use: language detectors that discard mixed-language records remove code-switching customers and regional variants.
- Trend and forecasting work: eligibility windows and retention gaps shift the period mix, so check counts by month before trusting any rate.
The purchase decision memo: what the ledger lets you decide
The memo turns the scored ledger into one of four decisions, recorded with its evidence. Proceed when the response scores 3 and no filter thins a segment your use depends on. Proceed with a rerun when a quality or completeness filter removed cases you need and the vendor kept the pre-filter data. Pilot on the pre-filter population when the ledger scores 2 and the largest filters are ambiguous. Stop when the vendor cannot produce a ledger or will not run the same filters on the delivery.
Record these fields in the memo:
- Intended use and acceptance test, written before any vendor answered.
- Ledger score for each vendor, with the questions it left unanswered.
- The filter that removed the most records in each stratum your use depends on, with the counts.
- Filters you asked to rerun or relax, and the rights limits that apply to any rerun.
- Disclosure limits the vendor cited, such as privacy, legal hold or contract, and whether counts were still provided.
- Decision, owner and date, plus the ledger change that would reopen it.
Putting a rejection question into a SourceX data request
SourceX does not keep datasets in stock: it sources each one on request from U.S. companies, starting from the buyer's description of scope, volume, language, format, timing and licensing needs on the request form. A buyer can write the attrition ledger into that request: the intended use, the acceptance test, the strata whose coverage must survive and the reason codes it wants reported. SourceX runs rights review and data preparation for each dataset, and de-identification is done by SourceX directly or through its processing partner, so some exclusions are fixed by rights and privacy rather than negotiable. Whether a particular ledger requirement can be met is settled request by request.
Methods note: the attrition ledger, the eight questions and the 0 to 3 rubric are editorial tools written for this guide on 2026-10-26, drawing on the C4 documentation study, The Stack's Croissant RAI record, the Pretrainer's Guide, CONSORT reporting guidance, Datasheets for Datasets, NIST AI 600-1 and the Data Provenance Initiative audit. They have not been tested on live RFP responses; typical attrition rates for business-records datasets were not measured, and no SourceX results are reported.
Frequently asked questions
What is rejection transparency in a dataset RFP?
It is a vendor's written disclosure of the records it filtered out before delivery: each rule, its version, the counts it removed and the segments it thinned. It covers criteria and counts, not the removed content. Buyers use it to see selection effects that the delivered records alone cannot reveal.
Will a vendor share the records it rejected?
Usually not, and often it should not, because removed records may contain personal data, privileged material or content restricted by contract. Ask for counts by stage and segment, reason codes and rule versions instead. Those expose selection effects without exposing anything that was removed.
Is a high rejection rate a sign of a good vendor?
Not on its own. A high rate can mean strict quality rules or a filter that removes the hardest and most useful cases. Only the criteria and the counts by segment tell you which, so treat the rate as a prompt for questions rather than a score.
How can filtering create selection bias in training data?
A filter creates bias whenever the records it removes differ systematically from those it keeps. The C4 study found its blocklist disproportionately removed text from and about minority individuals, and The Stack records that license screening left some programming languages less represented. Completeness or language filters on business records can do the same to escalations or regional customers.
What if records were removed for privacy or legal reasons?
Those records should stay removed, and the RFP should say so explicitly. Ask only for the rule, the approver and the counts by segment, so you can judge coverage without reversing the exclusion. Counsel should review any question that touches personal data or legal holds.
Sources
- Dodge and colleagues (EMNLP 2021) evaluated text removed from the C4 corpus and found its blocklist filtering disproportionately removed text from and about minority individuals. Source
- The Croissant RAI specification defines a data preprocessing protocol field and includes The Stack's record: 220.92M repository names to 137.36M accessible, about 81% of repositories without a detected license excluded, roughly 40% near-duplicates, and SQL, Batchfile and TypeScript less often permissively licensed (4% vs 10%). Source
- A Pretrainer's Guide to Training Data (NAACL 2024) pretrained 28 models of 1.5B parameters and found quality filtering raised both performance and toxic generation, toxicity filtering traded away toxicity identification, and no one-size-fits-all filter exists. Source
- Datasheets for Datasets asks dataset creators whether the raw data was saved in addition to the preprocessed or cleaned data. Source
- CONSORT guidance on losses and exclusions asks trials to report, for each group, losses and exclusions after randomisation with reasons, and to keep participants later found ineligible in the flow diagram. Source
- NIST AI 600-1, the Generative AI Profile, includes a suggested action to document training data curation policies. Source
- The Data Provenance Initiative's audit of more than 1,800 text datasets (2024) found license omission above 70 percent and error rates above 50 percent on popular dataset hosting sites. Source
Tell us what data you need
Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.