Skip to content

Leadership and readiness

Dataset RFP rejection transparency: 8 questions and an attrition ledger

By SourceX Editorial · Reviewed by Noah Loul ·

9 sections

Short answer

Dataset RFP rejection transparency means the vendor discloses what it filtered out before the records reached you: each rule, its version, how many records it removed and which segments it thinned. Ask for that as an attrition ledger with stratified counts, score each answer from 0 to 3, and read the result against your intended use, because a filter that helps one use can strip out the cases another needs.

Provenance trail

Dataset RFP rejection

This page covers dataset RFP rejection transparency: 8 questions and an attrition ledger.Illustrative

Key takeaways

  • Every delivered dataset is what survived a chain of filters, and an RFP that asks only about the survivors cannot see the selection effects.
  • Published corpora show the effect: the C4 blocklist filter disproportionately removed text from and about minority individuals (Dodge et al., EMNLP 2021).
  • Ask for an attrition ledger: stage, rule version, records in and out, reason codes and the segments each filter thinned.
  • A high rejection rate is not proof of rigor; without criteria and strata it cannot be interpreted.
  • Score answers from 0 (no ledger) to 3 (stratified, versioned and rerunnable), then choose proceed, rerun, pilot or stop.

Why should a dataset RFP ask about rejected records?

Because the records a vendor delivers are the survivors of every filter it ran, and those filters decide what the dataset can represent. Deduplication, language detection, quality scoring, privacy removal, license screening and completeness checks each drop records for a reason, and the reasons are rarely random. An RFP that asks only about the delivered records sees the result of those choices without the choices themselves.

Dataset RFP rejection transparency is the vendor's written disclosure of that selection: which rules ran, in what order and version, how many records each removed and which segments each one thinned. It does not mean handing over the removed content, which privacy law, contracts or legal holds may forbid. Criteria, counts and segment profiles are enough to see selection effects.

The RFP is the cheapest moment to ask. Before price is on the table, every vendor answers the same questions in writing, and a vendor that does not track its own attrition shows that before you have paid for a pilot to find out. The AI training data RFP template covers the full structure of a data RFP; this guide supplies the section on removed records, with an attrition ledger, eight questions, a scoring rubric and a decision memo.

How filtering changes what a dataset represents

A filter changes a dataset's composition whenever the records it removes differ from the records it keeps, and published corpora show this happening at scale. Documenting the C4 web corpus, Dodge and colleagues (EMNLP 2021) evaluated the text removed by its banned-word blocklist and found that the filter disproportionately removed text from and about minority individuals. Nobody set out to remove those voices; the effect came from what the word list happened to match.

Rights screens reshape data too. The Stack, a code dataset, records in its Croissant RAI example that a license detector found no license for about 81 percent of collected repositories, which were then excluded, and that roughly 40 percent of permissively licensed files were near-duplicates. The same record notes that languages such as SQL, Batchfile and TypeScript were less likely to be permissively licensed, 4 percent against an average of 10 percent, which may bias their representation. A legally necessary filter still shifted the language mix.

Filters also trade one use against another. After pretraining 28 models of 1.5 billion parameters on differently filtered data, Longpre and colleagues (A Pretrainer's Guide to Training Data, NAACL 2024) reported that quality filtering raised both downstream performance and toxic generation, that toxicity filtering cost the models some ability to identify toxic content, and that no single filtering approach suited every use. Business records can behave the same way: a completeness filter that drops tickets without a resolution code can also drop the escalations and abandoned cases an agent most needs to learn from.

The attrition ledger every vendor response should include

Ask each vendor for one ledger row per filter, from the source population to the records offered, with counts broken down by the strata in your specification. The structure borrows from clinical trial reporting: CONSORT guidance asks trials to report losses and exclusions in each group together with the reasons, and to keep participants later found ineligible in the flow diagram. Dataset attrition deserves the same discipline, where every removal is counted and nothing leaves the denominator unexplained.

A ledger like this is achievable even for very large datasets. The Stack's Croissant RAI record walks from 220.92 million repository names to 137.36 million public, accessible repositories before its license and deduplication steps. MLCommons' Croissant RAI vocabulary gives the description a machine-readable home in its data preprocessing protocol field, defined as the steps required to bring collected data to a state a model can process, such as filtering out incomplete entries.

The table lists the stages a business-records dataset usually passes through and what each row must report. The figure uses illustrative bars to show how a completeness filter can strip out hard cases while routine cases barely move.

The attrition ledger every vendor response should include
StageWhat the row recordsReason codes to requireCoverage effect to report
Source populationSystem, date range and record unit before any filter, with the total countNone; this row is the denominatorStarting mix by period, channel, product and outcome
Eligibility windowDate, account or region rules that define the scopeOut of window, out of region, test or internal accountWhich periods or regions the scope cuts off
Rights screenRule used to decide permission, its data source and versionNo permission found, contract restriction, opt-out, unknown statusSegments where permission was rarer, such as one product line or customer tier
Privacy and legal removalRule and approver for personal-data, legal-hold and sensitive-source removalsErasure request, legal hold, sensitive sourceCounts only; the content stays out
DeduplicationExact and near-duplicate method, threshold and which copy was keptExact duplicate, near duplicate, superseded versionWhether revisions or repeat cases were collapsed
Quality and completenessEach check with its threshold and versionMissing field, too short, failed parse, low scoreShare of hard, long or unresolved cases removed
Language and formatDetector, threshold and handling of mixed recordsNon-target language, mixed language, unreadable attachmentCode-switched or attachment-heavy records lost
Sample and offer selectionHow the offered set and any sample were drawn from the survivorsNot selected, reserved for another use, manual removalDifferences between the sample and the full offer
Stacked bars for six filter stages from source population to offered set, where routine cases shrink gradually but the hard-case segment nearly disappears at the completeness stage.
How one filter can erase a segment while the rest barely moves. Illustrative, not to scale.

Eight RFP questions about the records a vendor removed

Paste the eight questions into the data specification section and require answers in the ledger format, so responses can be compared line by line. Two deserve emphasis. Question six follows Datasheets for Datasets, which asks dataset creators whether the raw data was saved alongside the cleaned version; without it, no filter can ever be revisited. Question seven matters because permission metadata is often wrong at the source: the Data Provenance Initiative's 2024 audit of more than 1,800 text datasets found license omission above 70 percent and error rates above 50 percent on popular hosting sites, so a rights screen is only as good as the data it reads.

The questions, ready to paste:

  • Q1. List every filter between the source population and the records offered, in order, with the rule version and the date it ran.
  • Q2. For each filter, give records in and records out, broken down by the strata in our specification: period, channel, product line, language and outcome.
  • Q3. For each filter, give the reason codes used and the share of removals under each code.
  • Q4. Which thresholds were tuned, on what data, and did any tuning use our evaluation material or a public benchmark?
  • Q5. Were any records removed by hand? Name the written rule, the role that applied it and the count removed.
  • Q6. Is the pre-filter data retained, and could a quality or completeness filter be rerun at thresholds we set, within the same rights?
  • Q7. What data source did the rights screen use to decide permission, and how were records with unknown status treated?
  • Q8. Will the same filters, at the same versions, run on the full delivery and every refresh, and how will a change be notified?

Scoring the answers, from no ledger to a rerunnable one

Score each response from 0 to 3 on how complete its ledger is, then read the content. Completeness tells you whether the vendor can see its own selection; the content tells you whether that selection fits your use. A vendor that scores 0 has not necessarily delivered biased data, but neither of you can tell.

Apply the rubric question by question rather than on overall impression, and keep the scored responses in the RFP file. The request is not unusual: NIST's Generative AI Profile (NIST AI 600-1, 2024) includes documenting training data curation policies among its suggested actions, so a vendor running a disciplined pipeline should already hold most of this record.

Scoring the answers, from no ledger to a rerunnable one
ScoreWhat the response containsWhat you can concludeNext step
0: No ledgerGeneral statements about rigorous quality control, with no rules or countsSelection effects cannot be assessedMake the ledger a condition of shortlisting
1: Criteria onlyFilter names and rules, without countsYou know what was removed in principle, not how muchAsk for counts per stage before any sample
2: Counts without strataRecords in and out per stage, as totalsThe volume of attrition is visible; who was removed is notAsk for counts by your strata on the two largest filters
3: Stratified and versionedCounts by your strata, reason codes, rule versions and a rerun optionSelection effects can be read against your intended useCompare the segments removed with the cases your use depends on

Could a high rejection rate or a security badge tell you the same thing?

No. A high rejection rate sounds like rigor, but without criteria and strata it cannot be interpreted. An illustrative vendor that discards nine records in ten might be applying strict quality rules, or a completeness filter that removes every hard case, and the rate alone cannot tell the two apart.

Security attestations describe how a vendor protects its systems, not which records its filters removed. Catalog size and case studies are the other usual stand-ins: a large catalog says nothing about attrition inside any one dataset, and case studies report the buyers for whom the selection happened to fit. All of these are fair to weigh after the ledger, not instead of it.

There are legitimate limits on disclosure, and a fair RFP allows for them. Removed content may be personal data, privileged, under a legal hold or restricted by a customer contract, and it should stay removed. The request is never to send what was deleted; it is to state the rules and how many records each one took, by segment. A vendor can answer that without exposing anything it removed.

Which filters matter for your intended use?

Read the ledger against the use the data must serve, because the same filter can be harmless for one purpose and damaging for another. The pairings below are worth checking first; they follow from how each use consumes data, not from measured results.

  • Evaluation sets: any filter that removes ambiguous, long or unresolved cases makes the test easier than production and inflates scores.
  • Safety and abuse classifiers: toxicity and sensitive-content filters remove the very examples the classifier must recognize, the trade-off the Pretrainer's Guide measured.
  • Workflow and agent training: completeness filters that require a closed status or final code drop interrupted, escalated and abandoned cases.
  • Retrieval over document collections: deduplication that keeps one copy can drop the superseded versions a system must learn to tell apart.
  • Multilingual or regional use: language detectors that discard mixed-language records remove code-switching customers and regional variants.
  • Trend and forecasting work: eligibility windows and retention gaps shift the period mix, so check counts by month before trusting any rate.

The purchase decision memo: what the ledger lets you decide

The memo turns the scored ledger into one of four decisions, recorded with its evidence. Proceed when the response scores 3 and no filter thins a segment your use depends on. Proceed with a rerun when a quality or completeness filter removed cases you need and the vendor kept the pre-filter data. Pilot on the pre-filter population when the ledger scores 2 and the largest filters are ambiguous. Stop when the vendor cannot produce a ledger or will not run the same filters on the delivery.

Record these fields in the memo:

  • Intended use and acceptance test, written before any vendor answered.
  • Ledger score for each vendor, with the questions it left unanswered.
  • The filter that removed the most records in each stratum your use depends on, with the counts.
  • Filters you asked to rerun or relax, and the rights limits that apply to any rerun.
  • Disclosure limits the vendor cited, such as privacy, legal hold or contract, and whether counts were still provided.
  • Decision, owner and date, plus the ledger change that would reopen it.

Putting a rejection question into a SourceX data request

SourceX does not keep datasets in stock: it sources each one on request from U.S. companies, starting from the buyer's description of scope, volume, language, format, timing and licensing needs on the request form. A buyer can write the attrition ledger into that request: the intended use, the acceptance test, the strata whose coverage must survive and the reason codes it wants reported. SourceX runs rights review and data preparation for each dataset, and de-identification is done by SourceX directly or through its processing partner, so some exclusions are fixed by rights and privacy rather than negotiable. Whether a particular ledger requirement can be met is settled request by request.

Methods note: the attrition ledger, the eight questions and the 0 to 3 rubric are editorial tools written for this guide on 2026-10-26, drawing on the C4 documentation study, The Stack's Croissant RAI record, the Pretrainer's Guide, CONSORT reporting guidance, Datasheets for Datasets, NIST AI 600-1 and the Data Provenance Initiative audit. They have not been tested on live RFP responses; typical attrition rates for business-records datasets were not measured, and no SourceX results are reported.

Frequently asked questions

What is rejection transparency in a dataset RFP?

It is a vendor's written disclosure of the records it filtered out before delivery: each rule, its version, the counts it removed and the segments it thinned. It covers criteria and counts, not the removed content. Buyers use it to see selection effects that the delivered records alone cannot reveal.

Will a vendor share the records it rejected?

Usually not, and often it should not, because removed records may contain personal data, privileged material or content restricted by contract. Ask for counts by stage and segment, reason codes and rule versions instead. Those expose selection effects without exposing anything that was removed.

Is a high rejection rate a sign of a good vendor?

Not on its own. A high rate can mean strict quality rules or a filter that removes the hardest and most useful cases. Only the criteria and the counts by segment tell you which, so treat the rate as a prompt for questions rather than a score.

How can filtering create selection bias in training data?

A filter creates bias whenever the records it removes differ systematically from those it keeps. The C4 study found its blocklist disproportionately removed text from and about minority individuals, and The Stack records that license screening left some programming languages less represented. Completeness or language filters on business records can do the same to escalations or regional customers.

What if records were removed for privacy or legal reasons?

Those records should stay removed, and the RFP should say so explicitly. Ask only for the rule, the approver and the counts by segment, so you can judge coverage without reversing the exclusion. Counsel should review any question that touches personal data or legal holds.

Sources

  • Dodge and colleagues (EMNLP 2021) evaluated text removed from the C4 corpus and found its blocklist filtering disproportionately removed text from and about minority individuals. Source
  • The Croissant RAI specification defines a data preprocessing protocol field and includes The Stack's record: 220.92M repository names to 137.36M accessible, about 81% of repositories without a detected license excluded, roughly 40% near-duplicates, and SQL, Batchfile and TypeScript less often permissively licensed (4% vs 10%). Source
  • A Pretrainer's Guide to Training Data (NAACL 2024) pretrained 28 models of 1.5B parameters and found quality filtering raised both performance and toxic generation, toxicity filtering traded away toxicity identification, and no one-size-fits-all filter exists. Source
  • Datasheets for Datasets asks dataset creators whether the raw data was saved in addition to the preprocessed or cleaned data. Source
  • CONSORT guidance on losses and exclusions asks trials to report, for each group, losses and exclusions after randomisation with reasons, and to keep participants later found ineligible in the flow diagram. Source
  • NIST AI 600-1, the Generative AI Profile, includes a suggested action to document training data curation policies. Source
  • The Data Provenance Initiative's audit of more than 1,800 text datasets (2024) found license omission above 70 percent and error rates above 50 percent on popular dataset hosting sites. Source

Tell us what data you need

Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.

See if your company qualifies