Leadership and readiness
Do supplier intake questions predict buyer acceptance? Test each one
By SourceX Editorial · Reviewed by Noah Loul ·
9 sections
Short answer
Better intake questions raise buyer acceptance only when an answer changes what happens to a file and is later checked against a verified outcome. Test each question on a full cohort: its likelihood ratios, the acceptable files a screen on it would lose, and its net benefit against advancing everyone. In this article's illustrative cohort, a self-rated quality question barely beats no screen, while export and ownership questions earn one.
Checklist
Supplier intake buyer
Key takeaways
- An intake question cannot make a dataset more acceptable; it can only close files that would fail or order the queue so likely winners are worked first.
- Score each question three ways: likelihood ratios, acceptable files lost if a no closes the file, and net benefit against advancing every file.
- In the illustrative cohort, the export question cuts staff hours per accepted dataset from 50.0 to 25.6, while a self-rated quality question adds just 3.4 net accepted datasets per 1,000 over advancing every file.
- Questions a company already screens on cannot be validated from its own acceptance data, and acceptance rates on screened files overstate the pool.
- Validate answers against the nearest checked fact first; ask for counts by cohort and form version, not rates.
Can better supplier intake questions improve buyer acceptance?
Yes, but only in two ways, and neither makes a dataset better. A sharper intake question can close files that would have failed later, so staff hours go to suppliers a buyer might accept, and it can order the queue so likely winners are worked first. Whether a question does either is an empirical claim about later outcomes, and it can be tested before anyone credits it.
The decision an investor is judging is narrow: for each question on a supplier application, should it close a file on a bad answer, change the order in which files are worked, or be cut? Answering it takes four things a company either keeps or does not: every completed application in a dated cohort, the answers frozen as submitted, what happened to each file afterward (a checked fact, a rights decision, a buyer's acceptance or rejection), and the staff hours each file consumed, failures included.
A company that shows acceptance rates without those records is showing the result of its screen, not a test of its questions. The rest of this article builds the test: three conditions a question has to meet, an illustrative ledger that scores six questions, the measurement trap that hides a screen's best questions from its own data, and the evidence to request.
Three tests a question must pass before it changes a file's fate
A question earns a role only if it separates, if acting on it beats acting on nothing, and if it costs less than it saves. Each test uses a different number, and a question can pass one and fail another.
Separation is measured with likelihood ratios, a tool borrowed from diagnostic testing. The positive ratio is the share of acceptable files that answer yes divided by the share of other files that answer yes; the negative ratio is the same comparison for no answers. A question whose negative ratio sits near zero is safe to screen on, because acceptable files rarely answer no; a ratio near one means the answer carries almost no information.
Acting on the question is judged against the default of advancing every file. Decision-curve analysis in medicine does this with net benefit: true positives minus false positives weighted by an exchange rate set from the decision threshold, compared with treating everyone and treating no one, as Vickers, Van Calster and Steyerberg describe in the BMJ (2016). The paper suggests eliciting that rate with questions such as how many biopsies a doctor would accept to find one cancer. Here the rate is a business choice stated the same way, in files rather than money: this article assumes a company will work up to nine files that fail to find one that a buyer accepts, so each failed file costs one ninth of an accepted dataset.
Cost is partly hidden. Each added question lengthens the form, and in a web survey experiment Galesic and Bosnjak (Public Opinion Quarterly, 2009) found that longer stated questionnaires drew fewer people to start and finish, and that questions placed later received faster, shorter and more uniform answers. A question that predicts well but sits at the end of a long form may be answered carelessly by exactly the companies a business wants.
The intake question validity ledger: six questions, one cohort
In an illustrative cohort of 1,000 completed applications, of which 120 would end in a buyer-accepted dataset if fully worked, two questions about checkable facts earn a screen while a self-rated quality question barely beats screening on nothing. Every number in the ledger comes from a script and labeled assumptions, not from any company's records.
Advancing every file finds all 120 acceptable datasets at 50.0 staff hours per accepted dataset, for a net benefit of 22.2 accepted datasets per 1,000 applications. Closing every file that answers no to the export question keeps 108 of the 120, cuts the hours to 25.6 and lifts net benefit to 68.9. The self-rated quality question loses 24 acceptable files, twice as many, while saving far less: 45.6 hours per accepted dataset and a net benefit of 25.6, only 3.4 above advancing everyone.
The verdicts follow one rule, set before the numbers were run: a question earns a screen when it beats advancing every file by at least 10 per 1,000 and loses at most 10 percent of acceptable files; one that clears the benefit bar but loses more only orders the queue; one that adds less than 10 is cut or rewritten.
Two results surprise. The single question with the best positive ratio, whether records show how each case ended, loses 15 percent of acceptable files as a screen, so it ranks the queue rather than closing files. And the export and ownership questions, each acceptable alone, lose 14.5 percent of acceptable files when both are required, because loss compounds across screens.
Which verdict is right depends on what is scarce. The loss cap protects acceptable suppliers when applications are the constraint. When staff hours are the constraint instead, say 2,000 hours per cohort, enough to work about 333 files, advancing every file finds 40 accepted datasets, the export screen 78.3, ordering the queue by recorded outcomes 82.9, and requiring both export and ownership 102.6, because every file that pair advances can be worked. An investor should ask which constraint binds before judging a company's screens. The ledger, with applications as the constraint:
| Question (illustrative) | Says yes: of 120 acceptable / of 880 others | Likelihood ratio if yes / if no | If a no closes the file: acceptable lost, hours per accepted | Net benefit per 1,000 and verdict |
|---|---|---|---|---|
| A. Two or more years of history in the main system | 114 / 616 | 1.36 / 0.17 | 6 (5.0%), 38.4 hours | 45.6: earns a screen |
| B. Names an export route that keeps history and links | 108 / 352 | 2.25 / 0.17 | 12 (10.0%), 25.6 hours | 68.9: earns a screen |
| C. Records show how each case ended | 102 / 308 | 2.43 / 0.23 | 18 (15.0%), 24.1 hours | 67.8: orders the queue, does not close files |
| D. Records were created by the company, not its clients | 114 / 484 | 1.73 / 0.11 | 6 (5.0%), 31.5 hours | 60.2: earns a screen |
| E. Self-rated data quality: high | 96 / 633.6 | 1.11 / 0.71 | 24 (20.0%), 45.6 hours | 25.6: cut or rewrite as a checkable fact |
| F. Self-estimated volume above 50,000 records | 90 / 484 | 1.36 / 0.56 | 30 (25.0%), 38.3 hours | 36.2: orders the queue, does not close files |
| B and D both required | 102.6 / 193.6 | 3.89 / 0.19 | 17.4 (14.5%), 17.3 hours | 81.1: orders the queue; the pair breaks the loss cap |
| Advance every file (no screen) | 120 / 880 | not applicable | 0 lost, 50.0 hours | 22.2: the bar every question must beat |
Why the questions you already screen on vanish from your own data
A question that already decides who advances cannot be validated from the outcomes that follow it, because the files that answered badly never reached a buyer. Researchers call this the selective labels problem, and it means a company's strongest screening questions can look useless, or untestable, in its own acceptance data.
Lakkaraju and colleagues described it at KDD 2017 using bail decisions: a defendant's failure to appear is only observable if a judge released the defendant first, so outcome data exist only where a human chose to create them. Diagnostic research calls the same flaw verification bias; He and McDermott (Biostatistics, 2012) note that sensitivity and specificity computed only from verified subjects are typically biased.
Run the illustrative cohort through a process that advances only files answering yes to both the export and the ownership questions. About 296 files per 1,000 reach buyer review, 34.6 percent of them are accepted, and that rate is nearly three times the 12.0 percent in the full pool. Inside the advanced files, the export and ownership questions have no no answers at all, so their value cannot be estimated, and every other question shows acceptance rates more than double its full-cohort values, as the table shows. Ratios between yes and no hold up here only because the model assumes answers are independent once the outcome is known; real answers rarely are.
The investor trap is the headline rate. An acceptance rate measured on screened files describes the screen, not the supply, and a company that tightens its screen will report a rising rate even if it finds fewer acceptable datasets. In the ledger, the export screen lifts the acceptance rate among worked files from 12.0 to 23.5 percent while the count of acceptable datasets found falls from 120 to 108. What the advanced files can and cannot say about each question:
| Question (illustrative) | Full cohort: accepted if yes / if no | Advanced files only: accepted if yes / if no | What the company's own data can tell it |
|---|---|---|---|
| A. Two or more years of history | 15.6% / 2.2% | 41.8% / 8.1% | Direction holds; rates are inflated by the screen |
| B. Export route that keeps history | 23.5% / 2.2% | No file answering no was advanced | Nothing: the question cannot be tested without an audit slice |
| C. Records show how each case ended | 24.9% / 3.1% | 56.3% / 10.9% | Direction holds; rates are inflated by the screen |
| D. Created by the company, not clients | 19.1% / 1.5% | No file answering no was advanced | Nothing: the question cannot be tested without an audit slice |
| E. Self-rated data quality: high | 13.2% / 8.9% | 37.1% / 27.5% | A weak signal looks stronger in absolute terms |
| F. Self-estimated volume above 50,000 | 15.7% / 7.0% | 42.0% / 22.7% | Direction holds; rates are inflated by the screen |
How to validate a question that already decides who advances
Validate the answer against the nearest checked fact first, and only then link that fact to buyer acceptance. Pushing rejected files all the way to a buyer is too expensive to be the main method, as the illustrative numbers show.
An audit slice that advances a random 5 percent of the files answering no to the export question would work 27 files per 1,000 applications at 162 staff hours, and would be expected to contain 0.6 acceptable files. It would take many cohorts to estimate the negative ratio that way. The cheaper route is an outcome ladder: did the stated export route exist when the inventory was checked, did the rights review clear the records, did the prepared data pass the buyer's test. Most of a question's error shows up at the first rung, where checking costs minutes rather than weeks.
Three further practices make the test credible:
Checklist · 3 items
- Freeze the answers. Store each application as submitted, with the form version, so later edits by staff or applicants cannot leak the outcome back into the predictor. A reworded question starts a new cohort.
- Use differences among reviewers. Lakkaraju and colleagues proposed contraction, which compares a model with human decision-makers by exploiting differences among those decision-makers; a company that assigns files to reviewers without regard to answers can use the same variation to see what lenient reviewers let through.
- Count failures and open files. Withdrawn, dormant and still-open files stay in the cohort's denominator with their hours; dropping them flatters every question at once.
0 of 3 ticked. Tick each item as you go.
Strongest alternative: the form changed who applied, not who passed
The strongest competing explanation for a rise in acceptance after a form change is that the form changed who applied. A longer or stricter form can deter weak applicants and good ones alike, raise the acceptance rate among those who finish, and still lower the number of accepted datasets.
More applications pull the other way. A shorter form or a broader campaign can lift submissions while acceptable supply stays flat, so staff hours per accepted dataset rise even though the funnel looks fuller. Either way the rate moves for reasons that have nothing to do with the questions' predictive value.
Separate the two with counts, not rates. Ask for form starts, completions, files worked and accepted datasets for the periods before and after the change, by acquisition channel and with the same reviewers where possible. A question has improved buyer acceptance only if accepted datasets per period rise, or hold steady with fewer hours, once channel mix is held constant. Burden effects of the kind Galesic and Bosnjak measured also hide a loss nobody sees: companies that abandon a long form never become failed files, so they never appear in the cohort at all.
What could make an intake-led efficiency thesis wrong
An intake-led efficiency thesis says a data company will gain operating leverage because its questions route work better over time. Four developments could break it, and a diligence team should ask which one the company fears most.
- Answers drift once applicants learn the screen. When a question is known to decide advancement, applicants and the people who refer them learn the answer that advances a file. Questions about checkable facts, such as a named export route, resist this better than self-ratings, which is a reason to rewrite questions like E rather than keep them.
- Buyer requests move. A question validated against last year's requests can lose its value when buyers want different systems, outcomes or histories, so validity has to be re-estimated by request family and date.
- Small numbers mislead. With 120 acceptable files per 1,000 applications, the no answers to a screening question hold between 6 and 18 acceptable files per thousand in the illustrative cohort, so a single cohort cannot pin down a negative ratio. Pooling cohorts is valid only when the wording and buyer requests stayed the same.
- The checks that matter most are not on the form. Rights review, privacy preparation and the buyer's own test decide acceptance; an intake answer can only point at them. A company that treats strong intake answers as a substitute for the rights check moves cost downstream, where a failed file is far more expensive.
Intake validity questions for any data company, SourceX included
Ask for the cohort, not the rate, and ask how each question's value was measured. SourceX should be held to the same list.
Its live pages describe an intake built on a few questions and no files: the applicant lists its systems, years of history and what can be exported, and SourceX then checks fit, completes the inventory with the company, runs the rights check and sets the processing route. The application page puts the fit check at three screens and about seven minutes, and says that applying does not promise an offer or payment. The how-it-works page lists ten fit signals, among them historical depth, volume, whether records show how work ended and clear ownership. The ledger's illustrative questions echo several of those signals, but they are not SourceX's form, and SourceX publishes no figures on how its intake answers relate to later acceptance.
SourceX gives company and commercial metrics to investors directly rather than in articles, and it is raising from accredited investors under Rule 506(c), with materials available on request through its investors page. Questions worth putting to it, or to any data company, include:
- Which intake questions can close a file on their own, and what share of acceptable files did each screen lose in the last full cohort?
- How do you measure a question you already screen on: an audit slice, reviewer differences, or a checked fact at inventory?
- Is your quoted acceptance rate computed on all completed applications or only on files that reached a buyer?
- When did the form last change, and are cohorts before and after the change reported separately with their channel mix?
- Which answers are checked against a record during qualification, and how often does the checked fact contradict the answer?
- How many accepted datasets per period, with staff hours per accepted dataset, came from each version of the form?
Methods note: the ledger's inputs, formulas and limits
The intake question validity ledger, its verdict rule and the selective-labels comparison were built for this article on 2026-10-28 as an editorial model, computed by a short script (xs0186_intake_validity.py) kept with its output in the writer's working folder. Every input is illustrative and none is a SourceX figure: 1,000 completed applications, 120 of them acceptable if fully worked, six staff hours to work an advanced file, a decision threshold of one in ten (a weight of one ninth on each failed file), a 10 percent cap on acceptable files lost to a screen and a 10 per 1,000 net benefit margin. Each question's chance of a yes among acceptable and other files is set in the script: A 95 and 70 percent, B 90 and 40, C 85 and 35, D 95 and 55, E 80 and 72, F 75 and 55.
Formulas as the script applies them. Yes counts are the class size times the yes rate; likelihood ratios are the yes rate among acceptable files over the yes rate among others, and the same for no answers. Net benefit per 1,000 is acceptable files advanced minus one ninth of other files advanced. Hours per accepted dataset are six times files advanced, divided by acceptable files advanced. The pair B and D multiplies the two questions' rates, which assumes answers are independent once the outcome is known; the advanced-files table uses the same assumption. The advanced files in the selective-labels table are those answering yes to both B and D. The audit slice is 5 percent of files answering no to B at six hours each, and its expected acceptable files are the slice times the acceptable share of those no answers. The capacity case allows 2,000 staff hours (about 333 files) and works files in an order unrelated to the outcome, so accepted datasets found equal the files worked times the acceptance rate of the files advanced, with advanced files beyond capacity left unworked; a question used to order the queue has its yes files worked first, and at this capacity work stops before any no file is reached. All arithmetic uses exact fractions, and values are rounded once, half up, at output, so gains are computed from unrounded net benefits.
Public evidence: the selective labels paper (KDD 2017), He and McDermott on verification bias (Biostatistics, 2012), Vickers, Van Calster and Steyerberg on net benefit (BMJ, 2016) and Galesic and Bosnjak on questionnaire length (Public Opinion Quarterly, 2009), each checked through a search excerpt of its publisher's or index's own site, plus SourceX's how-it-works, application and investors pages fetched on 2026-10-28. Not measured: any real question's yes rates or validity, at SourceX or elsewhere; how answers drift over time; the drop-off a given question causes; and the size of the audit slice a real cohort would need.
Frequently asked questions
What is a likelihood ratio for an intake question?
It compares how often acceptable and unacceptable files give the same answer. A positive ratio of 2.25 means a yes is 2.25 times as common among files a buyer would accept as among the rest. A negative ratio near zero means acceptable files almost never answer no, which is what makes a question safe to screen on.
Why can't a data company validate intake questions from its own acceptance data?
Because outcomes exist only for files it chose to advance. A question that already closes files has no rejected cases with known outcomes, so its value cannot be estimated, and every acceptance rate among advanced files is inflated by the screen. An audit slice, reviewer differences or a checked fact closer to intake is needed.
Should a supplier intake form be shorter or longer?
As short as the questions that pass the tests allow. Longer stated questionnaires reduce how many people start and finish and degrade later answers, so a question must save more review effort than the completions it costs. Put questions about checkable facts first and cut self-ratings that barely separate.
How many applications does it take to validate one question?
More than one cohort usually supplies, because the cell that matters is small. In the illustrative cohort of 1,000 applications, the no answers to a screening question include only 6 to 18 acceptable files, so the negative ratio is uncertain. Pool cohorts only when the question wording and buyer requests were unchanged.
Does SourceX publish how its intake answers predict acceptance?
No. Its live pages describe the questions it asks and the checks that follow, but it publishes no validity, acceptance or cohort figures. It shares company and commercial metrics with investors directly, and this article's ledger is an illustrative method, not SourceX data.
Sources
- Vickers, Van Calster and Steyerberg (BMJ, 2016) describe net benefit, which weighs true positives against false positives using an exchange rate tied to the decision threshold and compares a model with treating everyone or no one, with the rate elicited as, for example, how many biopsies a doctor would accept to find one cancer. Source
- Galesic and Bosnjak (Public Opinion Quarterly, 2009) found that longer stated web questionnaire lengths reduced starts and completions, and that later questions drew faster, shorter and more uniform answers. Source
- Lakkaraju, Kleinberg, Leskovec, Ludwig and Mullainathan (KDD 2017) describe selectively labeled data, where outcomes are observed only because of a human decision such as a judge's release, and propose contraction, which exploits differences among decision-makers. Source
- He and McDermott (Biostatistics, 2012) note that sensitivity and specificity estimated only from verified subjects are typically biased, a problem known as verification bias. Source
- SourceX's how-it-works page describes its first step as a few questions with no files, in which the applicant lists systems, years of history and what can be exported before SourceX checks fit, completes the inventory, runs the rights check and sets the processing route, and it lists ten fit signals. Source
- SourceX's application page describes a fit check of three screens taking about seven minutes and states that applying does not ensure an offer or payment. Source
- SourceX's investors page says it shares company and commercial metrics with investors directly and offers investor materials on request. Source
Request SourceX investor materials
SourceX shares company and commercial metrics with investors directly. Any offering is made only to verified accredited investors.