Skip to content

AI data market

Will small language models raise domain data demand? A two-gate test

By SourceX Editorial · Reviewed by Noah Loul ·

10 sections

Short answer

Smaller models need more task-specific data per deployment, but most of it does not have to be bought. A large teacher model can write the target outputs and the deployer's own logs can supply the inputs. Outside business records matter at two gates: where the teacher fails the task and the deployer's records are too thin to fix it, and in held-out tests that neither model has seen.

Value drivers

Small language models domain

This page covers will small language models raise domain data demand? A two-gate test.Illustrative

Key takeaways

  • Small models trade stored knowledge for cheap inference: Microsoft's Phi-3 report says its 3.8B model cannot hold much factual knowledge and proposes search to compensate.
  • Distillation lets a large model write most training targets, so outside data demand moves from outputs to real inputs and independent tests.
  • Licensed business records become necessary only past two gates: the teacher fails the task, and the deployer's own records cannot cover the failing cases.
  • Benchmark parity is weak evidence for small models; the GSM1k study reports accuracy drops of up to 8 percent on fresh problems in its current version.
  • Count demand on a dated ladder from pilot to repeat order, and treat release notes and benchmark charts as the bottom rung.

Do smaller models need more specialized business data?

Per deployment, yes; across the market, only at two gates. A small language model reaches large-model quality on a narrow task by being adapted to it, and adaptation consumes task inputs, target outputs, a grader and often a source of facts. The question for investors is who supplies those inputs, because most of them can come from a larger teacher model or from the deployer's own logs rather than from a data license.

That distinction decides whether small language models create domain data demand or simply move work in-house. The evidence that small models need adaptation is strong and public: their developers say so in their own reports. The evidence that adaptation pays outside suppliers is thin. This article separates the two with a two-gate test, a demand-signal ladder tied to procurement documents and an interview design for the people who hold the budget.

The investor decision is concrete: whether the shift to smaller, cheaper models is a reason to expect more spending on licensed business records, and which evidence should arrive before that view is priced in.

Why small models move the data bill from compute to adaptation

Small models cut inference cost but give up stored knowledge, so the deployer pays for the difference in data. Stanford HAI's 2025 AI Index reports that the cost of GPT-3.5-level performance fell more than 280-fold between November 2022 and October 2024; its research chapter puts the price per million tokens at $20.00 at the start of that period and $0.07 at the end, reached by a small model, Gemini-1.5-Flash-8B. Cheap inference is why buyers try small models at all.

What they give up shows in the developers' own reports. Microsoft's Phi-3 technical report (April 2024) describes phi-3-mini, a 3.8 billion parameter model trained on 3.3 trillion tokens that scores 69 percent on MMLU, and concedes that it lacks the capacity to store much factual knowledge, citing weak TriviaQA results and proposing search augmentation as the fix. A model that cannot memorize a field's facts has to be handed them at run time or taught the narrow task directly.

Taught narrowly, small models can beat much larger ones. The LoRA Land report (April 2024) adapted 10 base models to 31 tasks with 4-bit low-rank adapters: 301 of the 310 adapted models beat their base model, 224 beat the GPT-4 benchmark, and on average they scored 34 points above their base models and 10 points above GPT-4. A June 2025 position paper by NVIDIA-affiliated researchers argues that small models are sufficiently powerful and more economical for many calls inside agent systems. Every LoRA Land gain came from task-specific training data.

The two-gate test: teacher ceiling first, record coverage second

A small model needs outside business records only when two gates both close: a large teacher model cannot already pass the deployer's test, and the deployer's own records do not cover the failing cases. If the teacher passes, distillation supplies the targets. If the deployer's records cover the cases, it tunes on them. Licensed records fill what is left, plus the held-out tests that every route needs.

Run the gates per task, not per company. Illustratively, a claims insurer might distill its email triage from a large model, tune a coverage-decision model on its own claim files and still lack enough examples of rare loss types, which is the one slice an outside supplier could fill. The table maps each route to the data it consumes and the dated public evidence behind it; note how little of that evidence concerns paid outside data.

The two-gate test: teacher ceiling first, record coverage second
Route after the gatesWhat the small model consumesUsual supplierRole for licensed business recordsDated public evidence
Teacher passes: distillReal task inputs, teacher outputs and a held-out testDeployer logs plus the large modelLow: rare inputs and independent test setsLlama 3.2 card, October 2024: 1B and 3B models trained on logits from Llama 3.1 8B and 70B
Teacher fails, own records cover the cases: tune in-housePast decisions with their outcomesThe deployerLow: the deployer already holds themLoRA Land, April 2024: 224 of 310 adapted models beat a GPT-4 benchmark on their tasks
Teacher fails, own records thin: licenseDecisions and outcomes from other companies, rare casesOutside suppliersHigh: the licensing caseNo public dated cohort of paid purchases found
Knowledge gap: retrieve at run timeDocuments, catalogs and policiesThe deployer, or licensed reference content for vertical vendorsMedium, mainly for vendors serving many customersPhi-3 report, April 2024: limited capacity for factual knowledge, search proposed
Every route: testReal tasks with graded outcomes the teacher never sawThe deployer or an outside supplierMedium to high, renewed at each model changeGSM1k, current version: accuracy drops of up to 8 percent on fresh problems
A flow from a task moving to a small model through two gates, whether a large teacher passes the test and whether own records cover the failing cases, ending in distillation, in-house tuning or outside business records, with a held-out test band under every route.
The two-gate test for small-model data demand. An editorial framework; no measured shares are shown.

Where distillation stops: the teacher's ceiling and the size floor

Distillation removes most demand for written answers but copies the teacher's limits, so outside records enter exactly where the teacher is weak. Meta's Llama 3.2 model card (October 2024) says its 1B and 3B models were pruned and trained with output logits from Llama 3.1 8B and 70B as token-level targets. OpenAI announced a distillation workflow for its API in October 2024 that stores a frontier model's input and output pairs and uses them to fine-tune a cheaper model.

Size still sets a floor. DeepSeek fine-tuned six open dense models, from 1.5 billion to 70 billion parameters, on reasoning samples generated by DeepSeek-R1, 800,000 of them for the Qwen-based models. On the AIME 2024 benchmark its 1.5B distilled model scores 28.9 percent pass@1 and its 32B model 72.6 percent, against 9.3 percent for GPT-4o, according to the DeepSeek-R1 repository. The same teacher data lifted every size, and the smallest still trailed the 32B model by more than 40 points.

The implication is narrow but useful. When the teacher already passes a task, the only outside data a small model needs is a stream of real inputs and a test the teacher has not seen. When the teacher fails, distillation reproduces the failure, and records of how experts actually resolved those cases become the scarce input.

Does a small model's benchmark parity survive a fresh test?

Often not fully, which makes evaluation data the most durable outside-data need in this market. The GSM1k study wrote fresh grade-school math problems in the style of the widely used GSM8k benchmark; its current version finds accuracy falling by as much as 8 percent, and several model families show signs of systematic overfitting. The paper points to the Phi and Mistral families, along with some Llama models, as likely overfitting, while many frontier models showed little sign of it.

Small models are marketed on benchmark parity, so a deployer has to re-test them on its own work before switching. That test needs real tasks with known outcomes that neither the teacher nor the student has seen, and it has to be refreshed whenever either model changes. It is the one input every route in the two-gate table consumes, and the one a deployer cannot generate from the teacher without contaminating the result.

A demand-signal ladder for small-model data, from release note to repeat order

Climb six rungs before counting small-model adaptation as demand for business data, and require a dated document for every rung above the second. Release notes and benchmark charts sit at the bottom: they show that a capability exists, not that anyone pays for data to use it.

The procurement evidence matrix is built into the last two columns: the document that proves each rung, and the false positive most often counted in its place. A small-model pilot that ends with the deployer tuning on its own records is real activity and real data work, but for an outside supplier it stops at the third rung.

A demand-signal ladder for small-model data, from release note to repeat order
RungWhat qualifies for small-model dataDocument that proves itFalse positive to discard
1. Release signalA small model claims parity on public benchmarksModel card or technical report with dated scoresTreating benchmark parity as parity on the deployer's tasks
2. Cost-driven pilotA deployer tests a small model against its large baselinePilot plan naming the baseline, test set and start dateA vendor demo on public prompts
3. Gap traced to dataThe miss survives prompting, retrieval and distillationError analysis naming the failing case typesA gap that closes with a stronger teacher or a larger student
4. Written outside requestA budget owner specifies records, volume, rights and an acceptance testDated request with scope and a named ownerAn exploratory call or a signed NDA
5. Paid commitmentAn order or license for data with payment termsContract or purchase orderPayment for model hosting or a tuning service, not data
6. Repeat at model changeA second purchase when the teacher or base model changesRenewal or second orderRe-running an old test set at no cost

Measuring it: an interview design for adaptation budget owners

Interview the people who decide whether a production task moves from a large hosted model to a small one, and classify every project as a request, an evaluation or a paid commitment by dated cohort. Market reports and vendor case studies cannot do this, because they count launches and capabilities rather than purchases.

At a glance

Measuring it: an interview design for adaptation budget owners

SourceX has not published results from this design. It sets out what an investor can ask any company in this market to show, SourceX included.

  • Population: independent budget owners at companies that considered moving a production task to a small model in a stated window; model vendors and their resellers are excluded.
  • Unit of analysis: one adaptation project, meaning one task and one model change, so a company with three projects counts three times and is flagged as clustered.
  • Cohort: the quarter in which the pilot started, reported separately so that later quarters cannot flatter earlier ones.
  • Commitment class: request, evaluation or paid commitment, with every class above a request confirmed by a dated document such as a scoped request, a test plan or a purchase order.
  • Variables: baseline model, small model, the route chosen at each gate, the acceptance threshold, and whether outside records were requested and why.
  • Denominator: every project started in the cohort, including those that stayed on the large model or closed using internal records only.
  • Uncertainty and limits: counts with intervals, nonresponse reported by cohort, and a plain warning that self-reported spending can overstate purchases until documents confirm them.

What could make the small-model data thesis wrong

The strongest alternative explanation is that cheaper models and better teachers reduce outside data spending rather than raise it. Capability announcements are not purchases, and every route in the two-gate table except one can be filled without a license. These are the risks to name before any upside.

  • Teachers keep improving. If each new large model passes more tasks at the first gate, distillation absorbs work that would otherwise need outside records, and the licensing slice shrinks with every release.
  • Own records are usually enough. Deployers that tune on their own logs never reach the fourth rung, so in-house adaptation can grow quickly while outside purchases stay flat.
  • Demand is concentrated. Reddit's Form 10-K for fiscal 2025 states that "substantially all of the contract value associated with our licensing revenue is derived from two of our partners." A market in which a few model developers buy most licensed data can absorb the small-model shift without adding buyers.
  • Synthetic tests may substitute. If teacher-written evaluation sets rank models as well as real tasks do, the evaluation slice weakens; GSM1k is evidence against that today, not proof that it will last.
  • Rights work may cost more than the slice is worth. The licensing case covers rare, cross-company cases that are small by construction, so rights review and contracting can outweigh what the slice supports.

Diligence questions for an adaptation-data seller, SourceX included

Ask which gate a seller's demand comes from, and ask for documents rather than logos. The same questions apply to SourceX.

SourceX turns selected business records into licensed AI datasets, and its work runs from finding and qualifying supply and reviewing rights to negotiating the contract, preparing the data, delivering it and settling payment. Nothing reaches SourceX until the company supplying the records has signed, on scope and terms it approved. Request, deal and commercial figures are unpublished; investors get company and commercial metrics from SourceX directly, and with a Rule 506(c) raise from accredited investors under way, materials are available on request through its investors page.

  • Of the data requests received in the last four quarters, how many came from deployers moving tasks to smaller models, and at which rung of the ladder does each sit?
  • For each of those requests, did a large teacher model already pass the task, and if so, why was outside data bought instead of distilled outputs?
  • Which requests could the buyer's own records have filled, and what did the buyer say about why they could not?
  • What share of paid commitments were held-out evaluation sets rather than training data, and how often were they bought again after a model change?
  • How many distinct, independent budget owners stand behind the paid commitments, and what share of value comes from the largest two?
  • If the next teacher model passed most of your buyers' current tasks, which of your products would still be needed?

Methods note: how the two-gate test and ladder were assembled

This article built four editorial frameworks on 2026-10-26: the two-gate test, the route table, the ladder and the interview design. They rest on the Phi-3 technical report (2024), the LoRA Land report (2024), Meta's Llama 3.2 model card (2024), OpenAI's distillation announcement (2024), the DeepSeek-R1 repository (2025), the current arXiv version of the GSM1k study, the NVIDIA-affiliated position paper on small models (2025), Stanford HAI's 2025 AI Index and Reddit's Form 10-K for fiscal 2025. The insurer example is illustrative and describes no real company.

Not measured: how many production tasks have moved to small models, how often each gate closes in practice, what deployers spend on outside records for small-model work, and how that spending splits between evaluation and training. None of these figures is public, and SourceX has not published its own.

Frequently asked questions

Is it cheaper to fine-tune a small model than to keep calling a large one?

It depends on volume and on where the adaptation data comes from. Inference on small models is far cheaper: Stanford HAI's 2025 AI Index tracked a more than 280-fold fall in the cost of GPT-3.5-level performance between November 2022 and October 2024. The saving holds only after paying for task data, a held-out test and re-testing at each model change, which can keep low-volume tasks on the large model.

What is knowledge distillation, and does it replace business data?

Distillation trains a smaller student model on the outputs of a larger teacher, as Meta did for the 1B and 3B Llama 3.2 models. It replaces most hand-written target answers. It does not replace real task inputs, independent test sets or records of the cases the teacher gets wrong.

Why would a small model need outside data if the company has its own logs?

Often it does not, because a company's logs cover the cases it sees regularly. Outside records help where that history is thin, such as rare case types, or where a vendor builds one model for many customers and needs coverage that no single customer holds.

Are small language model benchmark scores reliable for a business decision?

Treat them as a starting point. The GSM1k study found accuracy drops of up to 8 percent when models met fresh problems written in the style of a familiar benchmark, with several model families showing signs of overfitting. Re-test any small model on your own held-out tasks before switching.

Has SourceX published data on small-model adaptation demand?

No. Request, deal and commercial metrics stay unpublished; SourceX discusses company and commercial metrics with investors directly. Asking for materials on the SourceX investors page is also the way to ask how each rung of the ladder is defined, and any offering is limited to verified accredited investors.

Sources

  • Microsoft's Phi-3 report (April 2024): phi-3-mini has 3.8 billion parameters, was trained on 3.3 trillion tokens and scores 69 percent on MMLU; its authors say it lacks capacity to store much factual knowledge (low TriviaQA) and propose search augmentation. Source
  • LoRA Land (April 2024): 310 models from 10 base models and 31 tasks; 301 beat their base model, 224 beat the GPT-4 benchmark; on average 34 points above base and 10 points above GPT-4. Source
  • A June 2025 position paper by NVIDIA-affiliated authors argues small language models are sufficiently powerful, more suitable and more economical for many invocations in agentic systems. Source
  • GSM1k (current arXiv version): accuracy drops of up to 8 percent on fresh grade-school math problems, several model families showing systematic overfitting, Phi and Mistral among them, while many frontier models show little. Source
  • OpenAI announced Model Distillation in its API in October 2024: stored input and output pairs from frontier models such as GPT-4o are used to fine-tune cost-efficient models such as GPT-4o mini. Source
  • Meta's Llama 3.2 model card (released October 24, 2024): the 1B and 3B models used logits from Llama 3.1 8B and 70B as token-level targets, with knowledge distillation after pruning. Source
  • DeepSeek-R1 repository: six dense models from 1.5B to 70B distilled with DeepSeek-R1 samples (800k for the Qwen-based models); AIME 2024 pass@1 of 28.9 (1.5B), 72.6 (32B) and 9.3 for GPT-4o. Source
  • Stanford HAI's 2025 AI Index: the inference cost of a system performing at GPT-3.5 level fell more than 280-fold between November 2022 and October 2024. Source
  • AI Index 2025, research chapter: price per million tokens for GPT-3.5-level MMLU performance fell from $20.00 (November 2022) to $0.07 (October 2024), via Gemini-1.5-Flash-8B. Source
  • Reddit's Form 10-K for fiscal 2025 states that substantially all of the contract value associated with its licensing revenue is derived from two of its partners. Source

Request SourceX investor materials

SourceX shares company and commercial metrics with investors directly. Any offering is made only to verified accredited investors.

See if your company qualifies