Skip to content

AI data market

Post-training enterprise data demand: when model failures need records

By SourceX Editorial · Reviewed by Noah Loul ·

9 sections

Short answer

Post-training creates demand for company records only when a specific model failure clears four gates: it shows up on the buyer's own tasks, fixing it needs real decisions with outcomes that operating records hold, those records convert into demonstrations, preference pairs, graders or held-out tests, and cheaper fixes such as prompting, synthetic data or distillation have already failed. Count failures that clear all four, not announcements.

License terms

Post training enterprise

This page covers post-training enterprise data demand: when model failures need records.Illustrative

Key takeaways

  • Post-training demand for company records is failure-specific: count failures that clear four gates, not announcements about post-training.
  • Each format asks something different of a record: demonstrations need finished work, preference pairs need a better and a worse version, graders need a checkable outcome and tests need unseen cases.
  • Public evaluations name failures records could address: missed instructions and ignored reference files in GDPval, policy breaks and inconsistency in tau-bench.
  • Many failures close without outside records: one GDPval prompt change cut egregious PowerPoint formatting errors from 86 to 64 percent.
  • OpenAI's fine-tuning platform is winding down for new users, a reminder that the group of companies tuning models themselves can shrink.

When does post-training create demand for company records?

When a specific model failure clears four gates. It shows up on the buyer's own tasks, not only in a public benchmark. Fixing it needs evidence that operating records carry and public data does not: real requests, the decisions people made and the outcomes that followed. That evidence converts into a format post-training actually uses. And cheaper fixes, from better prompts to synthetic data, have been tried and failed.

Most claims about post-training enterprise data demand skip the gates and point at activity: new tuning methods, model releases, partnership news. Activity shows post-training is happening. It does not show that anyone will pay for a company's support tickets, claims files or project histories, because many failures can be fixed with data a developer already has or can generate.

This article builds the test from the failure end. It starts with failure modes documented in public evaluations, maps each to the business record that could supply the missing signal and the format that would use it, rules out the cheaper fixes, and then sets out how to measure whether budget owners actually buy.

What does each post-training format need from a business record?

Each format asks for a different kind of evidence, so a record that suits one may be useless for another. Demonstrations need finished work, preference data needs two versions and a judgment between them, reinforcement methods need an outcome a grader can check, and evaluation needs cases the model has never seen.

The formats are documented in primary sources. OpenAI's InstructGPT paper (arXiv, March 2022) fine-tuned GPT-3 on labeler-written demonstrations and then on rankings of model outputs, and labelers preferred outputs from the 1.3B-parameter InstructGPT model to those of the 175B GPT-3 despite it having 100 times fewer parameters. OpenAI's direct preference optimization guide trains on prompts paired with a preferred and a non-preferred response. Its reinforcement fine-tuning guide aims at expert-level performance within a domain and relies on a grader that scores each answer without a person in the loop, which requires experts to agree on the ideal output.

Reinforcement learning with verifiable rewards moves furthest from human-written data. AI2's open-instruct repository supports it for the Tülu 3 models, and the dataset named in its Tülu 3 instructions is built from math problems and precise instruction-following constraints that verifier functions can check. Business outcomes are rarely that clean, which is both the opening and the obstacle for company records.

What does each post-training format need from a business record?
FormatWhat the record must containOrdinary business examplePublic evidence of the formatSubstitute pressure
DemonstrationsA request and the finished work an expert acceptedA client brief paired with the final delivered reportInstructGPT (2022): labeler-written demonstrationsModel-written examples and distillation
Preference pairsTwo versions of an answer and which one wonA draft reply and the supervisor-approved rewriteOpenAI DPO guide: preferred and non-preferred responsesPreferences judged by another model
Graders and verifiable rewardsAn outcome a program or rubric can checkA claim decision later upheld or reversed on appealOpenAI RFT guide; Tülu 3 RLVR in open-instructPublic math and instruction-following sets
Held-out evaluationUnseen cases with a reference answerClosed support tickets with the resolution and the policy appliedAnthropic guide: turn the support queue into test casesPublic benchmarks, free but exposed

The failure-to-record map: which model failures point to company records?

Five failure modes documented in public evaluations point to evidence that ordinary business records hold. Each row names the failure, its public source, the record that could supply the missing signal, the format that would use it and the cheaper fix to rule out first. The record column is an editorial mapping, not a measured result.

Two evaluations carry most of the weight. OpenAI's GDPval paper (arXiv, October 2025) clustered the reasons experts preferred human deliverables and found instruction-following failures the most frequent cause of losses for Claude, Grok and Gemini, while GPT-5 at high reasoning effort lost mainly on formatting; Gemini and Grok also promised deliverables they did not provide, ignored reference data or used the wrong format. The tau-bench repository tests agents that work with tools under domain policy guidelines, and its leaderboard shows the best retail result falling from 0.692 on a single attempt to 0.462 when four attempts must all succeed.

Read the last column before the third. A failure that a prompt, a scaffold or a public dataset can fix creates no demand for company records, however striking the benchmark result.

The failure-to-record map: which model failures point to company records?
Failure modePublic evidenceRecord that holds the missing signalFormatRule out first
Misses instructions on long deliverablesGDPval (2025): most frequent loss cause for three model familiesBriefs, revision requests and the accepted final versionDemonstrations and preference pairsClearer prompts and self-checks
Ignores reference filesGDPval (2025): ignored reference data and missing deliverablesCases whose outcome depended on an attached documentGraders and held-out testsRetrieval and tool fixes
Breaks domain policy mid-conversationtau-bench: agents with tools and policy guidelinesTickets with the policy version and the action a supervisor approvedGraders and held-out testsPolicy text placed in the prompt
Inconsistent across repeated attemptstau-bench retail: 0.692 at pass^1, 0.462 at pass^4Many similar cases resolved the same way, with outcomesVerifiable rewardsSampling and voting at inference
Invents facts not in the inputInstructGPT (2022): 41 percent for GPT-3, 21 percent after tuning on human dataSource documents paired with summaries that stay within themDemonstrations and preference pairsRetrieval with citations
Five model failure modes on the left connected through a rule-out-cheaper-fixes band to four post-training formats on the right: demonstrations, preference pairs, graders and rewards, and held-out tests.
The failure-to-record map. Connections are an editorial mapping from public evaluations, not measured results.

Which failures close without anyone's records?

Many of them, which is the strongest alternative explanation for any post-training demand claim. Prompting, scaffolding, reinforcement learning on checkable public tasks and distillation from a stronger model can each close a failure while outside record purchases stay at zero.

GDPval shows the cheapest fix at work. Telling GPT-5 to check its output by rendering files as images eliminated black-square artifacts that had affected over half of its generated PDFs, and cut egregious formatting errors in PowerPoint files from 86 percent to 64 percent. No new training data was involved.

Reinforcement learning shows the next cheapest. DeepSeek's R1 repository describes R1-Zero, trained with large-scale reinforcement learning and no supervised fine-tuning step, developing self-verification, reflection and long reasoning chains. The same repository cautions the other way: R1-Zero suffered endless repetition, poor readability and language mixing, so the released R1 added cold-start data before reinforcement learning and used two supervised stages. DeepSeek also released six smaller models distilled from R1, a reminder that a capable model can become the data source for the next one.

The question for a supplier is whether the failure it targets survives these fixes. A record that teaches formatting loses to a prompt. A record that captures how one insurer applied one policy clause to one claim is harder to replace, because neither a public math set nor a stronger model holds that decision.

The failure-to-purchase ladder: from a benchmark miss to a renewed license

Rank evidence of demand by how close it comes to money for a single failure mode, from a public benchmark miss to a renewed purchase. Public sources fill the bottom rung; everything above it has to come from buyers and sellers under confidentiality.

At a glance

The failure-to-purchase ladder: from a benchmark miss to a renewed license

The one public paid commitment nearby is not failure-specific. Reddit's annual report for 2024 shows content licensing with contract value and states that "to date, substantially all of the contract value associated with our licensing revenue is derived from two of our partners." It proves developers pay for some outside data; it does not show which failures that data was bought to fix. The seven rungs are below:

  • Rung 1, public miss: a benchmark documents the failure, as GDPval and tau-bench do.
  • Rung 2, own-task miss: the buyer reproduces it on tasks drawn from its own workflow, for example cases converted from its support queue, as Anthropic's January 2026 evaluation guide recommends.
  • Rung 3, cheaper fixes exhausted: prompts, scaffolds, synthetic data and distillation were tried and the failure persisted.
  • Rung 4, record specified: a written request names the record type, the outcome field, the volume and the format it will feed.
  • Rung 5, sample tested: a sample went into an ablation or a held-out test and moved the failure rate.
  • Rung 6, paid commitment: a license with a value and a term.
  • Rung 7, renewal after a model change: the buyer bought again when its base model or policy changed.

How to verify post-training demand with the people who own the failure

Interview the people who own the failure and the budget to fix it, and record each failure mode's highest rung by dated cohort. This is a design only: SourceX has not conducted these interviews, and no results are published yet. It is laid out so anyone can repeat it and compare.

Checklist · 6 items

  1. Question: for each failure mode the team tracked last quarter, which fixes were tried, in what order, and did any require records from outside the organization?
  2. Population: post-training, applied AI and evaluation leads at model developers and at enterprises tuning models, plus the procurement staff who sign data contracts, with no commercial relationship to the interviewer.
  3. Unit of analysis: one failure mode at one organization, not one interview, so a team tracking three failures contributes three rows.
  4. Stages: requests, sample tests and paid commitments counted separately within the quarter of first request, with each stage's denominator stated.
  5. Variables: failure mode, format, record type requested, cheaper fixes tried first, rights obstacles, and whether a synthetic or distilled alternative won.
  6. Reporting: counts with ranges below 30 organizations, attribution by role and date only, and at least one failure that closed without outside data shown beside the headline.

0 of 6 ticked. Tick each item as you go.

Where the company-records thesis could fail

It fails if buyers stop tuning models themselves, if the records cannot be licensed, or if too few buyers pay. Each risk has a public marker an investor can watch.

At a glance

Where the company-records thesis could fail

  • Shrinking self-serve tuning: OpenAI's direct preference optimization guide, read October 26, 2026, says its fine-tuning platform is winding down, is no longer available to new users and lets eligible existing customers create training jobs until January 6, 2027. If customer-side tuning moves inside model vendors, fewer enterprises buy post-training records directly.
  • Rights friction: OpenAI's data partnerships page rules out sensitive or personal information and material owned by third parties, and business records often contain both. Records that cannot be cleared never turn into demand.
  • Grader gap: reinforcement fine-tuning needs experts to agree on the ideal output. Many business outcomes are disputed, delayed or confounded, so a record may describe a decision without proving it was right.
  • Substitution: distilled models, public verifiable-reward sets and prompt fixes can close a failure before a purchase is approved.
  • Buyer concentration: two partners account for substantially all of Reddit's licensing contract value in its annual report for 2024, and buyers of post-training records could be just as few.

What to ask a supplier of post-training records, SourceX among them

Ask which failure a supplier's records fix and which rung its evidence reaches. The questions apply to SourceX as much as to anyone.

SourceX is one application of this question. Acting for companies that license selected business records to AI developers, it qualifies and sources supply, clears rights, handles the contract, prepares and delivers the data and settles payment, and it locates each dataset on request from U.S. businesses instead of keeping an inventory. Its rule of thumb is that buyers expect two or more years of consistent history and records numbering in the tens of thousands or more within one system, judged deal by deal, which matters here because a failure mode needs many comparable cases with outcomes.

Request, deal and commercial metrics for SourceX are not public; investors see company and commercial metrics directly. SourceX's current raise follows Rule 506(c) and is restricted to accredited investors, and the investors page, where materials are requested, is the place to bring any question below.

Checklist · 6 items

  1. Which documented failure modes do your records address, and what shows each failure survived cheaper fixes?
  2. For each delivered dataset, which format did the buyer feed it into?
  3. How many written requests named a failure mode and an outcome field, by quarter of first request?
  4. How often did a sample test move the buyer's failure rate, and how was that measured?
  5. Which requests were lost to synthetic data, distillation or a prompt fix?
  6. Has any buyer bought again after changing its base model?

0 of 6 ticked. Tick each item as you go.

Methods note: sources behind the failure-to-record map

The four-gate test, the format matrix, the failure-to-record map, the failure-to-purchase ladder and the interview design are the article's own editorial frameworks, set out on 2026-10-26. Failure modes come from OpenAI's GDPval paper and the tau-bench repository; formats from the InstructGPT paper, OpenAI's preference and reinforcement fine-tuning guides, AI2's open-instruct repository and DeepSeek's R1 repository; task sourcing from Anthropic's January 2026 evaluation guide; rights and concentration markers from OpenAI's data partnerships page and Reddit's annual report for 2024. The GitHub repositories and Anthropic's guide were read directly; the rest were verified through searches restricted to each publisher's domain.

Not measured: whether any record type fixed any failure for any buyer, prices, purchase volumes, the share of failures closed by cheaper fixes, or any SourceX data. Benchmark figures describe the models and versions tested at the time and will change.

Frequently asked questions

Which business records are most useful for post-training?

Records that pair a real request with a decision and a checkable outcome: resolved tickets with the policy applied, claims with their final determination, drafts with the approved revision. Those can become demonstrations, preference pairs, graders or held-out tests. Records without an outcome describe activity but rarely fix a specific failure.

Does reinforcement learning reduce demand for human-written data?

For tasks with checkable answers, it can. DeepSeek's R1-Zero learned to reason through reinforcement learning without a supervised step, and AI2's Tülu 3 recipe uses verifiable rewards on math and instruction-following constraints. But R1-Zero's readability problems led DeepSeek to add cold-start data, and business outcomes rarely come with a program-checkable answer.

How can an investor tell real post-training demand from activity?

Ask for the failure mode, the cheaper fixes tried first and the highest rung reached: a written request naming the record type, a sample test that moved the failure rate, a paid commitment or a renewal. Model releases and benchmark misses sit at the bottom of that ladder.

Why does OpenAI's fine-tuning wind-down matter for data demand?

OpenAI's direct preference optimization guide, read October 26, 2026, says its fine-tuning platform is winding down, is closed to new users and accepts training jobs from eligible existing customers until January 6, 2027. Fewer self-serve tuning routes can mean fewer enterprises buying post-training records directly, so check current vendor documentation before sizing that buyer group.

Has SourceX published evidence of post-training demand?

No. None of those metrics is public, though investors can get company and commercial metrics from SourceX directly. Materials are available on request at sourcex.si/investors, and SourceX limits any offering to verified accredited investors.

Sources

  • OpenAI's InstructGPT paper (2022) fine-tuned GPT-3 on labeler demonstrations and output rankings; outputs of the 1.3B InstructGPT model were preferred to the 175B GPT-3, and InstructGPT invented information about half as often (21 percent versus 41 percent). Source
  • OpenAI's reinforcement fine-tuning guide targets expert-level performance within a domain, scores each response with a grader that runs without a person in the loop, and requires expert graders to agree on the ideal output. Source
  • OpenAI's direct preference optimization guide trains on prompts paired with preferred and non-preferred responses, and says the fine-tuning platform is winding down, closed to new users, with training jobs for eligible existing customers until January 6, 2027. Source
  • AI2's open-instruct repository provides code for DPO, preference fine-tuning and reinforcement learning with verifiable rewards, and its Tülu 3 instructions use an RLVR dataset of math and instruction-following constraint problems checked by verifier functions. Source
  • DeepSeek's R1 repository says R1-Zero was trained with large-scale reinforcement learning without supervised fine-tuning, suffered endless repetition, poor readability and language mixing, and that R1 added cold-start data with two RL and two SFT stages; six distilled models were released. Source
  • The tau-bench repository tests agents using domain-specific API tools and policy guidelines; on its retail leaderboard the best tool-calling result falls from 0.692 at pass^1 to 0.462 at pass^4. Source
  • The GDPval paper (October 2025) found instruction-following failures the most frequent loss cause for Claude, Grok and Gemini and formatting for GPT-5; a render-and-check prompt removed black-square PDF artifacts and cut egregious PowerPoint formatting errors from 86 to 64 percent. Source
  • Anthropic's guide to agent evaluations (January 9, 2026) recommends converting user-reported failures from the bug tracker and support queue into test cases. Source
  • OpenAI's data partnerships page seeks data expressing human intention, such as long-form writing or conversations, and is not seeking sensitive or personal information or information that belongs to a third party. Source
  • Reddit's Form 10-K for fiscal 2024 says that, to date, substantially all of the contract value associated with its licensing revenue is derived from two of its partners, with renewal risk. Source

Request SourceX investor materials

SourceX shares company and commercial metrics with investors directly. Any offering is made only to verified accredited investors.

See if your company qualifies