AI data market
Long context enterprise data demand: the work a 1M-token window leaves
By SourceX Editorial · Reviewed by Noah Loul ·
10 sections
Short answer
Long context removes one data job, packing a small corpus into a prompt, but it does not end enterprise data demand. Windows reached 1M to 10M tokens in 2025, yet the NoLiMa benchmark measured GPT-4.1's effective length at 16K tokens. Choosing the right records, proving their source and permission, and rebuilding tasks with checkable answers remain data work. Judge that demand by dated paid commitments, not capability announcements.
Value drivers
Long context enterprise data
Key takeaways
- A bigger window retires retrieval engineering only for small corpora; Anthropic's guidance puts the line near 200,000 tokens, about 500 pages.
- Advertised and usable length differ: NoLiMa lists Llama 4 Scout, sold with a 10M-token window, as effective to about 1K tokens on its test.
- Three data jobs survive long context: selecting records, carrying provenance and permission, and reconstructing tasks with checkable outcomes.
- The demand-signal ladder ranks evidence from announcement to renewed spend, and only its top two rungs show money committed.
- Public evidence stops at disclosed contracts, and the clearest one is narrow: Reddit's FY2024 10-K shows two partners holding substantially all of its licensing contract value.
Does a bigger context window shrink enterprise data demand?
It shrinks one slice of that demand and leaves the larger part standing. A long window lets a model read a small, already curated corpus in a single call, which removes chunking and retrieval work for that corpus. It does not decide which of millions of records belong in the call, whether the model is allowed to see them, or what the right answer to a piece of work was. Long context enterprise data demand now sits in those three jobs.
The ceilings did rise quickly. Meta's model card for Llama 4, released April 5, 2025, lists a 10M-token context for Llama 4 Scout and 1M for Llama 4 Maverick. OpenAI introduced GPT-4.1 on April 14, 2025 with up to 1 million tokens of context, up from 128,000 for its GPT-4o models. Anthropic opened a 1M-token window for Claude Sonnet 4 later in 2025.
The investor decision is narrower than those headlines. Does this capability progress reduce what model developers and enterprises spend on data work, or move the spending to different jobs? A useful answer needs dated evidence of money committed by independent budget owners, sorted by how firm the commitment is. Capability announcements are the weakest evidence of all, and this article ranks the stronger kinds.
Which data work does long context actually retire?
Long context retires retrieval engineering for small, stable corpora, and a credible thesis has to concede that first. Anthropic's 2024 guidance on contextual retrieval says a knowledge base smaller than about 200,000 tokens, roughly 500 pages, can be placed in the prompt whole, with no retrieval system at all. For a policy manual or a product catalog that fits, the chunking, embedding and index tuning a team once paid for becomes optional.
The open question is how much enterprise work fits inside that box. A company's support history, ERP transactions or engineering tickets span years and several systems, and they change every day. A window holds one selection at a time, however large it is, so someone still chooses what goes in, from which version and as of which date.
Cost and wait time also grow with what goes in. OpenAI reported about a minute to first token for GPT-4.1 when given a million tokens of context. At its 1M-token launch, Anthropic priced Claude Sonnet 4 prompts above 200,000 tokens at $6 per million input tokens, against $3 per million at or below that size. Rereading a whole archive on every call is a recurring cost, and selecting the right records once is how teams avoid paying it.
Advertised window versus usable window: what two benchmarks measured
Usable length is usually far shorter than advertised length, so selection keeps paying even as the ceiling rises. Two public benchmarks measure the gap by holding a task constant and stretching the input around it.
NVIDIA's RULER repository reports that while every model it tested claims at least 32K tokens, only half of them stay above its quality threshold at 32K, and almost all fall below that threshold before reaching their claimed length. Its results table lists GLM4, advertised at 1M tokens, with an effective length of 64K, and InternLM2.5, also advertised at 1M, at 4K. Gemini 1.5 Pro held above the threshold through 128K, the longest length the table reports.
NoLiMa, an ICML 2025 benchmark from Adobe Research, strips out the literal word overlap that lets models shortcut needle-in-a-haystack tests. It defines effective length as the longest context at which a model keeps at least 85 percent of its short-context score. On that test the repository lists GPT-4.1 at 16K effective against 1M claimed, Gemini 1.5 Pro at 2K against 2M, and Llama 4 Scout at 1K against 10M.
Anthropic's 2025 engineering guidance on context draws the practical conclusion: recall degrades as tokens pile up, so context is a finite resource with diminishing returns that has to be spent with care. For an investor, the read-through is that choosing, ordering and pruning what enters the window are durable jobs rather than a stopgap until windows grow.
| Model | Advertised window | Effective length on the test | Benchmark and its definition |
|---|---|---|---|
| GPT-4.1 | 1M tokens | 16K tokens | NoLiMa: keeps 85% of its short-context score |
| Gemini 1.5 Pro | 2M tokens | 2K tokens | NoLiMa: keeps 85% of its short-context score |
| Llama 4 Scout | 10M tokens | 1K tokens | NoLiMa: keeps 85% of its short-context score |
| GLM4 (9B) | 1M tokens | 64K tokens | RULER: stays above Llama 2 7B's score at 4K |
| InternLM2.5 (7B) | 1M tokens | 4K tokens | RULER: stays above Llama 2 7B's score at 4K |
Three data jobs a longer window leaves for someone to do
Selection, provenance and permission, and task reconstruction survive any window size, because each depends on facts the records do not state about themselves. Each one also has a different buyer and a different budget line, which matters when you test whether anyone pays for it.
Checklist · 3 items
- Selection: deciding which records, versions and periods belong in a call or a test set. A window reads what it is handed; it cannot know that a policy was replaced last quarter or that half of a customer's history lives in a second system. Retrieval test sets built from real near-duplicates and stale versions measure whether selection works.
- Provenance and permission: knowing where each record came from, which version controls and who may see it. Access rules have to be applied before records are assembled into a prompt, because a document that reaches the window has already been seen. These fields come from source systems and contracts, not from the text.
- Task reconstruction: rebuilding a real piece of work with its inputs, the decision taken and an outcome that can be checked. Anthropic's guide to agent evaluations separates the transcript, everything the agent said and did, from the outcome, such as whether a booking actually exists in the database; only the outcome shows success. OpenAI's GDPval (2025) holds 1,320 tasks across 44 occupations, written from real requests and reference files by professionals averaging 14 years of experience. Tasks like these are built, not found.
0 of 3 ticked. Tick each item as you go.
The demand-signal ladder: six rungs from announcement to renewal
Rank every claim about long-context data demand by the highest rung its evidence reaches, and remember that only rungs 5 and 6 show money changing hands. The ladder separates what a lab, a vendor or a pitch deck says from what a budget owner has committed to pay.
Rungs 1 through 4 are public and cheap to collect, which is why pitches lean on them. Rung 5 becomes public only when a seller has to file. Rung 6, repeat or renewed spending by budget owners who are independent of each other, is almost never public and has to come from diligence. A thesis resting on rungs 1 to 3 is a capability story; one resting on rungs 5 and 6 describes a market.
The figure stacks the six rungs in order and marks the top rung as available only through diligence.
| Rung | Signal | Dated public example | What it shows | What it does not show |
|---|---|---|---|---|
| 1 | Capability announcement | GPT-4.1 with up to 1M tokens, April 2025 | Capacity is on offer | That anyone pays for data to use it |
| 2 | Builder guidance on inputs | Anthropic on context as a finite resource, 2025 | The vendor says curation matters | Any budget, volume or buyer |
| 3 | Evaluation built from real work | OpenAI GDPval, 1,320 tasks in 44 occupations, 2025 | A lab invests in reconstructed tasks | Whether tasks were bought from outside |
| 4 | Public request for outside data | OpenAI Data Partnerships page, 2023 | Intent to acquire non-public data | Price, volume or follow-through |
| 5 | Disclosed paid commitment | Reddit FY2024 Form 10-K, licensing revenue | Money under contract | Breadth: two partners hold most of the value |
| 6 | Repeat spend by independent budget owners | Not public | A market wider than one or two buyers | Nothing until collected in diligence |
Procurement evidence matrix: which long-context data jobs show spending
Of the jobs that survive long context, task reconstruction has the strongest public evidence and permission metadata the weakest. The matrix places each job on the ladder and names the evidence that would move it up a rung.
Read it as a map of what is still unknown in the post-training data market. No job reaches rung 6 in public sources, and that gap is the reason to ask any data company for its own dated cohorts instead of accepting a market slide. Licensed human-written content is included as a reference row because it is the only category with a disclosed contract base, and that base is narrow.
The decision this supports: long context is a reason to re-weight a data thesis, not to drop it. Weight a company toward selection, permission and reconstructed-task work, and back it only when its own evidence for those jobs reaches rung 5 with more than two independent budget owners behind it. The next evidence to collect is the rung 6 cohort table from the interview design below.
| Data job | Highest public rung | Dated evidence | Missing evidence | Diligence question |
|---|---|---|---|---|
| Selection and retrieval test sets | 2 | Anthropic retrieval guidance, 2024; RULER and NoLiMa results | Any disclosed purchase of real retrieval sets | Who paid for a retrieval set built from real records, and did they reorder? |
| Provenance and permission fields | 2 | Anthropic context guidance, 2025 | A contract that prices permission fields | Which deals priced permission metadata apart from content? |
| Task reconstruction for evaluation | 4 | GDPval tasks, 2025; Data Partnerships request, 2023 | Disclosed contracts for reconstructed tasks | How many evaluation sets became paid orders within two quarters? |
| Licensed human-written content (reference) | 5 | Reddit FY2024 10-K: two partners hold substantially all contract value | Evidence of buyers beyond two partners | How many distinct budget owners renewed? |
How to interview budget owners without double-counting demand
Interview people who control separate budgets, code every item they report as a request, an evaluation or a paid commitment, and group items by the quarter they began. That turns anecdotes into a cohort table an investor can check line by line.
Population: data, evaluation and post-training leads at model developers, plus owners of enterprise AI platform budgets. Independence rules: one respondent per budget, no two respondents whose spending runs through the same vendor contract, and no respondent introduced by the company under evaluation. Record the employer type, the purpose of the budget and the respondent's signing authority.
Coding: a request is a written ask that names a scope; an evaluation is a sample or pilot run against a written acceptance test; a paid commitment is a signed order or purchase order. Each item also records the data job it concerns and whether a longer context window changed it. The cohort is the quarter of the first written request. The headline metric is the share of budget owners whose requests reached evaluation and then paid commitment within two and four quarters, with budget owners, not conversations, as the denominator.
Uncertainty and limits: show raw counts beside every rate, report intervals instead of single figures when a cohort holds fewer than about 30 budget owners, and keep open items open rather than dropping them. Respondents under confidentiality terms will describe categories, not counterparties, and self-report overstates commitment, so ask which document stands behind each paid item. The design is published here; SourceX has not published results from it.
Five questions anchor each interview:
Checklist · 5 items
- What did you request in writing over the last four quarters, and for which data job?
- Which requests reached a sample or pilot with a written acceptance test?
- Which became a signed order, and was any of them renewed or extended?
- Did a longer context window cancel, shrink or reshape any request?
- Who else in your organization holds a separate data budget?
0 of 5 ticked. Tick each item as you go.
The skeptic's case and five risks to the long-context data thesis
The strongest alternative explanation is that capability news never turns into data budgets at all. Announcements describe what models can accept, not what anyone has agreed to buy, and some long-context gains cut data spending outright.
Three facts support the skeptic. Anthropic's 200,000-token guidance shows long context replacing retrieval work for small corpora. Most enterprise data work happens on a company's own records, done by its own staff and software vendors, which creates labor and tooling spend but no purchase of outside data. And the visible paid market is narrow: Reddit's FY2024 Form 10-K reports that "to date, substantially all of the contract value associated with our licensing revenue is derived from two of our partners," and flags the risk that those agreements lapse or are renewed on less favorable terms.
The ladder is how to tell the explanations apart. If long context were cutting demand, cohorts that began after the 1M-token releases of April 2025 would show requests stalling at rungs 3 and 4 and fewer evaluations converting to orders. If demand were moving toward selection, permission and reconstructed tasks, the same cohorts would show requests naming those jobs. Only budget-owner data can say which pattern holds. Five developments would weaken the thesis even if the shift is real:
- Usable length catches up with advertised length. If benchmarks such as NoLiMa start showing effective lengths near 1M tokens, much of the selection work for mid-sized corpora becomes optional.
- Long context gets cheap. Anthropic's launch pricing charged a premium above 200,000 tokens; if premiums and wait times fall, rereading whole archives may beat curating them for many tasks.
- Generated tasks replace reconstructed ones. If synthetic tasks match real ones on a buyer's acceptance test, the premium for records with real outcomes falls.
- Buyer concentration persists. A market in which two partners hold substantially all licensing contract value, as Reddit's FY2024 10-K describes, carries renewal and pricing risk for every supplier in it.
- Labs keep task building in-house. GDPval was written by experts recruited and vetted for the project; if developers keep commissioning professionals directly, outside suppliers of records see a smaller share of the spend.
Questions for any company selling long-context data work, SourceX included
Ask for evidence at rung 5 and above, in cohorts, with denominators you set yourself. Ask SourceX each of these too.
SourceX is one labeled application of this thesis. It licenses selected business records for AI training and evaluation from the companies that hold them, and it handles the work in between, from qualification and rights review to contracting, preparation, delivery and payment. Because the supplying company signs, after approving the scope and terms, before any delivery takes place, provenance and permission are settled before a record moves, which is the job this article argues long context leaves open.
SourceX has not published request, evaluation or commitment data, and its company and commercial metrics are available to investors directly rather than in public. Its raise, under Rule 506(c), is from accredited investors. Investors can request materials on the SourceX investor page and bring any question below, for instance how its paid commitments are coded by quarter.
- Which rung does your strongest demand evidence reach, and how many independent budget owners stand behind it?
- Of requests that began after April 2025, how many named selection, permission or task-reconstruction work, and how many were paid?
- What share of paid commitments came from your two largest buyers?
- How do you show a buyer which records were selected, from which version and under which permission?
- Which benchmark result or price change would make your largest product line unnecessary?
Methods note: how the ladder and matrix were assembled
The demand-signal ladder and the procurement evidence matrix were built as editorial frameworks for this article on 2026-10-26. Their evidence comes from Meta's Llama 4 model card, OpenAI's GPT-4.1, GDPval and Data Partnerships pages, Anthropic's 1M-token announcement and two engineering guides, the RULER and NoLiMa repositories, and Reddit's FY2024 Form 10-K. Benchmark figures are copied from the published tables as they stood on that date. Effective length follows each benchmark's own definition and is not comparable from one benchmark to the other.
Not measured: what any buyer actually spends on selection, permission or task-reconstruction data; conversion from request to commitment at SourceX or any other company; and whether long context changed any specific purchase. The interview study above is designed to measure these, and its results are not published.
Frequently asked questions
Does a 1M-token context window replace retrieval for enterprise data?
Only for small, stable corpora. Anthropic's guidance says a knowledge base under about 200,000 tokens can go into the prompt whole, but enterprise archives span years and several systems and change daily. Someone still has to choose which records and versions enter the window, and cost and wait time rise with every token added.
What does effective context length mean?
It is the longest input at which a model still performs close to its short-input level on a given test. NoLiMa sets the bar at 85 percent of the short-context score; RULER uses a fixed score threshold. On those definitions several models sold with 1M-token or larger windows held up only to a few thousand or tens of thousands of tokens.
Why would model developers keep paying for enterprise data as windows grow?
Because the scarce part is not reading capacity but records that carry their own context: which version controlled, who could see it and what the outcome was. Evaluations such as OpenAI's GDPval had to be built from real requests and reference files by experienced professionals. Whether developers buy that work from outside or build it in-house is the open commercial question.
What counts as strong evidence of AI data demand for an investor?
Signed and renewed commitments from budget owners who are independent of each other, grouped by the quarter they began. Capability announcements, vendor guidance and public data requests are weaker signals because none of them shows money committed. Ask for counts of budget owners at each stage, not counts of conversations.
Does SourceX publish its demand or conversion figures?
No. Neither demand nor conversion figures are public. Investors who request materials through sourcex.si/investors receive company and commercial metrics directly, and any offering is limited to verified accredited investors.
Sources
- Meta's Llama 4 model card lists a release date of April 5, 2025, a 10M-token context for Llama 4 Scout and 1M for Llama 4 Maverick. Source
- OpenAI introduced GPT-4.1 on April 14, 2025 with up to 1 million tokens of context, up from 128,000 for GPT-4o, and reported about a minute to first token at a million tokens. Source
- Anthropic's announcement of a 1M-token context window for Claude Sonnet 4 priced prompts over 200K tokens at $6 per million input tokens, against $3 at or below 200K. Source
- Anthropic's contextual retrieval guidance says a knowledge base smaller than 200,000 tokens, about 500 pages, can be included whole in the prompt without retrieval. Source
- NVIDIA's RULER repository reports that only half of tested models handle 32K above its threshold, that almost all fall below it before their claimed length, and lists claimed and effective lengths per model. Source
- The NoLiMa repository (ICML 2025) defines effective length as the longest context keeping 85% of the base score and lists GPT-4.1 at 16K, Gemini 1.5 Pro at 2K and Llama 4 Scout at 1K effective. Source
- Anthropic's guide to context engineering says recall accuracy declines as tokens accumulate and that context is a finite resource with diminishing marginal returns. Source
- Anthropic's guide to agent evaluations distinguishes the transcript of a trial from its outcome, using a flight booking whose success is whether a reservation exists in the database. Source
- OpenAI's GDPval covers 1,320 tasks across 44 occupations, built from requests and reference files by industry experts averaging 14 years of experience. Source
- OpenAI's Data Partnerships page seeks large-scale datasets not already easily accessible online and says it is not seeking sensitive, personal or third-party information. Source
- Reddit's FY2024 Form 10-K says substantially all of the contract value associated with its licensing revenue is derived from two of its partners, and that the agreements may not be renewed or may be renewed on less favorable terms. Source
Request SourceX investor materials
SourceX shares company and commercial metrics with investors directly. Any offering is made only to verified accredited investors.