Skip to content

Deal economics

Dataset volume's diminishing value: 1,800x the lines, 3x the patterns

By SourceX Editorial · Reviewed by Noah Loul ·

9 sections

Short answer

Dataset volume shows diminishing value when new records repeat what an inventory already covers while every record still has to be cleared and prepared. In Loghub's public system logs, full logs held about 1,800 times the lines of their 2,000-line samples but only about 3 times the message templates, and the usable share of a growing sample fell from 59.5% to 24.7%. Value the next block, net of processing.

Value drivers

Dataset volume diminishing

This page covers dataset volume's diminishing value: 1,800x the lines, 3x the patterns.Illustrative

Key takeaways

  • Total value can rise while average value per record falls: repetition grows, marginal coverage shrinks and processing cost scales with every row.
  • Across 14 Loghub-2.0 systems, 1,801 times more annotated lines than the 2,000-line samples held 3.04 times more templates.
  • Hashing understates redundancy: 95.7% of 32,000 sample lines repeated an earlier template, but only 51.6% repeated an earlier line exactly.
  • With at most 20 examples credited per template, each line added in the last doubling contributed 0.173 usable units, down from 0.595 in the first block.
  • New patterns arrive in bursts, so ask for accumulation curves by period instead of extrapolating an early plateau.

Does more volume make a dataset worth more?

More volume raises a dataset's total value only while the new records add coverage a buyer does not already have, and average value per record usually falls as an inventory grows. Three curves pull apart: repetition rises, the coverage each new block adds shrinks, and the cost of clearing, preparing and delivering keeps climbing with every row.

Keep two measures apart. Average value is usable value divided by every raw record offered. Marginal value is the usable value the next block adds, divided by the records in that block. A seller can be right that the larger inventory is worth more in total and still be offering a final block that costs more to process than it adds.

Cost-based numbers hide the problem. When Statistics Canada built experimental estimates of the value of data in 2019, it valued data at the cost of producing it, because the agency had very little information on market sales of data. A cost figure rises with every row stored, whether or not the row teaches anything new. Use value does not, and an investor who prices an inventory by its row count inherits the cost logic without noticing.

This piece measures the coverage curve on public log data, turns it into a worksheet for valuing the next block of records, and tests the explanations that could make a falling average look like something else.

What 50 million annotated log lines say about coverage

In public system logs, coverage grew about three times while volume grew about 1,800 times. The Loghub project publishes labeled logs from real systems, and the Loghub-2.0 maintainers report, for 14 of them, how many distinct message templates appear in a 2,000-line labeled sample and how many appear in the full annotated log. A template is a message pattern with its variable parts masked, such as an HDFS line reporting that a packet responder for a given block is terminating.

Pooled across the 14 systems, 28,000 sample lines held 1,147 templates and 50,416,620 annotated lines held 3,488: 1,801 times the lines for 3.04 times the patterns. The median system grew 105 times in lines and 2.08 times in templates. HDFS is the starkest case, with 11,167,740 annotated lines, 5,584 times its sample, carrying 46 templates against the sample's 14. The spread runs from OpenStack, where 104 times the lines added 1.12 times the templates, to Thunderbird, where 8,301 times the lines added 8.33 times.

Two cautions keep the ratio honest. Loghub-2.0 re-annotated the full logs, so some template growth may reflect more careful labeling rather than volume alone. And a log template is a narrow proxy for a distinct situation: tickets, claims and work orders vary in free text and outcomes in ways a template count misses. The direction is still hard to argue with.

The eight systems with the most annotated lines, as the maintainers report them, ranked by annotated lines:

What 50 million annotated log lines say about coverage
SystemAnnotated lines (Loghub-2.0)Templates in 2,000-line sampleTemplates in full logLines grew vs templates grew
Thunderbird16,601,7451491,2418,301x vs 8.33x
Spark16,075,117362368,038x vs 6.56x
HDFS11,167,74014465,584x vs 3.29x
BGL4,631,2611203202,316x vs 2.67x
OpenSSH638,9462738319x vs 1.41x
HPC429,9874674215x vs 1.61x
HealthApp212,39475156106x vs 2.08x
OpenStack207,6324348104x vs 1.12x

Why do records that look unique still repeat each other?

Most repetition sits below the exact-match level, where a simple hash check cannot see it. Across the 16 Loghub 2,000-line samples, 30,637 of 32,000 lines (95.7%) repeated a template that had already appeared earlier in the same file, but only 16,513 (51.6%) repeated an earlier line character for character.

HDFS shows the gap at its widest. None of its 2,000 lines is an exact copy of an earlier one, because each carries its own block identifier, yet 1,986 of them repeat one of just 14 templates. A seller who reports zero duplicates after hashing would be telling the truth about bytes and nothing useful about coverage.

Repetition matters because it can subtract, not just fail to add. Hernandez and colleagues reported in 2022 that repeating 0.1% of a training set 100 times degraded an 800-million-parameter model to the performance of one half its size. The dose decides: Muennighoff and colleagues found in 2023 that up to four epochs of repeated data changed loss negligibly compared with unique data, while with heavier repetition the value of added compute decayed toward zero.

For diligence, ask for counts after near-duplicate or template-level collapse, not only after hashing, and ask which method produced them. Near-duplicate detection, for example with MinHash, is well documented, so a seller who has never run it is reporting a storage figure, not a coverage figure.

How fast does each added block stop paying?

In these logs, each doubling of history added fewer usable units per line than the one before, even though new templates kept arriving in bursts. Reading each sample in file order, which runs in ascending line number, the first 250 lines of the 16 logs introduced 497 templates; the last 250 introduced 186.

Coverage becomes a value proxy once depth is capped. Credit each template with at most 20 examples, an editable assumption standing in for the point where another instance of the same message pattern stops teaching a buyer's model much. On that basis the usable share of lines fell from 59.5% at 250 lines per log to 24.7% at 2,000. Each line added in the last doubling contributed 0.173 usable units, against 0.595 per line in the first block.

That marginal figure is the number to put against cost. If a usable unit is worth one value unit, an illustrative scale, the last doubling is worth preparing only when processing a raw line costs less than 0.173 value units, while the first block would justify up to 0.595. Swap in your own unit values; the ratio between blocks is the part the data supplies.

Novelty does not arrive smoothly, which is the main reason not to extrapolate. The Linux sample showed 23 templates in its first 1,000 lines and 95 new ones in the next 1,000; Thunderbird showed 25, then 124. A plateau fitted to the early lines would have written off exactly the records that held the new behavior.

A quick screen still helps. I. J. Good's 1953 estimate, credited in part to Turing, treats the share of patterns seen exactly once as the chance that the next record shows something new. At 1,000 lines it predicted 2.1% pooled across the 16 logs; the next 1,000 lines delivered 3.1%. In seven logs the next 1,000 lines brought more new templates than predicted, in Linux and Thunderbird more than ten times as many; four matched the prediction and five fell short of it. Use the singleton share to decide where to sample next, never as the forecast itself.

The pooled curve for the 16 samples, with the depth cap at 20 examples per template:

How fast does each added block stop paying?
Lines per log (16 logs)Distinct templatesUsable units at cap 20Usable share of linesUsable units per added line
250 (4,000 lines)4972,37959.5%0.595 (first block)
500 (8,000 lines)6533,46043.3%0.270
1,000 (16,000 lines)8645,13332.1%0.209
2,000 (32,000 lines)1,3637,89924.7%0.173
Bar chart of the usable share of log lines at four inventory sizes, falling from 59.5% at 250 lines per log to 24.7% at 2,000, with a callout comparing 1,801 times more lines with 3.04 times more templates.
Usable share with at most 20 examples credited per template, 16 Loghub samples; the cap is an editable assumption. Source: Loghub and Loghub-2.0 (logpai), SourceX count.

The marginal-block worksheet: rights first, then coverage, then cost

Value the next block of records rather than the inventory as a whole, and normalize rights and scope before doing any coverage arithmetic. A row the supplier cannot license, or one outside the buyer's written purpose, has no coverage value however novel it is, so steps one and two come before deduplication.

The last step charges processing to every raw row, including rows the earlier steps later discard, because someone has to read, clear and de-identify a row before learning that it repeats another. That is why heavy repetition lowers average value even after the duplicates are removed: their cost stays in the denominator.

Run each candidate block through the seven steps below and keep the block only when its marginal units, multiplied by your value per unit, exceed its processing cost:

The marginal-block worksheet: rights first, then coverage, then cost
StepWhat it removes or addsMeasure withLoghub reading or editable inputAsk the seller
1. Rights-cleared rowsRows outside the license grant, owned by customers or restrictedRows by system and record class after rights reviewNot measured in the logs; enter your own shareWhich record classes were excluded, and on what document?
2. Usable scopeRows outside the buyer's purpose, period or fieldsRows matching a written requestEditable inputWhich request was this block sized against?
3. Exact duplicatesIdentical copiesHash of normalized content51.6% of 32,000 sample linesWhat were the counts before and after hashing?
4. Near-duplicatesSame pattern with different identifiersTemplate or similarity clustering95.7% repeated an earlier templateHow many distinct patterns per 10,000 rows?
5. Depth capExamples beyond what the task needsExamples kept per patternCap of 20: 24.7% of lines usable at 2,000 per logHow many examples per pattern does the named task need?
6. Marginal blockCoverage the newest block addedUsable units added per row added0.173 per line in the last doublingCan we see the accumulation curve by period?
7. Processing costReview, de-identification and QA on every raw rowCost per raw row, discarded rows includedEditable input, in value unitsWhat does clearing a row you later discard cost?

Could falling per-record prices be a pricing habit rather than lost value?

Partly, and the worksheet is how to tell the two apart. Asking prices and published payout ranges are offers, not cleared transactions, and a per-record figure that falls with size can reflect negotiation, a thin pool of buyers or simple list-price convention rather than shrinking coverage.

Run the test in order. Normalize each deal for rights, usable scope, preparation work and the buyer's stated purpose, then compare value per usable unit instead of per raw row. If the per-unit figure still falls with size after normalization, coverage is doing the work; if it flattens, the discount was a pricing habit and the larger inventory may be underpriced.

Buyer purpose can also reverse the reading. A team training a broad model on general text may pay for scale itself, while a team building an evaluation set pays for distinct cases and hardly at all for the thousandth instance of a common one. The same inventory can sit on a steep curve for one buyer and a flat curve for another, so a valuation that names no buyer task has skipped the step that decides the answer.

What could make the volume discount wrong?

The thesis that volume lowers average value fails in five identifiable situations, and an investor should check each before applying a discount to a large inventory.

  • Rare events need long histories. Loghub's BGL log, which the repository lists as spanning 214.7 days, tags alert messages separately from routine ones; events like these are scattered across a long record, and a short, dense sample can miss them, which is the Linux and Thunderbird pattern at small scale.
  • Repetition carries base rates. Muennighoff and colleagues found modest repetition nearly as useful as unique data, and for forecasting or evaluation the frequency of a pattern is itself information that aggressive deduplication throws away.
  • Processing cost can fall. Sorscher and colleagues proposed a cheap self-supervised pruning metric that performed comparably to the best supervised ones, which were computationally intensive; if finding redundancy gets cheap enough, step seven shrinks and large blocks recover value.
  • Templates understate business variety. A support ticket or claim file differs from its neighbors in free text, attachments and outcome, so coverage curves measured on templated logs probably flatten faster than curves for richer records.
  • Breadth across suppliers resets the curve. A thousand records from a new company usually add more coverage than a thousand more from a supplier already in the inventory, so a portfolio of many mid-sized sources can escape the discount that a single giant source cannot.

Questions to ask any data company about volume, SourceX included

Ask for the curve, not the count. Any company that sources, prepares or resells business records should be able to answer these with its own numbers, and SourceX is no exception.

SourceX's how-it-works page lists volume as one of ten fit signals, next to historical depth and uniqueness, and marks that panel as not a valuation. The company shares company and commercial metrics with investors directly rather than publishing them. SourceX is raising from accredited investors under Rule 506(c), and its materials are available on request through the investors page.

Six questions that separate inventory from coverage:

Checklist · 6 items

  1. What does the accumulation curve look like by period: distinct patterns or cases added per quarter of history, per source system?
  2. How many records remain after exact hashing, and how many after near-duplicate or template-level collapse, and which method was used?
  3. For a named buyer task, how many examples per pattern does acceptance require, and what share of the inventory falls past that cap?
  4. What does it cost, in staff hours or processing units, to clear a raw record that is later discarded as repetitive or out of scope?
  5. Do rights exclusions vary across blocks, for example older records written under different customer terms?
  6. Are volume figures in investor materials raw rows, rights-cleared rows or delivered usable units?

0 of 6 ticked. Tick each item as you go.

Methods note: how the Loghub counts were made

Two public sources were counted on 2026-10-28. The first is the 16 files named <system>_2k.log_structured.csv in the logpai/loghub repository at commit dd61d0952749ee7963bde24220d1be5ede023033. Each file has 2,000 rows; files were read as UTF-8 with any byte-order mark dropped, every row was kept, and rows were taken in file order, which was checked to run in ascending line number. A row's template is its EventId value, compared exactly as written. A repeat line is a row whose EventId appeared in an earlier row of the same file; an exact duplicate is a row whose Content string, compared without trimming or case folding, appeared in an earlier row.

Prefix sizes were the first 250, 500, 1,000 and 2,000 rows of each file. New templates in a block are those whose first row falls inside it (rows 1-250 or 1,751-2,000). Usable units at a prefix sum, over templates, the smaller of the template's count and 20, and the usable share divides them by every line in the prefix. Marginal units per added line divide the pooled change in usable units by 16 times the change in prefix length. The singleton share at 1,000 rows divides the templates seen exactly once in rows 1-1,000 by 1,000; the observed rate counts templates first seen in rows 1,001-2,000, divided by 1,000. Pooled rates sum numerators and denominators before dividing, and each figure is rounded once, half up.

The second source is the dataset table in the logpai/loghub-2.0 README at commit ac4aad2ea86f561e5a0acbc6587fa28b487259f9, which covers 14 systems; Android and Windows are absent from it. Lines grew is annotated lines divided by 2,000; templates grew is full-log templates divided by sample templates, computed as exact fractions; the pooled figures divide summed annotated lines by 28,000 and summed full-log templates by summed sample templates, and the README's average row is not used. Medians use the mean of the two middle values. The template counts the script found in the 16 samples match the README's Loghub-2k column for every system it lists.

Not measured: rights, buyer value, processing cost or any business-record dataset; the cap of 20 and the value unit are assumptions. Loghub's terms make the data freely available for research or academic work and ask users to cite Zhu and colleagues' Loghub paper (ISSRE 2023); this analysis publishes only aggregate counts. The counting scripts read only these files and are available on request.

Frequently asked questions

Is a bigger dataset always more valuable?

Not per record. A bigger dataset is worth more in total only while new records add cases a buyer lacks; once they mostly repeat known patterns, value per record falls even if total value edges up. In Loghub's public logs, about 1,800 times more lines held about three times more templates.

How do you measure diminishing returns in a dataset?

Build an accumulation curve: count distinct patterns, cases or outcomes as records are added in time order, and compute the usable units each new block adds per record. Cap the examples credited per pattern at what the buyer's task needs, then compare the marginal figure with the cost of processing a raw record.

What is the difference between exact and near-duplicate records?

Exact duplicates match character for character and a hash check removes them. Near-duplicates share a pattern but differ in identifiers, timestamps or small edits. In the Loghub samples, 51.6% of lines were exact repeats but 95.7% repeated an earlier template, so hashing alone understated redundancy by a wide margin.

Should the price per record fall as dataset volume rises?

Often, but only a normalized comparison shows whether falling coverage or pricing habit drives it. Adjust each deal for rights, usable scope, preparation and buyer purpose, then compare value per usable unit. Asking prices and payout ranges are offers, not cleared transactions, so they cannot settle the question alone.

What does the Good-Turing estimate tell a dataset buyer?

It treats the share of patterns seen exactly once as the chance that the next record shows something new, a method from I. J. Good's 1953 paper. It is a useful screen for where to sample next, but in Loghub's logs it missed late bursts of new templates, so it should not replace sampling later periods.

Sources

  • The Loghub repository publishes 16 labeled 2,000-line system log samples, each line tagged with an event template, and states that the datasets are freely available for research or academic work. Source
  • The Loghub-2.0 maintainers report annotated line counts and template counts for 14 systems, next to the template counts of the original 2,000-line Loghub-2k samples. Source
  • Hernandez et al. (2022) report that repeating 0.1% of training data 100 times can degrade an 800M-parameter model to the performance of a 400M-parameter model. Source
  • Muennighoff et al. (2023) report that up to 4 epochs of repeated data changed loss negligibly compared with unique data, while with more repetition the value of added compute decays toward zero. Source
  • Sorscher et al. (2022) describe test error falling as a power of training set size, show that good data pruning can beat that scaling, and note that the best pruning metrics were computationally intensive. Source
  • I. J. Good (Biometrika, 1953) set out estimates, credited in part to Turing, of the share of a population represented by the species already seen in a sample. Source
  • Statistics Canada's 2019 experimental estimates valued data at the cost of producing it because the agency had very little information on market sales of data. Source
  • SourceX's how-it-works page lists volume as one of ten fit signals, alongside historical depth and uniqueness, and labels the panel as not a valuation. Source
  • SourceX's investor page says it shares company and commercial metrics with investors directly. Source

Request SourceX investor materials

SourceX shares company and commercial metrics with investors directly. Any offering is made only to verified accredited investors.

See if your company qualifies