Skip to content

Deal economics

Dataset valuation duplication adjustment: count unique cases first

By SourceX Editorial · Updated

10 sections

Short answer

Pick the unit a buyer would learn from, such as one job, claim or support issue, then link every record to its case and remove exact copies, migration duplicates, test records and near-duplicates. Report raw records, each removal and the unique cases left, by system and year, with the method and thresholds used. Discuss value only after that count exists, and only as a range that depends on it.

Value drivers

Dataset valuation

This page covers dataset valuation duplication adjustment: count unique cases first.Illustrative

Key takeaways

  • Count cases, the unit a buyer learns from, not rows or messages.
  • Remove exact copies, migration duplicates, test records and near-duplicates in that order, keeping each count.
  • Report a strict and a loose adjusted count, because near-duplicate detection is approximate.
  • Review a sample of merges so recurring events are not collapsed into one case.
  • The case definition and outcome completeness move the count more than any threshold.
  • Talk about value only after the inventory exists, and tie the range to it.

How do you count unique business cases before discussing dataset value?

Start from the case, not the row. Decide what a single business event is for the work a buyer would train or evaluate on, link every exported record to the case it belongs to, and strip out copies that add rows without adding cases. The result is a duplicate adjusted volume inventory: for each system and year, the raw record count, every removal step with its count and method, and the number of unique cases that remain, with how many of those have a recorded outcome.

This matters for valuation because volume is one of the first numbers anyone asks about, and raw counts from business systems are routinely inflated. A help desk that stores every reply as a record, an ERP migrated twice, or a CRM that logs automated emails can each multiply the apparent size of a dataset without adding a single new situation. A dataset valuation duplication adjustment done up front keeps the conversation on numbers that will survive a buyer's own checks.

Why raw record counts overstate what a buyer can use

Duplicates are common even in datasets built by specialists. Lee and colleagues found that widely used language modeling datasets contained many near-duplicate examples and long repeated substrings, including a single 61-word sentence repeated over 60,000 times in one corpus, and that deduplicating them reduced memorized output and evaluation overlap without hurting model quality. Business exports have their own sources of repetition: quoted email threads, cloned work orders, merged tickets and system copies left behind after migrations.

Repeats are also worth less than new cases to a model. Muennighoff and colleagues found that repeating training data for up to about 4 epochs made a negligible difference to loss compared with unique data, but that returns from further repetition fell toward zero. Hernandez and colleagues reported that repeating a small fraction of training data can cause disproportionately large harm to performance. A buyer who finds that a third of a dataset is repetition will discount it, and will also trust the rest of the description less.

Which unit counts as one case in your records?

Choose the unit that carries a full situation from start to outcome, because that is what makes a record teachable. The table gives common choices; the right one depends on the task a buyer has in mind, so record the unit you chose and why.

Which unit counts as one case in your records?
Business typeRaw records in the exportCase unitLinking key
Field serviceWork order lines, visits, technician notes, photosOne job from call to completionWork order or job number
Customer supportEmails, chat messages, ticket updatesOne customer issue, including reopened and merged ticketsTicket id plus merge history
Insurance claims administrationClaim notes, payments, correspondence, documentsOne claim through closureClaim number
B2B salesCRM activities, emails, calls, quote versionsOne opportunity from creation to win or lossOpportunity id
Manufacturing qualityInspection records, NCR updates, rework ordersOne nonconformance through dispositionNCR number
Legal operationsMatter notes, documents, time entriesOne matter or one contract negotiationMatter or contract id

Which kinds of duplicates should come out, and how do you find each?

Each kind of duplicate needs a different detector, and mixing them up either leaves copies in or merges cases that are genuinely different. Run them in order from cheapest and most certain to most approximate, and keep each step's count.

Which kinds of duplicates should come out, and how do you find each?
Duplicate kindWhat it looks likeHow to detect itMain risk
Exact copiesIdentical files or rows exported twiceHash of normalized contentLow; easy to remove
Migration copiesSame record under old and new system idsLegacy id mapping table, created and closed datesMissing mapping leaves both copies in
Test and internal recordsTraining tickets, demo accounts, internal jobsAccount flags, known test customers, creator roleReal records removed by a loose rule
Near-duplicate textTemplated replies, cloned work orders with small editsMinHash with locality-sensitive hashing at a set similarity thresholdOver-merging distinct cases that share boilerplate
Same case, many recordsThread messages, reopened tickets, split claimsRecord linkage on keys, dates and partiesLinking two cases that only look alike

What should the duplicate adjusted volume inventory include?

One inventory per source system, with one row per field below, filled for each year of history. Keep the numbers and the method together, so anyone who reruns the steps on the same export gets the same counts. Use composite keys where one id is not enough; the Table Schema standard, for example, lets a uniqueKey span several fields and excludes null values from the uniqueness check, which is worth knowing when legacy rows have blank ids.

Most of the work is ordinary data handling that an analyst can do with SQL or a notebook on a copy of the export. Keep the queries or scripts with the inventory, store the hashes you computed, and note who ran each step and when. If counts are later questioned, rerunning the same steps on the same export should reproduce them exactly, and that reproducibility is part of what makes the number credible.

What should the duplicate adjusted volume inventory include?
FieldWhat to recordHow to compute itWhy a buyer asks
System and export dateSource system, version, export date and scopeFrom the export jobConfirms which data the counts describe
Raw recordsRows or documents exportedDirect count per yearBaseline for every adjustment
Exact copies removedRecords with identical normalized contentContent hash, keep firstShows basic hygiene
Migration and system copies removedRecords duplicated across system idsLegacy id map, date matchCommon in multi-system histories
Test and internal records removedNon-customer recordsFlags and rules, listedShows nothing synthetic is counted
Near-duplicates removedRecords above the similarity thresholdMinHash and LSH, threshold stated, sample reviewedSeparates templated volume from real variety
Unique casesRecords linked into cases after removalsLinking key or record linkage, method statedThe volume a model can learn from
Cases with outcomeUnique cases with a recorded resultClosed status plus outcome field presentOutcome-bearing cases carry most of the value
Coverage by year and segmentUnique cases per year, product line or regionGroup countsShows consistency over time
Method notes and rangeThresholds and a strict and loose countRerun with two thresholdsShows how sensitive the count is

Setting thresholds without over-merging distinct cases

Normalize text before comparing it, or the detector measures formatting instead of content. Strip email signatures, quoted earlier replies, disclaimers, timestamps and auto-generated status lines, and compare the remaining text. Without that step, two different jobs written on the same template look identical, and the same job exported with different footers looks distinct.

Report a range, not a single adjusted count, because near-duplicate detection is approximate. The datasketch documentation for MinHash LSH states plainly that queries return false positives, pairs below your threshold, and false negatives, qualifying pairs that are missed, though pairs with higher similarity are always more likely to be returned. Run the step at a strict and a loose threshold and report both counts.

Then check the merges by hand. Draw a random sample of pairs the detector called duplicates and read them: two maintenance visits to the same unit a month apart are separate jobs even if the notes are almost identical. Record linkage tools make the same distinction formally. Splink, an open-source package for probabilistic record linkage, describes deduplication as linking records within one dataset, linkage as linking across datasets, and the overall aim as generating a unique identifier where the data lacks one. Both tasks need a reviewed sample before the counts are trusted.

Worked example: from export rows to unique jobs at a field service company

This example is illustrative; the company and every count are invented to show the method. A commercial HVAC contractor exports seven years of history from its field service system and an older system it replaced. The export contains 1,240,000 rows: work order lines, technician notes, status updates and attachments.

Grouping by work order number gives 286,000 work orders. Exact-copy removal drops 9,000 that were exported twice from the old system. The legacy id map shows 31,000 jobs that exist in both systems after the migration, so one copy of each is removed. Internal and training jobs account for 4,000 more. Near-duplicate detection at a strict threshold flags 18,000 cloned work orders; a reviewed sample of 200 flagged pairs shows that most are true clones but that recurring quarterly inspections were also caught, so the team keeps inspections with distinct visit dates and removes 11,000. The strict-to-loose range runs from 224,000 to 231,000 unique jobs, of which 198,000 to 204,000 have a recorded completion outcome.

The inventory now says something defensible: about 200,000 completed jobs over seven years, with method, thresholds and a by-year table. That is a smaller number than the raw export, and a far more credible one.

When repeated records are the signal, not noise

Do not deduplicate away the pattern a buyer may want. Recurring events, such as monthly inspections, repeat claims from the same policyholder or a customer who reopens the same issue, are separate cases with their own dates and outcomes, and the repetition itself can be what a model learns from. Collapse them only when the content and the event are both the same.

Some duplicates are useful when kept apart and labeled. Pairs of records describing the same entity in two systems are training material for matching and linkage tasks, and repeated template replies show how a team actually communicates. In those cases keep the copies, mark them with a group id, and count them separately from unique cases rather than inside them.

Which assumption changes the count and the buyer fit the most?

The case definition usually moves the number more than any threshold does. Counting support emails instead of support issues can multiply volume several times over, and a buyer who expects issues will re-count. Outcome completeness comes next, because cases without a recorded result are worth far less for most training and evaluation tasks. Keep these assumptions visible in the inventory so a discussion of value can say which of them it depends on.

Which assumption changes the count and the buyer fit the most?
AssumptionEffect on the unique countEffect on buyer fitEvidence to show
Case unit chosenLargest; can change counts severalfoldMust match the buyer's task unitUnit definition and linking key
Near-duplicate thresholdModerate; reported as a rangeToo loose removes real varietyStrict and loose counts with a reviewed sample
Migration mapping completenessModerate in multi-system historiesDuplicates found later erode trustLegacy id map coverage
Outcome field completenessSets the outcome-bearing subsetOften decides whether a task is possibleShare of cases with a recorded result by year
Rights over each system's recordsCan remove whole systems or yearsRecords without clear rights cannot be offeredContracts, notices and ownership for each source

Bringing a defensible count to a first licensing conversation

A duplicate adjusted inventory lets a company answer the volume question honestly the first time and spend the rest of the conversation on fit, rights and preparation. As a rough guide, buyers look for consistent history of two or more years and tens of thousands of records or more in one system, while smaller sets can still qualify when the work they describe is rare.

Owners who take the inventory to SourceX keep it in-house at first: qualification starts with a short company assessment that needs no data upload. If the records fit, SourceX handles the rights review, contracting, preparation and delivery for the cases the company selects.

Frequently asked questions

What is a duplication adjustment in dataset valuation?

It is the step that converts raw record counts into unique, usable cases before anyone discusses value. It removes exact copies, migration duplicates, test records and near-duplicates, links related records into cases, and reports what remains with the method used. The adjusted count is what a buyer's own checks will reproduce.

Should near-duplicates always be removed?

No. Remove near-duplicates that are copies of the same event, such as cloned work orders or quoted email threads. Keep recurring events with distinct dates and outcomes, and keep cross-system pairs when they are useful as matching data, labeled with a group id and counted separately.

How do I count cases when records are spread across several systems?

Pick a case key that exists in all of them, or build one through record linkage on ids, dates and parties. Deduplicate within each system first, then link across systems, and report the counts at both stages so a reviewer can see where overlap was found.

Will a smaller deduplicated count hurt a data licensing discussion?

It usually helps. A buyer will run its own duplicate checks, and a count that shrinks after delivery damages trust more than a modest number stated accurately at the start. Unique, outcome-bearing cases are what most training and evaluation work actually uses.

Can I use token or file counts instead of case counts?

They answer different questions. Token or byte counts help size compute and storage for text data, but they do not tell a buyer how many distinct situations the data covers. Report case counts for value discussions and add size measures alongside them.

Sources

  • Lee and colleagues found many near-duplicates in language modeling datasets, including one 61-word sentence repeated over 60,000 times, and showed deduplication reduced memorization and train-test overlap without hurting perplexity. Source
  • Muennighoff and colleagues report that up to about 4 epochs of repeated data changes loss negligibly compared with unique data, and that returns from more repetition decay toward zero. Source
  • Hernandez and colleagues report that repeating a small fraction of training data can cause disproportionately large harm to model performance. Source
  • The Table Schema standard lets unique keys span several fields and excludes null values from the uniqueness check. Source
  • The datasketch MinHash LSH documentation states that queries return false positives and false negatives, with higher-similarity sets more likely to be returned. Source
  • Splink's record linkage guide describes deduplication within datasets, linkage between datasets, and the aim of generating a unique identifier. Source

See if your company qualifies

A short company assessment. No data uploads are needed.

See if your company qualifies