Skip to content

AI uses for records

Dataset decision coverage rejection: test saturation before adding rows

By SourceX Editorial · Reviewed by Noah Loul ·

9 sections

Short answer

A dataset decision coverage rejection means the buyer counted distinct task structures, not rows, and found too few. Agree the structure key in writing, then measure distinct structures, concentration and the singleton share. When singletons are a tiny fraction of records, more rows from the same process add few new structures, so the remedy is a new source, a re-extraction, a narrower offer or a clean no.

License terms

Dataset decision coverage

This page covers dataset decision coverage rejection: test saturation before adding rows.Illustrative

Key takeaways

  • A coverage rejection is a claim about distinct task structures per stratum, so it can be argued only once both sides use the same written structure key.
  • Five numbers describe coverage: distinct structures, effective number, top-ten share, singleton share and the share of required strata that meet the buyer's floor.
  • The singleton share predicts saturation: it approximates the chance that the next record from the same process shows a structure not yet seen.
  • When a source is saturated, the remedies are re-extraction, a different source or a narrower offer; more rows from the same system is not one of them.
  • Test rights, terms and task fit before remediating, because the records that would add variety are often the least cleared for licensing.

What is a buyer saying when it rejects a dataset for decision coverage?

It is saying that the records hold too few distinct versions of the decision its model must learn or be tested on, however many rows there are. A support archive with 180,000 tickets can contain only a few hundred genuinely different situations, and if the buyer's task needs examples spread across a thousand, row count is not the problem and more rows of the same kind will not solve it.

That makes a coverage rejection different from the two complaints it is most often confused with. A duplicate finding says some records are copies of one event, which a uniqueness audit resolves record by record. A pricing objection says the coverage exists but is worth less than asked. A coverage rejection says the variety itself is missing, measured in the unit the buyer cares about.

Evidence that settles it has four parts: the structure key both sides used, the count of distinct structures that key produced, the buyer's floor for each stratum of the task, and the extraction and selection rules that turned source records into delivered rows. Without all four, neither side can say whether the gap is real, fixable or a misunderstanding.

Agree the task structure key before arguing about the count

Settle in writing what counts as one distinct structure, and version it, because every later number depends on it. A task structure is the combination of situation, options and result that should make a model act differently: the trigger, the facts available at the time, the action chosen from a known set of options and the class of outcome that followed.

In a Zendesk support archive, a workable key might be contact reason, product line, customer tier, action taken (refund, replacement, escalation or denial) and resolution outcome. In NetSuite credit memos it might be reason code, approval tier, amount band and whether the customer disputed afterward. The buyer's task decides the fields: if the model only chooses between refund and replacement, customer tier may not belong in the key; if it must apply tier-specific policy, it has to be there.

Two errors are common. A key that is too fine, for example one that includes free-text wording or exact amounts, makes nearly every record unique and hides repetition. A key that is too coarse, such as resolved versus unresolved, collapses real differences. Anthropic's January 2026 guidance on agent evaluations makes a related point about test sets: include cases where a behavior should occur and cases where it should not, because one-sided sets produce one-sided behavior. A key that cannot tell an approval from a denial hides exactly that imbalance.

Where the decision lives mostly in free text, an exact key will undercount. A similarity-based measure can stand in: the Vendi Score, which its authors describe as the effective number of unique elements in a sample given a pairwise similarity function, is one option. Agree the similarity function as carefully as you would agree a key, since it decides what counts as different.

Five numbers that show whether variety is really missing

Report five numbers on the agreed key, overall and for every stratum of the buyer's task, and both sides can see whether the shortfall is real. They come from ecology, where estimating how many species live in a habitat from a sample of individuals is the same statistical problem as estimating how many task structures a business process produces from a sample of records.

The iNEXT package from Anne Chao's group computes the core quantities: Hill numbers of order 0, 1 and 2 (richness, the exponential of Shannon entropy and the inverse of Simpson concentration) and a sample coverage estimate, with confidence intervals for smaller and larger sample sizes. The table gives each number, how an acceptance owner should read it and the red flag to look for:

Five numbers that show whether variety is really missing
NumberDefinitionHow to read itRed flag
Distinct structuresKey values with at least one recordRaw variety; sensitive to how fine the key isFar below the buyer's need, or close to the record count, which means the key is too fine
Effective number (Hill order 1)Exponential of the Shannon entropy of structure sharesHow many equally common structures the data behaves likeA small fraction of distinct structures: a few routine cases dominate
Top-ten shareShare of records in the ten most common structuresConcentration a buyer will see in any random sampleMost records sit in ten structures while the task needs breadth
Singleton shareStructures seen exactly once, divided by recordsRough chance that the next record from the same process is a new structureVery low while required strata are still below the floor
Strata at floorRequired strata with at least the agreed minimum of examplesThe number the acceptance decision should turn onAny must-have stratum below the floor

Will more records from the same source close the gap?

Usually not, and the singleton share shows how unlikely it is. In species estimation, the share of a sample made of species seen only once approximates the chance that the next individual belongs to a species not yet seen. iNEXT computes its sample coverage estimate as one minus the singleton count over the sample size, adjusted with the count of species seen exactly twice.

Read the singleton share as a rate. If 290 of 180,000 records are singletons, the share is about 0.16 percent, so the next 1,000 records from the same process should add only one or two structures not seen before (Illustrative). The same two counts give a second estimate: Chao1, which iNEXT uses inside its coverage calculation, approximates the structures the process holds but the sample has not shown as the singleton count squared over twice the doubleton count. With 290 singletons and 160 doubletons that is about 263, so a process with 1,140 observed structures holds about 1,400 in total (Illustrative). Chao1 is built as a lower-bound estimate with a wide interval, so treat it as an order of magnitude.

Both estimates assume new records come from the same process. A different business unit, period or company is a different process and can add structures the original never produced, which is why the remedy for a saturated source is a new source rather than more of the old one.

Web-scale training data shows the same gap between volume and variety. Lee and colleagues, in a 2022 ACL paper on deduplication, found a single 50-word sequence repeated 60,000 times in the C4 corpus and report that models trained on deduplicated data reach similar or better perplexity. The SemDeDup study from Meta researchers (2023) removed half of a LAION image-text subset as semantic duplicates with little loss in zero-shot accuracy while roughly halving training time. Business archives repeat themselves for duller reasons, such as templates, recurring orders and routine approvals, but the arithmetic is the same.

A saturation curve of distinct task structures against records from one support instance, flattening at 1,140 structures by 180,000 records, with a dashed extension toward a Chao1 estimate of about 1,400.
Illustrative: values from the worked example. More records from a saturated source add almost no new structures.

The coverage acceptance decision tree

Work the rejection through six questions in order; every exit is a decision, not a delay. The first two establish whether the shortfall is real, the middle two whether it can be fixed inside the current scope, and the last two whether anything is left to offer.

Record the answer and its evidence at each node. Run on the next offer, the same tree shows which rejections were measurement problems and which were real:

The coverage acceptance decision tree
QuestionEvidence to recordIf yesIf no
1. Is the structure key agreed, written and versioned?Key fields, version number, approver on each sideGo to question 2Agree a key from the buyer's task description before any rework
2. Does your recount reproduce the buyer's shortfall?The five numbers per stratum, the counting script and the data snapshotGo to question 3Send the recount and ask the buyer to rerun theirs on the same snapshot
3. Did extraction drop fields that define structure?Field map from source to delivery; null rates for key fieldsRe-extract the missing fields and recountGo to question 4
4. Is the source process far from saturation?Singleton share, Chao1 estimate, growth in the most recent trancheAdd records from the same system, then recountGo to question 5
5. Does another licensable source hold the missing structures?Candidate units, years or companies, with the rights status of eachWiden the scope after rights review and the company's approvalGo to question 6
6. Does the well-covered subset meet a smaller, real need?Strata at floor; uses the buyer named as alternativesNarrow the offer, for example to an evaluation setClose as a task mismatch and log the reason

A rejection reason log that makes the next offer better

Log each coverage rejection as a structured record, so patterns across buyers appear before the next offer goes out. A note saying the buyer wanted more variety teaches nothing; a row that names the key version, the stratum that failed and the node where the tree exited does.

Keep the buyer's own words beside your coding, and check that two people would code the same rejection the same way; if they would not, the codes are too loose. These fields are the minimum, with Illustrative example values:

A rejection reason log that makes the next offer better
FieldWhat to recordExample (Illustrative)
Rejection ID and dateOne row per buyer decisionR-031, 2026-10-12
Buyer's wordsThe stated reason, verbatimToo few denial and partial-refund cases
Key versionThe structure key used for every countSupport key v3, five fields
Failing strataStrata below the floor, with countsDenials: 38 structures at floor against a need of 120
Tree exitQuestion number and branch where the decision was madeQuestion 4: saturated
Root cause classMeasurement, extraction, saturation, scope, rights, terms or task mismatchSaturation
Remedy and ownerWhat was done, by whom, with what resultSecond business unit proposed, pending rights review
RecountThe five numbers after the remedy, same keyDenials: 131 structures at floor

When the real reason is rights, terms or a different task

Sometimes the coverage complaint is accurate but beside the point, and no technical remedy will change the outcome. Three alternative explanations deserve a test before anyone re-extracts a field.

Rights come first. The records that would add variety are usually the unusual ones: disputes, exceptions and large customers with negotiated terms, which are also the records most likely to sit under confidentiality clauses or unclear permissions. Documentation gaps are common even in public AI datasets: across the more than 1,800 text datasets audited by the Data Provenance Initiative for a 2024 Nature Machine Intelligence paper, popular hosting sites omitted licenses for about 70 percent of them and miscategorized licenses at rates above 50 percent. A recount cannot clear a right that was never granted.

Commercial terms come second. A buyer that wants exclusivity, a longer term or a lower price may cite coverage because it is easier to say. Ask directly whether a dataset that met the floor would be accepted on the offered terms; if the answer is no, the rejection belongs in the terms class of the log, not the coverage class.

Task mismatch comes third. If the buyer's task needs decisions the company does not make, such as approvals issued by a regulator rather than an internal team, no recount will create them. Record the gap precisely, because it is a sourcing fact about the buyer's need that can be routed to a business that does make those decisions.

Worked example: 180,000 tickets, too few denials

Illustrative and fictional. A software company offers 180,000 support tickets from four years of one Zendesk instance to a buyer training a refund-policy agent. The buyer needs at least 400 structures with 30 or more examples each, including at least 120 denial structures, and rejects the offer for decision coverage.

The recount on the agreed key finds 1,140 distinct structures, an effective number of 96 and a top-ten share of 62 percent. Only 210 structures reach 30 examples, and only 38 of those are denials. The singleton share is 290 in 180,000, about 0.16 percent, and Chao1 puts the unseen structures at about 263.

Question 3 finds a real extraction loss: the export dropped the policy-exception reason code, which splits several denial and partial-refund structures. Re-extraction raises distinct structures to 1,390 and structures at floor to 248, still short. Question 4 says the instance is saturated, so more tickets from it will not help. Question 5 finds a second business unit on a separate instance whose tickets would lift denial structures at floor to 131 and the total to 412; it goes to rights review and needs the company's approval of the wider scope before any records move.

The decision is explicit either way. If rights clear, the re-offer meets both floors. If they do not, the company offers the 248 well-covered structures as an evaluation set and logs the denial gap as a sourcing note. Every number in this example is invented to show the arithmetic, the 30-example floor is the buyer's assumption rather than a standard, and nothing here is a measured result from SourceX or any company.

Putting a structure floor into the data request

State the structure key and the floor at the start, and the coverage question is answered before anyone ships a sample. SourceX does not keep datasets for AI buyers on hand; it finds them among U.S. businesses when asked, and the request form is where a buyer lists scope, format, volume, language, timing and licensing needs.

A coverage-aware request adds three lines: the structure key the task depends on, the minimum examples per structure or stratum, and the acceptance test the buyer will run. As a general guide, buyers expect a system with at least two years of consistent history and tens of thousands of records, though smaller sets can qualify when the work is rare, assessed deal by deal. A request that names its key lets the supplier side judge early whether a company's process is varied enough, before rights review and preparation costs are spent.

Frequently asked questions

What is decision coverage in a dataset?

Decision coverage is how many distinct versions of a decision a dataset contains, counted with a key that combines the situation, the facts available, the action taken and the outcome. It is measured per stratum of the buyer's task. Two datasets with the same row count can differ by an order of magnitude in coverage.

How many distinct examples does an AI dataset need?

There is no universal number; the buyer's task sets a floor per structure or stratum. For agent evaluations, Anthropic's January 2026 guidance says 20 to 50 simple tasks drawn from real failures is a good start, with larger and harder sets needed as agents mature. Training sets usually need many more examples per structure than evaluation sets.

Can deduplication cause a decision coverage rejection?

Deduplication more often reveals one than causes it. Removing copies shrinks the row count but not the number of distinct structures, so a dataset that looked large can turn out to be narrow. If the deduplication removed legitimate repeat events, recount with uniqueness rules before accepting the result.

What does the singleton share tell an acceptance owner?

It is the number of structures seen exactly once divided by the number of records. It approximates the chance that the next record from the same process shows a structure not yet seen, so a very small value means more volume from that process will add little variety.

Does SourceX hold stock datasets that fill coverage gaps?

No. SourceX carries no inventory to fill gaps from; it goes looking among U.S. businesses once a request comes in. A request that states the structure key, the floor and the acceptance test helps show early whether a candidate company's records are varied enough.

Sources

  • Anthropic's January 2026 guidance on agent evaluations advises balanced problem sets that test when a behavior should and should not occur, and says 20 to 50 simple tasks drawn from real failures is a good start. Source
  • The Vendi Score takes samples and a pairwise similarity function and returns a number interpretable as the effective number of unique elements in the sample. Source
  • The iNEXT package computes Hill numbers of order 0, 1 and 2 and a sample coverage estimate equal to one minus the singleton count over sample size, adjusted using a Chao1 estimate built from singleton and doubleton counts. Source
  • Lee and colleagues found a single 50-word sequence repeated 60,000 times in C4 and report that models trained on deduplicated data reach similar or better perplexity. Source
  • SemDeDup (2023) removed half of a LAION subset as semantic duplicates with minimal loss in zero-shot performance and roughly half the training time. Source
  • The Data Provenance Initiative's audit of more than 1,800 text datasets (Nature Machine Intelligence, 2024) found license omission around 70 percent and license error rates above 50 percent on popular dataset hosting sites. Source

Tell us what data you need

Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.

See if your company qualifies