Systems and records
Dataset recurring utility review: is the buyer still getting value from the feed?
By SourceX Editorial · Updated
10 sections
Short answer
Review an ongoing data feed on a schedule and ask three things in order: does it still cover the slices the buyer needs, does each delivery add records the buyer does not already have, and has every repair, refresh or withdrawal reached the right recipients and versions. Close with a recorded decision to continue, re-scope, repair or pause. Service levels show the feed arrived; this review shows it still helps.
Value drivers
Dataset recurring utility
Key takeaways
- A feed can meet every service level and still lose value each quarter.
- Bring evidence to the review: delivery manifests, a coverage map, a novelty report, distribution comparisons and the buyer's direction of effect.
- Never change a released version in place; release fixes as new versions and trace them to recipients.
- Falling value can come from the buyer's changed needs, which is a scope conversation, not a defect.
- Pause when a blocking defect is live, when data falls outside the signed scope, or when nobody can say who holds which release.
How do you review whether an ongoing data feed still creates buyer value?
Hold a scheduled review, usually each quarter or after a set number of deliveries, that asks three questions in order: is the feed still covering the cases the buyer needs, is each new delivery adding something the buyer does not already have, and has every repair, refresh or withdrawal since the last review reached the right recipients and versions. Decide at the end of the meeting whether to continue, re-scope, repair or pause, and write the decision down with the evidence behind it.
Service levels answer a different question. A feed can arrive on time, pass every schema check and still be worth less each quarter because it keeps sending the same kinds of records, because the buyer's needs moved, or because a quiet change at the source shifted what the records mean. A utility review is how a supplier notices before the buyer does.
This guide is written for the supplier side: the operations or data lead at a business that sends records to an AI developer on a recurring schedule under a license. It gives the agenda, the evidence each side brings, an issue and recipient log, and an illustrative quarterly review.
On-time deliveries can still be losing value
Because the usual delivery contract measures arrival, not usefulness. A data contract written in the Open Data Contract Standard, for example, can declare service-level properties such as latency, frequency, retention and the time of day data becomes available, each with a value and a unit. Its data quality section adds rules such as row counts, null values and invalid values, written as text, library metrics, SQL or vendor-specific checks. Those are the right controls for a feed, and a supplier should meet them. None of them asks whether the records still help the buyer's model.
Value can drain away while every check stays green. A support-ticket feed might keep delivering the same five issue types after the buyer's model has learned them; a work order feed might lose a region when a branch moves to a new system; a product change might make old categories meaningless. The rows still arrive, complete and on time.
The dataset documentation literature anticipated part of this. Datasheets for Datasets includes maintenance questions asking whether a dataset will be updated, how often, by whom, and how updates will be communicated to users, and it suggests that updated versions come with updated documentation. A recurring utility review is the working meeting behind those answers.
What should each side bring to a utility and coverage review?
Bring evidence, not impressions. Most of what is needed already exists in the delivery logs and the buyer's own evaluation results; the review simply puts it side by side. Agree the list before the first review so neither side arrives with a different picture.
| Evidence | Who usually holds it | What it shows | If it is missing |
|---|---|---|---|
| Delivery manifest per release | Supplier | Record counts, date ranges and version IDs actually sent | Rebuild it from delivery logs before the review |
| Coverage map against agreed slices | Supplier, from the scope schedule | Counts per agreed category, region, record type or period | Agree the slice list first; without it coverage cannot be judged |
| Overlap and novelty report | Supplier or buyer | Share of new records that duplicate or nearly duplicate earlier ones | Run a simple duplicate check on stable keys and text hashes |
| Distribution comparison between releases | Supplier | Fields whose value mix moved between deliveries | Compare summary statistics for the main fields |
| Buyer utility signal | Buyer | Whether recent deliveries moved the buyer's own evaluation or use | Ask for a direction and confidence, not raw results |
| Issue and repair log | Supplier | Every defect found, its severity and its fix | Start the log now; past issues go in from memory, marked as such |
| Recipient and version log | Supplier | Which recipient holds which version of which file | Reconstruct from transfer records before any change notice |
The periodic utility and coverage review agenda
Run the agenda in this order and close each item with a recorded output. The order matters: a coverage question cannot be answered while an open defect is distorting the counts, and a renewal or scope decision cannot be made before coverage and novelty are known.
- Record for every review: review ID, period covered, releases covered, attendees by role, agenda outputs, decision, owner and next review date.
- Version the agenda itself and note the version in each record, so later reviews are comparable.
| Agenda item | Question to settle | Evidence used | Recorded output |
|---|---|---|---|
| 1. Open issues | Are any defects from the last period still unresolved? | Issue and repair log | Each issue closed, scheduled or escalated |
| 2. Source changes | Did anything change in the source systems, forms or processes? | Supplier change notes and distribution comparison | List of changes with the releases they affect |
| 3. Coverage | Which agreed slices are thin, missing or overfilled? | Coverage map | Slices to grow, hold or cap next period |
| 4. Novelty | How much of the new data repeats what the buyer holds? | Overlap and novelty report | Target for new versus repeated records |
| 5. Buyer utility | Did recent deliveries help the buyer's task, and which parts? | Buyer utility signal | Slices the buyer values most and least |
| 6. Repairs and refreshes | Which fixes must be re-sent, and to whom? | Recipient and version log | Change notice list and release versions |
| 7. Withdrawals | Must any records be withdrawn for rights, privacy or accuracy reasons? | Rights register and issue log | Withdrawal notice and confirmation steps |
| 8. Scope and terms check | Do the next deliveries still fit the signed scope? | Scope schedule in the agreement | Confirmed scope or a request for an amendment |
| 9. Decision | Continue, re-scope, repair first, or pause? | Outputs of items 1 to 8 | Decision, owner and date of the next review |
Is each new delivery adding coverage or repeating what the buyer already holds?
Measure it directly, in three ways. First, coverage: count new records per agreed slice and compare with the target for that slice. Second, novelty: the share of new records that are exact or near duplicates of records already delivered, checked on stable keys and on normalized text. Third, shift: which fields changed their mix of values between releases.
Shift checks can be automated with standard tools. TensorFlow Data Validation, for instance, compares statistics of different datasets to detect skew between training and serving data and drift between different days of training data, using comparators and thresholds set in a schema. A supplier does not need that library in particular, but it does need a repeatable comparison between this release and the last one, so that a change in the mix is noticed when it happens rather than at renewal.
Novelty and coverage tell the supplier what was sent. Only the buyer can say whether it helped. Research on data valuation shows why that is hard to measure precisely: Data Shapley, proposed by Ghorbani and Zou, scores each training point by its contribution to a predictor's performance, and the authors develop Monte Carlo and gradient-based approximations because the exact calculation is expensive. Most buyers will not run that for a supplier review. Ask instead for a direction and a confidence level per slice: helped, no visible effect, or hurt, and how sure they are.
Tracing a repair to every release and recipient that carries it
Find out from the recipient and version log, which is why that log has to exist before the first repair. Each delivered file should carry a version, and a released version should never be changed in place. Semantic Versioning states the principle plainly for software: once a versioned package has been released, its contents must not be modified, and any modification must be released as a new version. Apply the same rule to dataset releases, and a source change becomes traceable to exactly the releases and recipients that carry it.
When a defect or a source change is found, the issue entry should carry enough to act on without another meeting.
- Issue ID and date found, with who found it.
- Severity: blocks use, degrades a slice, or cosmetic.
- Source and derived links: the source records, the release versions and the derived files that carry the defect.
- Repair: what was changed, in which new release version.
- Regression checks: the tests re-run to confirm the repair did not break something else.
- Recipient and version log entries: who received an affected version, and when.
- Change notice: what each recipient was told, when, and what they must do, such as replace or delete.
- Reconciliation: confirmation from each recipient that the replacement or deletion happened, with the date.
Illustrative review: the third quarter of a work order feed
This example is illustrative; the business, counts and percentages are invented. A regional field-service company sends anonymized work orders to an AI developer each month under a license that names four agreed slices: heating repairs, cooling repairs, installations and maintenance visits.
At the quarterly review the coverage map shows installations at about 40 percent of the agreed share for the last two deliveries. The source change list explains it: one branch moved its installation jobs to a separate project tool in the second month, so they stopped appearing in the dispatch export. The novelty report shows that roughly 30 percent of maintenance visits in the latest delivery are near duplicates of earlier visits, because recurring service plans generate almost identical notes.
The buyer reports that heating and cooling repairs still help their diagnostic model with moderate confidence, maintenance visits show no visible effect, and installations would help if they returned. The decisions follow: add the project tool as a source for installations after checking that the signed scope covers it, cap near-duplicate maintenance visits per customer per period, and release the fix as a new version with a change notice to the one recipient.
One finding goes in the issue log rather than the decision list. A technician field was blank for one week because of an export error. It was repaired in a new release, the regression check confirmed the other fields were unchanged, and the buyer confirmed the replacement two days later.
When falling value comes from the buyer's side, not the feed
Sometimes the feed is fine and the buyer's use changed. A buyer may have moved to a new model, a new task or a new evaluation, and a slice that was valuable last quarter may simply no longer be the bottleneck. Falling utility on the buyer's side is a scope conversation, not a quality defect, and a supplier should not quietly change what it sends to chase it.
Tell the two cases apart with the evidence already on the table. If the coverage map, novelty report and distribution comparison are stable but the buyer's utility signal fell, the cause is probably on the buyer's side. If the supplier-side evidence moved at the same time, start there. When the cause is the buyer's changed need, record it, agree whether to re-scope, and put any change to the scope through the agreement rather than through the delivery pipeline.
Signals to pause the feed, cut its scope or bring in a specialist
A utility review sometimes shows that the right move is to stop sending data until something is fixed or clarified. The signals are usually visible in the agenda outputs.
At a glance
Signals to pause the feed, cut its scope or bring in a specialist
- Pause when a defect that blocks use is open in a release the buyer is actively using, until the repair and change notice are done.
- Pause when records arrive from a source system or region the signed scope does not cover, and bring in whoever manages the agreement.
- Narrow the scope when one slice keeps delivering mostly duplicates and the buyer reports no effect from it.
- Ask privacy or legal reviewers when a source change brings in new personal fields, new customer groups or records collected under a different notice.
- Escalate to the agreement owners when a withdrawal is needed and a recipient cannot confirm deletion or replacement.
- Stop and re-plan when the recipient and version log cannot say who holds which release, because no change notice can be trusted until it can.
Keeping a recurring license worth renewing
For a supplier, the review is what keeps a recurring arrangement useful to both sides, and it is evidence of a well-run program when the license comes up for renewal. It also protects the supplier: a clean recipient and version log is what makes a withdrawal or correction possible without guesswork.
SourceX handles sourcing, rights review, contracting, preparation and delivery for businesses that license operational records to AI developers. The business approves the scope and terms and signs before anything is delivered, and large deliveries stay in its own storage or travel on encrypted drives rather than being hosted by SourceX. A business considering a recurring arrangement can start with a short assessment that needs no data upload.
Frequently asked questions
How often should a recurring data feed be reviewed for utility?
Quarterly works for most monthly feeds, with an extra review after any major source change. Daily or weekly feeds may need a monthly review of coverage and novelty, while the full agenda can stay quarterly.
Is a utility review the same as an SLA review?
No. An SLA review checks whether deliveries met agreed latency, frequency, completeness and quality targets. A utility review asks whether the data still covers what the buyer needs and adds something new, which an on-time, complete delivery can fail to do.
What if the buyer will not share evaluation results?
Ask for a direction and a confidence level per agreed slice instead of numbers: helped, no visible effect or hurt. That is usually enough to decide which slices to grow or cap, and it avoids exposing the buyer's model results.
How should a supplier handle a repair to data the buyer already has?
Release the fix as a new version rather than changing the old one, send a change notice to every recipient of an affected version, and get confirmation that they replaced or deleted it. The recipient and version log makes that list exact.
Can a supplier add a new source to improve coverage?
Only if the signed scope covers it or the agreement is amended first. A new source can also bring new personal data or different notices, so privacy review should check it before the first delivery.
Sources
- The Open Data Contract Standard lets a contract declare service-level properties such as latency, frequency, retention and time of availability. Source
- The Open Data Contract Standard expresses data quality rules as text, library metrics such as row counts and null values, SQL queries or vendor-specific checks. Source
- Datasheets for Datasets asks whether a dataset will be updated, how often, by whom and how updates will be communicated. Source
- TensorFlow Data Validation detects skew between training and serving data and drift between days of training data by comparing dataset statistics. Source
- Data Shapley quantifies the value of each training datum to predictor performance, estimated with Monte Carlo and gradient-based methods. Source
- Semantic Versioning requires that a released version not be modified and that any modification be released as a new version. Source
See if your company qualifies
A short company assessment. No data uploads are needed.