Rights and contracts
AI dataset refresh SLA: what to settle before the first delivery
By SourceX Editorial · Reviewed by Noah Loul ·
10 sections
Short answer
A dataset refresh SLA should settle seven things before the first delivery: the refresh unit and cadence, the timeliness clock and the timestamp it reads, a completeness baseline tied to a source control total, change notice, incident classes with detect, notify and repair clocks, the rejection and replacement path, and end-of-feed terms. Otherwise batch one becomes the standard by default.
License terms
AI dataset refresh SLA
Key takeaways
- Without a written baseline, the first accepted batch becomes the standard for every later one.
- Latency measured per record catches stale batches that a freshness check on the youngest record misses; name the timestamp field either way.
- Completeness needs a control total from the source system's own reports and an expected range per stratum set from pre-delivery history.
- Incidents that grow by the hour, such as personal data in a batch or misdelivery, need their own detect, notify and repair clocks.
- A pilot should prove every clause can be measured, including a tabletop drill for one severe incident.
What should a dataset refresh SLA define before the first delivery?
A dataset refresh SLA should fix seven things before the first batch moves: the refresh unit and cadence, which clock measures timeliness, the completeness baseline and the control total it is checked against, the change notice for schema or scope, the incident classes with a detect, notify and repair clock for each, the replacement rule for a rejected batch, and what happens to the feed when either side stops. Anything left undefined at delivery one gets defined by whatever delivery one happened to contain.
This page covers the part of a recurring data agreement that is easiest to postpone and hardest to retrofit: the decisions that have to exist on paper before a baseline is set. The day-to-day metric formulas for a running feed are covered in the guide to data feed SLA metrics; here the question is what to settle while both parties still have leverage and nothing has been accepted yet.
The outline below is written for AI procurement teams buying recurring operational records (support tickets, work orders, claims notes, transaction histories) for fine-tuning, retrieval or evaluation refreshes. It is a contracting checklist, not legal advice, and counsel should review the final schedule.
Why the first accepted batch quietly becomes the contract
Once a buyer accepts batch one without a written baseline, that batch becomes the de facto standard: its field list, its volume, its timestamp meaning and its gaps. If the second batch drops a column or arrives with half the records, the supplier can fairly ask where the expectation was written down.
Site reliability practice draws a useful line here. Google's SRE book treats a service level indicator as a carefully defined quantitative measure of some aspect of the service, an objective as a target for that indicator, and an agreement as an explicit or implicit contract that carries consequences for meeting or missing the objectives. A refresh SLA signed before delivery one makes the contract explicit; skipping it leaves an implicit one that nobody can measure.
The practical move is a baseline record attached to the first acceptance: the schema version, the record count by stratum, the source control total, the earliest and latest source timestamps, and the list of known gaps. Every later batch is compared with that record, not with memory.
Freshness or latency: which clock belongs in the agreement?
Put latency in the agreement when the age of each record matters, and freshness when the question is only whether the feed is still alive. The two are different measurements, and confusing them is the most common reason a refresh SLA cannot be enforced.
The Data Contract Specification defines freshness as the maximum age of the youngest entry, and latency as the maximum time from the source timestamp to the point the data is processed and available, which includes waiting for the next batch run. The Open Data Contract Standard, which the Data Contract Specification now points users toward, lists latency as preferred to freshness among its SLA properties. A feed can satisfy a freshness target with one new record while most of the batch is months old; latency measured per record catches that.
Whichever clock you choose, name the timestamp field it reads. A creation time, a last-modified time and a closed date give very different answers for the same work order. If the supplier's system cannot populate the named field reliably, find out during the pilot, not at the first dispute.
Two thresholds work better than one. dbt's freshness configuration, for example, separates warn_after from error_after, each with a count and a period. Mirroring that in a contract gives the supplier an early warning level that triggers a conversation and a breach level that triggers the remedy ladder.
How do you set a completeness baseline when there is no history yet?
Set completeness against a control total produced by the source system's own reporting, not against the exported file, and agree an expected range per stratum instead of a single number. A count taken from the export only proves the export agrees with itself.
The Open Data Contract Standard's quality library includes a rowCount metric alongside null and duplicate counts, which is a convenient way to state the check in a machine-readable contract. The harder part is choosing the range. Before delivery one, ask the supplier for monthly counts over a recent period from the source system's reports, broken down by the strata you care about, and set the expected range from that history with the supplier's explanation for any unusual month.
Completeness also has to survive preparation. When the supplier removes records during de-identification, deduplication or scope filtering, the manifest should report exclusions per step with a reason code. Otherwise a privacy exclusion and an extraction failure look identical on the buyer's side.
The freshness, completeness and incident SLA outline
Use this outline as the schedule that sits behind the main license or supply agreement. Each row names the clause, what to write, how it is evidenced and why it has to be settled before delivery one rather than after.
| Clause | What to write | Evidence each batch | Why settle it before delivery one |
|---|---|---|---|
| Refresh unit and cadence | Snapshot or delta, the cut-off rule, the schedule, and whether late records roll into the next batch | Manifest with batch ID, cut-off and delivery time | The first batch fixes the unit by default; a delta buyer handed a snapshot has no basis to object |
| Timeliness clock | Latency or freshness, the named timestamp field, a warning level and a breach level | Per-record lag distribution or youngest-record age | The timestamp field must exist in the first export or the clause is unmeasurable |
| Completeness baseline | Control total source, expected range per stratum, exclusion reason codes | Delivered count, control total and exclusions per step | The range comes from pre-delivery history; afterwards it comes from whatever arrived |
| Field completeness and validity | Required fields, allowed values, rule set version | Rule results attached to the manifest | A rule added after acceptance looks like a new demand |
| Schema and scope change notice | Notice period, what counts as breaking, a parallel period | Versioned dictionary and change log | Without it, the supplier's next system upgrade becomes the buyer's outage |
| Incident classes and clocks | Severity classes with detect, notify and repair targets and named contacts | Incident log with timestamps | Notice duties are hard to agree in the middle of an incident |
| Rejection and replacement | Who may reject, within what window, and the replacement or withdrawal path | Acceptance record per batch | A rejected first batch otherwise has no defined route |
| Withdrawal and deletion propagation | How records removed at source are flagged in later batches and deleted downstream | Tombstone or withdrawal list with record IDs | Stable record IDs must be designed into batch one |
| Continuity and end of feed | End-of-support notice, final batch, retention and destruction duties | Notice letter and destruction confirmation | Exit terms cost little to agree at the start and a lot at the end |
Which incidents need their own clocks?
Give a separate clock to any incident whose cost grows by the hour: personal data found in a delivered batch, a batch delivered to the wrong recipient, a corrupted or truncated batch already loaded into training, and a rights problem discovered in records already delivered. Late or thin batches can share an ordinary service clock.
At a glance
Which incidents need their own clocks
The Open Data Contract Standard names three properties that map neatly onto incident handling: timeToDetect, timeToNotify and timeToRepair. Writing all three avoids a common gap, where an agreement sets a repair window but says nothing about how quickly the party that found the problem must tell the other.
Privacy incidents carry outside deadlines that the schedule should not contradict. Under GDPR Article 33, a processor must notify the controller without undue delay after becoming aware of a personal data breach, and the controller notifies the supervisory authority without undue delay and, where feasible, within 72 hours. Contract notice windows are a commercial choice layered on top, and counsel should align them with the roles each party actually holds.
For structure, NIST SP 800-61 Revision 3, finalized in April 2025, frames incident response through the CSF 2.0 functions rather than a standalone procedure, which is a reasonable reference when both security teams need a shared vocabulary for detect, respond and recover.
- Severity 1, privacy or misdelivery: notify on discovery with named contacts on both sides, quarantine the batch, agree deletion or replacement.
- Severity 2, integrity: a batch that fails reconciliation or is truncated after loading; notify within the service day, replace, and record which downstream jobs consumed it.
- Severity 3, rights: records found outside the licensed scope; withdraw by record ID and confirm deletion downstream.
- Severity 4, service: late, thin or schema-drifted batch; handled through the warning and breach levels and the normal remedy ladder.
What must the pilot prove before the buyer commits to the larger purchase?
The pilot should prove that every clause in the outline can actually be measured, not only that the sample looks useful. A pilot that shows good records but no source timestamps, no control total and no manifest has tested the data and skipped the SLA.
Ask for at least two pilot batches on the real cadence, so the second can be compared with the first. Run the completeness check against the supplier's control total, compute the timeliness clock from the named field, apply the validation rule set and record where the parties' numbers disagree. Then run a short tabletop drill for one severity 1 scenario: who calls whom, which batch IDs are quarantined, how deletion is confirmed.
The pilot result is a go, a narrowed go, or a stop. A narrowed go is common: the buyer drops a stratum whose control totals cannot be produced, or accepts freshness on one system and latency on another.
Worked example (illustrative): a monthly work-order feed with a migration in the middle
Illustrative only. A buyer licensing field service work orders for a scheduling model agrees a monthly delta feed. The schedule names latency on the closed_at field with a warning level of 21 days and a breach level of 35 days at the 95th percentile, a completeness range of 90 to 110 percent of the source report's monthly closed-order count for each of three regions, and a 30-day notice period for breaking schema changes.
Pilot batches one and two reconcile within the range for two regions. The third region lands at 62 percent because the supplier's report counts orders closed by subcontractors that the export does not include. The parties narrow scope: the third region moves to a separate stratum with its own range, documented in the baseline record.
The exception arrives in month five. The supplier migrates to a new field service system, and closed_at is now written when the invoice posts, several days after the job ends. Latency jumps past the warning level although the work is just as recent. Because the schedule names the field and requires notice of semantic changes, the supplier flags it in advance, both sides agree a mapped timestamp for the new system, and the baseline record gains a version note instead of a dispute.
Signals that the agreement needs narrowing or specialist review
Stop and narrow the scope when the supplier cannot produce a control total independent of the export, when the named timestamp field is missing for a large share of records, or when no one on the supplier's side can own the incident clocks. Each of these makes a clause unmeasurable, and an unmeasurable clause is worse than none because it creates false comfort.
Bring in counsel when the feed contains records about people, when records may need to be withdrawn after delivery, or when the parties' privacy roles are unclear. Bring in security reviewers when the incident classes involve misdelivery or credentials. Bring in the research team when a refresh changes the evaluation set, because a contaminated eval refresh is an incident even if every SLA number is green.
Refresh terms for records sourced on request
Because SourceX holds no stock and finds each dataset among U.S. businesses only after a buyer asks, refresh expectations belong in the request itself, next to the scope, volume, language, format, timing and licensing needs the request-data form already asks for. Saying up front that you need a recurring feed, and which clock and baseline you expect, lets the supplying business judge whether its systems can produce the evidence.
The supplying company signs off on scope and terms before the first record moves. Each deal's signed Dataset Deal Confirmation then fixes the delivery date, exclusivity and term, and de-identification sits with SourceX through approved processing routes.
Frequently asked questions
Is a refresh SLA the same as acceptance criteria?
No. Acceptance criteria decide whether one batch is accepted; a refresh SLA governs the series, including timing, change notice, incident clocks and what happens at the end of the feed. Most recurring agreements need both, with the acceptance criteria applied to every batch under the SLA.
Should a dataset refresh SLA use freshness or latency?
Use latency when the age of each record matters to the model, for example in retrieval or continual fine-tuning, and freshness when you only need to know the feed is alive. Whichever you choose, name the timestamp field it reads and confirm in the pilot that the field is populated.
What if the supplier cannot provide a control total?
Then completeness cannot be measured against the source, only against the export. Either narrow the scope to systems that can report counts, accept a weaker check such as an expected range from historical volumes, or record the gap in the baseline so neither side treats it as met.
Who should own the incident clocks on each side?
Name a role, not only a person, on both sides for each severity class, with a backup. The owner on the supplier side needs authority to quarantine or withdraw a batch; on the buyer side, authority to pause the jobs that consume it.
How often should the outline be revisited?
Revisit it when either side changes systems, scope, schema or legal role, and at a fixed interval such as each renewal. Version the baseline record so a later reviewer can see which terms applied to which batches.
Sources
- The Open Data Contract Standard's SLA section lists properties including latency (preferred to freshness), frequency, retention, timeToDetect, timeToNotify and timeToRepair. Source
- The Open Data Contract Standard's data quality library includes rowCount, nullValues and duplicateValues metrics. Source
- The Data Contract Specification defines freshness as the maximum age of the youngest entry and latency as the maximum time from source to destination, and is deprecated in favor of ODCS. Source
- dbt freshness configuration separates warn_after and error_after thresholds, each with a count and a period. Source
- Google's SRE book defines SLIs as quantitative measures of service and SLAs as explicit or implicit contracts with consequences for meeting or missing SLOs. Source
- GDPR Article 33 requires processors to notify controllers without undue delay and controllers to notify the supervisory authority without undue delay and, where feasible, within 72 hours. Source
- NIST SP 800-61 Revision 3 (April 2025) presents incident response recommendations as a CSF 2.0 Community Profile. Source
Tell us what data you need
Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.