Incremental Deliveries vs Full Refreshes for Ongoing Dataset Purchases

What actually differs between a snapshot and a delta

A full refresh ships the complete current state of the licensed records each cycle, while an incremental delivery ships only what changed since the last cycle. In warehouse terms, a full refresh usually truncates and reloads the target, which is why integration vendors call it a destructive load [1]. An incremental load moves only new or changed rows and must reconcile them against what you already hold [1][3].

For a buyer of licensed operational data, the difference matters beyond file size. A snapshot tells you what is true now but not what changed, so you must diff consecutive snapshots yourself to find deletions or edits. A delta tells you exactly what changed but is only correct if every previous delta was applied, in order, without loss.

Full loads stay useful for small tables and where the source system cannot reliably track changes [3]. Practitioners commonly run incremental pipelines with periodic full rebuilds to correct accumulated drift [2], and the same logic applies to supplier deliveries.

How to signal inserts, updates and deletes

The safest delta format carries one row per change event with an explicit operation code, rather than relying on you to infer intent. Two mature open-source conventions are worth borrowing in your spec. Delta Lake's change data feed adds a _change_type column with the values insert, update_preimage, update_postimage and delete, plus _commit_version and _commit_timestamp. Debezium change events carry an op field alongside before and after row images and source metadata.

You do not need the supplier to run Delta Lake or Kafka. Ask for the same semantics in flat files:

• Operation column. op with a closed set such as I, U, D. A delete row needs only the record ID, the operation and the change timestamp.
• Full row on update. Ship the complete post-update record, not just changed columns; partial updates force column-level merge logic and break when the schema evolves.
• Pre-image only when useful. For eval sets where you track label corrections, a before image (as in update_preimage) lets you see exactly what was corrected without querying history.
• One file vs separate files. A single change file with an operation column keeps ordering intact. Separate inserts/, updates/, deletes/ folders are easier to eyeball but lose the ordering between an insert and a later delete of the same record within a cycle.

Deletes deserve special attention because they often carry licensing or privacy weight, for example a supplier withdrawing records or correcting a consent error. See propagating deletions and corrections through recurring deliveries for how to carry those into derived training sets.

Stable record IDs and upsert keys

Incremental delivery only works when every record has a key that never changes across deliveries. The receiving side applies each change with an upsert (MERGE INTO ... WHEN MATCHED THEN UPDATE WHEN NOT MATCHED THEN INSERT), which is the same pattern Salesforce exposes when it uses an External ID field to decide whether an incoming record is inserted or updated [4].

Specify the key explicitly in the data contract: which column, whether it is composite (for example ticket_id plus message_seq), and what happens when the source system merges two records. Common failure modes:

• Surrogate keys regenerated per export. Row numbers or export-time UUIDs make every delta look like all-new inserts, producing duplicates in your training set.
• Pseudonymized keys that change. If de-identification replaces IDs with a salted hash and the salt rotates between cycles, prior records become unreachable. Ask that the pseudonym mapping stay consistent for the life of the license.
• Merged or split source records. CRM and ticketing systems merge duplicates. Ask the supplier to emit a delete for the absorbed ID and an update for the survivor.

The stable record IDs and join keys page covers key design in depth.

Watermarks, late-arriving records and ordering

A watermark is the boundary that defines which changes a delivery covers, and it should be set by the supplier's change timestamp, not by file arrival time. Typical choices are an updated_at column, a database commit sequence, or a version number like Delta's _commit_version. Each delivery manifest should state the low and high watermark, and the next delivery's low watermark must equal the prior high watermark.

Timestamp watermarks fail in predictable ways. Long-running transactions commit after rows with later timestamps were already exported. Records backfilled from another system arrive with old updated_at values. Clock skew across application servers reorders events. Two mitigations work well in a spec:

1. Overlap window. The supplier re-exports a trailing window (for example the last 24 hours before the low watermark) every cycle, and your upsert is idempotent so replays are harmless.
2. Monotonic sequence. Where the source exposes a log sequence number or version counter, use it instead of wall-clock time.

Also note that change tracking typically only captures changes after it is enabled; Delta's change feed, for instance, does not record earlier history, and its change files are removed by VACUUM under the table's retention policy. If a supplier turns on change tracking for your deal, the first delivery must be a full snapshot.

When to re-baseline with a full snapshot

Schedule a full snapshot at onboarding, after any schema break, after any gap in the delivery sequence, and on a fixed cadence such as every fourth cycle. The re-baseline is your reconciliation point: load it into a staging table, diff it against the state you built from deltas, and investigate any difference before swapping it in. Periodic full rebuilds are the standard practitioner correction for drift in incremental pipelines [2], which in supplier deliveries usually traces to a missed delete, a key collision or an out-of-order file.

Size the cadence by cost of drift, not cost of transfer. For an eval set where a wrong label distorts every benchmark run, re-baseline more often. For a large pretraining-style corpus where a handful of stale rows barely matter, a quarterly or semiannual snapshot is often enough. Coordinate the schedule with schema evolution rules, since a breaking change is easiest to absorb at a snapshot boundary.

A delta delivery spec you can adapt

The artifact below is a delivery manifest plus change-file record layout for a recurring delta of support-ticket records. It sits within the wider dataset delivery formats and schemas hub. Adapt the field names to your own technical delivery specification.

Illustrative example: invented to show structure; it does not describe an available dataset.

delivery:
  sequence: 14                    # strictly increasing; gaps trigger re-baseline
  type: incremental               # incremental | full_snapshot
  watermark:
    column: source_updated_at     # or source_commit_seq
    low:  "2026-09-01T00:00:00Z"  # must equal previous high
    high: "2026-10-01T00:00:00Z"
    overlap_replayed: "PT24H"
  files:
    - path: changes/part-00000.parquet
      rows: 48210
      sha256: "<hex>"
  counts: { insert: 31004, update: 16890, delete: 316 }
  schema_version: "2.3.0"
record_layout:
  record_id:          string  # stable across all deliveries, never reused
  op:                 enum[I,U,D]
  source_updated_at:  timestamp_utc
  source_commit_seq:  int64   # optional, preferred for ordering
  payload:            struct  # full post-change record; null when op = D
  delete_reason:      enum[source_deleted, withdrawn, correction]  # when op = D

Validate each delivery on arrival with a short checklist:

• Sequence number is exactly prior + 1; otherwise halt and request a snapshot.
• Low watermark equals the previous high watermark.
• File checksums and row counts match the manifest (see manifests and checksums); Parquet footers make truncated files detectable because metadata is written after the data [5].
• Operation counts sum to total rows, and no record_id appears twice with conflicting operations at the same sequence position.
• Every D references a record_id you already hold, or is logged as an orphan delete.
• Insert count is plausible against the prior cycle; a spike often means regenerated keys.

Commercial terms that depend on the delivery mode

The delivery mode affects how a recurring license is priced, measured and audited, so settle it before the agreement is drafted. Questions worth putting in your requirements:

• Is volume measured as rows in the full licensed set, or as rows changed per cycle?
• Do updates to existing records count as new records for pricing?
• Must delete rows be applied downstream within a set period, and how do you evidence that?
• Who pays for an out-of-cycle re-baseline after a supplier-side gap?

The ongoing data supply agreements page covers contract structure, and SLAs for recurring deliveries covers freshness and completeness metrics. If you are still choosing between file drops and an API, start with API and streaming feeds vs batch files. For a broader view of cadence, see how often AI buyers want fresh data and delivering fresh data each quarter.

SourceX sources operational datasets from US companies and manages the commercial process, including licensing agreements and ongoing purchases. You can describe the recurring data you need to SourceX, including the refresh pattern your pipeline expects; data is sourced on request, and a request does not guarantee a match.

Plan your recurring delivery with SourceX

SourceX looks for US businesses that hold the data you describe, rights-reviews every dataset, and delivers it under a license defining records, uses, term and delivery, with each release approved by the supplying company. Nothing is contracted until a supplier agrees. Tell SourceX what recurring data your team needs.