Skip to content

Systems and records

Synthetic data generation reproducibility: can you rebuild it months later?

By SourceX Editorial · Updated

10 sections

Short answer

To make synthetic dataset generation reproducible months later, decide first which rebuild you can promise: exact, statistical or archive-only. Then capture a recipe and dependency manifest at generation time: generator weights by commit hash or hosted model id with its retirement risk, inference engine and hardware, framework versions, every random seed, the grounding record snapshot, prompts, filters and a locked environment, plus hashes of the outputs and an untouched real holdout for checking the rebuild.

System export

Synthetic data generation

This page covers synthetic data generation reproducibility: can you rebuild it months later.Illustrative

Key takeaways

  • Reproducibility is a promise with levels: exact rebuild, statistical rebuild that passes agreed checks, or archive-only when the generator can no longer be called.
  • Hosted generators retire on the provider's schedule, and requests to a retired model fail, so a dataset made through one is archive-only unless the outputs were kept.
  • Seeds alone do not make generation deterministic; inference engines, frameworks and random number libraries each limit determinism to the same versions and often the same hardware.
  • The manifest pins model weights and dependencies by hash, not by name or branch, because names and branches move.
  • A rebuild is checked against output hashes for exact claims and against an untouched real holdout and negative controls for statistical claims.
  • Buyers should ask which level a synthetic data vendor can deliver before relying on a dataset they may need to regenerate.

How do you make synthetic dataset generation reproducible months later?

You make it reproducible by deciding at generation time which kind of rebuild you will need, then recording every input that rebuild depends on in a manifest stored with the outputs. Most teams record the prompt and the model name and discover six months later that neither is enough.

A synthetic dataset is the output of a program whose inputs include a model checkpoint, an inference stack, random number streams, grounding records, filters and a software environment. Change any of them and the output changes, sometimes subtly. The manifest exists so that a person who did not run the job can say which of those inputs still exist and whether the rebuild matched.

The questions usually arrive for practical reasons: a regulator or customer asks how training data was made, a bug in a filter means one slice must be regenerated, or a licensed seed record has to be removed and everything derived from it rebuilt. Each of those is easier when the answer is written down before it is needed.

Write the manifest in the same job that generates the data, not afterwards from memory. Fields filled in by the pipeline itself, such as resolved commit hashes, image digests and response fingerprints, are the ones that hold up when someone checks them later; fields typed in by hand drift toward what people intended to run rather than what actually ran.

Which level of rebuild can you actually promise?

Promise the highest level your generator and infrastructure can support, and say plainly when that is archive-only. The three levels below are an editorial ladder; they differ in what they require and in what a successful rebuild proves.

Choose the level per dataset, not per team. A small evaluation set generated on owned weights and fixed hardware can reasonably aim for exact; a large instruction set made through several hosted models over a quarter is usually statistical at best and archive-only in practice. Stating the level in the manifest stops a later reader from assuming more than was ever possible.

Mixed GPU fleets deserve a note of their own. If shards were generated on two hardware types, record which shard ran where, because an exact claim can then hold for some shards and not others, and the validator report should say so shard by shard.

Which level of rebuild can you actually promise?
Rebuild levelWhat it requiresWhat a successful rebuild provesTypical blocker
ExactSame weights, engine version, hardware type, seeds, environment and inputsOutput hashes match record for recordHosted generator, mixed GPU types or unpinned engine
StatisticalSame recipe and grounding snapshot; compatible generatorRebuilt set passes agreed distribution, rule and holdout checks within toleranceGenerator retired or replaced by a different model
Archive-onlyPreserved outputs, manifest and hashesThe delivered set is the one that was generated and documentedOutputs not kept, or kept without hashes
NoneNothing beyond the files themselvesNothing about how the data was madeNo manifest written at generation time

Where a rebuild breaks between generation day and month six

Rebuilds most often break because a dependency changed or disappeared, not because someone lost the prompt. The common failure points come from the documentation of the tools themselves.

Hosted generators retire. Anthropic's model lifecycle documentation, for example, defines a retired model as no longer available, with requests to it failing, and commits to at least 60 days of notice for publicly released models. The same page lists temperature, top_p and top_k as deprecated for its newer models, returning an error when set to non-default values, so even the sampling settings of an older recipe may not be replayable on a successor.

Seeds are best effort on hosted APIs. OpenAI's official Python SDK describes its seed parameter as a beta feature that makes a best effort to sample deterministically, states that determinism is not assured and points to the system_fingerprint response field for monitoring backend changes. Record that fingerprint per request; a change between generation and rebuild explains a mismatch before anyone suspects the recipe.

Self-hosted stacks have limits too. The vLLM documentation says it does not promise reproducible results by default and, even with its deterministic settings, only provides reproducibility on the same hardware and the same vLLM version. PyTorch's notes say it does not promise completely reproducible results across releases, individual commits or platforms, or between CPU and GPU runs with identical seeds.

Smaller pieces of the pipeline matter as well. NumPy's compatibility policy expects the same random stream only with the same bit generator, seed, call sequence, NumPy build, environment and machine. Python randomizes string hashing per process unless PYTHONHASHSEED is set to an integer, which can change the order of set iteration and with it the order in which prompts are built or sampled.

The generation recipe and dependency manifest

The manifest is one file per generation run, stored next to the outputs and versioned with them. Each field below names what it is for, an illustrative value and the check a validator runs against it. Values are invented examples, not real identifiers.

At a glance

The generation recipe and dependency manifest

Two fields need particular care. Model weights should be pinned by commit hash: the huggingface_hub library downloads from the main branch by default and accepts a revision such as a tag or commit hash, and only a commit hash cannot move. Software should be locked by version and hash: pip documents generating fully pinned requirements with pip freeze and a hash-checking mode that guards against a package changing without its version number changing.

The generation recipe and dependency manifest
FieldPurposeIllustrative valueValidator check
generatorIdentify the exact model that wrote the dataexample-org/example-8b at revision 3f9a2c (full 40-character hash stored)Revision is a full commit hash, not a branch or tag
generator_accessShow whether the model can be called againself-hosted weights archived; or hosted model id with lifecycle status on generation dateHosted generator forces archive-only unless weights are held
inference_stackPin the engine and its deterministic settingsengine name and exact version, batch invariance on, tensor parallel size 2Version and settings present; matches rebuild host
hardwareRecord what determinism depends onGPU model, count, driver and CUDA versionsSame hardware type for an exact rebuild claim
environment_lockFreeze every Python packagerequirements lock file with hashes, plus container image digestEvery package pinned with a hash; image referenced by digest
seedsFix every random streamglobal seed, per-shard seeds, PYTHONHASHSEED=0, sampler seedNo stream left unseeded; hash seed set
samplingReplay how tokens were drawnsampling settings sent, samples per prompt, max length, stop rulesSettings accepted by the generator as recorded
grounding_snapshotFix the real records the prompts usedseed set id, snapshot date, record count, content hashHash of the snapshot matches; removed records listed
prompts_and_codePin templates, filters and selection logicrepository commit for templates, filters and dedup rulesCommit exists and builds; filter versions listed
outputsProve what was deliveredrecord count and per-shard SHA-256 hashesHashes recomputed and matched

A valid manifest and a failing one (illustrative)

The two summaries below are invented to show how the validator reads a manifest; they describe no real dataset or vendor.

  • Valid, statistical level: open-weight generator pinned by full commit hash with weights archived; inference engine version, hardware and deterministic setting recorded; lock file with hashes and image digest; all seeds including PYTHONHASHSEED; grounding snapshot of 4,000 licensed records with content hash; 120,000 output records hashed in 24 shards; an untouched real holdout of 600 records stored separately. A rebuild on a different GPU type is not hash-identical, so the claim is statistical, and the rebuilt set passes the agreed holdout and rule checks.
  • Failing: generator recorded as a hosted model alias with no dated id or fingerprint; sampling settings that the current model rejects; requirements file with unpinned ranges; grounding records pulled live from a ticketing system with no snapshot; no output hashes. Nothing can be rebuilt or even confirmed, so the honest level is none, and the dataset should be treated as undocumented until its outputs are at least hashed and archived.

What should the rebuild validator check?

The validator should check the manifest first and the outputs second, because a manifest gap explains an output mismatch faster than any statistic. Run it on the original build, store the report, then run it again on every rebuild.

At a glance

What should the rebuild validator check

  • Manifest completeness: every field present, revisions are commit hashes, lock file has hashes, image referenced by digest.
  • Exact claim: per-shard hashes of the rebuild match the originals; any mismatch is reported by shard.
  • Statistical claim: length, label and topic distributions within the tolerance agreed before the rebuild.
  • Rule consistency: generated records still obey the business rules or policy version dated in the manifest.
  • Real holdout: models trained on original and rebuilt sets score within tolerance on the same untouched real records.
  • Negative controls: a deliberately broken recipe (wrong seed set or filter off) must fail the checks, proving they can detect a difference.
  • Privacy screen: the rebuilt set passes the same near-duplicate and personal data checks against the grounding snapshot.

A perfect rebuild can still miss the real exceptions

A rebuild that matches the original perfectly can still be wrong in the same way the original was, so reproducibility has to sit next to a comparison with untouched real cases. Keep a holdout of real records that never entered prompts, filters or tuning, and record its identifiers in the manifest so later rebuilds use the same one.

Check the holdout for the exceptions generators tend to smooth away: rare categories, malformed or partial inputs, records that changed policy mid-thread, and edge cases where staff overrode the normal rule. If the synthetic set has none of these and the holdout has many, the dataset is reproducible but thin, and the manifest should say so in a known-gaps field rather than leaving the discovery to the buyer.

Date the policy too. When generated records encode business rules, such as refund limits or escalation thresholds, write the policy version and its effective date into the manifest. A rebuild made after the policy changed should either use the old version on purpose or be labeled as a new dataset, never a silent mix of both.

Rebuilding after a grounding record has to be removed

A removal request is the rebuild most teams eventually face, and the manifest decides whether it takes an afternoon or a quarter. When a licensed seed record must be deleted, because a license ended or a person asked for erasure, you need to find every generated record that drew on it and regenerate or drop exactly those.

That requires a link from each output record to the grounding records in its prompt, kept as stable identifiers or salted hashes, and a grounding snapshot that can be re-cut without the removed records. With both, the rebuild is targeted: regenerate the affected shard from the trimmed snapshot, rerun the validator on that shard, and publish a new manifest version that lists the removal and the new output hashes.

Without the link, the only safe option is to rebuild everything or withdraw the dataset. If the generator has since retired, even the full rebuild becomes a new dataset made with a different model, which is why the removal path should be tested on a small sample while the original generator is still available.

  • Keep: output record id to grounding record ids, stored privately with the manifest.
  • Keep: the snapshot cut script, so a trimmed snapshot can be rebuilt with the same filters.
  • Record: each removal as a manifest version with date, reason code and affected shard list.
  • Test: one removal drill per dataset before the generator's announced retirement date, if it has one.

What should a buyer ask a synthetic data vendor before relying on a dataset?

Ask which rebuild level the vendor can support and what evidence backs it, before the dataset becomes a dependency of your models. A vendor that generated through a hosted model it does not control may honestly offer only archive-only reproducibility; that can be acceptable if the outputs, hashes and manifest are delivered with the data.

Useful requests include the manifest itself, the validator report from the original build, the identifiers of the real holdout used, and a statement of what happens if a grounding record must later be removed. The provenance and rights questions about the generator's output terms and the seed records are separate, and they are covered in the linked provenance and derived-rights guides.

Write the agreed level into the acceptance criteria for the purchase. If the vendor promises a statistical rebuild, name the checks and tolerances now; if it offers archive-only, require hashed outputs and the manifest at delivery, so a later question about how the data was made has a documented answer rather than a recollection.

Where licensed real records fit in a rebuildable mixture

Licensed real records make a mixture easier to rebuild because they are a fixed snapshot with known terms, while the synthetic share depends on a generator that may retire. They also supply the untouched holdout that every statistical rebuild needs.

A team planning a mixed real and synthetic set can request the real share through SourceX, which sources it from U.S. businesses for that request instead of from a standing collection; the request records scope, volume, language, format, timing and licensing needs. Because the supplier approves scope and signs before delivery, the grounding snapshot named in the manifest comes with a dated agreement.

Frequently asked questions

Does setting a random seed make LLM data generation reproducible?

Not on its own. Hosted APIs describe seeds as best effort, and self-hosted engines and frameworks limit determinism to the same versions and often the same hardware. A seed is one field in the manifest, alongside pinned weights, engine, environment and inputs.

What happens to reproducibility when a hosted generator model is retired?

Requests to a retired model fail, so the exact recipe can no longer run. The dataset drops to archive-only unless you kept the outputs and hashes, and any regeneration with a successor model is a new dataset that must pass statistical checks.

Is a statistical rebuild good enough?

Often, if the tolerances were agreed before the rebuild and the checks include an untouched real holdout and negative controls. It is not enough when a contract, audit or deletion request requires proof that specific records were or were not generated.

Where should the manifest be stored?

Store it next to the outputs in the same versioned location, with its own hash, and include it in any delivery. A manifest kept in a separate wiki or notebook tends to drift from the data it describes.

How do I show that a rebuild check can detect a difference?

Run a negative control: rebuild once with a deliberate change, such as the wrong grounding snapshot or a filter switched off, and confirm the validator fails it. A check that passes everything proves nothing about the real rebuild.

Sources

  • Anthropic's lifecycle documentation says requests to retired models fail, gives at least 60 days of notice for publicly released models and lists temperature, top_p and top_k as deprecated for newer models. Source
  • OpenAI's SDK describes the seed parameter as a beta, best-effort feature without assured determinism and points to system_fingerprint for backend changes. Source
  • vLLM does not promise reproducibility by default and provides it only on the same hardware and vLLM version. Source
  • PyTorch does not promise reproducible results across releases, commits or platforms, or between CPU and GPU runs. Source
  • NumPy's compatibility policy expects the same random stream only with the same bit generator, seed, call sequence, build, environment and machine. Source
  • Python seeds string and bytes hashing randomly unless PYTHONHASHSEED is set to an integer. Source
  • huggingface_hub downloads the main branch by default and accepts a revision such as a branch, tag or commit hash. Source
  • pip documents fully pinned requirements via pip freeze and hash-checking that guards against packages changing without a version change. Source

Tell us what data you need

Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.

See if your company qualifies