Skip to content

AI data market

Data sourcing mix: combining synthetic, custom and existing data

By SourceX Editorial · Updated

10 sections

Short answer

Combine the three routes by task requirement, not by volume. Existing business records supply the real case mix, context and outcomes; expert-created examples fill named gaps such as rare cases, rationales and format; synthetic data widens coverage around real or expert seeds. Keep a locked real test set, let synthetic data accumulate beside real data rather than replace it, and tag every record with its route.

License terms

Data sourcing mix

This page covers data sourcing mix: combining synthetic, custom and existing data.Illustrative

Key takeaways

  • Write task requirements first; each one usually points to a different primary route.
  • Existing records are the source of truth for distribution and outcomes; outcomes cannot be generated.
  • Expert-created data should fill a named gap and stop when its acceptance test passes.
  • Synthetic rows need a named seed, should accumulate beside real data, and should be raised only when real-data tests show a gain.
  • Tag every record with route, source, generator, rights basis and split status so the mix can be audited and changed.

How do you deliberately combine existing, expert-created and synthetic data?

Allocate sources requirement by requirement, not by volume. Use existing business records for the distribution, context and outcomes of real work; commission expert-created examples for the requirements those records cannot supply, such as rare cases, explicit rationales or a fixed output format; and use synthetic data to widen coverage around real or expert seeds, never to replace them and never as the only evaluation evidence. Then tag every record with its route so the mix can be measured and changed.

Most sourcing comparisons treat the three routes as competitors and ask which one to buy. In practice, a capable data plan uses all three for different jobs, and the failures come from mixing them without a plan: synthetic examples that drift from reality, expert tasks that are cleaner than the work the model will see, and licensed records that never covered the edge cases the product depends on.

This guide gives the worksheet for making that allocation explicit, the decision rules that set proportions without inventing a ratio, and the record-level tags that keep the mix auditable after training.

What each sourcing route can and cannot carry

Each route carries a different kind of truth, so start by being clear about what you are actually buying. The table below is the comparison the worksheet relies on.

What each sourcing route can and cannot carry
RouteWhat it carries wellWhat it cannot carryMain failure if overused
Existing business recordsReal distribution of cases, messy inputs, context across systems, actual outcomesCases the business never saw; rationales nobody wrote down; a target formatModel inherits the business's gaps and habits
Expert-created examplesChosen difficulty, explicit reasoning, rare cases on demand, a clean formatThe frequency and texture of real work; outcomes that took months to observeModel is fluent on tidy tasks and brittle on real ones
Synthetic dataVolume and variation around a seed, paraphrase, format practice, controlled perturbationsFacts or cases absent from its seeds and generator; trustworthy labels without checksNarrowing toward the generator's habits; lost tail cases
Mixed without tagsNothing extraAny way to measure which route helped or hurtProblems cannot be traced or removed

Break the task into requirements before choosing any source

A source decision only makes sense against a specific requirement, so write the requirements first. A requirement is a capability the model must show, stated so that someone could check it on a sample, together with the evidence that would prove the capability.

At a glance

Break the task into requirements before choosing any source

Useful requirement types for most business tasks are listed below. Each one tends to point toward a different route, which is why one task usually needs a mix.

Checklist · 7 items

  1. Coverage of the normal case mix: what share of inputs look like routine work, and in what forms they really arrive.
  2. Rare but costly cases: exceptions, escalations, fraud patterns or safety-relevant situations that appear seldom.
  3. Reasoning or rationale: the explanation a competent person would give, which records often lack.
  4. Output format and policy: the structure, tone and rules the product must follow.
  5. Grounding in outcomes: whether a decision turned out right, known only from later records.
  6. Robustness: performance on noisy, incomplete, misspelled or reordered inputs.
  7. Freshness: how quickly the underlying process changes and how often data must be replaced.

0 of 7 ticked. Tick each item as you go.

The task requirement and source mix planning worksheet

Fill one row per requirement. The primary route is the one that supplies the ground truth; the supporting route adds volume or variety without becoming the source of truth. The acceptance test is what a reviewer checks on a sample before that portion of the data is accepted.

The task requirement and source mix planning worksheet
RequirementEvidence that proves itPrimary routeSupporting routeAcceptance test
Normal case mixSample matches the live input distributionExisting recordsSynthetic paraphrase of real inputs, capped and taggedDistribution check against a held-out real sample
Rare costly casesEach defined case type appears often enough to learn and to testExisting records, filtered for the casesExpert-created cases where records are too fewExpert review of every case; real cases reserved for test
RationaleExplanations an expert would acceptExpert-created rationales on real inputsSynthetic variants checked by expertsBlind expert rating against a rubric
Output format and policyOutputs follow the specificationExpert-written exemplarsSynthetic format practiceAutomated schema and policy checks
Outcome groundingLabels tied to observed later outcomesExisting records with outcome fieldsNone; outcomes cannot be generatedLabel audit on a random sample
RobustnessStable performance on degraded inputsExisting noisy recordsSynthetic perturbations of real inputsPerformance gap between clean and noisy test slices
FreshnessData reflects the current processRecent existing recordsExpert updates when rules changeDate coverage check and rule change log

Rules that set the proportions without guessing a ratio

Set proportions from evidence about each requirement, not from a target percentage chosen in advance. The rules below keep synthetic and expert data in their supporting roles and make the mix adjustable.

Two findings from the research literature explain why the anchoring rules matter. A 2024 Nature study by Shumailov and colleagues found that training models on recursively generated data causes them to lose information about the true distribution, starting with the tails. Gerstgrasser and colleagues then reported that the collapse they observed when synthetic data replaced real data was avoided in their experiments when synthetic data accumulated alongside the original real data. Neither result sets a safe ratio for your task, but both point the same way: keep real data in the mix and keep it identifiable.

Self-Instruct is a useful illustration of seeding. Its pipeline starts from a seed set of manually written tasks, 175 of them in the released data, and bootstraps a far larger instruction set from them. Whatever the generator adds is shaped by those seeds, which is why the seeds deserve the most expert attention in any synthetic plan.

  • Every synthetic record names the real or expert seed it was generated from; no seed, no record.
  • Synthetic data accumulates beside the real data; it never replaces real records in a later training round.
  • Expert-created data fills a named gap from the worksheet, and stops when that gap's acceptance test passes.
  • Raise the synthetic share for a requirement only after an ablation on real test data shows it helps that requirement.
  • If a requirement has no real or expert evidence at all, treat it as unproven rather than generating around the gap.

Keeping the evaluation set out of the mix

Build the test set first, from real records that no other route has touched, and lock it before any synthetic or expert data is produced. Synthetic examples generated from the same seeds as the test cases will flatter the model, and expert tasks written by the same people who wrote the training examples share their blind spots.

A practical split keeps three slices: a real holdout that mirrors the live distribution, a real rare-case slice for the costly exceptions, and an optional expert slice for capabilities that real records cannot show, such as written rationales. Report results per slice and per training route. A model that improves on the expert slice while slipping on the real holdout is learning the experts' style, not the task.

Run the comparison as an ablation whenever the mix changes: train once with the new rows and once without them, then compare on the locked real slices only. If the change helps on synthetic or expert test items but not on real ones, it has not earned its place in the mix, however good the generated examples look on inspection. Keep the ablation results with the worksheet, so the next person to change a proportion can see why the current one was chosen.

Tagging every record with its route, generator and rights basis

Tag each record at creation with its route and lineage, because a mix that cannot be decomposed cannot be fixed. Datasheets for Datasets, proposed by Gebru and colleagues, ask creators to document a dataset's motivation, composition, collection process and recommended uses; a mixed dataset needs those answers per route, not only for the whole.

At a glance

Tagging every record with its route, generator and rights basis

Rights differ by route too. Generated data inherits the terms of its seeds and, in some cases, of the generator. The Stanford Alpaca project, for example, released 52K instructions generated with the Self-Instruct approach under CC BY NC 4.0 and states the dataset is intended and licensed for research use only. For high-risk systems in the EU, Article 10 of the AI Act requires data governance practices covering data collection processes and the origin of data, and preparation operations such as annotation, labeling, cleaning, updating, enrichment and aggregation, which per-record route tags make far easier to show.

  • route: existing, expert, synthetic or derived, with derived meaning a transformed real record.
  • source_ref: the licensed dataset and record ID, the expert task order and author role, or the seed IDs.
  • generator: model or tool name and version, prompt template ID and the generator's terms version, for synthetic rows.
  • rights_basis: license, statement of work or generator terms that allow this use, with any field-level limits.
  • review_status: who checked the row, against which acceptance test, and when.
  • split_lock: whether the row or any of its seeds sits in a test slice, so derived rows can be excluded from training.

Worked scenario (illustrative): a warranty claim triage model

Consider an illustrative plan for a model that triages warranty claims for an equipment maker into approve, request information, inspect or deny, with a short rationale. The figures below are illustrative shares of the training set, not a recommendation.

The normal case mix comes from three years of licensed claim records with their final dispositions, which also supply outcome grounding. A distribution check shows the records cover routine claims well, so existing records make up roughly 70% of training rows in this plan, with paraphrased claim text adding a capped share for robustness.

Suspected fraud and safety-related failures are rare in the records. Experts write cases for the patterns that appear fewer than a set number of times, around 10% of rows, and every real instance of those patterns is reserved for the rare-case test slice rather than training.

The records hold dispositions but almost no written reasoning, so experts write rationales for a sample of real claims, and synthetic variants of those rationale examples, checked by an expert, make up most of the remaining share. Format practice rows are generated freely because the schema check catches every failure.

The exception is the deny label. Synthetic variants of denied claims kept producing plausible but invented policy reasons, so the team stopped generating them and relied on real denials plus expert-written examples. That is the worksheet working as intended: the acceptance test caught a requirement where generation hurt, and that row of the mix changed.

Separating provider claims from evidence, and showing what is still unknown

Mark each input to the worksheet as observed, provider-claimed or assumed, because a mix built on a vendor's description is only as good as that description. A data provider saying its synthetic data is privacy-safe or representative is a claim; a test you ran on its sample against your real holdout is evidence.

At a glance

Separating provider claims from evidence, and showing what is still unknown

Show uncertainty where it changes the decision. The assumption that most often flips the mix is the coverage of rare cases in existing records: if a records sample shows more rare cases than expected, expert creation shrinks; if fewer, it grows. State that number as a range from the sample, and rerun the plan when the range moves.

  • Observed: measured on a sample you hold, with the sample size and date.
  • Provider-claimed: taken from a provider's documentation or proposal, dated and attributed, not yet tested.
  • Assumed: your team's estimate, with the reason and what would change it.
  • Unknown: left blank on purpose, with the step that would fill it, such as a paid pilot or a records sample.

When the worksheet points to existing business records

When several rows of the worksheet name existing business records as the primary route, the next step is finding a company that holds them and can license them. SourceX looks for those records at U.S. businesses request by request and keeps no inventory of ready-made datasets.

Buyers describe the scope, volume, language, format, timing and licensing needs in a data request, and those answers map directly onto the requirement, evidence and acceptance columns of the worksheet. Bringing the completed worksheet to that request makes it clear which requirements the records must cover and which will be filled by other routes.

Frequently asked questions

Is synthetic data better than real data for training?

Neither is better in general; they do different jobs. Real records carry the true distribution and outcomes, while synthetic data adds volume and variation around seeds. Research on recursive training found models lose tail information when generated data replaces real data, so keep real data in the mix.

What should a source mix planning worksheet include?

One row per task requirement, with the evidence that proves it, the primary route that supplies ground truth, a supporting route, and an acceptance test. Add whether each input is observed, provider-claimed or assumed, so reviewers can see which parts of the plan are untested.

Can synthetic data be used in the evaluation set?

Keep the core evaluation set real and untouched by any route used for training. Synthetic tests can probe format or robustness, but scores on them should be reported separately, since examples generated from the same seeds as training data inflate results.

When is expert-created data worth commissioning?

When a requirement cannot be met from existing records, such as rare cases, written rationales or a strict output format. Commission against a named gap with an acceptance test, and stop once it passes, so expert tasks support real data rather than replacing it.

Does generated data carry license restrictions?

It can. Generated data may inherit terms from its seed data and its generator, and some released datasets are limited to research use. Record the generator, its terms version and the seed sources on every synthetic row, and have counsel review commercial use.

Sources

  • Shumailov and colleagues (Nature, 2024) found that models trained on recursively generated data lose information about the true distribution, beginning with its tails. Source
  • Gerstgrasser and colleagues report that replacing real data with each generation's synthetic data tends toward model collapse, while accumulating synthetic data alongside the original real data avoided it in their experiments. Source
  • The Self-Instruct repository describes bootstrapping instructions from a seed set of manually written tasks and releases 175 seed tasks with the generated data. Source
  • The Stanford Alpaca repository releases 52K instruction-following examples generated with Self-Instruct techniques under CC BY NC 4.0 and states the dataset is licensed for research use only. Source
  • Article 10 of the EU AI Act requires data governance for high-risk systems' training, validation and testing data, covering collection processes, data origin and preparation operations such as annotation, labeling, cleaning, updating, enrichment and aggregation. Source
  • Datasheets for Datasets proposes documenting each dataset's motivation, composition, collection process and recommended uses. Source

Tell us what data you need

Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.

See if your company qualifies