AI uses for records
SourceX dataset refresh value: four tests before you request new records
By SourceX Editorial · Reviewed by Noah Loul ·
9 sections
Short answer
SourceX dataset refresh value comes from what newer records add that the first delivery lacks, not from how many there are. Because SourceX sources datasets on request, a refresh is a new request the supplying company must approve. Before asking, test four things: whether the business changed, which task cells are thin, which outcomes have matured since the cutoff, and how much of a new window would repeat old patterns.
Value drivers
SourceX dataset refresh
Key takeaways
- A SourceX refresh is a new, scoped request the supplying company approves, so specify what the new records are for.
- Three value sources justify a refresh: a dated business change, thin task cells and outcomes that matured after the cutoff.
- More of the same rarely pays; measure the repeat rate of your latest quarter before asking for a later window.
- A re-extraction of still-open case IDs can add more labels than a batch of new cases.
- Keep a frozen, version-pinned evaluation anchor and date every rule change, or old and new scores stop being comparable.
What makes a SourceX refresh worth requesting?
A refresh is worth requesting when newer records would change what your model learns or how you score it, not when there are simply more of them. SourceX sources datasets on request from U.S. businesses instead of selling from stock, so a refresh is not a subscription switch: it is a new, scoped request that the company holding the records has to agree to before anything moves.
That structure shapes the decision. The supplying company approves the scope and terms and signs before it delivers any data to SourceX, de-identification is handled by SourceX or its processing partner, and exclusivity and term are written into each deal's own signed agreement, so every round is scoped on its merits. A buyer who asks for the next six months gets whatever those months happen to contain. A buyer who asks for named cells, a window that starts at a dated business change, or a re-pull of cases that were still open gets records chosen for value.
This page gives a buyer-led way to choose among those requests, or to decide that none is needed: four tests run on the delivery you already hold, a worked example with labeled assumptions, the hazards that make newer records harmful, and a card to send with a second request.
Four sources of refresh value, and the one that rarely pays
New records add value in four ways, and only three reliably justify a new sourcing round: they capture behavior after a business change, they fill task cells the first delivery left thin, or they complete outcomes that were still open at the first cutoff. The fourth, more examples of patterns you already have, is what a plain later window mostly delivers, and it is usually the weakest.
Run the tests in this order, because each can settle the request on its own. A confirmed change sets where the window starts, thin cells set the scope, matured outcomes may need no new cases at all, and a high repeat rate argues for a smaller refresh or none.
| Value source | What new records add | Test on the delivery you hold | What to ask for |
|---|---|---|---|
| Change | Behavior under the policy, product or system in force now | Date every business change since your window closed; check whether production errors rise after it | A window that starts at the change date, sized to your evaluation need |
| Coverage | Cases in task cells the first delivery under-filled | Count records per cell against a minimum set before counting | Named cells with per-cell minimums, from the same company or a different one |
| Maturity | Final outcomes for cases still open at the first cutoff | Share of first-delivery cases with no final status at extraction | A re-extraction of the same case IDs with a new as-of date |
| Volume | More examples of patterns already covered | Repeat rate of the latest quarter against earlier quarters | Usually nothing; at most a capped sample |
The change test: did the business move after your window closed?
Recency pays only when something in the business changed after your first window ended and your task depends on it. Without a dated change, a later window mostly repeats the earlier one; with one, the old records may teach a rule the business no longer follows.
Banking supervisors wrote this principle down long before AI datasets were traded. The Federal Reserve's 2011 model risk guidance, SR 11-7, says ongoing monitoring is essential to evaluate whether changes in products, exposures, activities, clients or market conditions require a model to be adjusted, redeveloped or replaced. Put dataset where it says model and the test carries over: name the change, then decide whether it reaches your task.
How much a change costs is task-specific. Wild-Time, a benchmark of five datasets with real shifts over time, reported an average performance drop of 20% from in-distribution to out-of-distribution data across the 13 methods it tested, and found that none closed the gap. Your own number comes from your own errors: compare production failure rates before and after the dated change, by case type. If they do not move, recency is not the problem, and a refresh bought for it would be wasted.
The coverage test: count thin cells before asking for more months
Coverage pays when specific task cells are too thin to train or score reliably, and the fix is to name those cells rather than extend the window. A cell is one combination your task cares about, such as case type by outcome by customer segment.
Set the minimum per cell before you count, from your evaluation plan: enough cases to estimate an error rate within the margin you would act on. Then list every cell below it. If a cell is thin because the work is rare at that company, a later window will fill it slowly; the same scope from a different company may fill it faster, which a request-based model allows. SourceX's general guide is that buyers look for years of consistent history and tens of thousands of records in one system, while smaller sets can qualify when the work is rare, and every dataset is assessed on its own.
Choosing between the same company and a new one is a trade between continuity and breadth. The same company keeps definitions, systems and labeling habits stable, which makes the new records easy to merge; a second company brings different customers and workflows, which tests whether your model generalizes but adds a mapping job for every field. If the thin cells are rare everywhere, neither choice fills them quickly, and a smaller evaluation-only target may be the honest answer.
Write the cells into the request. A line such as warranty disputes resolved by partial refund, enterprise accounts, at least 150 cases can be checked on delivery; a line asking for more recent data cannot.
The maturity test: outcomes that finished after the cutoff
Sometimes the most valuable refresh contains no new cases at all. If a meaningful share of the first delivery was still open when it was extracted, re-pulling those same case IDs months later adds the outcomes your labels were missing.
Measure, by cell, the share of first-delivery cases with no final status at extraction. A dispute, claim or ticket that closed after the cutoff turns an unlabeled case into a labeled one, and those are often the long, contested cases a model most needs to see resolved. Ask for a re-extraction keyed to the original IDs, with the new as-of date stated, so you can see which fields changed after the first delivery and treat later edits with care.
The novelty test: how much of a new window repeats the old one?
Volume is the weakest reason to refresh, because operational systems repeat themselves by design. Help desks are a clear case: Zendesk documents macros as the way to create repetitive ticket responses and actions, and one macro can set comment text, ticket fields and the subject line at once. A later window from the same help desk will hold many cases that differ only in names and dates.
Repetition is not free. Anthropic researchers reported in 2022 that repeating 0.1% of training data 100 times could degrade an 800M-parameter language model to the performance of a model half its size, even though the other 90% of training tokens stayed unique. Working the other way, the SemDeDup authors removed 50% of a LAION image-text subset as semantic near-duplicates with minimal loss in performance. Neither result transfers directly to support records, but both point the same way: records that repeat existing patterns add little and can cost something.
Estimate the repeat rate on data you already hold. Split the first delivery by quarter, normalize each record's text (lower-case it and replace numbers, names and IDs with placeholders), and measure what share of the latest quarter's records match an earlier-quarter record exactly or above a similarity threshold you fix in advance. That share forecasts how much of the next window would repeat what you have. Write down the normalization rules and the threshold, so the same test can be rerun on the refresh when it arrives.
Worked example (illustrative): a support dataset, two quarters later
The numbers below are illustrative; they describe no real company and no SourceX deal. A buyer holds 48,000 support cases covering 24 months, organized into 36 cells (6 case types by 3 outcomes by 2 customer segments), and runs the four tests before deciding what to request.
The change test finds that the returns policy changed after the window closed, and production errors on return requests rose after that date while other case types stayed flat, so recency matters for 1 of 6 case types. The coverage test finds 9 of 36 cells below the buyer's preset minimum of 150 cases, 6 of them enterprise-segment disputes. The maturity test finds 6% of cases (2,880) with no final status at extraction, concentrated in chargeback disputes. The novelty test finds that 41% of the latest quarter's records match an earlier-quarter record after normalization, mostly macro-driven password and shipping-status cases.
The request that follows from those four results:
- Ask for a post-change window covering return requests only, starting at the policy change date.
- Ask for a targeted fill of the 9 thin cells, with the 150-case minimum written in for each.
- Ask for a re-extraction of the 2,880 open cases by original ID, with a new as-of date.
- Skip the general six-month extension: on these numbers about two in five of its records would repeat what the buyer already has, and most of the rest would land in the 27 cells that already meet their minimums.
When newer records make your comparisons worse
A refresh can make a dataset worse even when every record is genuine. Three hazards recur: rule changes that alter what a label means, task changes that make old and new scores incomparable, and overlap between new records and the cases you evaluate on.
Rule changes are the quietest. If the company redefined a status, merged two categories or changed a resolution code inside the new window, a model trained across that boundary learns two meanings for one label. Ask for the date of any change to labels, exclusion rules or de-identification, and treat records on each side as separate versions until you have compared them.
Task changes are louder but easy to misread in a single score. A new product line, channel or customer segment shifts the case mix, so a blended accuracy figure can move even when the model is no better or worse at any one kind of case. Report results per cell and weight them to the mix you intend to serve, not to whatever mix the refresh happened to contain.
Comparability needs a frozen anchor. Keep the first delivery's evaluation set unchanged and pinned to a version identifier, the way the Hugging Face datasets library lets a user load a specific dataset version by Git tag, branch or commit through its revision parameter. Score each refresh against that anchor and against a separate post-change set, and report both numbers. Assign evaluation cases by a fixed rule on customer, case or incident ID, so that a refreshed record about the same customer cannot slip into training.
The refresh request card to send with a second request
Send a one-page card with any second request so the supplying company, SourceX and your own team agree on what the new records are for. It works as a time-aware data card: every field ties the request to a date, a version or a count that can be checked when the records arrive.
Pair the card with a change-impact checklist, the six checks a refresh must clear before it reaches training or evaluation, and submit both through the request form at sourcex.si/request-data with the intended use and the acceptance test written in.
- Every refreshed record carries its source window and its as-of extraction date.
- No refreshed record shares a customer, case or incident ID with the frozen evaluation set.
- Label and status distributions are compared across the boundary, and any definition change is dated.
- The repeat share is measured with the pre-set rules and reported per cell.
- Thin cells are checked against their minimums, and shortfalls are reported rather than padded from other cells.
- Scores are reported on both the frozen anchor and the post-change set.
| Card field | What to state | Why it matters |
|---|---|---|
| Prior delivery | Reference, window start and end, as-of extraction date | Defines what counts as new |
| Change log | Each business change since the window closed, with its date | Sets where a post-change window starts |
| Target cells | Named cells with a minimum for each | Turns a vague refresh into a checkable scope |
| Open-case re-extraction | Original case IDs and the new as-of date | Adds outcomes without adding cases |
| Repeat ceiling | Normalization rules, similarity threshold and maximum repeat share | Limits how much of the refresh may duplicate the first delivery |
| Rule versions | Label, exclusion and de-identification versions on both sides of any change | Keeps one label from carrying two meanings |
| Frozen anchor and acceptance test | Pinned evaluation set version and the test the refresh must pass | Keeps old and new scores comparable |
Frequently asked questions
Does SourceX sell automatic dataset refreshes?
SourceX describes its datasets as sourced on request from U.S. businesses rather than held in stock, so a refresh is a new request scoped like the first one. The supplying company approves it and signs before delivering, and each deal's terms sit in its own signed agreement. Describe what the new records are for when you submit the request.
How do I know if newer data will improve my model?
Check whether a dated business change affects your task, whether production errors rise after it, and whether your thin cells or open outcomes are what limit you. If none of those apply, newer records mostly repeat what you have, and a refresh is unlikely to move your scores.
Should a refresh replace or extend the first delivery?
Usually extend, and keep the first delivery's evaluation set frozen. Replace only the parts a dated rule change made wrong, and keep both versions labeled so scores before and after the change can be compared.
Can model scores be compared across a dataset refresh?
Only against a frozen anchor set pinned to a version, with evaluation cases assigned by a fixed rule on customer or case ID. Report the anchor score and a separate post-change score; a single blended number hides whether the refresh helped.
What should a refresh request to SourceX include?
The prior delivery's window and as-of date, any dated business changes, the named cells and their minimums, the open case IDs to re-extract, a repeat ceiling, rule versions and the acceptance test. Submit it through the request form with the intended use.
Sources
- The Federal Reserve's SR 11-7 guidance says ongoing monitoring is essential to evaluate whether changes in products, exposures, activities, clients or market conditions necessitate adjustment, redevelopment or replacement of a model. Source
- Wild-Time benchmarks five datasets with temporal shift and 13 methods, reporting an average 20% performance drop from in-distribution to out-of-distribution data that existing methods do not close. Source
- Zendesk documents macros for repetitive ticket responses and actions; a macro can add comment text, update ticket fields and set the subject. Source
- Hernandez et al. (Anthropic, 2022) found that repeating 0.1% of data 100 times can degrade an 800M-parameter model to the performance of a 400M-parameter model while 90% of training tokens remain unique. Source
- SemDeDup removed 50% of a LAION subset as semantic near-duplicates with minimal performance loss. Source
- The Hugging Face datasets library loads a specific dataset version by Git tag, branch or commit through the revision parameter. Source
Tell us what data you need
Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.