Skip to content

Systems and records

Dataset acquired business distribution shift: comparing coverage before and after a deal

By SourceX Editorial · Reviewed by Noah Loul ·

10 sections

Short answer

To detect workflow distribution changes after an acquisition, tag every record with the company and system it came from and whether it predates the deal close, define the population the model will serve, then compare the mix of case types, channels, customers, process steps and outcomes across those groups. Check what migration left out, and decide whether to reweight, stratify or keep the companies as separate slices.

License terms

Dataset acquired business

This page covers dataset acquired business distribution shift: comparing coverage before and after a deal.Illustrative

Key takeaways

  • Label every record with its originating company, source system and pre- or post-close period before analysis.
  • Define the population the model will serve first; shift only means something relative to a target.
  • Compare coverage across segments, job types, channels, process steps, field completion and outcome maturity.
  • Judge differences by effect size and decision impact, since large datasets make almost any difference significant.
  • Inventory what migration skipped, because missing records after an acquisition are rarely missing at random.

How do you detect a workflow distribution shift after an acquisition?

Treat the acquisition as a known change point and compare the data on both sides of it. Every record needs three labels before any analysis: the originating company, the source system, and whether it was created before or after the deal closed. With those in place, a coverage comparison across case types, channels, customer segments, regions, process steps, field completion and outcomes shows what the combined business now does that the legacy business did not.

This matters for anyone building or buying a dataset from a company that has grown by acquisition. A model trained on six years of one company's service tickets learns that company's work. If the last eighteen months include a newly acquired business with different customers and procedures, the training set is a blend of two populations whose proportions changed partway through, and a random split will hide it.

The rest of this guide covers what kinds of change an acquisition introduces, how to define the population you care about, the comparison table itself, which statistics to run, the records migration tends to drop, a worked example and the cases where the shift is the point rather than the problem.

What actually changes in the data when one company buys another?

Usually several things at once, and they need different fixes. The mix of work shifts, because the acquired company brings its own customers and job types. New categories appear that the legacy taxonomy never had. Codes get remapped during integration, so the same label can mean something narrower or broader after close. Procedures change as the acquired team adopts, or resists, the parent's process. And recording itself changes: fields become mandatory, statuses are renamed, channels are consolidated.

Machine learning research groups these under distribution shift. The 2016 Concrete Problems in AI Safety paper lists robustness to distributional shift among its core accident risks, and the WILDS benchmark of in-the-wild distribution shifts attaches domain metadata to every example, such as which hospital a patient's data came from, so models can be trained and evaluated by group. An acquired company is a domain in exactly that sense, and the originating-company label plays the role of the hospital ID.

What actually changes in the data when one company buys another?
Change typeWhat it looks like in recordsUsual fix
Mix shiftSame categories, very different proportions after closeReweight or stratify to the target population
New categoriesJob or case types with no legacy equivalentExtend the taxonomy; evaluate the new slice separately
Code remappingOld codes collapsed or split during integrationKeep original codes beside mapped ones; audit the crosswalk
Process changeDifferent step order, approvals or handoffs in acquired recordsLabel process version; do not pool for workflow models
Recording changeFields newly required, renamed statuses, merged channelsCompare field completion by period before reading content shifts

Who is the model for, and which cases are underrepresented relative to those users?

Decide the deployment population before measuring anything, because shift is only defined relative to a target. If the model will serve the combined company from next quarter on, the post-close mix is the target and six years of legacy-only history over-represents the old business. If it will serve only the legacy division, acquired records are out of scope. If it will be sold to other companies, neither business's mix is the target and both should be described honestly.

Datasheets for Datasets asks the same question of any collection: whether it holds every possible instance or a sample of a larger set, what that larger set is, and whether and how representativeness was checked. Answer it in writing for the acquisition: name the population, the sampling frame you actually have, and the groups that frame misses. Customers the acquired company served but who left before migration, channels it used that the parent does not, and job types it handled in a separate system are the usual gaps.

The pre and post acquisition coverage comparison

Build one table per dataset, with a row for each dimension the model's inputs or outputs depend on and a column for each group. The illustrative version below uses four groups: legacy company before close, legacy company after close, acquired company after close (migrated or native), and the stated target population. Values are shares of records and are illustrative, not from any real company.

The pre and post acquisition coverage comparison
DimensionLegacy, pre-closeLegacy, post-closeAcquired, post-closeCheck before use
Customer segment: commercial shareIllustrative 12 percentIllustrative 14 percentIllustrative 71 percentCompare with target mix; reweight if target differs
Top job type shareResidential repair, illustrative 48 percentResidential repair, illustrative 45 percentRefrigeration service, illustrative 52 percentFlag categories with no legacy equivalent
Intake channel: phone shareIllustrative 80 percentIllustrative 65 percentIllustrative 30 percentSeparate channel change from portal rollout timing
Steps per completed job, medianIllustrative 4Illustrative 4Illustrative 7Check whether extra steps are real work or system artifacts
Resolution code completionIllustrative 95 percentIllustrative 96 percentIllustrative 60 percentLow completion may reflect mapping, not missing work
Records with mature outcomesIllustrative 98 percentIllustrative 85 percentIllustrative 55 percentExclude or flag jobs inside the callback window
History depth availableSix yearsEighteen monthsTwo years migrated, open items only before thatRecord what migration cut and why
Region coverageTwo statesTwo statesThree states, one newTreat the new region as a separate slice

Which statistics show whether the difference is real and large enough to matter?

Use simple two-sample tests per dimension, then judge by effect size, not by p-value alone. For categorical fields such as job type or channel, a chi-square test of independence on a contingency table of group by category is the standard choice; SciPy's chi2_contingency computes it from observed counts with expected frequencies derived from the marginal sums. For continuous fields such as handle time or steps per job, a two-sample Kolmogorov-Smirnov test compares the two empirical distributions; SciPy's ks_2samp implements it.

With tens of thousands of records, almost any difference will be statistically significant. Report the size of the change, such as the difference in category shares or the KS statistic, and set in advance how large a change needs to be before it changes a decision. For many fields at once, a domain classifier is a useful summary: train a model to predict legacy versus acquired from the record's features. The Failing Loudly study of shift detection found two-sample tests on reduced representations worked well and that domain-discriminating approaches help characterize what shifted and whether it is harmful. If the classifier separates the groups easily, its most important features name the shift.

  • Categorical dimensions: chi-square on group by category counts, plus the share difference for each category.
  • Continuous dimensions: two-sample KS test plus medians and the 90th percentile for each group.
  • Text fields: compare category labels first; embedding comparisons come after you know which slices differ.
  • Many dimensions at once: a domain classifier, reading its feature importances rather than its accuracy alone.
  • Over time: the same comparisons by quarter, so integration milestones show up as steps rather than averages.

What did the migration leave out, delay or record selectively?

Integration projects rarely move everything, and what they skip is not random. Common patterns: only open work orders and a recent window of closed ones were migrated; attachments and free-text notes stayed in the old system; records were re-keyed with new IDs, breaking links to invoices; customers who churned during integration were never created in the new system; and a cutover period of weeks was handled in spreadsheets.

Each of these produces missing data that correlates with the outcome. Closed jobs from before the window are missing, so the acquired company's history looks younger and less resolved than it was. Churned customers are absent, so the acquired slice looks more loyal. Inventory them before any analysis.

What did the migration leave out, delay or record selectively?
Evidence itemQuestion to answerWhere to look
PopulationWhich companies, divisions and periods does the target include?Use case owner; deployment plan
Sampling frameWhich records actually exist in the extract, by company and period?Extract counts by origin and month
Company, customer and channel weightsHow do group shares in the extract compare with the target?Coverage comparison table
ExclusionsWhat was filtered out, by rule, and how many records each rule removed?Extraction log
MissingnessWhich records or fields migration skipped, and is the gap related to outcomes?Migration scope documents; legacy system counts
Temporal shiftsWhere do integration milestones fall, and do metrics step at those dates?Quarterly comparisons; project timeline
Outcome maturityHow many recent records are too new to have a final outcome?Outcome lag by job type

Illustrative worked example: a home services company adds a refrigeration contractor

This example is illustrative. A residential HVAC company with six years of field service records acquires a commercial refrigeration contractor eighteen months before a dataset is prepared. The acquired company's jobs were migrated into the parent's field service system with two years of closed history and all open work orders; older closed jobs stayed in its retired system.

The coverage comparison shows the shift at once. Commercial customers are about an eighth of legacy jobs and about seven in ten acquired jobs. Refrigeration service, a category with no legacy equivalent, is the acquired slice's largest job type. Acquired jobs have more steps, partly because commercial sites require check-in procedures and partly because the migration split some single jobs into parent and child work orders. Resolution codes are complete for only about three in five acquired jobs because its old codes were mapped many-to-one and unmapped values were left blank.

Decisions that follow, given a buyer who wants a model for residential and light commercial dispatch: keep the acquired slice, labeled by origin, but report it separately; exclude refrigeration-only job types from the residential evaluation; recover original resolution codes from the migration crosswalk before relying on them; drop jobs closed in the last sixty days from outcome labels, since callbacks are still possible; and state in the dataset documentation that acquired history before the migration window is absent. A model evaluated only on the pooled data would have looked fine on average and been weakest on exactly the commercial jobs the combined company is growing.

When is the acquisition shift the thing you want rather than a flaw?

When the target population is the post-acquisition business. If the model will serve the combined company, the acquired slice is the most representative part of the data, and the legacy history is what needs down-weighting. The mistake to avoid is the reverse: treating the newer records as noise because they are fewer.

Two other exceptions are common. If the acquired company was never integrated and still runs its own systems and procedures, there is no shift inside one dataset; there are two datasets, and they should be documented and split as two sources. And if a buyer wants variety, for instance to test whether a model generalizes across companies, the acquisition is a ready-made second domain, valuable precisely because it differs. In both cases the origin label still has to be present, or nobody can tell which situation they are in.

What should be disclosed to a data buyer, and when does a statistician need to look?

Disclose the origin labels, the coverage comparison, the migration gaps and any code crosswalks with the dataset itself. A buyer cannot correct for a shift it cannot see, and describing the data as one company's history when it is two is a material misdescription. Where a model built on the data falls under the EU AI Act's rules for high-risk systems, Article 10(3) requires training, validation and testing data to be relevant, sufficiently representative and, as far as possible, free of errors and complete for the intended purpose, which makes this documentation part of the compliance file; counsel decides whether and how that applies.

Bring in a statistician or the business's operations lead when the shift is large on a dimension the model depends on, when missingness looks related to outcomes, when the target population is itself uncertain, or when the decision is between reweighting and collecting more data. These are judgment calls about the business and the use, and the coverage table is the evidence they need, not the answer.

Licensing records from a company that grew by acquisition

Companies built through acquisitions often hold unusually broad operating histories, which is part of their value for AI training, provided the origin of each record is kept. Before licensing, the selling company should confirm it has rights over the acquired company's records, which may depend on the deal structure and the acquired company's own customer terms.

Acquisitive companies that license operating records through SourceX get help with rights review, contracting, preparation and delivery. A buyer who wants origin labels and a pre and post close coverage table attached can say so on the request form, alongside scope, volume, format and timing; such datasets are sourced from U.S. businesses for each request rather than held ready.

Frequently asked questions

Does an acquisition always cause distribution shift in a company's data?

Not always, but it usually changes something: the mix of customers or job types, the codes used, or how records are kept. If the acquired company ran very similar work in a similar way, the shift may be small. The coverage comparison is how you find out instead of assuming.

Should acquired-company records be removed from a training set?

Only if they fall outside the population the model will serve. If the model is for the combined business, the acquired records are often the most representative part of the data. Keep them labeled by origin so they can be weighted, stratified or evaluated separately.

Is this the same as data drift in production monitoring?

It uses the same tests, but the question is different. Production drift compares live traffic with training data over time. An acquisition comparison looks inside a historical dataset for a known change point and the populations on each side of it, before any model is trained.

What label matters most when preparing data from an acquisitive company?

The originating company and source system on every record, kept through migration and de-identification. Without it, neither the seller nor the buyer can measure the shift, correct for it, or split evaluation by origin.

How long after an acquisition does the data settle?

There is no fixed period. Look at quarterly comparisons of the acquired slice on process and recording dimensions; when steps per job, field completion and code use stop moving, integration effects have likely settled. Outcome fields lag further, by the time it takes jobs or cases to reach a final result.

Sources

  • Concrete Problems in AI Safety (2016) lists robustness to distributional shift among its core accident risk problems. Source
  • WILDS is a benchmark of in-the-wild distribution shifts whose examples carry metadata such as domain identity, for example which hospital the data came from, with Grouper objects for group-aware loading and evaluation. Source
  • Datasheets for Datasets asks whether a dataset contains all possible instances or a sample of a larger set, what that set is, and whether and how representativeness was validated. Source
  • SciPy's chi2_contingency performs a chi-square test of independence on a contingency table, with expected frequencies computed from the marginal sums. Source
  • SciPy's ks_2samp performs the two-sample Kolmogorov-Smirnov test comparing the distributions of two independent samples. Source
  • Failing Loudly (Rabanser, Gunnemann and Lipton) found two-sample testing on reduced representations performed best for detecting dataset shift and that domain-discriminating approaches help characterize shifts and judge whether they are harmful. Source
  • Article 10(3) of Regulation (EU) 2024/1689 requires training, validation and testing data sets for high-risk AI systems to be relevant, sufficiently representative and, to the best extent possible, free of errors and complete in view of the intended purpose. Source

Tell us what data you need

Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.

See if your company qualifies