Skip to content

Systems and records

How to oversample rare cases and keep real-world frequency weights

By SourceX Editorial · Updated

10 sections

Short answer

Draw rare cases at a higher, recorded inclusion probability and ship the sampling details with each record: the stratum, the probability, a design weight equal to its inverse, and the full archive count for every stratum. Those dataset exception oversampling weights let a team train on an enriched sample, then reweight, correct priors or recalibrate to real-world frequency, and evaluate on the mix the model will meet in production.

System export

Dataset exception

This page covers how to oversample rare cases and keep real-world frequency weights.Illustrative

Key takeaways

  • Oversampling is safe only when each record keeps its stratum, inclusion probability and design weight.
  • The archive count for every stratum, including ones you barely sampled, is what makes reweighting possible later.
  • Class weights and balanced metrics are modeling choices; design weights reconstruct the real population, and the two should be stored separately.
  • Keep at least one evaluation at the natural frequency and never resample test data.
  • Weights cannot recover cases the archive never captured, so disclose channels, units and periods that are missing.

How do you oversample rare cases without misrepresenting real-world frequency?

Select rare cases at a higher, written-down inclusion probability, and ship that probability with every record. Each record carries its stratum, the chance it had of being selected and a design weight equal to the inverse of that chance, and the dataset carries the full count of records in each stratum of the source archive. Anyone can then train on the enriched sample and still reweight back to the frequency the business actually saw.

This is the same bookkeeping survey statisticians have used for decades. A 1997 Census Bureau working paper on the American Community Survey describes a base weight as the inverse of the probability that an address was selected. A training dataset that oversamples exceptions is a stratified sample with unequal selection probabilities, and it needs the same record of how each case got in.

Without that record, oversampling quietly rewrites reality. A model trained on a set where half the invoices are exceptions learns that exceptions are common, an evaluation on the same set reports numbers that will not hold in production, and nobody downstream can undo it because the original frequencies were never written down.

Pin down the deployment population before choosing what to enrich

Start by naming the population the model will serve, then compare it with the archive you can actually sample. They are rarely the same. The deployment population might be every supplier invoice a payables team receives next year; the archive might hold only invoices keyed into NetSuite, missing those settled by card, those handled by a subsidiary on another ERP and those purged under a retention policy.

Datasheets for Datasets, the documentation framework proposed by Gebru and colleagues, asks dataset creators to state whether the data is a sample of a larger set, what that larger set is and how the sample was drawn. Answer those questions in writing before any oversampling decision, because the strata you choose only make sense relative to a defined frame.

Check four things about the frame: which periods it covers, which channels feed it, which business units or acquired companies are missing, and how long outcomes take to settle. An exception that is still open has no final resolution, so a recent month can look cleaner than it will be once disputes close.

Then choose strata that match decisions the model must make, not whatever categories the source system happens to offer. In a payables archive that might mean clean matched invoices, price or quantity variances, suspected duplicates and changes to vendor bank details. Each stratum needs a rule that assigns every record in the frame to exactly one group, written so a second analyst would assign the same record the same way. Overlapping or vague strata make inclusion probabilities impossible to state.

What dataset exception oversampling weights must record per case

Every record in an enriched dataset needs the fields below. The example values are illustrative and describe no real dataset. Keep them as plain columns rather than burying them in a readme, so that weighting survives every filter and join.

What dataset exception oversampling weights must record per case
FieldDefinitionIllustrative valueWhy it matters
stratum_idThe selection group the record was drawn fromprice_variance_exceptionWeights are only meaningful within a defined stratum
inclusion_probabilityChance this record had of being selected from its stratum0.2Lets anyone recompute the design weight
design_weightInverse of the inclusion probability5How many archive records this one stands for
selection_methodRandom within stratum, all cases taken, or rule-based pickrandom_within_stratumRule-based picks cannot be reweighted honestly
duplicate_groupShared ID when a case appears more than once because of sampling with replacementdup_0007Stops copies from leaking across train and test
source_entityTokenized business unit, customer or channel the record came frombu_03 / channel_emailReveals concentration in one entity
event_dateDate the case occurred2025-03-14Supports checks for temporal shift
outcome_matureWhether the final resolution is known, and the date it settledtrue / 2025-05-02Separates open cases from resolved ones
exclusion_reasonBlank, or why a record in the frame was left outblankMakes exclusions countable rather than invisible

The dataset-level disclosure that travels with the records

Record-level fields tell you how each case got in; the disclosure tells you what the whole sample can and cannot stand for. Write it as a short card delivered with every version of the dataset.

At a glance

The dataset-level disclosure that travels with the records

  • Population: who or what the model will be used on, in one sentence.
  • Sampling frame: the systems, periods, channels and business units the records were drawn from, and the ones that were not.
  • Strata: each stratum's definition, the count in the frame, the count sampled and the inclusion probability.
  • Exclusions: the count removed for each reason, such as privacy, legal hold, corrupt files or immature outcomes.
  • Missingness: fields that are often blank, and whether blanks cluster in particular strata or periods.
  • Concentration: the share of records from the largest few business units, customers or channels, before and after weighting.
  • Temporal profile: counts by quarter in the frame and in the sample, and any known process change in the period.
  • Reconstruction recipe: how to compute population-frequency estimates from the design weights, with one worked check that recovers the frame counts.

Worked example: supplier invoice exceptions (illustrative)

The counts below are invented to show the arithmetic. A payables archive holds 432,400 invoices for a two-year period. Clean invoices dominate, while the cases a model most needs to learn from are rare.

  • Unweighted, clean invoices are 8,000 of 16,400 sampled records, about 49 percent of the dataset.
  • Weighted, they are 8,000 times 50, or 400,000 of 432,400, about 92.5 percent, which matches the archive.
  • Suspected duplicates are about 12 percent of the sample but under half a percent of the archive; the disclosure has to make both numbers visible.
  • The check that weighted counts reproduce frame counts exactly is the first acceptance test a buyer should run.
  • A model trained on the unweighted sample will predict exceptions far more often than they occur; any rate it reports needs reweighting or recalibration before a finance team reads it as a forecast.
Worked example: supplier invoice exceptions (illustrative)
StratumRecords in frameRecords sampledInclusion probabilityDesign weight
Clean, matched invoices400,0008,0000.0250
Price or quantity variance exceptions30,0006,0000.25
Suspected duplicate invoices2,0002,0001.01
Vendor bank-detail change before payment4004001.01
Total432,40016,400Varies by stratumVaries by stratum

Training on enriched data: duplicates, class weights and prior correction

You have three common ways to use the enrichment in training, and each needs the frame counts to stay honest. The simplest is to keep the sample as delivered and let the model see more rare cases. The imbalanced-learn library's RandomOverSampler goes further by sampling minority records with replacement, which duplicates real records; if you do that, the duplicate group field keeps copies of one case on the same side of every split.

The second is to weight the loss. scikit-learn's balanced class weight sets each class's weight to the number of samples divided by the number of classes times that class's count. That is a modeling choice for balance, not a reconstruction of real frequency, so record which weights were used and keep the design weights separately.

The third is to correct the model's output afterwards. King and Zeng describe prior correction for logistic regression fitted on case-control samples: fit as usual, then adjust the intercept using the fraction of events in the population and the fraction in the sample. It needs the population fraction from an outside source, which is exactly what the frame counts in the disclosure provide. Whatever you choose, check calibration on data at the real mix before anyone reads a predicted probability as a rate.

Testing on the mix the model will meet in production

Keep at least one evaluation set at the natural frequency, or weight the enriched test set back to it, and never resample test data. The imbalanced-learn documentation warns that resampling the whole dataset before splitting means the model is not tested on a distribution similar to the real use case, and that resampling can leak information from future test samples into training.

Report three views and label each. Weighted results estimate what production will look like. Per-stratum results show whether the rare cases are handled. Unweighted pooled results only describe the sample and should not be compared across dataset versions.

Be careful with metrics that reweight on their own. scikit-learn's balanced accuracy weights each sample by the inverse prevalence of its true class, which is useful when classes matter equally but is not the same as your design weights. State which kind of weighting a reported number uses.

Which cases are underrepresented relative to the people the model will serve?

Compare the weighted sample with the deployment population along every axis that changes behavior: business unit, customer segment, channel, region, product line and period. Oversampling by outcome fixes rarity of the event; it does nothing for a channel that never reached the archive.

Survey practice offers a disclosure standard worth copying. OMB's 2006 Standards and Guidelines for Statistical Surveys call for documentation that describes the sample weights, including adjustments for nonresponse, and how to apply them, along with weighted and unweighted response rates. The training-data equivalent is a table of coverage by entity and channel, weighted and unweighted, with a note on every known gap.

Cap the share any single customer or business unit can contribute, and disclose the cap. When one large account generates most of the exceptions, an uncapped sample teaches that account's quirks as if they were the industry's.

When real-world frequency is the wrong target, and other limits

Sometimes the deployment population is not the archive's population. A triage model that only ever sees invoices already flagged by a rules engine should be weighted to the flagged mix, not to all invoices. A process change that will double a case type next year may justify weighting to an expected mix instead of the historical one. In both cases, write down the target distribution and why it was chosen.

Weights also have limits. They correct for cases you chose not to sample; they cannot recover cases the archive never captured, such as disputes settled by phone with no record. Very unequal weights make weighted estimates less precise, so heavily enriched strata need enough records of their own to report on. Rule-based selection, such as taking only the exceptions an analyst found interesting, has no inclusion probability and cannot be reweighted at all.

Request a statistician's review when the reconstruction check fails, when exclusions remove a large share of any stratum, or when outcomes in recent periods are still maturing.

Specifying an enriched exception sample from a supplier

When the rare cases come from another company's records, write the template into the request: strata, target counts, the record-level fields above and the disclosure card. Ask for frame counts even for strata you do not want, because they are what make reweighting possible later. The supplier's right to license the records, privacy permission and the de-identification plan are separate checks from the sampling design.

Enriched samples like this are a natural fit for SourceX's model of finding records at U.S. businesses on request rather than selling from stock: the buyer's request form carries scope, volume, language, format, timing and licensing needs, and the strata and frame counts can be written into it.

Frequently asked questions

Does oversampling rare cases bias a model?

It shifts what the model sees as normal, which biases predicted probabilities toward the rare class unless you correct for it. The fix is to keep inclusion probabilities and frame counts so you can reweight, apply prior correction or recalibrate. The bias is a problem only when the sampling design is lost.

What is the difference between a design weight and a class weight?

A design weight is the inverse of the probability that a record was selected, and it reconstructs the source population. A class weight is a training choice that changes how much each class counts in the loss. They can be combined, but they answer different questions and should be stored separately.

Should the test set be oversampled too?

Keep at least one test set at the natural frequency, or weight an enriched test set back to it. Oversampled test sets are useful for per-stratum results on rare cases, but their pooled scores do not predict production. Never resample test data with the same procedure used for training.

How do I check a supplier's weights are right?

Sum the design weights within each stratum; the totals should match the frame counts in the disclosure. Then compare weighted shares by period, channel and business unit with any independent counts you hold. Mismatches usually point to undisclosed exclusions.

Can synthetic records replace oversampled real cases?

Synthetic records generated by interpolation have no inclusion probability, so they cannot be weighted back to the population. Keep them in a separately labeled slice if you use them. Real rare cases with known selection probabilities remain the reference for evaluation.

Sources

  • The American Community Survey base weight is the inverse of the probability that an address was selected for the sample. Source
  • Datasheets for Datasets proposes documenting a dataset's motivation, composition, collection process and recommended uses, including whether it is a sample and how it was drawn. Source
  • RandomOverSampler generates new minority samples by randomly sampling the available samples with replacement. Source
  • With class_weight set to balanced, scikit-learn sets weights to n_samples divided by n_classes times each class count. Source
  • Prior correction for case-control samples subtracts a term built from the population and sample event fractions from the logit constant, and needs the population fraction from outside information. Source
  • Resampling the whole dataset before splitting means the model is not tested on the natural imbalanced distribution and can leak test information. Source
  • Balanced accuracy is raw accuracy with each sample weighted by the inverse prevalence of its true class. Source
  • OMB survey standards call for documentation describing sample weights, including nonresponse adjustments and how to apply them, and weighted and unweighted response rates. Source

Tell us what data you need

Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.

See if your company qualifies