Skip to content

Systems and records

Dataset exceptional year coverage: include, exclude or label a crisis year?

By SourceX Editorial · Updated

8 sections

Short answer

Keep a crisis year in the archive, label it as its own period, and decide per use case: hold it out of training for models meant for normal operations, keep it with a label where patterns persisted, and train on it where the model must handle disruption. Date the period by month from evidence, test whether each attribute reverted or shifted, check what it under-recorded, and always report a separate crisis evaluation slice.

System export

Dataset exceptional year

This page covers dataset exceptional year coverage: include, exclude or label a crisis year.Illustrative

Key takeaways

  • Never delete an exceptional period silently; label it as a regime and decide its role per use case.
  • Date the period by months from external and internal evidence, because crises rarely match calendar years.
  • Decide for each attribute whether the crisis was a reverting shock or a lasting shift; that choice sets the treatment.
  • Inspect what the period under-recorded, such as closed sites, deferred work and blank fields, before judging what it shows.
  • Validate with time-ordered splits and report a separate crisis slice alongside the main result.

How should a dataset treat a crisis year with unusual business behavior?

Keep the crisis period in the archive, label it as its own regime, and decide per use case whether a model trains on it, trains on it with a marker, or only sees it in a separate evaluation slice. Never delete it silently. The right choice depends on the population the model will serve: if deployment looks like normal operations, the crisis months distort the baseline; if deployment includes disruption, they are the most valuable records in the set.

Exceptional periods are not only pandemics. A supplier failure, a regional storm season, a system migration, a recall or a price war can each change who contacted the business, what they asked for and how staff resolved it. The same decision applies to all of them: define the period, characterize what changed, then choose a treatment that matches the task and write the choice down where every downstream user will see it.

Three treatments cover nearly every case. Include means the period trains the model like any other. Include with a label means the records carry a regime field, and the model or the analyst can use it, weight it or condition on it. Exclude from training means the records stay in the delivery but out of the training split, usually reserved as a stress-test slice.

Dating the exceptional period by months, not by calendar year

Date the exceptional period from evidence at monthly or weekly resolution, then trim the label to fit the business, because crises rarely respect calendar years. Use an external reference for the shock and internal data for its local shape.

For the 2020 U.S. downturn, the NBER Business Cycle Dating Committee placed the peak of monthly economic activity in February 2020 and the trough in April 2020, a two-month recession it called the shortest on record. That dates the macro shock. It does not date your business's shock, which may have started earlier with supply problems or lasted years longer in a sector that stayed disrupted.

Inside the archive, look for change points in record volume per week, mix of request types, channel shares, average handling time, cancellation and no-show rates, and the share of records with key fields left blank. Record both the external reference and the internal evidence for the start and end dates you choose, so a reviewer can disagree with the boundary and see why it was drawn there.

Shock that reverted or a new normal: the question that sets the treatment

Before choosing a treatment, decide whether the period was a temporary shock that reverted or a level shift that became the new normal. That single judgment changes most downstream choices.

Forecasters have formal names for the two shapes. Hyndman and Athanasopoulos describe a spike variable, equal to one only in the affected period, for effects that last a short time, and a step variable, equal to zero before an intervention and one from then on, for a permanent change in level. The U.S. Census Bureau used a related idea when it seasonally adjusted services data in early 2020: it changed its additive outlier treatment so that the pandemic quarter's unusual movements would not distort the estimated seasonal factors.

The lesson transfers directly to training data. A shock that reverted should be isolated so it does not teach the model a pattern that no longer exists. A shift that persisted, such as a permanent move of customer contact from phone to chat, should not be excluded, because the post-crisis world is built on it. Many crises are both: volume reverted while channel mix did not. Label each affected attribute, not just the dates.

The period selection decision guide, by use case

Use the table to choose a default treatment, then look for the evidence in the third column that would overturn it. The defaults are reasoned starting points, not measured results.

Treatments can be combined inside one delivery. The same records can sit in the training split for a triage model and in a held-out slice for a pricing model, as long as the regime label and the split assignment per use are both recorded. Down-weighting the period is a middle path when a team cannot choose, but it hides the decision inside a number; if you weight, publish the weight and the reason, and still report the crisis slice separately so the effect of the weight can be seen.

The period selection decision guide, by use case
Use caseDefault treatmentEvidence that changes the defaultDisclose with the dataset
Demand or workload forecasting for normal operationsInclude with a label; model the period with spike or step termsPost-crisis volume never returned to the old level, so the step applies, not the spikePeriod dates, chosen shape and the evidence for it
Classifying requests, tickets or claims into categoriesInclude with a labelCrisis-only categories or temporary codes that were later retired or remappedCategory list in force per period and any remapping
Triage or routing under disruption and backlogInclude and oversample the crisis periodPolicies used then are now prohibited or no longer operationally possibleWhich rules applied during the period
Pricing, discount or credit decisionsExclude from the baseline fit; keep as a labeled sliceTemporary policies became permanent termsPolicy start and end dates and affected segments
Anomaly or fraud detectionInclude with a labelCrisis-era behavior patterns were rare and reverted, which would make them noisy inliersRate of flagged events by period
Agent or assistant training on resolved workInclude if resolutions were correct under rules still validWorkarounds that violated or bypassed standard procedureWhich resolution paths were crisis exceptions
Headline evaluation set for normal operationsExclude from the main test; report a separate crisis sliceDeployment population explicitly includes disruptionSlice sizes and how they were drawn

Which cases does the crisis year itself under-record?

An exceptional year is often thin exactly where it matters, so inspect what is missing before judging what is present. Ask which cases the archive lost or delayed and whether those cases resemble the people the model will serve.

Common gaps include closed locations that stopped generating records entirely, deferred work that was booked late or never booked, outcomes that were still open when the export was taken, and channels that changed so requests arrived through systems the export does not include. Staffing shortages tend to leave optional fields blank, which can look like a change in customer behavior when it is a change in recording behavior.

Compare each period's case mix against the deployment population, segment by segment. The EU AI Act frames this well for high-risk systems: Article 10 asks that training, validation and testing data take into account, to the extent the intended purpose requires, the characteristics of the specific geographical, contextual, behavioural or functional setting where the system will be used. Even outside the Act's scope, that is the right test. The question is never whether the crisis year was normal; it is whether its records match the setting the model is meant for.

Worked case (illustrative): a field service archive that spans 2019 to 2023

This case is illustrative; the business, dates and figures are invented to show the reasoning. A commercial HVAC service company offers five years of dispatch records, 2019 through 2023, to train a model that predicts job duration and recommends technician skill level.

Starting state: weekly job counts fall sharply from late March 2020 and recover by the third quarter of 2021. Internal evidence shows three changes. Planned maintenance visits dropped to roughly a third of their usual share (illustrative), emergency calls held steady, and a new job type for ventilation upgrades appeared in mid-2020 and stayed. The completion-notes field is blank on about twice the usual share of jobs during the period (illustrative).

Linked evidence: the dispatch system's job type table shows the ventilation job code created in June 2020; HR records show the technician headcount fell for eleven months; work orders show many maintenance visits rescheduled rather than cancelled. Rule applied: duration prediction for normal operations, so the shock in maintenance volume is treated as a spike, the new job type as a step, and blank completion notes as a recording defect.

Decision: the March 2020 to September 2021 window gets a regime label. Its maintenance jobs stay in training with the label; its ventilation jobs stay without restriction because the work continues; jobs with blank notes are excluded from the duration-label subset only. The crisis window becomes a separate evaluation slice. Outcome check: a time-ordered evaluation on 2023 jobs is compared with and without the labeled window, and the slice result is reported separately.

Counterexample: a sister model that recommends staffing levels during surges reverses the decision. For that task the crisis window is the core training signal, and excluding it would leave the model with no examples of the situation it exists to handle.

Testing the choice with time-ordered splits and a separate crisis slice

Validate the treatment with splits that respect time, then report the crisis slice on its own. Random splits mix pre- and post-crisis records into both train and test and hide the question you are trying to answer.

Scikit-learn's TimeSeriesSplit documents the reason: standard cross-validation is inappropriate for time-ordered data because it would train on the future and evaluate on the past, so each split trains on earlier folds and tests on the next one. Run the comparison you care about inside that structure, for example a model trained with the crisis window included, one with it labeled and one with it excluded, all tested on the most recent period.

Hold the evaluation period fixed across all three variants so differences come from the treatment, not from a different test window. If the archive ends soon after the crisis, the most recent period may still carry its effects; in that case say so and treat the result as provisional until a cleaner period accumulates.

Expect the gap to be real. The Wild-Time benchmark, built from datasets with natural shifts over time, found a large gap between in-distribution and out-of-distribution performance under temporal shift that existing robustness methods did not close. That is a reason to measure your own gap rather than assume one treatment generalizes. Report the test period, the slice sizes and the variance across folds alongside any comparison.

What a supplier should disclose, and a buyer should ask, about an exceptional year

The period treatment is only useful if it travels with the data. Datasheets for Datasets, the documentation framework from Gebru and colleagues, already asks creators to describe composition and the collection process; an exceptional period belongs in both answers.

Companies that license operational records to AI developers can expect this question, and buyers should ask it whenever a delivery spans a disrupted year. For suppliers working with SourceX, which helps businesses turn selected records into licensed datasets and runs the rights review, preparation and delivery around them, the period label, its evidence and its known gaps are worth preparing once so they ship with every release. Buyers generally look for years of consistent history in one system, and each dataset is assessed individually.

  • Start and end dates of each exceptional period and the external and internal evidence used to draw them
  • Whether each affected attribute behaved as a reverting shock or a lasting shift
  • Record counts per period and segment, so gaps are visible as numbers
  • Fields whose completeness or meaning changed during the period, including temporary codes
  • Policies, prices or procedures in force only during the period
  • Open outcomes at export time and when they are expected to mature
  • The treatment recommended for each intended use and the evaluation that supports it

Frequently asked questions

Should I remove 2020 from my training data?

Usually not. Keep the records, label the affected months as their own period and decide per use case: exclude them from a baseline that should reflect normal operations, keep them for tasks that must handle disruption, and always report a separate evaluation slice. Deleting them throws away evidence about the conditions models most often fail under.

How do I know whether a crisis caused a temporary shock or a permanent shift?

Compare the attribute's level before the crisis, during it and well after it. If the post-crisis level returned near the old one, treat the period as a reverting shock; if it settled somewhere new, treat it as a step change from the date it began. Judge each attribute separately, because volume can revert while channel mix or job types do not.

Is labeling a crisis period enough to protect a model?

A label lets models and analysts condition on the period, but it does not fix missing cases or recording defects inside it. Check what the period under-recorded, such as closed locations or deferred work, and evaluate on a time-ordered split with a separate crisis slice to see whether the label actually helps.

Who underrepresented during an exceptional year matters most?

The segments the model will serve after deployment. Compare the period's mix of customers, channels, request types and locations against the expected deployment population and look for segments that disappeared or were recorded late. A segment absent during the crisis can be fully present again once the model is in use.

What should a data buyer ask about a disrupted year before licensing?

Ask for the period dates and the evidence behind them, record counts by period and segment, any temporary codes or policies, which fields changed completeness, and the supplier's recommended treatment for the buyer's use. Those answers decide whether the year helps or hurts the intended model.

Sources

  • The NBER Business Cycle Dating Committee determined a trough in monthly U.S. economic activity in April 2020, following the February 2020 peak, making the recession two months long and the shortest on record. Source
  • Hyndman and Athanasopoulos describe spike variables for effects lasting one period and step variables that are zero before an intervention and one afterward for permanent level changes. Source
  • The U.S. Census Bureau changed its additive outlier treatment in X-13ARIMA-SEATS for first-quarter 2020 services estimates so that pandemic effects would not distort the seasonal factors. Source
  • Article 10(4) of the EU AI Act requires that data sets for high-risk systems take into account, to the extent required by the intended purpose, the characteristics of the specific geographical, contextual, behavioural or functional setting of use. Source
  • Scikit-learn documents TimeSeriesSplit as a cross-validator for time-ordered data where standard methods would train on future data and evaluate on past data. Source
  • The Wild-Time benchmark reports a large gap between in-distribution and out-of-distribution performance under real temporal distribution shift that existing methods do not close. Source
  • Datasheets for Datasets proposes documentation questions organized by lifecycle stage, including motivation, composition and the collection process. Source

Tell us what data you need

Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.

See if your company qualifies