AI uses for records
Seasonal business dataset evaluation: grade by conditions, not months
By SourceX Editorial · Updated
10 sections
Short answer
Seasonal business dataset evaluation should report results by the operating conditions that make work harder, such as disruption, load, case mix and temporary rules, and treat the calendar month as a residual check. In a public test on 2013 New York flight records, a calendar-aware model stayed within 5.5 points of the true late rate every month, yet missed it by 24 points in bad-weather hours.
Evaluation split
Seasonal business dataset
Key takeaways
- A month label averages calm and disrupted days together, so a monthly score can look stable while a recurring condition breaks the model.
- In 159,698 held-out 2013 New York departures, adverse-weather hours were 7.8% of flights but 15.5% of late arrivals, and the model underpredicted lateness there by 23.7 points.
- Harder can mean a higher base rate, a calibration break or weaker ranking; bad weather produced the first two while the model's ranking ability rose.
- Origin-airport conditions left most of the month-to-month gap unexplained, so keep the month as a residual stratum and hunt for the conditions it stands in for.
- Report every seasonal comparison with per-condition counts, base rates, calibration gaps and day-clustered intervals instead of one blended score.
How should you evaluate a business dataset when the workload is seasonal?
Evaluate it by operating condition first and by calendar period second. A seasonal score change has three possible causes, the model, the mix of work or the conditions under which the work was done, and only a breakdown by condition can tell them apart.
The decision this supports is narrow. When a model trained or tested on business records scores worse in a peak period, should the team retrain, resample or simply report the period separately? A useful answer needs four numbers for each condition stratum: how many records it holds, the base rate of the outcome, the gap between predicted and actual rates, and how well the model ranks cases. If those numbers hold steady across conditions, the month was noise. If one stratum breaks, the month was hiding it.
Calendar periods still matter, because some conditions are tied to dates by policy or law. What changes is the order of work: name the conditions, measure them in the records, and only then ask what is left for the calendar to explain.
Why does a month label make a poor proxy for difficulty?
A month is a container for conditions, not a condition. It mixes calm days with disrupted ones, and the dates that drive business load move from one year to the next.
Official statistics handle the moving part explicitly. In the Census Bureau's X-13ARIMA-SEATS seasonal adjustment program, the monthly Easter effect can land in March or April depending on the year, and the reference manual says holidays outside its built-in set, such as Chinese New Year or Ramadan, need user-defined regression variables. A retailer whose returns surge after a holiday that drifts between months will see that surge land in a different calendar bucket each year.
Machine learning research has a name for the failure that follows. Oakden-Rayner and colleagues (CHIL 2020) showed that a model can score well overall while failing on subsets nobody labeled, with relative performance differences above 20% on clinically important subsets, and argued that checking for this hidden stratification belongs in any deployment. Seasonal business data carries the same risk: the hard stratum comes back every year, but it rarely has a column of its own.
A public test: one model, 327,346 flights, two ways to slice the year
To see what a month-level report hides, we ran a reproducible test on nycflights13, a public dataset of all 336,776 flights that left JFK, LaGuardia and Newark in 2013, with hourly weather for each airport. Airline operations stand in for a business queue here: seasonal volume, a measurable outcome and conditions recorded next to the work.
The task was to predict whether a completed flight would arrive late, using the Bureau of Transportation Statistics threshold of 15 minutes or more after schedule. The model, a logistic regression, saw only schedule fields, the month included: carrier, origin, destination, departure hour, weekday, month and distance. It was fit on the 167,146 completed flights from odd days of each month and scored on the 160,200 from even days, so every test day was unseen. We then sliced the test half two ways, by calendar month and by whether the origin reported precipitation or visibility under 3 miles in the scheduled departure hour.
By month, the model looked healthy. Its predicted late share stayed within 5.5 points of the actual share in every month, from 4.96 points too high in February to 5.48 points too low in March. By condition, it did not. In adverse-weather hours the actual late share was 48.1% against a predicted 24.4%, a 23.7-point miss with a 95% interval of 17.1 to 30.5 points when whole days are resampled. Those hours held 7.8% of test flights but 15.5% of late arrivals.
The table puts the two views side by side; condition rows cover the 159,698 test flights with a weather match:
| Slice of the test half | Flights | Late share | Model's predicted share | Ranking (AUC) |
|---|---|---|---|---|
| All months, all hours | 160,200 | 24.3% | 24.4% | 0.672 |
| Best month (September) | 13,475 | 13.9% | 13.5% | 0.688 |
| Worst month (December) | 13,114 | 33.2% | 34.0% | 0.605 |
| Clear hours, all months | 147,185 | 22.3% | 24.4% | 0.673 |
| Adverse-weather hours, all months | 12,513 | 48.1% | 24.4% | 0.705 |
Is the season raising the base rate, breaking calibration or blurring the ranking?
Separate the three, because each calls for a different fix. A higher base rate calls for reweighting or a condition feature, a calibration break calls for recalibration by condition, and weaker ranking calls for more informative inputs or more training examples from the hard stratum.
The flight test shows why the split matters. Bad weather roughly doubled the base rate, from 22.3% in clear hours to 48.1%, and broke calibration, yet the model ranked flights slightly better in those hours, with an AUC of 0.705 against 0.673. Winter showed a different pattern: ranking was weakest in January, February and December, at 0.598 to 0.605, and strongest in June and July, at 0.698 and 0.695, while calibration in every month stayed inside the 5.5-point band.
Recalibrating once on the whole year would not repair the weather stratum. Ovadia and colleagues (NeurIPS 2019) found that post-hoc calibration that works when test data look like training data can fail under even a mild shift in the inputs, so a seasonal evaluation should report calibration per condition, not only per period.
Inside single months, the condition gap was often wider than the whole seasonal swing. Monthly late shares ranged from 13.9% to 33.2%, a 19.3-point spread, yet in 8 of 12 months the gap between adverse and clear hours was larger. Clear and adverse late shares were 23.4% and 32.9% in January, 19.9% and 49.4% in February, 22.0% and 57.7% in March, 27.9% and 62.4% in April, 21.6% and 32.3% in May, 28.3% and 62.5% in June, 29.5% and 67.6% in July, 22.1% and 65.0% in August, 12.7% and 56.1% in September, 15.1% and 8.7% in October, 16.9% and 28.1% in November, and 28.9% and 62.3% in December. October is the exception worth keeping in view: only 462 test flights fell in adverse hours that month, and they were late less often.
How much of the month-to-month gap do measured conditions explain?
Less than you might hope, and that is a finding in its own right. Reweighting each month to the year's mix of weather, departure load and time of day narrowed the spread in monthly late shares only from 19.3 to 17.1 points, so most of the seasonal difference came from something the origin-airport fields did not capture.
The method is direct standardization, the logic the National Center for Health Statistics applies in age adjustment when one population is compared at different points in time: hold the mix fixed and see what difference remains. Here the cells were weather (adverse or clear) by load (peak or off-peak hour) by part of day (before noon, noon to 4:59 p.m., and later).
The missing conditions most likely sit elsewhere in the network. The Bureau of Transportation Statistics sorts delay causes into air carrier, National Aviation System, weather, late-arriving aircraft and security, and files non-extreme weather, heavy traffic volume and air traffic control under the National Aviation System. Weather at the destination, congestion across the system and late inbound aircraft are all invisible to an origin-hour stratum.
Not every intuitive condition mattered, either. In clear weather, flights scheduled in origin-hours with 24 or more departures, the busiest quarter by flights, were late 21.4% of the time against 22.6% in quieter hours. A condition earns its place in the evaluation by moving the numbers in your own records, not by sounding plausible.
So keep the month, but demote it. Report condition strata first, then show how much difference by month survives standardization, and read a large remainder as a pointer to a condition nobody has measured yet.
The seasonal evaluation card: what to fill in before comparing two periods
Fill in one card per evaluation period, with a row for every candidate condition and the evidence that it does or does not change difficulty. Drop a row only after the numbers show it does not matter in your records.
The rows translate the flight test into signals that business systems often record; confirm which of these fields your own systems hold before filling the card in:
| Condition | Signal in business records | How to stratify | Report per stratum |
|---|---|---|---|
| External disruption | Weather, carrier or vendor outages, system incidents logged against dates | Flag records created or handled inside a logged disruption window | Records, base rate, calibration gap, ranking metric |
| Load against capacity | Open backlog at intake, cases per scheduled agent, queue age | Quartiles of load measured when each case arrived | The same four numbers; keep the stratum only if it moves them |
| Case mix | Shares by category, product line, channel and customer segment | Hold category fixed and compare within it | Mix-standardized score beside the raw score |
| Temporary rules | Holiday return windows, deadline extensions, promotional terms with effective dates | Tag each record with the rule regime in force when it was decided | Scores by regime, with regime-only categories marked |
| Moving calendar effects | Holiday dates, trading days, fiscal close, filing deadlines | Days from the nearest event instead of the month name | Score by distance to the event |
| Residual period | Month or week after the conditions above are held fixed | Direct standardization to a reference mix | Raw against standardized spread; a large remainder signals a missing condition |
Change-impact checklist before calling a seasonal dip a model regression
Run these checks in order before anyone retrains; the first one that fails usually explains the dip:
Checklist · 8 items
- Recount records per condition stratum in both periods; a stratum with only a few hundred records, clustered on a handful of days, gives intervals too wide to compare.
- Compare base rates per stratum before comparing scores, since a doubled base rate moves accuracy even when the model has not changed.
- Compute the calibration gap per stratum, predicted minus actual rate, and flag any stratum outside the band seen in calm periods.
- Compare ranking metrics per stratum separately from calibration; in the flight test they moved in opposite directions.
- Resample whole days or weeks when building intervals, because conditions cluster in time and record-level resampling overstates precision.
- Standardize the period comparison to a fixed condition mix and report the raw and the standardized difference together.
- Check whether labels, policies or system fields changed between the periods; if they did, treat the dip as a task change, not a difficulty change.
- Write the conditions, cut points and reference mix into the dataset documentation so the next period is graded the same way.
0 of 8 ticked. Tick each item as you go.
What if the newer season is a different task, not a harder one?
Sometimes it is, and the evaluation should say so instead of averaging it away. Newer records can add noise, reflect a changed process or measure a different outcome, and each of those breaks historical comparison in a way no amount of stratification repairs.
A short test separates the cases. If the outcome definition, the policy that decides it and the fields that record it are unchanged, a condition-driven score change is a difficulty change, and old and new periods can be compared within strata. If any of the three changed, the periods hold different tasks: version the label and the policy, and compare only records decided under the same rules.
The flight test passes the first branch by construction, because the 15-minute threshold did not change during 2013. Business records seldom make it that easy. A holiday return policy or a temporary approval limit can shift both the operating conditions and the correct answer in the same weeks, and only the rule version on each record lets you tell the two apart.
Limits of the flight test, and the evidence that should come next
The test demonstrates a method; it measures no business. It covers one metro area in one year, uses weather at the origin only, fits one simple logistic model, and excludes the 9,430 flights that were cancelled or diverted, which carry no arrival delay. Flights with no arrival delay peaked in February at 5.4% of scheduled departures, so some of the hardest operating days are thin in the labels.
The next evidence has to come from business records: the same stratified report on two or more years of history, with load and disruption fields captured when the work happened. SourceX has not published measurements of seasonal difficulty in licensed business datasets; this article publishes the method and a public replication. The data are tidyverse/nycflights13 at commit df98ef2, and every number above can be recomputed from the definitions given here.
Asking for records that carry their operating conditions
A seasonal evaluation can only be as good as the condition fields delivered with the records. When specifying a dataset, name the conditions the evaluation will stratify by and require the fields that measure them, recorded at the time of work rather than rebuilt afterward.
SourceX does not keep an inventory of datasets: it finds U.S. businesses whose records fit a buyer's request, and the request itself carries the scope, volume, format, language, timing and licensing terms. As a general guide, buyers look for two or more years of consistent history in one system, which also gives an evaluation at least two passes through each season. A request that lists the strata, the minimum records per stratum and the acceptance test lets candidate records be judged against the evaluation they have to support.
Frequently asked questions
Is a seasonal holdout enough to evaluate a model on peak periods?
Not on its own. Holding out a peak month shows how the model did on that month's particular mix of conditions, which will not repeat exactly next year. Pair the holdout with condition strata so you can tell whether the score reflects the peak itself or the disruptions that happened to fall inside it.
Which operating conditions should be recorded with business records?
Start with what changes the work: disruption windows, load at the moment a case arrived, case category and the rule regime in force, all captured at the time of work. Keep the ones that move base rates or scores in your own records. In the public flight test, departure load barely mattered while weather at the origin mattered a great deal.
How many records does each condition stratum need?
Enough that the interval around its score is narrower than the differences you care about. In the flight test, 12,513 adverse-hour flights still produced a 95% interval of 17.1 to 30.5 points around the calibration gap, because bad weather clusters on a limited number of days. Count days or weeks of exposure, not just records.
Should peak-season records be removed to keep an evaluation clean?
Removing them hides the stratum most likely to fail in production. Keep them, report them as their own stratum, and weight them to the share of work they represent when a single summary figure is needed.
Does calibration or ranking matter more in a seasonal evaluation?
It depends on how the output is used. Decisions that use the predicted rate, such as staffing or reserves, depend on calibration per condition; decisions that use the ordering, such as which case to handle first, depend on ranking. Report both, because a season can move them in opposite directions, as bad weather did in the flight test.
Sources
- The nycflights13 dataset holds all 336,776 flights that departed New York's three airports in 2013 with hourly weather per airport; all test results in this article were computed from it. Source
- BTS counts a flight as late when it arrives 15 or more minutes after schedule and files non-extreme weather and heavy traffic under National Aviation System delays. Source
- Models can perform well overall while failing on unlabeled subsets, with relative differences above 20% on important subsets. Source
- Post-hoc calibration that works on data like the training set can fail under even a mild dataset shift. Source
- NCHS uses direct age adjustment to compare one population at two or more points in time. Source
- The Census Bureau's X-13ARIMA-SEATS treats Easter as a moving holiday falling in March or April and needs user-defined regressors for holidays such as Ramadan. Source
Tell us what data you need
Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.