Skip to content

AI uses for records

Dataset rare branch utility experiment: how to measure the benefit

By SourceX Editorial · Updated

10 sections

Short answer

To measure the benefit of rare workflow branches, train matched models on equal-size datasets, one with extra common-path cases and one with rare-branch cases instead, then score both on a locked, branch-stratified holdout across several seeds with paired confidence intervals. The benefit is the rare-branch gain net of any common-path loss. This is a how-to-measure protocol; it reports no results.

Evaluation split

Dataset rare branch utility

This page covers dataset rare branch utility experiment: how to measure the benefit.Illustrative

Key takeaways

  • Swap rare-branch examples for common ones at equal volume; adding them on top confounds coverage with quantity.
  • Define branches at decision points, with a written rarity threshold and checked labeling rules.
  • Lock a grouped, branch-stratified test set before building any arm, and score it once per frozen arm.
  • Report per-branch success with paired intervals across seeds, plus common-path and untouched-task guardrails.
  • This page publishes a protocol only; it reports no measured results.

How do you measure the incremental benefit of rare workflow branches?

Run a controlled experiment: train one model on a base set plus extra common-path examples and another on the same base set plus the same number of rare-branch examples, then compare both on a held-out test set stratified by branch, across several random seeds, with paired confidence intervals. The benefit is the per-branch gain on rare branches, net of any loss on common ones. What follows is a method for running that comparison, and no measured outcome appears anywhere on the page.

Holding the budget equal is the key design choice. Adding rare-branch examples on top of a base set confounds two things, more data and different data, so the comparison must swap examples rather than add them. Everything else, from the base model and training recipe to the test set and the scoring code, is fixed in a written plan before the first run.

The protocol suits teams buying or curating workflow data for agents and fine-tuned models who need to know whether paying for coverage of uncommon paths is worth it compared with buying more of what they already have. It does not tell you the answer in advance; every count shown is a planning figure.

Defining a rare branch before you count anything

A branch is one outcome of a decision point in a workflow, and it is rare when its share of cases falls below a threshold you set in the plan. Process mining gives the vocabulary: a variant is the sequence of activities one case followed. The pm4py library, for example, retrieves variants as a mapping from activity sequences to their counts, and offers filters that keep the top k variants or only those above a coverage percentage.

Those filters show why rare branches go missing. A dataset built by keeping the variants that cover most cases drops the tail by construction, so a model trained on it may never see the path where an approval is escalated, a shipment is split, or a refund waits on a carrier claim. Define rare branches at the decision point, not the full variant, or every case with one unusual loop becomes its own branch and the counts fall too low to test.

Write the definition into the plan: the decision points, the branch labels, the source field that decides each label, the frequency threshold and the period of records it was measured on. Two reviewers should label a sample of cases independently; if they disagree often, the branch is not yet well enough defined to experiment on.

The targeted coverage intervention and evaluation plan

The plan below is the asset. Fill in every row and freeze it, with a date and version, before training. Rows left vague are where post hoc choices creep in, and post hoc choices are what turn a protocol into a story.

The targeted coverage intervention and evaluation plan
Plan elementWhat to fix in advanceIllustrative entry
Research questionOne sentence naming the branches, the model and the outcomeDoes swapping common-path examples for rare-branch examples raise task success on rare branches without lowering it on common ones?
Population and periodSource systems, date range, inclusion and exclusion rulesClosed return cases from one order system over a stated period, excluding test accounts
Branch definitionDecision points, labels, deciding field, rarity thresholdCarrier-claim refund, split shipment, manager override; each below a stated share of cases
Unit and splitWhat one example is and how cases are grouped when splittingOne case; split by customer account so no account appears on both sides
ArmsControl, treatment and any extra arm, with exact example countsBase plus common examples; base plus rare examples; base plus reweighted existing rare cases
Fixed settingsBase model, recipe, hyperparameters, seeds, scoring code versionSame checkpoint and recipe for all arms; five seeds each
Primary metricOne metric per branch, with its scorerTask success on the rare-branch test slice, graded against the recorded outcome
Guardrail metricsMetrics that must not get worseCommon-branch success; refusal rate; format errors
Uncertainty methodInterval type, pairing and how seeds are combinedPaired bootstrap interval over test cases, reported per seed and pooled
Decision ruleWhat result counts as a benefit, a harm or no conclusionBenefit only if the rare-branch interval excludes zero and the common-branch guardrail holds

Three training arms with the same example budget

Use at least two arms with identical example counts and, where the budget allows, a third that tests a cheaper fix. The third arm answers a question buyers ask: could reweighting the few rare cases already on hand do the same job as sourcing new ones?

Three training arms with the same example budget
ArmTraining dataWhat it isolatesWatch for
ControlBase set plus extra common-path casesEffect of more data of the usual kindCommon cases drawn from the same accounts as test cases
Rare-branch treatmentBase set plus the same count of rare-branch casesEffect of coverage at equal volumeRare cases that are also longer or cleaner than the control cases
Reweighting armBase set with existing rare cases repeated or upweightedWhether new sourcing beats reuseOverfitting to the handful of repeated cases
Optional dose armBase set plus half the rare-branch countWhether benefit scales with amountToo few rare cases to separate from noise

Test sets the intervention is never allowed to see

Hold out a test set before any arm is built, stratify it by branch, and never let it inform training or the choice of what to source. The scikit-learn documentation describes data leakage as using information at training time that would not be available at prediction time and states the general rule never to fit on test data. In workflow records, leakage usually arrives through the same customer, ticket thread or order appearing on both sides of a split, so group cases before splitting.

Keep three slices: rare-branch cases, common-branch cases and a set of untouched tasks the intervention has nothing to do with. The untouched slice catches regressions that a branch-focused metric misses, such as a model that gets better at carrier claims and worse at simple returns because the treatment arm displaced some common-path examples.

Limit how often the team looks at the test set. Dwork and colleagues showed that reusing a holdout adaptively, deciding the next step from the last score, can overfit to the holdout itself. Tune on a separate validation split and score the test set once per frozen arm.

Per-branch metrics, seeds and paired uncertainty

Report task success per branch, not one blended score, because a small slice can improve or collapse without moving the average. Oakden-Rayner and colleagues describe this as hidden stratification: unlabeled subsets of a class can perform much worse than the whole, and held-out measurements can be falsely reassuring when those subsets are not labeled.

At a glance

Per-branch metrics, seeds and paired uncertainty

Train every arm with several random seeds. Dodge and colleagues fine-tuned BERT hundreds of times on four GLUE tasks varying only the seed and found that weight initialization and data order alone produced substantially different results. A gain smaller than the spread between seeds of one arm is not evidence of anything.

Compare arms on the same test cases. When both models are scored on identical cases, the outcomes are paired, and a paired method is more sensitive than comparing two independent percentages. SciPy's bootstrap function accepts a paired option for resampling matched observations together, and the statsmodels McNemar test checks whether two models' binary outcomes on the same cases differ, using the cases where one succeeds and the other fails.

  • Primary: success rate on each rare-branch slice, per arm, with a paired interval for the treatment minus control difference.
  • Guardrail: success on common branches and on untouched tasks, with the same interval method.
  • Stability: the range of each metric across seeds within an arm.
  • Business outcome: where the recorded outcome allows, whether the model's action would have reached the same resolution as the case did.
  • Counts: the number of test cases behind every rate, printed next to it.

What alternative explanation could account for the apparent improvement?

Assume any improvement has a dull explanation until the design rules it out. Each confounder below has a control you can build into the plan; list them in the pre-registration so a reviewer can check which ones were handled.

What alternative explanation could account for the apparent improvement?
Alternative explanationHow it shows upControl in the design
More tokens, not more coverageRare-branch cases are longer, so the treatment saw more textMatch arms on token count as well as case count
Cleaner recordsRare cases were hand-picked and better documentedApply the same quality filter to both arms; log rejection rates
LeakageTest cases share customers or threads with training casesGroup split by account or thread; check overlap before training
Seed luckOne lucky run drives the differenceSeveral seeds per arm; report the spread
Grader biasScorer favors the phrasing rare-branch examples taughtGrade against recorded outcomes; blind graders to arm
Test set tuningChoices made after looking at test scoresFreeze arms before scoring; tune on validation only
Period shiftRare cases come from a newer period than the test setSample both arms and the test set from the same period

What sample, baseline and uncertainty would make the result defensible?

The number of rare-branch test cases sets the floor on what the experiment can detect, and it is usually the binding constraint, not training volume. Estimate in the plan the smallest difference worth acting on, check whether the rare slice is large enough to detect it with the chosen interval method, and if it is not, either source more test cases or state up front that only large effects can be detected.

At a glance

What sample, baseline and uncertainty would make the result defensible

The baseline is the control arm, not the model before training: the question is whether rare-branch data beats the same amount of ordinary data. Report the reweighting arm against both, since a buyer weighing new sourcing against reuse needs that comparison too. Report every arm and every seed, including runs that failed or were stopped.

Use this pre-registration checklist before the first training run.

  • Research question, branches and decision points written and dated.
  • Population, period, inclusion rules and branch-labeling rules fixed, with reviewer agreement checked on a sample.
  • Test set drawn, grouped by account or thread, stratified by branch and locked.
  • Arms defined with exact case and token counts; quality filters identical across arms.
  • Base model, recipe, seeds and scorer version recorded.
  • Primary metric, guardrails, interval method and decision rule stated.
  • Smallest effect worth acting on stated, with a check that the rare slice can detect it.
  • Deviation policy: any later change is reported with its reason, and exploratory analyses are labeled as such.
  • Statement that the published page will separate protocol from results and report null outcomes.

Illustrative design: a returns workflow with an uncommon carrier-claim path

The following design is illustrative; the counts are planning figures, not results. A team fine-tunes a model to propose the next action on retail return cases. The common path is a return received and refunded. Three branches are rare in the team's records: refunds that wait on a carrier damage claim, returns split across two shipments, and refunds approved by a manager override.

The plan locks a test set of 150 cases per rare branch and 300 common cases, grouped by customer account. The base training set is fixed; the control arm adds 600 common cases, the treatment adds 600 cases drawn evenly from the three rare branches, matched on token count, and the reweighting arm repeats the rare cases already in the base set. Each arm runs with five seeds.

The decision rule says the team pays for more rare-branch records only if the treatment beats the control on at least two of the three rare slices, with paired intervals excluding zero, and common-path success does not fall outside the control's seed range. If the reweighting arm matches the treatment, reuse wins and no sourcing is needed. If the carrier-claim slice cannot reach 150 test cases from the records available, that slice is reported as underpowered rather than dropped.

Finding rare branches in licensed operational records

Rare branches are rare in every single company's records, which is why they are expensive to collect and why a test like this matters before buying. Operational records that keep the decision field, the outcome and the case history together make both the branch labels and the outcome grading possible; records that keep only the final status do not.

Companies holding records like these can license selected ones to AI developers through SourceX, which carries out rights review, contracting, preparation and delivery. On the buying side, AI teams describe the workflows, branches, volume, format and licensing needs they want in a data request, and SourceX sources matching records from U.S. businesses on request. A rough yardstick for supply is two or more years of consistent history and tens of thousands of records or more in one system, with smaller sets still in play when the work they capture is rare.

Frequently asked questions

What should the targeted coverage intervention and evaluation plan include?

It should fix the research question, population and period, branch definitions, unit and split, training arms with equal budgets, fixed settings, primary and guardrail metrics, the uncertainty method and the decision rule. Every element is written and dated before training. The plan also states how deviations will be reported.

Why not just add rare-branch examples to the existing training set?

Adding examples changes two things at once, the amount of data and its coverage, so any gain cannot be attributed to coverage. Swapping the same number of common-path examples for rare-branch examples holds volume constant. Matching token counts as well removes the effect of longer records.

How many rare-branch test cases are enough?

Enough to detect the smallest difference the team would act on, given the interval method and the variation between seeds. That number depends on the baseline success rate and the effect size, so compute it in the plan. If the records cannot supply it, report the slice as underpowered.

Does this page report whether rare-branch data helps?

No. It publishes a method only and reports no results. Any measured outcome would need its own population, period, sample sizes and uncertainty, published separately from this protocol.

Can process mining tools find rare branches automatically?

They can count variants and show which activity sequences are uncommon, which is a good starting point. A person still needs to define branches at the decision points that matter, because full variants split on minor loops and produce counts too small to test.

Sources

  • The pm4py library retrieves process variants from an event log as a mapping from activity sequences to their counts or traces. Source
  • pm4py offers filters that keep the top k variants of a log or only the variants above a minimum coverage percentage, which removes rare variants. Source
  • The scikit-learn documentation defines data leakage as using information unavailable at prediction time and states the rule never to fit on test data. Source
  • SciPy's bootstrap function computes a two-sided bootstrap confidence interval of a statistic and supports paired resampling of matched samples. Source
  • The statsmodels McNemar test compares paired binary outcomes in a 2x2 table, using the cases that are positive in one sample and negative in the other. Source
  • Dodge and colleagues fine-tuned BERT hundreds of times on four GLUE datasets varying only random seeds and found weight initialization and data order caused substantially different results. Source
  • Oakden-Rayner and colleagues describe hidden stratification, where unlabeled subsets perform worse and held-out measurements can be falsely reassuring. Source
  • Dwork and colleagues showed that reusing a holdout set adaptively can lead to overfitting to the holdout set itself. Source

Tell us what data you need

Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.

See if your company qualifies