AI uses for records
Enterprise AI evaluation sampling plan: a stratified task worksheet
By SourceX Editorial · Updated
10 sections
Short answer
Sample AI evaluation tasks from a written frame of real work: list every completed case in a recent window from the system of record, group it into strata by task family and business severity, allocate tasks proportionally with minimum floors for high-severity strata, draw at random within each stratum, and keep the weights. Report workload-weighted and per-stratum results with intervals, plus a log of work the frame misses.
Evaluation split
Enterprise AI evaluation
Key takeaways
- Draw evaluation tasks from a deduplicated frame of completed work in the system of record, not from memorable examples.
- Use strata that change a decision, usually task family crossed with business severity, each with a rule a second person can apply.
- Allocate proportionally, raise high-severity strata to a floor, and keep a weight for every task.
- Report workload-weighted and per-stratum results with intervals, and rerun with an alternative window to test the assumptions.
- Log the work the frame misses and keep hand-written cases in a separate, labeled set.
How do you sample AI evaluation tasks to reflect real business workloads?
Start from the work, not from the test cases people remember. Build a sampling frame of every completed unit of work in a recent window, split it into strata that match the decisions you will make, give each stratum a share of the evaluation proportional to its volume, raise the share for strata where a mistake is expensive, and draw tasks at random inside each stratum.
Two outputs make the plan inspectable: a worksheet that shows how many tasks came from each stratum and why, and a weight for every task so results can be reported either as the business experiences them or stratum by stratum. Without the weights, an evaluation that deliberately oversamples rare, high-risk work will overstate how often those cases occur; without the floors, it will barely test them at all.
This page gives the worksheet, the allocation rules, a severity rubric, a worked scenario with illustrative counts and the checks for work the frame misses. It is written for teams evaluating a model or agent against their own operations before buying or deploying it.
Define the workload and the sampling frame before picking any task
Write the target workload as a population statement: the unit of work, the systems it is recorded in, the time window, and what is excluded and why. For a support team the unit might be a resolved ticket; for accounts payable, an invoice that reached a final state; for field service, a closed job.
The frame is the actual list you sample from, so it should come from the system of record rather than from memory or a hand-picked folder. Export it completely and deduplicate it. Zendesk's own guidance on its incremental exports API is a useful warning here: polling windows that overlap can return the same ticket more than once, and results can include records whose updated time precedes the requested start time, so the export needs a deduplication and filtering step before it becomes a frame.
Choose the window to match how the system will be used. Frameworks point the same way: the NIST AI Risk Management Framework asks that performance be demonstrated for conditions similar to the deployment setting, and the EU AI Act requires high-risk systems' training, validation and testing data to take into account the geographical, contextual, behavioral or functional setting in which the system is intended to be used, to the extent the intended purpose requires. A frame from a season, region or product line the system will not serve fails that test even if it is large.
Which strata should an enterprise AI evaluation sampling plan use?
Use strata that change a decision. A stratum earns its place if you would act differently when the model fails there: route that work to people, add a control, or reject the vendor. Task family and business severity almost always qualify; channel, customer segment, language or document type qualify when the workflow genuinely differs between them.
Keep the grid small enough to fill. Every extra dimension multiplies the number of cells, and empty or tiny cells produce results nobody can interpret. A practical design crosses task family with severity tier and holds other dimensions as recorded attributes that can be checked for balance after sampling rather than forced into the allocation.
Write a defining rule for each stratum that a second person can apply to any record without asking you, for example the ticket form, category field and escalation flag that place a case in S4. Ambiguous membership is a sampling error that no amount of statistics later can repair.
Decide in advance what happens to strata that are smaller than their floor. If the frame holds fewer records than the floor, take every record in that stratum (a census of that cell), note it on the worksheet with a weight of one, and record the shortfall in the gap log rather than padding it with invented cases.
The stratified task sampling worksheet
Fill one row per stratum. Every input has a named source, so a reviewer can rebuild the allocation from the frame without trusting the analyst's arithmetic.
| Worksheet field | What to enter | Where it comes from | Check before freezing |
|---|---|---|---|
| Stratum id and name | Short id and plain-language name | Stratum definitions | Unique, stable across reruns |
| Defining rule | Field values or flags that place a record in the stratum | System of record schema | Every frame record falls in exactly one stratum |
| Frame count | Number of records in the window for this stratum | Deduplicated export | Strata counts sum to the frame total |
| Workload share | Frame count divided by frame total | Calculated | Shares sum to 100% |
| Severity tier | Low, medium, high or critical | Severity rubric and task owner | Signed off by the business task owner |
| Floor | Minimum tasks for this stratum | Severity tier rule | Floors sum to less than the total budget |
| Allocated tasks | Final count after floors and rescaling | Allocation rule | Counts sum to the total budget |
| Weight per task | Frame count divided by allocated tasks | Calculated | Weighted counts reproduce the frame total |
| Completion evidence | Which record proves the real outcome of each task | System of record | Present for every sampled task |
| Draw seed and date | Random seed and date of the draw | Sampling script | Draw can be reproduced exactly |
How to allocate tasks: proportional shares, severity floors and weights
Apply the allocation in three passes. First, give each stratum its proportional share of the total budget. Second, raise any stratum below its floor to the floor. Third, take the tasks added by the floors back from the strata above their floors, in proportion to their size, and round so the total is exact.
Then compute each stratum's weight as frame count divided by allocated tasks. Weighted results estimate performance across the workload as it actually arrives; unweighted per-stratum results answer whether the model can be trusted on each kind of work. Report both, because they answer different questions.
The scenario below is illustrative and not drawn from any real company: a support desk with a frame of 48,000 resolved tickets over two quarters, an evaluation budget of 400 tasks and a floor of 40 tasks for high and critical strata. Proportional shares would give S4 and S5 only 32 and 8 tasks; the floors raise both to 40, and the 40 extra tasks come out of S1 to S3 in proportion to their size.
Draw inside each stratum with a recorded random seed, and sort the stratum by date before drawing systematically if you want the sample spread evenly across the window instead of clustered in a busy month. Keep a numbered reserve list per stratum so that replacements, when a task has to be dropped, come from the same random order rather than from someone's choice. Freeze the draw, the reserve list and the worksheet together before any model sees a task.
| Stratum (illustrative) | Frame count | Severity tier | Allocated tasks | Weight per task |
|---|---|---|---|---|
| S1 Billing questions, routine | 19,200 (40%) | Low | 142 | 135.2 |
| S2 Order changes and returns | 14,400 (30%) | Medium | 107 | 134.6 |
| S3 Multi-step technical troubleshooting | 9,600 (20%) | Medium | 71 | 135.2 |
| S4 Account access and security | 3,840 (8%) | High | 40 | 96.0 |
| S5 Regulatory complaints and legal escalations | 960 (2%) | Critical | 40 | 24.0 |
| Total | 48,000 (100%) | Mixed | 400 | Varies by stratum |
How to score business severity before sampling
Severity belongs to the task, not to the model's answer, and it should be fixed before any results exist so nobody can promote a stratum after seeing where the model failed. Have the business task owner assign tiers using consequences they already track.
| Tier | What a wrong result causes | Illustrative support example | Floor rule in this plan |
|---|---|---|---|
| Low | Minor rework, caught in normal review | Wrong answer to a billing date question | Proportional share only |
| Medium | Customer-visible error, correctable within the workflow | Return opened with the wrong reason code | Proportional share only |
| High | Loss of access, money movement or data exposure for one customer | Password reset to an unverified requester | Floor of 40 tasks |
| Critical | Regulatory, legal or safety consequence | Formal complaint closed without required escalation | Floor of 40 tasks and full expert review of every response |
Which business tasks are missing from the frame, and how do you log them?
Every frame misses some work, and the plan should say which. Common gaps are work done outside the system of record (phone calls, side email threads, spreadsheets), task types too new to appear in the window, rare events that happened zero times in it, and steps that a person completes in another tool before closing the record.
Keep a gap log beside the worksheet with three columns: the missing work, how you know it exists, and what the evaluation therefore cannot say. The Holistic Evaluation of Language Models paper argues that an evaluation taxonomy should make explicit what is missing, not only what is covered. GDPval offers a published example of stating coverage: its authors describe the benchmark as covering the majority of U.S. Bureau of Labor Statistics work activities for 44 occupations across the top nine sectors contributing to U.S. GDP, which tells a reader both the scope and its edges.
If a gap covers high-severity work, do not quietly backfill it with hand-written cases inside the random sample. Build a separate, clearly labeled supplementary set and report it apart from the weighted results.
Which sampling choices could flip the conclusion?
Three assumptions move results most: the time window, the stratum definitions and the floor sizes. Rerun the weighted estimate with an alternative window and with any borderline stratum rule changed, and publish the range alongside the main figure. If the conclusion flips, the evaluation is not ready to support a deployment decision.
Show uncertainty for every reported number. Miller's statistical treatment of language model evaluations recommends standard errors on scores, paired comparisons when two systems answer the same questions, and clustered standard errors when questions come in related groups, which business tasks usually do because several tickets can share one customer or incident. A floor of 40 tasks makes a stratum testable, not precise, so per-stratum intervals will be wide and should be shown as such.
Set thresholds before results arrive. The NIST AI RMF Playbook suggests establishing thresholds and alert procedures for dataset representativeness within the context of use; in this plan that means writing down, in advance, the minimum per-stratum success and the maximum share of critical-tier failures that would stop a rollout.
What completion evidence must each sampled task carry?
A sampled task is only useful if its real outcome is known from the records, so the model's answer can be scored against what actually resolved the case. For each task, store the starting state, the inputs a person had, the final state in the system of record and the field or event that proves completion, such as a closed status with a resolution code, a posted credit memo or a passed inspection.
Drop and replace tasks whose outcome cannot be verified, using the next random draw from the same stratum, and count the replacements per stratum. A stratum where many draws lack completion evidence is a finding in itself: it says the workflow is poorly recorded there, which often matters more than the model's score.
Where outcomes were later reversed, such as a refund approved and then clawed back or a ticket reopened within days, record the final state and keep the reversal as an attribute. Those cases are often the ones a reviewer most wants to see, and dropping them makes the workload look cleaner than it was.
Sourcing real business tasks when your own frame falls short
Some teams have the model but not the workload, or have it for only part of the strata they care about. In that case the worksheet doubles as a data specification: the stratum rules, frame window, floors and completion evidence describe exactly what records an outside supplier would need to provide.
Where your own frame falls short, SourceX can be asked to find U.S. businesses whose records fill the missing strata; it holds no stock, and its request-data form covers scope, volume, language, format, timing and licensing needs, fields a completed sampling worksheet already answers.
Frequently asked questions
What is a sampling frame for an AI evaluation?
It is the complete, deduplicated list of real work units, such as resolved tickets or closed jobs, from which evaluation tasks are drawn. It is defined by the system of record, a time window and written exclusions. If the frame is wrong, no allocation rule can make the sample representative.
Why not just sample tasks proportionally to volume?
Proportional sampling mirrors the workload but gives rare, high-severity work very few tasks, sometimes none. Floors fix that, and weights then let you report workload-level results without the oversampling distorting them.
How many tasks should a high-severity stratum get?
Enough that a failure there is likely to show up and that the stratum's interval is informative for your decision. The 40-task floor in this page is illustrative; size real floors with the business task owner and an evaluation statistician against the failure rate you need to detect.
Should hand-written test cases go into the sample?
Not into the random sample. Keep hand-written or synthetic cases in a clearly labeled supplementary set and report them separately, so the weighted results still describe real work.
How often should the sampling plan be redrawn?
Redraw when the workload mix, task families or success criteria change, such as a new product line, a policy change or a new channel. Keep the old draw and its seed so results remain comparable over time.
Sources
- Zendesk's guide to its incremental exports API notes that overlapping polling windows can return the same ticket more than once and that results can include records updated before the requested start time. Source
- The NIST AI Risk Management Framework (AI RMF 1.0) MEASURE 2.3 calls for AI system performance to be measured and demonstrated for conditions similar to the deployment setting, with measures documented. Source
- The NIST AI RMF Playbook suggests establishing thresholds and alert procedures for dataset representativeness within the context of use. Source
- Article 10(4) of the EU AI Act requires training, validation and testing data sets for high-risk AI systems to take into account, to the extent required by the intended purpose, the specific geographical, contextual, behavioral or functional setting of use. Source
- The HELM paper argues that holistic evaluation should provide a top-down taxonomy and make explicit the major scenarios and metrics that are missing. Source
- GDPval covers the majority of U.S. Bureau of Labor Statistics work activities for 44 occupations across the top nine sectors contributing to U.S. GDP. Source
- Miller (2024) treats language model evaluations as experiments and gives methods for standard errors, paired model comparisons and clustered standard errors. Source
Tell us what data you need
Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.