AI uses for records
Dataset pilot human time savings: time the reviewers, not the model
By SourceX Editorial · Updated
10 sections
Short answer
Dataset pilot human time savings are the reviewer minutes a dataset change removes per accepted output: less checking, fewer corrections, fewer outputs redone by hand. Observe them in a blinded crossover in which the same reviewers handle current and dataset-improved outputs on matched tasks while tools log active time and edits. Do not infer them from model speed; OpenAI says GDPval's 100x speed figure leaves out human oversight.
Agent run
Dataset pilot human time
Key takeaways
- Measure reviewer minutes per accepted output, not per output, so rejected drafts and hand-made replacements count against the arm that caused them.
- Split the minutes into checking, correcting, redoing and escalating; a better dataset usually cuts the last three more than checking.
- Model speed excludes oversight and self-reports can point the wrong way, as METR's 2025 developer trial showed, so log time in the tools.
- Use a blinded, counterbalanced crossover with the same reviewers, an unchanged acceptance rubric and a second-reviewer audit.
- Field studies show savings vary by task and by worker experience; report each stratum before averaging.
What counts as human time saved by a dataset improvement?
It is the reduction in reviewer minutes per accepted output once the model has been trained or grounded with the new dataset, compared with the same model without it, on the same tasks and under the same acceptance standard. The unit matters: minutes per accepted output, not per output, because a fast draft that gets rejected still costs the minutes spent checking it and the minutes spent doing the work by hand.
That definition makes the decision concrete. A buyer is not asking whether the new model answers faster, but whether the people who stand behind its answers spend less time reaching work they will sign. If the dataset raises accuracy but reviewers still check every output just as carefully, the saving is smaller than the accuracy gain suggests; if it cuts the share of outputs redone by hand, the saving can be larger.
Evidence that supports a purchase has four parts: observed active time for each component of review, the size of edits on accepted outputs, rejection and escalation rates, and an interval from a paired design. Everything below is built to produce those four.
Why model speed and self-reports both overstate the saving
Neither the model's speed nor people's impressions measure human time, and both have been shown to mislead. The saving lives in the reviewer's workflow, so it has to be observed there.
Model speed leaves out the human. OpenAI reported in 2025 that frontier models completed its GDPval tasks roughly 100 times faster and 100 times cheaper than industry experts, and said in the same announcement that those figures reflect pure model inference time and API billing rates, so they do not capture the human oversight, iteration and integration that real work needs.
Impressions can point the wrong way. In METR's randomized trial of early-2025 AI tools, 16 experienced open-source developers working on 246 tasks expected AI to cut their completion time by 24 percent and afterward believed it had cut it by 20 percent; measured completion time was 19 percent longer with AI allowed. METR changed its study design in February 2026 after more developers declined to take part because they did not want to work without AI, which it said likely biased its newer speedup estimates downward. Both lessons carry over to a dataset pilot: time has to be logged, not recalled, and who agrees to take part is part of the result.
The review ledger: where a reviewer's minutes go
Split every reviewed output into four timed components, because a dataset improvement usually moves some of them and leaves others alone. Checking time rarely falls much, since a reviewer still has to read an output before signing it; correcting and redoing are where a better dataset tends to show up.
Correction effort can be counted as well as timed. Machine translation research has long used edit rate for this: translation edit rate counts the edits a person would need to turn system output into a reference, and its human-targeted variant, HTER, uses a reference made by a person editing that very output, as Snover and colleagues described in 2006. The same idea works for business outputs: save each output before and after the reviewer's edits and count the changes. The ledger is below:
| Component | Clock starts | Clock stops | How to capture it | Often missed |
|---|---|---|---|---|
| Check | Output opens in the review tool | Reviewer accepts, edits or rejects | Active-time log with an idle cutoff | Re-reading after an interruption |
| Correct | First edit to the output | Reviewer accepts the edited version | Time log plus a saved before-and-after diff | Edits made later in another system |
| Redo | Output rejected | Hand-made replacement accepted | Time log for the manual work, linked to the task | Work passed to a colleague |
| Escalate | Reviewer asks for help or a second opinion | Question answered | Ticket or message timestamps for both people | The second person's minutes |
What field studies of AI assistance show, and why they do not transfer directly
Randomized studies of AI assistance report large time effects that vary by task and by worker, which is the main reason a dataset pilot has to stratify. They measured access to a tool, not a change of dataset, so their numbers are context for a pilot, not a forecast of its result.
In a preregistered experiment published in Science in 2023, Noy and Zhang gave half of 453 college-educated professionals access to ChatGPT for occupation-specific writing tasks; average time fell by 40 percent and rated quality rose by 18 percent. Brynjolfsson, Li and Raymond studied 5,179 customer support agents and found that an AI assistant raised issues resolved per hour by 14 percent on average and by 34 percent for novice and low-skilled workers, with minimal effect on the most experienced.
The Boston Consulting Group experiment reported by Dell'Acqua and colleagues shows why averages hide the risk. On tasks inside the AI's capability frontier, consultants with AI finished 12.2 percent more tasks, 25.1 percent faster, with quality rated more than 40 percent higher; on a task outside it, they were 19 percentage points less likely to reach a correct solution. A dataset improvement can produce the same jagged shape: real savings on routine work, slower or riskier review on the hard cases.
So stratify by task difficulty and by reviewer experience before the pilot, and report each cell. An average saving driven by novices on easy tasks says little about the senior reviewers who handle the cases that carry the risk.
The preregistered protocol: hold everything fixed except the dataset
Use a blinded crossover: each reviewer handles outputs from both the current model and the dataset-improved model, on matched tasks, in a randomized order, without knowing which model produced which output. Fix every setting below before any output is generated.
Blinding and order are the two settings most often skipped. Reviewers who know an output came from the new model may check it differently, and reviewers speed up over a session as they learn the task, so a fixed order credits the second arm with their learning. Analyze the pairs: Evan Miller's 2024 paper on error bars for evals recommends inference on paired differences and clustered standard errors when questions come in related groups, and in a review study the natural groups are reviewers and task families. The protocol is below:
| Field | Fixed setting | Why it is fixed |
|---|---|---|
| Models | Current and dataset-improved models share base, prompt and tools | Only the dataset may differ |
| Tasks | Real tasks sampled by difficulty stratum and matched across arms | Easy-task samples inflate savings |
| Reviewers | One pool for both arms, stratified by experience, recruited before arms are revealed | Opt-outs and skill mix change the result |
| Blinding and order | Arm labels hidden; order randomized and counterbalanced per reviewer | Expectation and learning look like savings |
| Acceptance standard | The production rubric, unchanged for the whole pilot | A looser bar is a cheaper bar, not a saving |
| Time capture | Tool-logged active time with a preset idle cutoff; no self-reports | Recall is biased |
| Quality audit | A second reviewer re-checks a random share of accepted outputs in both arms | Faster sign-off can mean rubber-stamping |
| Primary metric | Reviewer minutes per accepted output, with a minimum saving set in advance | One number decides; the rest explains it |
Success, stop and inconclusive rules for a time-savings pilot
Fix the minimum saving, the quality floor and the stop triggers before the first session, in the units the business pays for. A saving that depends on a lower acceptance standard is not a saving.
Derive the minimum saving from volume rather than intuition. A saving per accepted output matters only when multiplied by the number of outputs the team accepts in a year and set against the dataset's cost and the work of running a new model. In illustrative terms, 2.5 minutes saved on 40,000 accepted outputs a year is about 1,670 reviewer hours; the same saving on 2,000 outputs is about 83 hours, which may not justify a license at all.
- Success: reviewer minutes per accepted output fall by at least the preset minimum, the paired interval is clear of zero, and the second-reviewer audit finds no more defects in the new arm than in the current one.
- Stop: the audit finds a severe defect in an accepted new-arm output that reviewers missed, or rejection rates rise in any difficulty stratum by more than a preset margin.
- Inconclusive: the interval covers both zero and the minimum saving, or the saving appears only in one reviewer-experience group too small to estimate on its own.
- Void: reviewers learned which arm was which, the rubric changed mid-pilot, time logging failed for more than a preset share of sessions, or the base model changed.
The result table: reading minutes, edits and rejections together
Read the four ledger components and the audit together, because one headline saving can come from very different mechanisms. SourceX has published no pilot results; the table gives the reading and the decision for each pattern.
An illustrative example shows the arithmetic. Suppose 200 matched tasks reviewed by six people give 9.0 reviewer minutes per accepted output in the current arm (3.0 checking, 3.5 correcting, 2.0 redoing, 0.5 escalating) and 6.5 in the dataset arm (3.0, 2.0, 1.0 and 0.5). The 2.5-minute saving, about 28 percent, comes entirely from fewer corrections and fewer outputs redone by hand; checking time did not move. The figures are invented to show the reading and describe no real pilot. The patterns are below:
| Pattern | What it means | Decision | Next evidence |
|---|---|---|---|
| Less correcting and redoing, same checking, clean audit | The dataset fixed real errors | Count the saving; license if it covers the cost | A later task sample to confirm it holds |
| Faster checking, more audit defects | Reviewers trusted the new arm and looked less | No saving; tighten the review step | A rerun with stricter blinding |
| Fewer edits but more rejections | More outputs fail outright | Net effort may rise; compare minutes per accepted output | Rejection reasons by task type |
| Saving only for novices or easy tasks | The gain is jagged | Deploy to that stratum only | A larger sample of hard tasks |
| Saving shrinks across sessions | Early speed was novelty or learning | Use the later sessions as the estimate | A longer pilot window |
| No difference beyond the interval | The dataset does not change review work | Do not buy on time savings | Check accuracy or risk benefits instead |
Could the saving be an artifact of the setup?
Yes, in at least four ways, and each has a check that belongs in the plan rather than in the post-mortem.
- A weak comparison arm. If a better prompt or retrieval over the buyer's own documents would remove the same minutes, the dataset is not the cause. Add the strongest no-new-data configuration as a third arm when the budget allows.
- Learning and novelty. Reviewers speed up as they learn a task and may slow down once novelty fades. Counterbalanced order handles the first; reporting time by session number exposes the second.
- Selective participation. METR's experience shows that who declines a study can bias it. Recruit reviewers before the arms are revealed and report how many declined or dropped out.
- Single-run luck. Outputs vary between trials, which is why Anthropic's guide to agent evaluations runs several per task, and the same guide treats reading transcripts as the way to catch unfair grading. Read a sample of the review sessions with the largest time differences before accepting the average.
What a time-savings pilot cannot settle
It measures review effort on the tasks sampled during the pilot window. It does not capture errors that surface after sign-off, which need a downstream rework window of their own, long-run effects on reviewer skill, or how the saving turns into cost, which depends on whether freed time is redeployed.
It also tests one dataset change against one base model. A new base model can deliver part of the same saving with no new data, so a saving that justified a purchase this quarter should be retested when the model changes. Logging active time is also monitoring of staff: tell participants what is recorded and why, keep the logs to the pilot's purpose, and follow the workplace rules that apply.
Putting reviewer effort into a data request
A request that names the review ledger tells suppliers which records could matter: outputs paired with the corrections reviewers made, rejection reasons and the final accepted version. State the task, the strata, the acceptance rubric and the minimum saving the buyer would pay for.
Datasets reach buyers through SourceX by request, sourced from U.S. businesses rather than drawn from stock; a buyer describes scope, volume, language, format, timing and licensing needs in the request form. Between the two sides SourceX handles the deal itself, qualifying and sourcing supply, checking rights, writing the contract, getting the data ready, delivering it and settling payment, and the supplying company holds on to every record until it has approved scope and terms and signed. Whether a given dataset saves reviewer time is what the pilot tests; nothing here promises that it will.
Frequently asked questions
How do you measure the human time saved by an AI dataset?
Log reviewer active time in the review tool, split into checking, correcting, redoing and escalating, and divide by the number of accepted outputs. Compare the same reviewers on matched tasks with and without the dataset change, blinded and in randomized order. Self-reported time is no substitute: in METR's 2025 trial, developers believed AI had cut their time by 20 percent when measured completion time was 19 percent longer.
Why not estimate time savings from model latency?
Because the model's time is rarely the bottleneck once a person has to stand behind the output. OpenAI's GDPval announcement says its 100x speed figure reflects pure inference time and excludes human oversight, iteration and integration. The reviewer's checking and correcting time is where the saving is won or lost.
How many reviewers does a time-savings pilot need?
Enough that the result does not hinge on one person. Each reviewer's speed is correlated across sessions, so analyze with standard errors clustered by reviewer; a pilot with very few reviewers will have wide intervals however many tasks it covers. Cover each experience level you plan to deploy to.
What is HTER, and why is it useful here?
HTER, human-targeted translation edit rate, counts the edits needed to turn a system output into a version a person produced by correcting that same output. Snover and colleagues described it in 2006 for machine translation. Applied to business outputs, it measures correction effort independently of how fast a reviewer types.
Can a better dataset make review slower?
Yes. If outputs become more fluent but fail more often on hard cases, reviewers can spend longer finding subtle errors or redo more work by hand. The BCG experiment reported by Dell'Acqua and colleagues found AI users 19 percentage points less likely to be correct on a task outside the AI's frontier, which is why the protocol stratifies by difficulty.
Sources
- OpenAI reported that frontier models complete GDPval tasks roughly 100x faster and 100x cheaper than industry experts, and that these figures reflect pure model inference time and API billing rates, not human oversight, iteration and integration. Source
- METR's 2025 randomized trial of 16 experienced open-source developers on 246 tasks found completion took 19 percent longer with AI allowed, although developers had forecast a 24 percent reduction and afterward estimated a 20 percent reduction. Source
- METR said in February 2026 that more developers were choosing not to participate because they did not wish to work without AI, which likely biased its estimate of AI-assisted speedup downward, and that it was changing its study design. Source
- Snover and colleagues (AMTA 2006) described translation edit rate, the edits needed to change system output into a reference, and a human-targeted variant (HTER) that uses a reference made by editing that output. Source
- Noy and Zhang (Science, 2023) found in a preregistered experiment with 453 college-educated professionals that ChatGPT access cut average time on writing tasks by 40 percent and raised output quality by 18 percent. Source
- Brynjolfsson, Li and Raymond found, using data from 5,179 customer support agents, that an AI assistant raised issues resolved per hour by 14 percent on average and by 34 percent for novice and low-skilled workers. Source
- Dell'Acqua and colleagues (HBS Working Paper 24-013) found consultants using AI completed 12.2 percent more tasks, 25.1 percent faster, with over 40 percent higher quality inside the AI frontier, but were 19 percentage points less likely to be correct on a task outside it. Source
- Evan Miller (2024) recommends inference on paired differences when comparing two models on the same questions and clustered standard errors when questions come in related groups. Source
- Anthropic's guide to agent evaluations runs multiple trials per task because outputs vary between runs and says reading transcripts is how to find unfair grading or ambiguous tasks. Source
Tell us what data you need
Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.