AI uses for records
Why AI evaluations moved to real business tasks
By SourceX Editorial · Updated
Short answer
AI evaluations moved to real business tasks because public benchmarks saturate, leak into training data and test narrow formats that say little about paid work. A real-world task evaluation needs three parts: the original inputs, the instruction a professional received and a result an expert judged. Business records that keep all three linked are the raw material.
Key takeaways
- Public benchmarks lose value once leading models score near the ceiling or once test items appear in training data.
- A real-world task evaluation pairs actual work inputs and instructions with an outcome a qualified person accepted or corrected.
- Private task sets resist contamination because they never appeared on the public web, as long as the license keeps them out of training.
- Records that link a request, the work product and a review or outcome convert into evaluation tasks with the least effort.
- Evaluation-only use can be licensed on different terms from training use, so the permitted use should be stated explicitly.
What changed in how AI models are evaluated?#
AI evaluation has shifted from academic question sets toward tasks drawn from real occupations, where a model must produce work a professional would recognize and an expert judges the result. Model developers and the companies deploying their tools both want to know whether a system can do the job, not whether it can pick the right letter on a quiz.
Several public evaluations published in 2024 and 2025 made the shift visible. GDPval, released by OpenAI on September 25, 2025, covers 44 occupations from the nine industries that each contribute more than 5% of US GDP. Its task writers needed at least four years of professional experience and averaged 14 years, and each task is tied to a real work product such as a legal brief, an engineering blueprint or a customer support conversation.
The headline results are launch-time snapshots that change as models improve. What matters for a business owner is the design: every one of these evaluations starts from realistic inputs and judges a finished piece of work, which is exactly what operating records contain.
| Evaluation | Published | What the tasks are | What it reported at launch |
|---|---|---|---|
| GDPval | September 2025, OpenAI | 1,320 tasks across 44 occupations, tied to real work products and reference files | A 220-task gold subset was open-sourced for others to use |
| Remote Labor Index | October 2025, Scale AI and the Center for AI Safety | Real paid freelance projects; the median project took a professional about 11.5 hours | The highest-performing agent achieved an automation rate of 2.5% |
| TheAgentCompany | NeurIPS 2025 | Tasks inside a simulated small software company with internal websites and data | The most competitive agent completed 30% of tasks autonomously |
| OSWorld | NeurIPS 2024 | 369 computer tasks in real web and desktop applications | Humans completed 72.36% of tasks; the best model completed 12.24% |
Why public benchmarks stopped being enough#
Public benchmarks stopped being enough for a few connected reasons: saturation, contamination, narrow formats and missing context. Each one weakens what a high score can tell a buyer about real work.
None of this makes benchmarks useless. Developers still use them to catch regressions and compare versions. The shift is about what counts as convincing evidence that a model or an agent is ready for a specific kind of work.
| Problem | What happens | Why real tasks help |
|---|---|---|
| Saturation | Leading models score near the ceiling, so the test no longer separates strong systems from weaker ones | Real work has open-ended outputs with room to be better or worse |
| Contamination | Questions and answers circulate online and end up in training data, inflating scores | Private task sets drawn from company records were never public |
| Narrow format | Multiple choice or short answers miss planning, tool use and judgment | Real tasks require several steps, files and decisions |
| Missing context | Questions arrive self-contained, unlike work that depends on history and policy | Business records carry the context a professional actually used |
| No accountable grader | Answers are matched to a key rather than reviewed by someone who does the job | Expert review or a recorded outcome judges quality |
- 01Your work records
- 02Personal details removed
- 03You approve the release
- 04Licensed to AI developers
What a real business task evaluation contains#
A real business task evaluation contains the inputs a worker actually had, the instruction they were given and a way to judge the result. Without all three, a task is either impossible to attempt or impossible to grade.
The grading basis is usually the hardest part to create from scratch and the easiest to find in a mature company. Code reviews, QA checklists, approval comments, customer acceptance and reopened tickets are all records of a qualified person judging work.
- Inputs: the files, messages, records and system state available at the start, such as a customer email with an attached purchase order.
- Instruction: what the worker was asked to do, in the words of the request, the ticket or the SOP.
- Context: policies, prior history and constraints a competent professional would consult.
- Reference result: the work product that was accepted, such as the shipped fix, the sent reply or the approved estimate.
- Grading basis: a rubric, a reviewer's comments or a recorded outcome showing whether the result worked.
Which business records convert into evaluation tasks#
Business records convert into evaluation tasks when they link a request to a work product and to a judgment of that work. The table shows typical examples across the kinds of companies SourceX focuses on, using records the company itself owns rather than client deliverables.
Notice that the judgment signal often arrives later than the work. A callback, a reopened ticket or a recurrence after a corrective action is the record that says whether the original answer was right, so archives that keep that later history are worth more than snapshots.
| Company type | Task input | Reference result | Judgment signal |
|---|---|---|---|
| B2B software | Bug report or support ticket with logs | Merged fix and release note | Code review comments; ticket resolved or reopened |
| Engineering or architecture firm | Contractor RFI referencing drawings | Issued response or revised sheet | Principal review; follow-up RFIs |
| Consulting firm | Internal proposal request and scoping notes | Submitted proposal and staffing plan | Partner review notes; proposal won or lost |
| Home services contractor | Service call notes and equipment history | Diagnosis, estimate and completed job | Callback or warranty claim, or none |
| Distributor or 3PL | Order exception or carrier delay notice | Resolution steps and customer message | Exception closed, credit issued or dispute |
| Manufacturer | Nonconformance report with inspection data | Disposition and corrective action | CAPA verification; recurrence or none |
Why private task sets matter to model developers#
Private task sets matter to model developers because they are far less exposed to contamination. A developer that evaluates on tasks drawn from a company's internal records can be much more confident the model has not seen those exact items during training, provided the license keeps them out of training pipelines, so the score carries more weight.
Private sets also cover work that public data barely touches. Little of the public web shows how a controller resolves a price variance, how a dispatcher reorders a day after a no-show, or how an engineer answers a contractor's RFI. Those are exactly the tasks enterprise agents are being built to do.
For the supplier, evaluation use can carry a different risk profile from training use. Evaluation sets can be held under tighter access and kept apart from training pipelines by contract, but those terms are negotiated deal by deal and belong in the license rather than in assumptions.
Illustrative: a vertical software company scopes an evaluation-only license#
Illustrative: a fictional company that sells scheduling software to commercial cleaning contractors keeps its Zendesk history connected to Jira issues, GitHub pull requests and release notes going back many product versions. Its CEO is open to licensing but uneasy about customer conversations ending up in a training corpus.
The team scopes a narrower package: resolved bug reports where the ticket, the engineering discussion, the merged fix and the reviewer's comments all link together. Customer names, email addresses and account identifiers are removed in preparation, and the proposed license limits use to evaluation, excludes training on the tasks and requires deletion at the end of the term.
The CEO approves the package because the permitted use is narrow and documented. The engineering team also keeps its own copy of the prepared task set as a regression suite for the AI features in its product.
What owners should do about the shift#
Owners should treat the shift as a reason to keep work records linked and judged, not as a reason to rush into a deal. The same habits that make records useful for outside evaluation also make them useful for testing your own automation.
- Keep the link between request, work product and review, such as ticket to issue to pull request, or RFI to response to approval.
- Preserve reviewer comments and rejection reasons, not only final approvals.
- Record outcomes after delivery: reopened tickets, callbacks, credits, warranty claims and corrective actions.
- Protect history during system migrations, because exports often break the links between records.
- Decide in principle whether you would allow evaluation use, training use or both before any buyer asks.
How SourceX approaches evaluation-set requests#
SourceX treats an evaluation request as a licensing transaction with its own scope and permitted use. In the SourceX five-step transaction, Supply identifies which linked record families could form tasks, Rights checks customer contracts and vendor terms, Preparation removes personal and confidential details, Approval puts the final package in front of the company's signer, and Delivery follows only after that sign-off.
For an evaluation package, the permitted-use entry in the SourceX Evidence Packet carries most of the weight: it says in plain words whether training is excluded, next to the package's provenance, licensing rights, privacy record and release authorization.
Frequently asked questions
Is evaluation data worth less than training data?
Not necessarily. Value depends on the drivers in the SourceX Enterprise Data Value Framework, such as uniqueness, domain expertise, human-generated signal and AI utility, more than on the intended use. A well-judged set of tasks can matter to a developer precisely because it is hard to build any other way.
Does an evaluation set need a large volume of tasks?
It usually needs fewer items than a training set, but each task must be complete and gradable. The buyer specifies what it needs. Linkage and a clear judgment signal matter more than raw count, and incomplete tasks are typically dropped during preparation.
Who grades the model on our tasks?
The buyer runs the evaluation. Grading may use your recorded outcome, a rubric derived from your review criteria, or the buyer's own expert reviewers. As the supplier, your role is to provide complete tasks and accurate reference results, not to score anyone's model.
Can a developer keep our tasks after the evaluation?
That depends on the license. Terms can limit retention, require deletion at the end of the term, restrict access to named teams and prohibit training on the tasks. Agree these points before delivery and record them in the permitted-use section of the agreement.
What if some of our recorded outcomes were wrong?
Imperfect outcomes are normal in real work, and later corrections such as a reopened ticket or a callback are useful signals in their own right. Preparation can flag or exclude tasks whose reference result was later reversed, so an evaluation does not reward a mistake.
Sources
- OpenAI's GDPval benchmark, released September 25, 2025, covers 44 occupations from the nine industries that each contribute more than 5% of U.S. GDP, with 1,320 tasks in the full set and a 220-task gold subset that is open-sourced. Source
- GDPval task-writing experts needed at least 4 years of professional experience and averaged 14 years; tasks are tied to real work products such as a legal brief, an engineering blueprint or a customer support conversation, and can include reference files. Source
- Scale AI and the Center for AI Safety's Remote Labor Index (October 2025), built from real paid freelance projects, found that the highest-performing AI agent achieved an automation rate of 2.5%. Source
- Each Remote Labor Index project takes a human professional a median of about 11.5 hours. Source
- TheAgentCompany benchmark (NeurIPS 2025) mimics a small software company with internal websites and data; with the most competitive agent, 30% of its tasks could be completed autonomously. Source
- OSWorld (NeurIPS 2024) has 369 computer tasks in real web and desktop apps; at launch humans completed 72.36% of tasks while the best model achieved 12.24%. Source
Related resources
See if your company qualifies
A short company assessment. No data uploads are needed.