Build an internal AI evaluation set from your own records before buying AI tools

Why test AI tools on your own records?#

Testing AI tools on your own records shows how they perform on your product, your customers' language and your codebase, rather than on examples a vendor picked. Public benchmarks and polished demos rarely include your legacy modules, your internal jargon or the odd configuration behind many of your escalations.

An internal set also makes the decision repeatable. When a second vendor arrives, or the first ships a new model, you rerun the same tasks and compare results directly. Without a fixed set, every evaluation becomes a fresh round of impressions from whoever attended the demo.

The five-step method#

The five-step method turns resolved work into a fixed test: define the job, select records with known outcomes, strip and freeze the inputs, write the rubric, then run every tool on identical tasks. Give each step a single owner and a written output.

• 1Step 1, define the job: name the task the tool must do, such as drafting first replies, triaging bugs or reviewing pull requests.
• 2Step 2, select resolved records: choose closed items whose correct outcome is documented and agreed.
• 3Step 3, strip and freeze: remove the answer and later events from each task, de-identify it, and save a fixed snapshot.
• 4Step 4, write the rubric: define pass, partial and fail for each task before any tool runs.
• 5Step 5, run and compare: give each tool identical inputs and context, score without knowing which tool produced which answer, and log failure reasons.

Which records make good evaluation tasks?#

Good evaluation tasks come from records where the work finished and the right answer is written down: a ticket resolved and confirmed by the customer, a bug closed by a merged fix, a review that led to a change. Open, abandoned or disputed items make poor tests because nobody can say what correct looks like.

Mix routine work, where a tool must be reliably right, with a smaller group of difficult cases that show where it breaks. Starting with only the hardest cases makes every tool look weak and tells you little about daily use.

How to keep the answer out of the test#

Keeping the answer out of the test means each task contains only what a person had when the work began. For a ticket, that is the customer's first message and the account context, not the agent's reply. For a bug, it is the report and the code at the commit before the fix.

Leakage is easy to miss. Ticket titles get edited after resolution, bug reports get updated with the root cause, and linked items point straight at the fix. Use the original text from the history, strip later edits and remove links that point forward in time.

Protect the set itself. Run trials under terms that bar the vendor from training on or keeping your inputs, check the tool's data settings, and never upload the answer key. A set a vendor has already seen stops being a fair test of that vendor.

Scoring: what a usable rubric looks like#

A usable rubric defines, before any tool runs, what pass, partial and fail mean for each task type, plus a separate category for answers that would cause harm. Written criteria stop reviewers from grading on tone or confidence instead of correctness.

Record a failure reason for every fail and partial, such as missing context, invented behavior or a wrong product version. Two tools with similar pass rates can fail in very different ways, and the pattern of failures usually says more about fit than the headline score. It also tells you what context a tool would need, such as log access or account data, before it could handle a task type.

Illustrative: twenty closed tickets for a support copilot decision#

Illustrative: a fictional fleet maintenance software company is choosing between two AI support tools. The head of support and the CTO pull twenty closed Zendesk tickets: routine how-to questions, billing setting changes, sync errors linked to Jira bugs, and a few escalations that took several handoffs.

Each task keeps the customer's first message and the product version, with names and account numbers replaced. The rubric is written before either vendor sees a ticket, both tools run under trial terms that bar training on inputs, and two support leads score the answers without knowing which tool wrote them. Where the leads disagree, the disagreement is logged rather than averaged away.

One tool handles how-to questions well; both fail on sync errors that required log access neither had. The company pilots the stronger tool on how-to tickets only, keeps sync errors with human agents, and reruns the same twenty tasks when the vendor releases a new version.

What an evaluation set reveals about your records#

An evaluation set reveals which of your records carry verified outcomes and linked context, the same qualities AI developers look for when they license business records for training and evaluation. If building the set was easy, your records are probably well connected; if every ticket needed detective work to find its answer, that is useful to know too.

Using records internally and licensing them are separate decisions. Licensing needs a rights review, privacy preparation and your approval for each package. SourceX describes what makes records useful with the SourceX Enterprise Data Value Framework and handles licensing through the SourceX five-step transaction: Supply, Rights, Preparation, Approval and Delivery, beginning with a fit check that collects metadata, not files.