AI uses for records
What should a reproducible agent evaluation report include?
By SourceX Editorial · Updated
10 sections
Short answer
A reproducible agent evaluation report records enough for another team to rerun the test and land within the stated uncertainty: the versioned task set, the dated model and generation settings, the tool policy including approvals and mocks, seeds and trials per task, harness and grader versions, and results by task family with intervals, coverage gaps and exclusions. A headline success rate alone cannot be checked.
Evaluation split
Agent evaluation
Key takeaways
- Pin what you control, such as the dated model id, prompts and generation settings, and record what you do not, such as backend fingerprints and request ids.
- The tool policy, including mocked tools, limits and approval rules, defines the system under test as much as the model does.
- A seed fixes only the randomness you own; run several trials per task and report all of them, including an all-trials view.
- A coverage matrix of task families against failure types shows what the score says nothing about.
- Intervals should be clustered by task, and the report should state which assumption, often the grader, could flip the decision.
What should a reproducible agent evaluation report include?
It should include everything another team needs to rerun the evaluation and land within the reported uncertainty: the task set and its version, the exact model and generation settings, the tool policy the agent ran under, the seeds and number of trials, the environment and grader versions, and results broken down by task family with intervals and a list of what was left out.
Most agent evaluation write-ups stop at a headline success rate and a model name. That is a claim, not a report. An agent run depends on far more moving parts than a single-turn benchmark: which tools existed, what they returned, whether a human or a policy approved risky calls, how the sandbox was reset between attempts, and how many times each task was tried. Change any of these and the number moves, sometimes more than the gap between two models.
The test of an agent evaluation reproducibility report is simple. Hand it to an engineer who was not in the room, give them access to the same model and task files, and ask them to rerun it. If they have to email you to ask what the timeout was, which version of the refund tool was mocked or whether failed runs were retried, the report is incomplete.
Why can't a score be rerun without its run conditions?
Because hosted models and agent loops are not deterministic by default, and the providers say so. A score produced under unrecorded conditions cannot be told apart from noise when a second team gets a different number.
OpenAI's cookbook on its seed parameter describes the Chat Completions API as non-deterministic by default. Setting a seed makes the service attempt deterministic sampling on a best-effort basis, and the response carries a system_fingerprint that changes when the backend configuration changes. The practical reading for an evaluation report: record the seed you sent and the fingerprint you received on each call, and treat a fingerprint change between two runs as a reason the scores may differ. OpenAI's current API reference still calls seed best effort but now marks system_fingerprint as deprecated, so log it while responses still return it and do not build a rerun check that depends on it alone.
Agent frameworks add their own variation. Tool responses can depend on wall-clock time, rate limits and retries; a sandbox that is not fully reset leaks state from one attempt into the next; a model-graded scorer is itself a model call with its own settings. Stanford's HELM project frames its goal as holistic, reproducible and transparent evaluation for exactly this reason: a published number is only as useful as the record of how it was produced.
Which model settings must be pinned, and which only recorded?
Pin everything you control and record everything you do not. You control the model identifier you request, the generation parameters, the system prompt and the chat template. You do not control the provider's backend build, which is why the fingerprint or equivalent field belongs in the log rather than the configuration.
The distinction matters when a rerun fails to match. If a pinned value differs, the rerun was not the same experiment. If only a recorded value differs, such as a backend fingerprint or a library patch version, you have a candidate explanation to test.
| Item | Pin or record | Why it changes results | Illustrative entry |
|---|---|---|---|
| Model identifier | Pin the dated version, never an alias | Aliases move to newer builds without notice | provider model id with release date suffix |
| Generation parameters | Pin temperature, top_p, max tokens, stop sequences | Sampling settings change both success and variance | temperature 0.2, max output 2,048 tokens |
| System prompt and template | Pin as a hashed file in the repo | Small wording changes shift tool-use behavior | prompts/agent_v7.txt plus its hash |
| Backend fingerprint | Record per call | Provider changes can alter outputs at fixed seed | fingerprint string logged with each response |
| Provider request ids | Record per call | Lets the provider or you trace a specific failing call | request id list kept in the run log |
| Harness and grader versions | Pin package versions, record commit | Scoring code changes move scores without any model change | harness version plus grader commit |
What belongs in the tool policy block?
The tool policy block says exactly what the agent could do and what stopped it: each tool's name, schema version and side effects, whether it was live or mocked, timeouts and retry rules, and which calls needed approval and from whom. Two runs with the same model and different tool policies are different evaluations.
At a glance
What belongs in the tool policy block
Approval is the item teams forget most often. Inspect, the UK AI Security Institute's evaluation framework, supports approval policies in which some tool calls are approved by a human operator and the rest run automatically, and in which any tool call that no approver handles is rejected. An agent evaluated with a human approving every browser call and the same agent evaluated with automatic approval are measuring different systems, and the report has to say which one produced the number.
Record the state the tools started from as well. If the agent works against a ticketing sandbox, the report should name the fixture snapshot and the reset step between attempts, so that the third trial of a task does not see the refund the first trial already issued.
- Tool inventory: name, schema version and a one-line description of side effects (reads, writes, sends, pays).
- Mode: live system, recorded replay or hand-written mock, per tool.
- Limits: per-call timeout, retry count and backoff, maximum tool calls or turns per task.
- Approval: which calls need approval, who or what approves, and what happens to an unapproved call.
- Starting state: fixture or snapshot identifier and the reset procedure between trials.
What can a seed fix, and how many trials do you need?
A seed fixes the randomness you own, such as task order, few-shot selection and local sampling; it does not make a hosted model deterministic. Report the seeds anyway, then run each task several times and report every trial, because the variation across trials is part of the result.
Harnesses differ in what a seed covers, so name the harness's own semantics. EleutherAI's lm-evaluation-harness takes separate seeds for Python, NumPy, Torch and few-shot sampling, with defaults of 0, 1234, 1234 and 1234, and a value of None skips one. Inspect passes a seed to the model only for some providers; its options reference lists OpenAI, Google, Groq, Mistral, Hugging Face and vLLM. A report that says seed 42 without saying which of these it reached is ambiguous.
For agents, repeated trials matter more than any single seed. The tau-bench authors introduced pass^k, the chance that an agent succeeds on all k independent trials of a task, averaged across tasks, and showed that it falls well below the single-trial success rate. Report the number of trials per task, the per-task success counts and both the average and an all-trials view, and never report the best of several runs as the result.
What does a filled-in model, tool policy and seed template look like?
Fill in one row per field for every run you publish internally or hand to a vendor. The rerun check column is what a second team does to confirm the field was enough. Values shown are illustrative placeholders, not recommendations.
At a glance
What does a filled-in model, tool policy and seed template look like
Keep the completed template next to the raw run logs. Inspect's log format, for example, stores the eval specification with task, model and creation time, the plan with solvers and generation config, per-sample inputs, outputs and scores, and reductions across epochs; lm-evaluation-harness can save every model input and output with its log_samples option. The template points to those artifacts rather than restating them.
| Field | What to report | Illustrative value | Rerun check |
|---|---|---|---|
| Task set | Name, version, item count, hash of the task file | AP exceptions set v3, 240 tasks | Hash of local copy matches |
| Model | Dated identifier, provider, generation parameters | dated model id, temperature 0.2 | Same id accepted by the provider |
| Prompt | System prompt file and chat template, with hashes | agent_v7 prompt, provider default template | Hashes match before the run |
| Tool policy | Inventory, mode, limits, approval rule, starting state | 6 tools, 2 mocked, writes need approval | Same tool list loads with same schemas |
| Seeds | Each seed and which component it reached | harness 0,1234,1234,1234; model seed sent | Same seeds logged in rerun |
| Trials | Trials per task and how failures and errors were handled | 5 trials, errors rerun once then scored fail | Same counts per task |
| Grader | Scorer type, grader model and prompt if model-graded, version | state check plus rubric judge v2 | Grader commit matches |
| Environment | Harness version, container image, region, run dates | harness version, image digest, run dates | Image digest matches |
| Results | Per task family: successes, trials, rate, interval, all-trials rate | Refunds 61 of 100 trials | Rerun inside reported interval |
| Exclusions | Tasks dropped, why, and the count | 8 tasks dropped for broken fixtures | Same tasks excluded |
Which business task or failure type is absent from the evaluation?
Ask it of every report, and answer it with a coverage matrix: task families down the side, failure types across the top, and a count of tasks in each cell. Empty cells are findings, not formatting.
A support agent evaluation can score well on refunds and address changes while containing no task where the customer is not the account holder, no task where the right answer is to escalate, and no task where a tool returns an error. Each of these is a real failure type in production, and an aggregate score says nothing about them if the set has none.
Mitchell and colleagues made the parallel argument for models in Model Cards for Model Reporting: quantitative results should be broken down by relevant groups and conditions rather than reported as one average. For agents, the relevant conditions are task families and failure types. Publish the matrix, list the empty cells under Exclusions or Known gaps, and state which of them the buyer of the system cares about most.
The cheapest fix is often to admit the gap rather than fill it badly. A handful of invented tasks in an empty cell gives a misleading rate with a very wide interval; a stated gap tells the reader where not to rely on the result.
Would a different grader or timeout flip the winner, and how wide is the interval?
Show an interval for every rate, cluster it by task when trials repeat, and run at least one sensitivity check on the assumption most likely to flip the decision, usually the grader, the tool timeout or the approval rule. If the decision survives the check, say so; if it does not, the report should say that too.
Inspect's metrics reference shows what a careful report can include without custom code: standard error, bootstrap standard error, confidence intervals, a Wilson interval for binary scores, and clustered standard errors when several scores share a cluster such as a task. Repeated trials of the same task are not independent observations, so an interval that treats 5 trials of 240 tasks as 1,200 independent draws is too narrow.
Illustrative worked scenario: a team compares two agent configurations on the 240-task accounts-payable set, 5 trials each. Configuration A succeeds on 64 percent of trials and configuration B on 59 percent; the task-clustered intervals overlap. Rerunning with the rubric judge replaced by a stricter state check moves A to 57 percent and B to 58 percent. The honest report states that the observed gap depends on the grader, recommends neither configuration on this evidence, and names the stricter grader as the one to use going forward. All figures in this scenario are illustrative.
A rerun came back different: what do you check first?
Work from the cheapest explanation to the most expensive. Most mismatches come from a changed input that the first report did not pin, not from the model itself.
- Compare hashes of the task file, prompt and grader; a mismatch ends the investigation.
- Compare tool policies, including mocked versus live tools, limits and the approval rule.
- Check that the reset step ran between trials and that fixtures match the named snapshot.
- Compare per-call backend fingerprints or equivalent fields; a change points to the provider side.
- Compare per-task success counts, not only the average, to find the tasks that moved.
- Only then treat the gap as model variance, and widen the trial count before drawing a conclusion.
Where do real business tasks for the set come from?
A reproducible report is only as useful as the tasks behind it, and the hardest cells of a coverage matrix to fill are the rare, high-severity ones that come from real operations: disputed invoices, account changes by the wrong person, escalations that should not be automated. Those cases live in business records, and using them for evaluation needs the right to do so, not only access to the systems.
A team short of rare cases can put the empty cells of its coverage matrix into SourceX's request-data form as scope, along with volume, language, format, timing and licensing needs, and SourceX sources matching records from U.S. businesses rather than from a standing catalog. Each supplying company approves the scope and terms and signs before delivery, and that signed agreement is the documented origin to cite in the Task set row of the template.
Frequently asked questions
What is an agent evaluation reproducibility report?
It is the record that lets another team rerun an AI agent evaluation and get a result inside the reported uncertainty. It covers the task set version, the model and generation settings, the tool policy, seeds and trials, the environment and grader, and results by task family with intervals and exclusions.
Does setting a seed make a hosted model deterministic?
No. Providers that accept a seed, such as OpenAI, describe it as a best-effort attempt at deterministic sampling; OpenAI's backend fingerprint field, which signals configuration changes, is now marked deprecated in its API reference. Report the seed, log any fingerprint the response returns and run several trials per task.
How many trials per task should an agent evaluation run?
Enough to see variation per task, which in practice means several trials rather than one. Report per-task success counts, the average rate and an all-trials view such as pass^k, and cluster intervals by task because trials of one task are not independent.
Should the tool policy really be part of the report?
Yes. The same model under a human approval rule, an automatic approval rule or mocked tools is a different system. Record each tool's mode, limits, approval rule and starting state, or a second team cannot rebuild the conditions.
What if the rerun differs but nothing obvious changed?
Compare file hashes, tool policies, sandbox resets and backend fingerprints in that order, then compare per-task results to find which tasks moved. Treat the remaining gap as variance only after those checks, and increase the number of trials before deciding.
Sources
- OpenAI describes its Chat Completions API as non-deterministic by default, the seed parameter as best-effort determinism, and system_fingerprint as the indicator of backend changes. Source
- HELM is an open-source framework from Stanford CRFM for holistic, reproducible and transparent evaluation of foundation models. Source
- Inspect eval logs record the eval spec, plan and generation config, per-sample results, epoch reductions and provider request ids. Source
- Inspect approval policies can route selected tool calls to a human and approve the rest automatically; unhandled calls are rejected. Source
- Inspect's seed option is passed only to some model providers. Source
- Inspect provides standard errors, bootstrap and Wilson intervals, and clustered standard errors for evaluation metrics. Source
- lm-evaluation-harness sets separate Python, NumPy, Torch and few-shot seeds and can log every model input and output. Source
- The tau-bench paper defines pass^k as the probability that an agent succeeds on all k independent trials of a task. Source
- Model Cards for Model Reporting recommends disaggregating quantitative evaluation results by groups and conditions. Source
- OpenAI's current Chat Completions reference describes seed as best effort and marks the system_fingerprint response field as deprecated. Source
Tell us what data you need
Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.