Leadership and readiness
Agent benchmark environment reproducibility: an 89-task audit for investors
By SourceX Editorial · Reviewed by Noah Loul ·
10 sections
Short answer
Agent benchmark environment reproducibility rests on five pinned items: the task commit, the start image, resource limits, network policy and the grader. Our audit of Terminal-Bench 2.0's 89 task files found dated image tags but no digests, internet open in every task, graders that download packages while grading, and five tasks whose limits or tests changed under the same 2.0 label. Ask for the run configuration before crediting a gain.
Agent run
Agent benchmark environment
Key takeaways
- A benchmark score is a joint result of model, harness and environment; without the environment record, a gain cannot be attributed to data.
- Terminal-Bench 2.0 names every image with a dated tag and pins 88 of 89 graders' Python dependencies exactly, but uses no digests and allows internet in all 89 tasks.
- Between the commit Harbor's registry pins for 2.0 and the April 2026 main branch, five tasks changed memory, time limits or a grader test.
- Anthropic measured a 6-point Terminal-Bench 2.0 swing from resource settings alone, so limits and enforcement belong in every comparison.
- A data claim needs a paired ablation on a frozen environment in which only the data changes; anything less is a configuration claim.
Can an investor credit an agent score without the environment behind it?
No. An agent benchmark result is the joint output of a model, a harness and an environment, so a score is only reproducible when the starting image, the resource limits, the network policy, the reset rule and the grader are pinned to a version someone else can run. Without that record a diligence team cannot tell whether a reported gain came from new data, a new model, a bigger budget or a different box.
The decision this article serves is narrow: whether a benchmark gain cited in a pitch, a data room or a model card should count as evidence in an investment memo. Useful evidence has three parts. The environment is specified tightly enough to rebuild, a rerun lands within a stated tolerance of the claim, and any claim that data caused the gain comes from an ablation in which only the data changed.
To show what that record looks like in practice, we audited the public task files of Terminal-Bench 2.0, a terminal-agent benchmark that its maintainers say gave each task several hours of human and model-assisted validation. It is a careful benchmark, which makes it a fair test: the gaps it leaves are the ones a less careful benchmark will also have.
What environment evidence means: start state, tools, resets and the final check
Environment evidence is everything outside the model that decides whether a run passes. Harbor, the framework Terminal-Bench 2.0 now runs on, defines a task as one or more instructions, a sandbox environment and a verifier that produces the reward, and a trial as one agent's attempt at one task. Each of those parts can change a score without the model changing at all.
Five layers cover it. The start state is the container image or snapshot the agent wakes up in. Tools and resources are the software installed in it plus the CPU, memory, disk and time it gets. The network policy says what the agent and the grader can reach outside the box. The reset rule says how each trial starts and whether anything carries over from the last one. The final-state check is the grader: which files or states it inspects, which dependencies it needs, and what counts as a pass.
A sixth item ties them together: the version pin, meaning the exact commit of the task set that produced the number. Harbor's environment documentation says most providers prefer a prebuilt image named in the task file over building one from scratch, citing speed and reproducibility. That preference is sound, and it moves the question to how the image is named.
What Terminal-Bench 2.0's 89 task files pin, and what they leave open
Terminal-Bench 2.0 pins a great deal, but not everything. At commit 2fd12b8 (April 29, 2026), every one of the 89 task files names a prebuilt image with a dated tag, every base image in the 90 FROM lines of the Dockerfiles carries a version tag rather than latest, and 88 of 89 graders pin every Python test dependency to an exact version, with the remaining one pinning a library to a git commit hash.
Three gaps remain. No image is named by digest. Docker's own build guidance explains why that matters: a version tag can point to a newer build months later, while a digest reference keeps the same image even if the publisher replaces the tag. Every task sets allow_internet to true, so the agent can reach the open network during the task. And all 89 graders fetch software while grading: 82 run apt-get install with no version pins, mostly to add curl, and the others install pinned Python packages from the network.
None of this means scores are wrong. It means a rerun depends on outside servers and on registry tags staying put, which a diligence team should confirm rather than assume. Harbor's current verifier documentation points the same way, telling task authors to install test dependencies in a dedicated tests/Dockerfile or in the agent image. The figure plots each task as one dot per property; the counting rules are in the methods note.
| Layer | What the 89 task files state (commit 2fd12b8) | Count | What stays open |
|---|---|---|---|
| Start image | Prebuilt image with a dated tag such as :20251031 | 89 tagged, 0 by digest | Whether the tag still points to the same build |
| Base images | Version tags such as ubuntu:24.04 and python:3.13-slim-bookworm | 90 of 90 FROM lines | Patch-level changes if the image is rebuilt |
| Agent network | allow_internet = true | 89 of 89 | What the agent fetched or looked up during the run |
| Resources | One value each for CPUs, memory and disk, no separate ceiling | 84 tasks at 1 CPU; 69 at 2,048 MB | How the runner enforced those values |
| Grader dependencies | apt-get install at grading time, no version pins | 82 of 89 | Mirror availability and package drift on rerun |
| Pass rule | pytest exit code written as reward 1 or 0 | 89 of 89; median 3 test functions | Whether the tests can be satisfied without doing the task |
| Leakage marker | Canary GUID string in the Dockerfile | 88 of 89 | Whether the canary was actually checked in training data |
Five tasks changed while the label stayed 2.0
Version labels are not version pins. Harbor's registry.json, which lists each dataset's tasks by repository and commit, pins [email protected] to commit 69671fb, dated October 31, 2025. The repository's main branch had moved to 2fd12b8 by April 29, 2026. Comparing the two, five tasks changed in ways that can move a score: three got more memory, one got a longer agent time limit, two got longer grader time limits, and one grader test was edited.
Everything else held still. No Dockerfile, environment file or instruction changed, the prebuilt image tag stayed the same in all 89 tasks, and the remaining differences were a reformatted task-file schema, added README files and two edited reference solutions, which the grader does not use. That is good maintenance practice, and the registry pin means a careful team can still run the October 2025 version exactly.
The changes matter because resources are a known score lever. In a post published February 5, 2026, Anthropic reported that across six resource configurations on Terminal-Bench 2.0, the gap between the most- and least-resourced setups was 6 percentage points (p < 0.01), and that infrastructure error rates fell from 5.8% under strict enforcement to 0.5% uncapped. Two runs that both say 2.0 can therefore differ in the exact variable that study found moves results. The ledger below lists every scoring-relevant change:
| Task | Field | Registry pin 69671fb (Oct 31, 2025) | Main 2fd12b8 (Apr 29, 2026) |
|---|---|---|---|
| crack-7z-hash | Memory | 2,048 MB | 4,096 MB |
| crack-7z-hash | Agent time limit | 900 s | 1,800 s |
| filter-js-from-html | Memory | 2,048 MB | 4,096 MB |
| filter-js-from-html | Grader time limit | 900 s | 1,800 s |
| gpt2-codegolf | Memory | 4,096 MB | 8,192 MB |
| query-optimize | Grader time limit | 900 s | 1,800 s |
| install-windows-3.11 | Grader test file | Reads one QEMU process ID | Takes the first of several process IDs |
Was the gain the data, or the model, harness and budget?
Not necessarily the data. The strongest alternative explanation for any benchmark gain is that the model snapshot, the prompt and scaffold, or the inference budget changed at the same time as the training data, and public evidence shows each of those can move scores on its own.
Budget and scaffold first. The Holistic Agent Leaderboard paper (October 2025) ran 21,730 agent rollouts across 9 models and 9 benchmarks and reported that higher reasoning effort reduced accuracy in the majority of runs, so more inference spend is not even a reliable direction of change. Infrastructure second: Anthropic's resource study held the model, harness and task set fixed and still moved Terminal-Bench 2.0 by 6 points, and it notes anecdotally that pass rates can drift with time of day as API latency changes.
Then the grader itself. Bercovich and colleagues (April 2026) ran more than 40,000 adversarial trials over 1,860 tasks drawn from five public benchmarks and confirmed 331 environments whose verifiers an agent could satisfy without doing the intended work. A frozen environment makes a score repeatable; it does not make it valid.
The test that separates data from everything else is an ablation on a frozen environment: same task commit, same image digests, same resource enforcement, same network policy, same model snapshot, same harness and budget, enough trials to measure spread, and only the training data changed between arms. A company that cannot show that design has a configuration claim, not a data claim.
The benchmark underwriting worksheet
Use the worksheet to decide whether a cited benchmark result enters the memo as evidence, as a lead to verify, or not at all. Each row names what to request, what Terminal-Bench 2.0's public record shows for that item, and how to read the answer.
Score it simply. A result with the first six rows documented and an exploit review on file can be weighed as evidence. A result missing the version pin, the resource enforcement or the run conditions is a lead to verify through a rerun. A result missing three or more rows should not support a valuation argument. The thresholds are editorial judgment, stated so a reader can change them.
| Evidence item | What to request | Terminal-Bench 2.0 public record | Underwriting read |
|---|---|---|---|
| Version pin | Task-set commit for every reported run | Registry pins 2.0 to 69671fb; main has since moved | Do not accept a version label without a commit |
| Start state | Image digest each task pulled at run time | Dated tags on 89 of 89 tasks, no digests | Ask for digests from the run logs |
| Resources | Per-task floor and ceiling plus the enforcement mode used | One value per resource; runners enforce it differently | Compare scores only at equal limits |
| Network | Policy for the agent phase and the grading phase | Internet open to the agent in 89 of 89 tasks | Ask what was fetched; treat online steps as non-repeatable |
| Reset | How each trial starts and whether state carries over | Not stated in the task files | Require a fresh sandbox per trial |
| Final-state check | Grader code, frozen grader dependencies, pass rule | Binary pytest reward; 82 graders install packages at grading time | Prefer a frozen grader image |
| Run conditions | Model snapshot, harness, reasoning budget, number of trials | Outside the task files | Hold all of them fixed in any data ablation |
| Exploit review | Transcript audit for grader bypass | Third-party hackability audits exist for terminal benchmarks | Discount results with no review |
Replication checklist: rerun before the number reaches a memo
Before a benchmark number reaches a memo, have someone outside the company rerun a sample of it. The steps below turn a reported score into a measured one, and the order matters, because each step removes one explanation before the next is tested:
Checklist · 8 items
- Fix the task set by commit, not label, and record the commit in the replication log.
- Resolve every image tag to a digest on the day of the run and save the list; rerun later against the same digests.
- Set CPU and memory enforcement explicitly and record both the reserved amount and the hard ceiling, since Anthropic recommends specifying both rather than one pinned value.
- Choose a network policy per phase; Harbor documents public, allowlist and no-network modes and notes that authors restrict access for security, reproducibility or to limit reward hacking.
- Run the reference solutions first; a task the oracle cannot pass is an environment fault, not an agent failure.
- Run the company's configuration for several trials per task and report the spread, not only the mean.
- Read a sample of passing transcripts for shortcuts before accepting any pass rate.
- For a data claim, rerun with and without the data on the identical setup and compare the paired results task by task.
0 of 8 ticked. Tick each item as you go.
What could make environment evidence a weak screen
Environment evidence is necessary, not sufficient, and it can mislead in four ways. Name them in the memo next to the score.
- Freezing has a cost. Docker's guidance notes that pinning a digest opts out of automated security fixes, so some teams deliberately accept tag drift; ask how they balance the two rather than treating every unpinned tag as a defect.
- Realistic agent work often needs the internet, and all 89 Terminal-Bench 2.0 tasks allow it. Sealing every environment would make scores repeatable but less representative, so online tasks need logging rather than prohibition.
- A perfectly reproducible grader can still be gamed. The Terminal Wrench audit confirmed 331 hackable environments across five public benchmarks, so repeatability must be paired with an exploit review.
- Private environments cannot be published. Companies building on proprietary business systems may only be able to share environment evidence under confidentiality, which shifts the work to a third-party rerun instead of a public check.
Questions for any company that cites agent benchmarks, SourceX included
Ask the same questions of every company, and expect written answers: Which task commit and image digests produced the cited number? What resource floor, ceiling and enforcement mode were used, and does the score hold at the benchmark's published limits? What could the agent and the grader reach on the network? How many trials per task, and what was the spread? Which transcripts were reviewed for grader shortcuts, and by whom? If data is credited for the gain, where is the ablation in which only the data changed?
The same standard applies when the claim is about data rather than models. SourceX licenses selected operating records from companies to AI developers and handles the steps in between, from qualifying a company and reviewing its rights to contracting, preparing the data, delivering it and paying the supplier; this article makes no benchmark claims on its behalf. When a buyer tests whether licensed operating records improve an agent, the fixed-environment ablation above is the design that would show it.
SourceX is raising from accredited investors under Rule 506(c) and shares company and commercial metrics with investors directly. Investor materials can be requested at sourcex.si/investors, where a diligence team can put any of the questions above to the company.
Methods note: how the Terminal-Bench audit was counted
We counted from the public repository laude-institute/terminal-bench-2 at commit 2fd12b8 (April 29, 2026) and compared it with commit 69671fb (October 31, 2025), the commit Harbor's registry.json pins for [email protected] at Harbor commit e77a18b. A task is a top-level directory containing task.toml; there are 89. An image counts as digest-pinned only if its reference contains @sha256:. A base image is every Dockerfile line beginning with FROM.
A grader downloads at grading time if tests/test.sh contains any of these strings: "apt-get install", "curl ", "wget ", "pip install", "pip3 install", "uvx", "uv pip install", "uv run", "npm install" or "git clone". Apt packages are the tokens after apt-get install on the same line, minus flags and everything from the first shell operator on; a package counts as pinned only if its token contains an equals sign. Python test dependencies are the tokens after -w in a uvx call plus package tokens after pip install or pip3 install; each counts as pinned only if it contains ==. A task has a binary reward if test.sh contains both echo 1 > /logs/verifier/reward.txt and echo 0 > /logs/verifier/reward.txt. Test functions are lines in the tests folder's .py files that begin, after indentation, with def test_. A task states a resource ceiling only if an [environment] key contains limit, max, ceiling or request; none does. Drift compares memory, storage, CPUs, image and both time limits after converting values such as 2G to megabytes, plus a file-level diff.
Not measured: we did not pull or build any image, check whether a dated tag was ever re-pushed, run any agent, or estimate how much each change moves scores. The audit covers one benchmark and does not rank it against others. Counts were taken on October 27, 2026; no random sampling was used.
Frequently asked questions
What is agent benchmark environment reproducibility?
It is the ability to rerun an agent benchmark in the same starting state, with the same tools, resource limits, network access, reset rule and grader, and get results within a stated tolerance. It requires a version pin for the task set and an exact reference, such as a digest, for each image. Without those, two runs with the same benchmark name can be different tests.
Does a dated Docker image tag make a benchmark reproducible?
Not on its own. Docker's build guidance explains that a tag can be pointed at a newer image, while a digest reference always resolves to the same image. Dated tags, as Terminal-Bench 2.0 uses, are good practice; asking for the digests a run actually pulled closes the remaining gap.
How much can infrastructure change an agent benchmark score?
In Anthropic's February 2026 study, the gap between the most- and least-resourced setups on Terminal-Bench 2.0 was 6 percentage points (p < 0.01) with the model, harness and tasks held fixed. Leaderboard leads are often smaller than that, so resource settings belong in any comparison.
Which Terminal-Bench 2.0 version should a company cite?
A commit, not a label. Harbor's registry pins [email protected] to commit 69671fb from October 31, 2025, while the repository's main branch later raised memory or time limits on four tasks and edited one grader test. Citing the commit lets a reviewer rerun the same tasks.
Does SourceX publish benchmark results or company metrics?
No. This article makes no benchmark claims for SourceX, and its company and commercial metrics are shared with investors directly rather than published. Investor materials can be requested at sourcex.si/investors; any offering is made only to verified accredited investors through SourceX's offering documents.
Sources
- Terminal-Bench 2.0 task files at commit 2fd12b8 (2026-04-29): 89 task.toml files, Dockerfiles, graders and README text on task validation, counted by SourceX on 2026-10-27. Source
- Terminal-Bench 2.0 at commit 69671fb (2025-10-31), the commit pinned by Harbor's registry; differences from main in memory, time limits, one grader test and two solutions. Source
- Harbor's registry.json pins all 89 terminal-bench 2.0 tasks to commit 69671fb. Source
- Harbor verifier documentation: tests/test.sh entrypoint, reward file, separate versus shared verification, and advice to install test dependencies in tests/Dockerfile or the agent image. Source
- Harbor environment documentation: providers prefer a prebuilt docker_image over building from the environment directory for speed and reproducibility. Source
- Harbor network policy documentation: public, no-network and allowlist modes, set per phase, used for security, reproducibility or reward hacking mitigation. Source
- Harbor core concepts: a task is instructions, a sandbox environment and a verifier; a trial is one agent's attempt at one task. Source
- Docker build best practices: a version tag can resolve to a newer image later; pinning a digest keeps the same image and opts out of automated security fixes. Source
- Anthropic, Feb 5, 2026: resource configuration moved Terminal-Bench 2.0 results by 6 percentage points (p < 0.01) with model, harness and tasks fixed; recommends specifying allocation and hard limit per task. Source
- Holistic Agent Leaderboard (October 2025): 21,730 rollouts across 9 models and 9 benchmarks; higher reasoning effort reduced accuracy in the majority of runs. Source
- Terminal-Bench paper (January 2026): Terminal-Bench 2.0 has 89 tasks selected from 229 submissions, each with a container environment, instruction, tests and oracle solution. Source
- Terminal Wrench (April 2026): over 40,000 adversarial trials on 1,860 tasks from five public benchmarks confirmed 331 reward-hackable environments. Source
Request SourceX investor materials
SourceX shares company and commercial metrics with investors directly. Any offering is made only to verified accredited investors.