Skip to content

Systems and records

Coding dataset environment reconstruction: making old tasks rebuildable

By SourceX Editorial · Updated

9 sections

Short answer

To make a coding task independently rebuildable, freeze what the patch depended on at its base commit: base image by digest, runtime version, hashed lock files, mirrored copies of every dependency, build and test commands, and service fixtures. Then a reviewer who did not prepare the task rebuilds it offline from that bundle alone and must reproduce the same failing-then-passing tests.

System export

Coding dataset environment

This page covers coding dataset environment reconstruction: making old tasks rebuildable.Illustrative

Key takeaways

  • A patch is only usable as a task if someone else can rebuild the environment and rerun the tests.
  • Pin by content, not by name: image digests, hashed lock files and mirrored package artifacts.
  • Decide per task whether you reconstruct the historical stack or a modernized one, and label it.
  • An independent reviewer rebuilds offline from the bundle alone and must reproduce the fail-to-pass result.
  • Tasks that depend on live services, non-redistributable toolchains or production data are mocked, narrowed or excluded with a written reason.

How do you make an engineering task's environment independently rebuildable?

Make each task rebuildable by freezing everything the patch depended on at the commit it was written against: the base image by digest, the language toolchain version, a lock file with hashes, a copy of every dependency artifact, the build and test commands, and the services the tests call. Then have someone who did not prepare the task rebuild it from that record alone and get the same test outcome.

Coding dataset environment reconstruction is the work of recovering that state after the fact. A repository holds the code, but the environment that made the code pass its tests was spread across a CI runner, a package registry, an internal artifact store and an engineer's laptop. None of those were designed to be replayed years later.

A buyer evaluating a coding agent needs the before state, the patch and a test that flips from failing to passing. If the environment cannot be rebuilt, the test cannot run, and the patch is just text. This guide is for the software company preparing those tasks, not for the buyer writing a runtime contract.

Why does a patch that once passed stop building months later?

Builds decay because almost every input is fetched by name rather than by content, and names get reassigned or removed. A Dockerfile that says FROM python:3.9 pulls whatever image carries that tag today. Docker's own guidance is to pin the base image to a digest so a rebuild keeps using the same image even after the publisher moves the tag.

Package registries also change underneath you. PyPI supports yanking, under PEP 592: installers skip a yanked release when another version satisfies the constraint, although an exact pin still installs it. npm allows a new package to be unpublished within 72 hours when nothing depends on it, and a deleted version number can never be reused. Go modules can be retracted by their authors. Each of these quietly changes what a loose constraint resolves to.

Then there are inputs nobody wrote down: an apt-get install without versions, a private registry that was decommissioned, a TLS certificate that expired, a feature flag service the integration tests called, and the system clock. A test that asserts a date is in the future will pass on the day the patch merged and fail the next year.

The practical consequence is that the cheapest time to capture an environment is when the pull request merges, and the most expensive is years later during dataset preparation. If you are preparing tasks now, start with the CI history: archived job logs often print the resolved image digest, the runtime version and the full install output, which is enough to reconstruct most of the checklist without guessing.

Are you rebuilding the environment as it was, or a modernized one?

Decide up front whether each task reproduces the historical environment or runs the same code on a newer, supported stack, and record the choice per task. The two answer different questions and should not be mixed in one delivery without a label.

A historical reconstruction keeps the exact versions the code ran on, including end-of-life operating systems and runtimes. It is the faithful option for evaluating whether an agent can fix the bug as it existed, but it may need archived package mirrors and old images that carry known vulnerabilities, so it runs offline in an isolated sandbox.

A modernized rebuild upgrades the toolchain until the tests pass again. It is easier to run and maintain, but the test outcome now reflects a different environment, and some original failures may disappear or new ones appear. If you choose it, keep the original lock file alongside the new one and mark the task as modernized so the buyer can filter it out.

Are you rebuilding the environment as it was, or a modernized one?
ChoiceUse whenMain riskLabel in the manifest
Historical, exactThe task is an evaluation of the original bug and fixOld images and packages may be unavailable or vulnerableenvironment_mode: historical
Historical, patched baseExact OS packages are gone but language dependencies are intactOS-level behavior can differ from the original runnerenvironment_mode: historical-os-substituted
ModernizedThe original stack cannot be recovered at allTest outcomes no longer describe the original environmentenvironment_mode: modernized, with both lock files
ExcludedNeither version reproduces the fail-to-pass resultLosing rare tasksExclusion record with reason

The dependency and environment reproducibility checklist

The checklist below is the asset: one row per layer of the environment, what to capture, how to prove it is pinned, and the failure it prevents. A task passes only when every row has evidence or a written exception.

At a glance

The dependency and environment reproducibility checklist

Treat the base commit as the anchor. Every value in the checklist should be read from the repository at that commit, from the CI configuration that ran it, or from artifacts stored at the time, not from the current state of the main branch.

The dependency and environment reproducibility checklist
LayerCaptureEvidence it is pinnedFailure it prevents
SourceRepository tree at the base commit, plus the reference patch and test patch as separate diffsCommit hash and a content hash of the archived treePatch applied to the wrong revision
Base imageOS image used by CI at that commitImage reference by digest, not a tagA moved tag silently changes the OS
OS packagesName and exact version of every system package installedVersion list plus a dated archive source such as a Debian snapshot URLapt resolving to a newer package
Language runtimeInterpreter or compiler version and build flagsVersion string recorded from the running containerBehavior differences between minor versions
Direct and transitive dependenciesFull lock file for each ecosystemLock file with integrity hashes (package-lock.json, go.sum, or pip requirements with hashes)Resolver picking a different transitive version
Dependency artifactsCopies of every downloaded package, including internal onesWheelhouse, vendor directory or registry mirror with checksumsYanked, unpublished or decommissioned packages
Build and test commandsExact commands, environment variables and flags from CICommands stored in the task manifest and run unmodifiedHidden flags such as legacy peer dependency settings
Services and fixturesDatabases, queues and mocks the tests call, with seed dataService image digests and fixture files in the task bundleTests calling a live system that no longer exists
Time, network and randomnessClock assumptions, network access and random seedsBuild runs with network disabled; frozen clock where tests depend on datesTests that pass only on the merge date
Test oracleTests that fail before and pass after the patch, plus tests that must keep passingResult files from runs at base and after the reference patchGrading on tests that never exercised the change

Keeping dependencies installable after registries change

Store a copy of every dependency artifact with the task rather than trusting a public registry to keep serving it. A lock file records what was installed; it does not make those files available again.

Each ecosystem has a native way to do this. pip can build a wheelhouse of all dependencies and install from it with no index, and hash-checking mode rejects any file whose contents differ from what was recorded. Its documentation notes that a wheelhouse is usually specific to an OS and CPU architecture, so record the platform with it. npm ci installs exactly what package-lock.json lists, fails if the lock and package.json disagree, and never rewrites either file, which makes it the right install command for a rebuild. Go records hashes for every module in go.sum and refuses a download that does not match; go mod vendor copies the modules into the repository.

Operating system packages need their own archive. For Debian-based images, snapshot.debian.org serves past package sets by date, so an apt source can point at a timestamp near the commit date. Internal packages are the usual gap: if your company ran a private registry, export the exact versions the lock file names before the registry is retired, and check whether any of them contain third-party code under terms that limit redistribution.

  • Python: pip wheel into a directory, then install with --no-index and --require-hashes from that directory.
  • Node.js: commit package-lock.json, keep the .npmrc flags used to create it, and install with npm ci against a mirrored cache.
  • Go: keep go.mod and go.sum, run go mod vendor, and build with the vendor directory and network disabled.
  • Java and others: export the resolved artifacts from the build cache or artifact repository with their checksums.
  • OS packages: record name=version pairs and the dated archive URL they came from.

Can a separate reviewer rebuild the environment and confirm the result?

Yes, and the rebuild only counts when it is done by someone who did not prepare the task, on a clean machine, with network access off, using nothing but the delivered bundle. If they need to ask a question or fetch a file, the record is incomplete.

Define what counts as the same result before the rebuild starts. The Reproducible Builds project defines reproducibility as any party recreating bit-by-bit identical artifacts from the same source, environment and instructions. That is the strongest standard and is worth meeting for the built image where possible, but for a coding task the decision usually rests on test outcomes: the same tests fail at the base commit, the same tests pass after the reference patch, and the pass-to-pass set is unchanged.

The SWE-bench harness is a useful structural reference: it separates a base image, environment images and a thin per-task instance image. Layering your tasks the same way means a reviewer can rebuild one shared environment and then verify many tasks against it.

Run the rebuild on a different host from the one used for preparation, ideally a different machine type, and record the CPU architecture. Differences that only appear on another architecture, such as a wheel compiled for one platform, are exactly what a buyer will hit first.

  • 1. Reviewer receives only the task bundle and manifest.
  • 2. Build every image from the recorded recipe with network disabled; note any step that tries to reach the internet.
  • 3. Run the full test command at the base commit three times and record results.
  • 4. Apply the reference patch, run again three times, and compare with the recorded results.
  • 5. Mark any test whose result differs between runs as flaky and remove it from grading.
  • 6. Sign a rebuild record: reviewer, date, host platform, image digests produced, and any differences found.

Illustrative case: reconstructing three years of tasks from a billing service

The following example is illustrative and invented to show the method; it does not describe a real company or dataset. A software company selects 120 merged pull requests from a Python billing service, each linked to an issue and changing at least one test.

The engineer preparing tasks finds that CI used a floating python:3.8 tag for the first eighteen months, then a digest-pinned image. For the early tasks they recover the image digest from archived CI logs for 70 of the pull requests; the rest have no log, so they rebuild the base from the Debian snapshot closest to each merge date and mark those tasks historical-os-substituted.

Two internal packages lived in a registry retired last year. The team restores them from a backup of the registry storage and adds them to each task's wheelhouse with hashes. One dependency version had been yanked upstream; because the lock file pins it exactly, it still installs from the mirror.

A separate engineer then rebuilds every task offline. In this example 96 tasks reproduce the recorded fail-to-pass result, 14 have flaky tests that are removed from grading, and 10 call a payment sandbox that no longer exists and are excluded with a written reason. The delivered manifest states those counts, which are illustrative only.

Which tasks to drop when the environment cannot be recovered

Drop or narrow a task when its tests depend on something you cannot capture or are not allowed to ship. The rebuild record should say which, because a buyer reads an unexplained gap as a quality problem.

Common reasons to stop are tests that call a live third-party API, a commercially licensed compiler or SDK whose license does not allow redistribution, hardware or GPU drivers the reviewer cannot reproduce, and fixtures built from production customer data. The last one is a privacy question before it is a technical one; replace the fixture with a synthetic one or exclude the task.

Secrets are a second stop condition. Build scripts often fetch tokens from the environment, and old commits may contain credentials in configuration files. Scan and remove them before the bundle leaves your systems, and rotate anything found, since removal from a copy does not revoke a key.

Keep the exclusion list as part of the delivery rather than deleting the tasks silently. A buyer comparing the selection with the repository history will notice missing pull requests, and a reason per task, with the count by reason, shows the gaps are deliberate rather than a sign of weak preparation.

Which tasks to drop when the environment cannot be recovered
ConditionDecisionWhat to record
Tests call a live external serviceMock it with recorded responses or excludeService name and how responses were captured
Licensed toolchain cannot be redistributedShip a build recipe only, or excludeLicense term relied on, reviewed by counsel
Fixtures contain production personal dataReplace with synthetic fixtures or excludePrivacy reviewer sign-off
Result differs between clean rebuildsRemove flaky tests or excludeRun counts and differing tests
Original environment unrecoverableModernize with a label, or excludeBoth lock files and the reason

Rebuildable tasks and the licensing review

A rebuildable environment proves the task is technically usable; it does not establish that the company may license the code, the bundled dependencies or the test fixtures. Ownership of the repository, the terms of any vendored third-party code and the privacy status of fixtures are separate checks, and counsel reviews them on the specific facts.

Engineering history is one of the record types SourceX helps companies license to AI developers, with SourceX handling rights review, contracting, preparation and delivery. Scope and terms are the company's to approve, and it signs before any data is delivered; large repositories or image bundles can stay in the company's own storage or ship on encrypted drives rather than being hosted by SourceX. A short company assessment, with no uploads, is the first step.

Frequently asked questions

What should a dependency and environment reproducibility checklist include?

It should cover the source tree at the base commit, the base image by digest, OS package versions, the language runtime, lock files with hashes, copies of every dependency artifact, build and test commands, service fixtures, clock and network assumptions, and the test oracle. Each item needs evidence that it is pinned, not just a version name.

Is a lock file enough to rebuild an old commit?

No. A lock file records which versions were installed and, in most ecosystems, their hashes, but it does not keep those files available. Packages can be yanked, unpublished or lost when a private registry is retired, so the artifacts themselves need to be mirrored with the task.

Should the rebuilt environment match the original bit for bit?

Bit-for-bit identical images are the strongest evidence and worth pursuing for shared base images. For grading a coding task, the practical test is that the same tests fail before the patch and pass after it across repeated clean runs, with the remaining tests unchanged.

Who should perform the independent rebuild?

Someone who did not prepare the task, working on a clean machine with network access disabled and only the delivered bundle. If that person has to ask the preparer anything, the answer belongs in the manifest.

What if the original CI image no longer exists?

Recover its digest from archived CI logs if you can. Otherwise rebuild the base from a dated package archive near the commit date, label the task as OS-substituted, and confirm the fail-to-pass result still holds before including it.

Sources

  • The npm ci documentation says the command requires an existing package-lock.json, exits with an error if the lock does not match package.json, removes any existing node_modules and never writes to package.json or the lock file. Source
  • pip's repeatable-installs guide describes pinning with ==, hash-checking that protects against a package changing without a version change, and a wheelhouse that allows installation when index servers are unavailable but is usually OS and architecture specific. Source
  • Docker's build best practices say pinning a base image to a digest keeps builds on the same image version even if a publisher replaces the tag with a new image. Source
  • PEP 592 lets a package index mark a file as yanked; installers must ignore yanked releases when a non-yanked version satisfies the constraints, but an exact pin to a yanked version still installs. Source
  • The Go modules reference says go.sum holds cryptographic hashes of a module's direct and indirect dependencies and the go command reports a security error when a downloaded file does not match; it also documents vendoring and retracted versions. Source
  • The SWE-bench documentation describes an evaluation harness that builds Docker images in three layers: a base image, environment images and per-task instance images. Source
  • Reproducible Builds defines a build as reproducible when, given the same source code, build environment and build instructions, any party can recreate bit-by-bit identical copies of the specified artifacts. Source
  • snapshot.debian.org is an archive of past and current Debian packages that can be addressed by date and version, which lets an older apt configuration point at a timestamped archive. Source
  • The npm unpublish policy allows removing a new package within 72 hours if nothing depends on it and, later, only when several conditions are met; a deleted version number can never be reused. Source

See if your company qualifies

A short company assessment. No data uploads are needed.

See if your company qualifies