Systems and records
Software dataset pilot acceptance: how buyers test private engineering history
By SourceX Editorial · Reviewed by Noah Loul ·
9 sections
Short answer
A software dataset pilot passes acceptance when a small sample of private engineering history clears three written gates: every file is licensable, an independent reviewer can rebuild each task from a pinned base commit and reproduce the test results, and each task has a requirement that predates the fix, discriminating tests and release evidence. Agree the pass rules and sign-offs before the sample is cut.
Checklist
Software dataset pilot
Key takeaways
- A pilot of private engineering history is accepted on three gates: rights, reproducibility and task utility.
- Clear rights first with a per-file license map, because a task that cannot be licensed is worthless however well it runs.
- Ship the base commit, an image pinned by digest and explicit FAIL_TO_PASS and PASS_TO_PASS test lists so a stranger can replay each task.
- A useful task has a requirement written before the fix, tests that fail on the unpatched code and review and release evidence.
- Keep valuable chains that cannot be replayed as a labelled non-executable class instead of scoring them as tasks.
- Agree pass rules and sign-offs in writing before the sample is cut, and keep the rejection log.
What does a buyer pilot of private engineering history have to prove?
It has to prove three things on a small sample before anyone scales: that the supplier has the right to license every file and record in it, that an independent reviewer can rebuild each task and get the same result, and that the tasks teach or test something a coding agent actually needs. A pilot that checks only one of those is a demo, not an acceptance test.
Private engineering history means the linked trail behind shipped software: the requirement or issue, the discussion, the commits and pull request, the review, the tests that ran, and what happened after release. Buyers want it because public repositories are widely crawled, so tasks drawn from them are harder to trust for evaluation. That same privacy is why the pilot has to be stricter, not looser.
The useful framing for a software company leader is that the buyer is not judging the code. It is judging whether each chain of records is complete, licensable and replayable. Agree the acceptance plan in writing before the sample is cut, so both sides know what a pass looks like and nobody argues about it after delivery.
Which rights questions must clear before code leaves the building?
Rights clear first, because a task that fails a rights check is worthless however well it runs. Read access to a repository is not ownership, and ownership of most of the code does not mean the company can license every file in the tree.
Most private repositories mix four kinds of material: code employees wrote, code contractors or agencies wrote under their own agreements, vendored open source or third-party libraries, and code built for one customer under terms that may restrict reuse. Each needs a different answer. Contractor code depends on the assignment language in the services agreement. Vendored code carries its own license. Customer-specific modules depend on the customer contract, and counsel should read that contract rather than rely on memory.
A per-file inventory makes this tractable. The REUSE specification from the Free Software Foundation Europe sets out a standard way to declare copyright and licensing for each individual file in machine-readable form, using SPDX license identifiers. A supplier does not have to adopt REUSE, but a file-level license map built on the same idea lets a reviewer see at a glance which paths are owned, which are third-party, and which must be excluded from the sample.
Secrets and personal data are a separate gate from licensing. Commit history keeps credentials that were later deleted, and issue threads carry customer names and email addresses. Scrub or exclude them before the sample is shared, and record what was removed so the reviewer can tell an intentional gap from a broken chain.
Keep four questions apart in the rights review: whether staff can technically access the code, whether privacy rules permit sharing what is in the threads, whether the company owns the code, and whether its contracts let it license that code for AI use. A yes to the first says nothing about the other three.
Can a separate reviewer rebuild the environment and replay the result?
Yes, if the supplier ships the exact starting state and a pinned environment, and no if it ships a diff and a description. Reproducibility means a reviewer who has never seen the codebase starts from the commit before the change, applies the reference patch, runs the named tests and gets the same pass and fail pattern the supplier reported.
The public SWE-bench benchmark is a useful reference design. Each task instance records the repository, the base commit, the problem statement, the reference patch, a separate test patch, and two test lists: FAIL_TO_PASS tests that should fail before the fix and pass after it, and PASS_TO_PASS tests that must keep passing. The project moved to a containerized evaluation harness built on Docker for more reproducible evaluations. A private pilot can borrow that structure field for field.
Pin the environment by content, not by name. Docker documents that an image digest is an immutable SHA-256 identifier, so pulling by digest returns the same image every time, while a tag can be moved to a different image. Record the digest, the dependency lockfile, the build command and the test command for every task. SLSA, the supply-chain framework, describes provenance as verifiable information about where, when and how an artifact was produced; the pilot needs that kind of record for each task's environment, even if the supplier never adopts SLSA itself.
- Base commit hash, recorded before the change was merged.
- Environment image digest, lockfile and runtime version.
- Build and test commands exactly as the reviewer should run them.
- FAIL_TO_PASS and PASS_TO_PASS test lists, with the reference run log.
- Known flaky tests, named and excluded from the pass rule rather than silently ignored.
What makes a task useful rather than merely runnable?
A task is useful when the requirement existed before the fix, the tests actually discriminate between a correct and an incorrect change, and the record shows what the change was for. Plenty of runnable tasks fail this: a one-line version bump, a test that passes on an empty patch, or a fix whose only description is the commit message written afterward.
Check the link from requirement to change first. On GitHub, a pull request is linked to an issue when its description uses a closing keyword such as fixes or resolves followed by the issue reference, and GitHub notes that these keywords only take effect when the pull request targets the default branch. Teams on Jira or Linear have equivalent links. Chains that rely on someone remembering which ticket went with which branch should be marked as inferred, not linked.
Then check the review and the release. Review comments that explain why an approach was rejected are some of the most valuable text in the chain. Post-release evidence tells the buyer whether the change held. DORA, the research program on software delivery, defines change fail rate as the ratio of deployments that require immediate intervention after release, such as a rollback or hotfix, and deployment rework rate as the ratio of unplanned deployments caused by a production incident. A pilot does not need those rates for the company, but it should flag any sampled change that was later rolled back or hotfixed, because that changes what the task teaches.
The rights, reproducibility and task utility acceptance plan
Use this plan as the written acceptance criteria for a first sample. Each row is one check, the evidence the supplier hands over, the rule that decides pass or fail, and who signs. Adjust the owners to your organization; keep the pass rules explicit so the result cannot be argued after the fact.
Run the gates in order. Rights checks come first because they are cheap and remove material the reviewer should never see. Reproducibility comes second because it is mechanical and either works or does not. Utility comes last because it takes judgment, and that judgment is wasted on chains that will be removed anyway. Record a reason code for every rejection, not just a count, and have each signer date their sign-off.
Agree up front what happens to borderline chains. A common choice is a third outcome, held for repair, with a deadline: the supplier gets one chance to fix a missing lockfile or a mislabelled link, and the chain is rejected if the fix does not land. Without that rule, pilots drift into open-ended back and forth.
| Gate and check | Evidence the supplier provides | Pass rule | Signs off |
|---|---|---|---|
| Rights: file ownership | Per-file license map with owned, third-party and excluded paths | Every file in the sample is owned or carries a license that permits the agreed use | Supplier counsel |
| Rights: contractor and customer code | List of modules written under services or customer contracts, with the governing clause | Excluded unless counsel has confirmed the clause allows licensing | Supplier counsel |
| Rights: secrets and personal data | Scan report and removal log for history, issues and review threads | No live credential or unredacted personal data remains; every removal is logged | Security lead |
| Reproducibility: starting state | Base commit hash and repository snapshot for each task | Reviewer checks out the base commit with no missing files | Buyer reviewer |
| Reproducibility: environment | Image digest, lockfile, build and test commands | Reviewer builds without network guesswork and gets the reference run log | Buyer reviewer |
| Reproducibility: test behavior | FAIL_TO_PASS and PASS_TO_PASS lists, flaky tests named | Reference patch flips every FAIL_TO_PASS test and keeps every PASS_TO_PASS test green | Buyer reviewer |
| Utility: requirement first | Issue or ticket created before the first commit, with link type | Requirement predates the change; inferred links are labelled | Engineering lead |
| Utility: discriminating tests | Result of running the tests against the unpatched base | At least one test fails on the base commit; an empty patch does not pass | Buyer reviewer |
| Utility: review and release | Review thread, merge record, release tag, any rollback or hotfix | Rationale is present; later rollbacks or hotfixes are flagged on the task | Engineering lead |
| Documentation | Datasheet for the pilot sample | Motivation, composition, collection, exclusions and maintenance are answered | Both parties |
Worked example: an illustrative first sample from one product line
This example is illustrative; the company, the counts and the outcomes are invented to show how the plan behaves. A mid-sized vertical software company offers 60 candidate chains from one product's repository, all from the last three years, each starting with a tracked issue and ending in a merged pull request.
The rights pass removes 9 chains: 6 touch a payments module a contractor built under an agreement counsel has not yet reviewed, and 3 modify a vendored library whose license the company cannot confirm. The secrets scan finds a deleted API key in the history of 2 more chains; the supplier rewrites the history for those and logs the removal, so they stay in.
The reproducibility pass first fails 11 chains: 7 depend on an internal package registry the reviewer cannot reach, and 4 have tests that depend on wall-clock time. The supplier vendors the packages into the image, 4 of the 7 then rebuild cleanly, and 7 chains are removed. The utility pass removes 8 more: 5 where the issue was opened after the fix was written, and 3 where the tests also pass on the unpatched base.
The result, still illustrative, is 36 accepted tasks out of 60, about 60 percent, each with a known reason for every rejection. That rejection log is as useful to the buyer as the accepted tasks, because it predicts how the full archive will behave.
| Stage (illustrative) | Chains in | Removed | Main reason |
|---|---|---|---|
| Rights review | 60 | 9 | Unreviewed contractor module and unconfirmed vendored license |
| Secrets and personal data | 51 | 0 | Two chains scrubbed and logged rather than removed |
| Reproducibility | 51 | 7 | Private registry and time-dependent tests, after 4 were repaired |
| Task utility | 44 | 8 | Issue written after the fix, or tests pass on the base commit |
| Accepted | 36 | Not applicable | Every rejection has a logged reason |
When a chain fails replay but still deserves a place
Some of the most instructive engineering history cannot be replayed: a production-only bug that was diagnosed from logs, a performance fix with no deterministic test, or a migration whose effect shows only in later incidents. Rejecting all of them biases the sample toward easy, well-tested changes.
The exception is to accept such chains as a separate, labelled class: non-executable records with full requirement, discussion, review and release evidence, but no claim of reproducible tests. They can support training or qualitative evaluation but should never be scored as if they were executable tasks. Keep them out of any pass-rate statistic and name the class in the datasheet.
The limit of this whole plan is that a reproduced patch proves only that the reference change satisfies the recorded tests. It does not prove the tests were complete or that the change was the best one. Post-release flags help, but the buyer should treat the pilot as evidence about the archive's quality, not as proof that any single task is flawless.
Which pilot findings should pause the program instead of scaling it?
Pause when a finding would repeat across the full archive and cannot be fixed by excluding a few chains. Three patterns usually qualify, and each calls for a different specialist rather than more engineering effort.
- Rights gaps that are structural, such as a core module written under a customer contract nobody has read: stop and send the contract to counsel before cutting a larger sample.
- Reproducibility failures rooted in infrastructure the company cannot ship, such as builds that require a proprietary internal service: narrow the scope to repositories that build standalone.
- Utility failures that show the process itself, such as issues routinely written after the code: scope the offer as non-executable history rather than benchmark tasks.
- Secrets found in more than an isolated handful of chains: have security review the whole history rewrite before anything else moves.
- Disagreement about the pass rules after the sample is delivered: rewrite the acceptance plan first, then rerun it on the same sample.
Getting your engineering history pilot-ready before a buyer asks
Most of the plan above is work the supplier can do before any buyer conversation: a per-file license map, a list of contractor and customer-specific modules, a secrets scan of the full history, and a record of which repositories build from a pinned image. Document the candidate sample with a datasheet; the Datasheets for Datasets paper proposes that every dataset ship with one covering motivation, composition, collection process and recommended uses, and those questions map well onto an engineering archive.
Engineering history is one of the record types SourceX helps companies license to AI developers, with SourceX handling rights review, contracting, preparation and delivery. A company's fit is judged first from a short assessment of the business, not from a code upload. The repository stays put until the company has approved the scope and terms and signed, and very large archives need not move to SourceX at all: they can remain in the company's storage or ship on encrypted drives.
Frequently asked questions
How large should a first engineering history pilot sample be?
There is no standard size. Pick enough chains, from more than one repository or team where possible, that every gate in the acceptance plan is exercised several times and the rejection reasons are meaningful. The pilot agreement should state the number and how it was drawn so the result can be compared with the full archive.
Does the buyer need the whole repository or only the changed files?
For replayable tasks the reviewer needs the full repository at the base commit, because tests and builds depend on files the change never touched. Excluded paths can be removed only if the build and tests still run without them. Record any removal in the sample's documentation.
What should we do about flaky tests in our history?
Name them. A test that passes or fails at random should be listed as flaky and excluded from the pass rule for that task rather than quietly dropped. If a task's only discriminating test is flaky, the task belongs in the non-executable class.
Who owns code that contractors wrote inside our repositories?
It depends on the assignment and license terms in each contractor or agency agreement, and some agreements leave ownership with the contractor. Counsel should review those agreements before any contractor-written module enters a sample. This is general information, not legal advice.
Is a reproduced fix the same as a verified production outcome?
No. Replaying a fix shows the reference change satisfies the recorded tests. Whether it held in production is a separate question answered by release records, rollbacks, hotfixes and later incidents, and the pilot should flag those on each task.
Can tasks from private history leak into a model's training data later?
They can if the license allows training use or if the buyer's handling is loose. If the buyer wants the tasks for evaluation, the license should say so and restrict training use, and the supplier should keep its own record of which chains were delivered.
Sources
- SWE-bench task instances record the repository, base commit, problem statement, reference patch, test patch and FAIL_TO_PASS and PASS_TO_PASS test lists. Source
- SWE-bench moved to a fully containerized evaluation harness using Docker for more reproducible evaluations. Source
- A Docker image digest is an immutable SHA-256 content identifier, so pulling by digest returns the exact same image every time. Source
- SLSA defines provenance as verifiable information describing where, when and how a software artifact was produced. Source
- The REUSE specification defines a standard, machine-readable way to declare copyright and licensing for each individual file, using SPDX license identifiers. Source
- GitHub links a pull request to an issue when the description uses a closing keyword, and the keywords only work when the pull request targets the default branch. Source
- DORA defines change fail rate as deployments requiring immediate intervention, and deployment rework rate as unplanned deployments caused by a production incident. Source
- Datasheets for Datasets proposes that every dataset be accompanied by a datasheet documenting its motivation, composition, collection process and recommended uses. Source
See if your company qualifies
A short company assessment. No data uploads are needed.