Skip to content

AI uses for records

Coding agent feature requirement dataset vs bug-fix task data

By SourceX Editorial · Reviewed by Noah Loul ·

10 sections

Short answer

Training data for feature development should differ from bug-fix data by carrying the requirement as it stood at the start, numbered acceptance criteria, the new interfaces, tests mapped to each criterion, the review that shaped the design and evidence that the feature shipped and held up. Bug-fix tasks lean on a test that fails before the patch; feature tasks need a traced chain from requirement to production instead.

Agent run

Coding agent feature

This page covers coding agent feature requirement dataset vs bug-fix task data.Illustrative

Key takeaways

  • A bug-fix task is defined by a failing test; a feature task has to be defined by a versioned requirement and acceptance criteria.
  • Feature tasks should name the new interfaces, or a correct solution under another name will fail every test.
  • Map each acceptance criterion to at least one test, and reject records with uncovered criteria.
  • Prove each link with a record: issue to pull request, CI on the merge, review approvals, deployment status.
  • Use the requirement version from before development started; edits made after merge describe the code, not the ask.

How should training data for feature development differ from bug-fix data?

Bug-fix data is anchored on broken behavior: an issue describes what goes wrong, and a test that fails before the patch and passes after it defines done. Feature data has no such anchor. A feature task has to carry the requirement as it stood when work began, its acceptance criteria, the new interfaces the change introduces, tests written for the new behavior, the review discussion that shaped it and evidence that it shipped and held up in production.

So a feature dataset needs more links, not just more examples. The schema on this page connects a versioned requirement to code, tests, review and deployment, and adds validators that reject records where any link is missing or out of order. Without those links, a model learns to produce plausible code that passes the tests someone happened to write, which is a weaker skill than building what was asked.

The audience here is engineering leaders at software companies deciding whether their own feature history, held in tools such as Jira, Linear, GitHub, GitLab and their CI and deployment systems, can be shaped into training or evaluation data for coding agents.

What does a bug-fix task get for free that a feature task has to build?

The bug-fix format is well understood because the test does most of the specification work. SWE-bench's dataset guide lists the fields of each task: the repository, a base commit, the issue text as the problem statement, the gold patch, a test patch, and two test lists, FAIL_TO_PASS and PASS_TO_PASS. The first list proves the fix; the second proves nothing else broke.

Commit conventions draw the same line from the other side. The Conventional Commits specification says a fix commit patches a bug and correlates with a PATCH release, while a feat commit introduces a new feature and correlates with a MINOR release. Teams that follow it already label much of their history, which makes a first split between the two task types cheap.

What a feature lacks is a pre-existing failing behavior. Before a feature exists, there is nothing to fail except tests written for it, and those tests were written by the same team, often in the same pull request, with the same blind spots. The requirement and its acceptance criteria have to stand in for the bug report, and they have to be checkable.

Why must a feature task declare its new interfaces?

Public feature benchmarks show what the task needs beyond an issue and a test list. FEA-Bench, from Microsoft researchers, describes a test set of 1,401 task instances from 83 GitHub repositories, collected from pull requests whose purpose was new feature implementation and filtered by rules and by intent. Each instance records the repository, a base commit, an environment setup commit, the pull request number and the unit test status lists.

The benchmark's prompt builder also gives the solver the definitions of new functions or classes to implement, with their signatures and docstrings. That is a telling design choice: without the interface, a correct implementation under a different name or signature fails every test, and the benchmark would measure guessing rather than engineering.

The dataset card also states that FEA-Bench is not intended as training data, to avoid contamination. Private feature history is valuable precisely because it is not already in public training corpora, which is why it should be kept out of any public mirror and why evaluation splits from it need strict separation from training splits.

Where should the acceptance criteria come from?

Acceptance criteria should come from the requirement record, written before the code, not reconstructed from the merged diff. Cucumber's Gherkin reference describes a feature's free-text description as a place to list business rules, which it calls general acceptance criteria, and its Rule keyword groups the scenarios that illustrate one business rule. Its example mapping practice puts each rule on its own card with concrete examples underneath.

You do not need Gherkin files for this to work. A Jira story or Linear issue with numbered acceptance criteria is enough if each criterion is specific, testable and timestamped. What matters for training data is that every criterion maps to at least one test and that the requirement version used for the task is the one that existed when development started.

Requirement edits are the main trap. Product managers often update a ticket after merge to match what shipped, which turns the requirement into a description of the code. Keep the edit history and use the version at the base commit's date; if the tool does not keep history, record that the requirement is unversioned and treat the task as lower confidence.

Which fields does a feature task record need, and how is each one validated?

The asset is a field-level task schema. Each record is one feature, from requirement to production, with validators that a script or a reviewer can run. Records that fail any required check go to a repair queue, not into the dataset.

Which fields does a feature task record need, and how is each one validated?
FieldPurposeRequired contentValidator check
requirementStates what was askedRequirement ID, text and version as of the base commit dateVersion timestamp is not later than base_commit date
acceptance_criteriaDefines done in the requirement's own termsNumbered, testable criteria with IDsAt least one criterion; each has an ID
interfacesNames the new or changed public surfaceFile, name, signature and docstring per new componentEvery listed signature exists in the merged code
repo_snapshotFixes the starting stateRepository, base_commit, environment_setup_commit or container image digestCheckout and build succeed in a clean environment
patchThe implementationDiff from base_commit, split into source, test and documentation partsApplies cleanly to base_commit
testsProves the new behavior and protects the oldNew tests mapped to criterion IDs, plus a regression listNew tests fail or are absent at base; all pass after patch
criterion_coverageShows each criterion is testedMap of criterion ID to test IDsNo criterion without a test; no orphan new test
reviewCaptures why the design changedReview comments, requested changes, approvals with rolesAt least one approval by someone other than the author
deploymentShows it shippedEnvironment, deployment status and timestampA success status in a production environment after merge
production_evidenceShows it held upPost-release signal: rollback absence, follow-up bugs linked, feature flag stateWindow stated; linked follow-up bugs listed even when empty

Each arrow in the chain needs a record that a machine can follow. From requirement to pull request, the cleanest evidence is an explicit link. GitHub's documentation says a pull request can be linked to an issue with keywords such as closes, fixes or resolves in the pull request description or a commit message, and that merging a linked pull request into the default branch closes the issue automatically. Jira and Linear offer similar development panels that record branch and pull request links.

From pull request to tests, the evidence is the CI run on the merge commit and the test files touched by the patch. From review to design, it is the review thread with requested changes and the commits that answered them. From merge to production, GitHub's deployment status API lets external services mark a deployment as error, failure, inactive, in progress, queued, pending or success, with an optional log URL. GitHub environments can also require named reviewers to approve a deployment and can prevent the person who started it from approving it.

When a link exists only in someone's memory, the record should say so. A task with a missing deployment link can still be used for implementation training, but not for any claim about production outcomes.

Can someone outside the team rebuild the feature and rerun every check?

Yes, if the record pins the environment and the validators are run by someone other than the original team. The reviewer checks out base_commit, builds from environment_setup_commit or the stored image, confirms the new tests fail or do not exist, applies the patch, and confirms that the new tests and the regression list pass. This mirrors the FAIL_TO_PASS and PASS_TO_PASS split used for bug fixes, with the difference that a feature's new tests are often absent at base rather than failing.

Then the reviewer checks what tests alone cannot. Does every acceptance criterion map to a test that exercises it? Do the interfaces in the record match the merged code? Does the documentation part of the patch describe the behavior the criteria require? A record that passes tests but leaves a criterion uncovered is a partial implementation, and labeling it complete teaches a model to stop early.

Record the reviewer's result in the task itself, with the date and the toolchain versions used. Rebuilds drift as dependencies age, so a task verified once should be re-verified before each dataset release.

Valid and failing records, side by side (illustrative)

The two records below are illustrative, written to show the schema, and do not come from any real repository or company.

At a glance

Valid and failing records, side by side (illustrative)

  • Valid record: requirement REQ-41 version 2, dated the day before the base commit, asks for CSV export of invoice lists with three acceptance criteria (column order, date format, a 10,000-row limit with a warning). The interfaces field lists one new function signature. Three new tests map one to one to the criteria and are absent at base; all pass after the patch, as does the regression list. Review shows one requested change to the date format, answered in a follow-up commit, and an approval by a second engineer. A deployment status of success in production follows the merge, and no linked follow-up bugs appear in the stated 60-day window.
  • Failing record: the same feature, but the requirement text was edited after merge to drop the row limit, and the stored version is the edited one. The validator flags the requirement timestamp as later than the base commit. The criterion_coverage map has no test for the row limit in the original version, so even after restoring the earlier text the record fails coverage. The record goes to repair, not to training.
  • What the pair teaches: a feature record is only as good as its earliest requirement version and its criterion-to-test map. Passing tests did not make the failing record correct.

What must be removed or cleared before feature history leaves the company?

Feature history is more sensitive than bug history. Requirements carry roadmap plans, pricing logic and customer names, and review threads often quote customer complaints or internal debates. Decide which requirement fields can leave the company, and generalize customer references before any export. Give every task record a rights and redaction note: who owns the repository, which third-party paths were excluded and what was removed, so the validators can confirm that no excluded path appears in the patch.

Code rights need the same care as for any repository license. Exclude vendored third-party code and files under licenses that do not permit the intended use, scan history for secrets, and confirm the company owns the code it is offering, including work by contractors. Ownership of the code, permission to share the requirement documents and the right to license both for AI training are separate questions, and counsel should confirm each one for the specific repositories.

Can a software company license its feature history?

It can, once ownership and redaction are settled. SourceX works with software companies whose requirement-to-production history could become a licensed dataset for AI, running the rights review, contract, preparation, delivery and payment. Engineering leaders learn whether their company qualifies from a short assessment that asks for no code or tickets, and no repository content moves until the company has approved the scope and terms and signed.

Frequently asked questions

Can bug-fix datasets teach a coding agent to build features?

Partly. They teach reading a codebase, locating the right files and satisfying tests, which features also need. They do not teach working from a requirement with no failing test, designing a new interface or covering every acceptance criterion, so feature tasks need their own records.

Do feature tasks need tests written before the code?

No, but they need tests mapped to acceptance criteria that were written before the code. Tests written in the same pull request are normal; the check is that each criterion from the original requirement has at least one test that exercises it.

Why include deployment and production evidence in a training record?

Merged code is not the same as shipped code. Deployment status and a post-release window separate features that went live and held up from features that were rolled back or quietly reworked, which matters for evaluation and for weighting examples.

Can we use public feature benchmarks as training data?

Check each benchmark's own terms first. FEA-Bench's dataset card, for example, says it is not intended for training, to avoid contamination. Private feature history avoids that conflict but needs its own rights review.

What is the most common reason a feature record fails validation?

Usually a requirement that changed after the work started, or an acceptance criterion with no test behind it. Both are fixable when the requirement tool keeps edit history and the team is willing to add the missing test mapping during preparation.

Sources

  • SWE-bench task instances include the repository, base commit, problem statement from the issue, gold patch, test patch, FAIL_TO_PASS and PASS_TO_PASS test lists. Source
  • Conventional Commits 1.0.0 defines fix as patching a bug (PATCH) and feat as introducing a new feature (MINOR). Source
  • FEA-Bench contains 1,401 task instances from 83 GitHub repositories collected from feature-implementation pull requests, records base and environment setup commits, and is not intended for training. Source
  • FEA-Bench prompts give the solver the definitions of new functions or classes to implement, including signatures and docstrings. Source
  • Cucumber's Gherkin reference describes a feature description as a place for business rules (general acceptance criteria) and the Rule keyword as grouping scenarios for one business rule. Source
  • Cucumber's example mapping practice records each acceptance criterion or rule on its own card with illustrating examples beneath it. Source
  • GitHub links a pull request to an issue through keywords such as closes, fixes or resolves, and merging into the default branch closes the linked issue. Source
  • GitHub deployment statuses can be error, failure, inactive, in_progress, queued, pending or success, with an optional log URL. Source
  • GitHub environments can require named reviewers to approve deployment jobs and can prevent self-review by the person who triggered the deployment. Source

See if your company qualifies

A short company assessment. No data uploads are needed.

See if your company qualifies