Systems and records
Engineering history task density: count complete tasks in your repo
By SourceX Editorial · Updated
10 sections
Short answer
To count complete engineering tasks in repository history, sample merged pull requests on the default branch year by year, grade each against a written definition (requirement, base commit, patch, fail-before and pass-after tests, review decision), replay the strongest candidates in a rebuilt environment, and extrapolate the verified share back to each year with a confidence interval. Raw commit or pull request totals overstate what a reviewer can verify.
Record timeline
Engineering history task
Key takeaways
- Engineering history task density is verified complete tasks per year of history, not commits or pull requests.
- Use merged pull requests on the default branch as the frame and join issues, reviews, CI and deployments to them.
- Grade every sampled change on a ladder and count only items a second person replayed: tests fail before the patch and pass after.
- Report a rate and a Wilson interval per year, because process changes make some years far thinner than others.
- The definition of test evidence moves the count more than any statistical choice, so state it first.
How do you count complete engineering tasks in repository history?
Count them by sampling, not by totals. Take every pull request merged into the default branch during a fixed window, draw a random sample per year, and grade each sampled change against a fixed definition: a traceable requirement, a code change, test evidence that fails before and passes after, a recorded review decision and, ideally, evidence it shipped. Then replay the strongest candidates in a rebuilt environment and extrapolate the verified share back to the whole window with an interval, year by year.
Raw counts mislead in both directions. A repository with 40,000 commits may hold only a few hundred changes a reviewer can verify end to end, while a team that squash-merges everything can look thin in the commit log and still hold a dense record in its pull requests. Engineering history task density is the number that survives this grading: verified complete tasks per year of history, with the uncertainty stated.
The rest of this page gives the definition, the sampling frame, a worksheet you can copy, the replay check and an illustrative scenario. It does not tell you what any count is worth, and every number in the scenario is invented.
What counts as one complete engineering task?
A complete task is a change a stranger could re-run and judge without asking anyone who was there. That means five linked pieces: the requirement as it was written before the work, the repository state the work started from, the code change itself, a test that demonstrates the requirement was met, and the review decision that accepted it. Deployment or production evidence is a sixth piece that turns a verified patch into a verified outcome.
Public benchmark construction uses a close version of this definition. The SWE-bench collection code keeps a pull request only if it was merged and resolves at least one issue, splits each change into a code patch and a test patch by file path, and treats instances that also carry tests as candidates for evaluation. Its tutorial then runs execution-based validation and keeps instances where at least one test changes from failing to passing.
Your definition can be stricter or looser, but it has to be written down before anyone starts grading. The commonest source of an inflated count is a definition that quietly drifts from "has a test" to "touches a test file" halfway through a sample.
Which unit should you sample: commits, pull requests or issues?
Use merged pull requests or merge requests on the default branch as the sampling frame, because that is where a requirement, a review and a final change meet. Commits are too fine-grained and issues include work that never shipped; both are better used as joins than as the unit you count.
At a glance
Which unit should you sample: commits, pull requests or issues
Three platform behaviors decide how the frame is built. GitHub links a pull request to an issue through closing keywords such as closes, fixes and resolves only when the pull request targets the default branch, and a keyword in a commit message closes the issue without listing the pull request as linked. Squash merging, if your repository used it, loses the original commit timestamps and authors inside each pull request. And in plain Git, following only the first parent of each merge gives a mainline view that ignores commits brought in from side branches.
- Frame: every pull request merged into the default branch between two dates, one row per pull request, with merge date, merge method and base commit.
- Join 1: issue or ticket keys found by closing keywords, manual links, branch names or a ticket key in the title.
- Join 2: review events (approve, request changes, comment) and the CI runs attached to the head commit.
- Join 3: deployment or release records that reference the merge commit, if they exist.
- Exclusions decided up front: bot-authored dependency bumps, release-version commits, reverts and changes to vendored third-party code.
The task completeness sampling and extrapolation worksheet
The worksheet turns the method into lines a team can fill in. Each line names the input, where it comes from, the rule that applies and what to record so a second person can audit the result. Keep it in a spreadsheet next to the sample list itself.
| Line | Input or step | How to obtain it | Rule or formula | What to record |
|---|---|---|---|---|
| 1 | Window and frame size per year (N_y) | Platform API or export of merged pull requests on the default branch | Count after the up-front exclusions, per calendar year | Query text, export date, N_y for each year |
| 2 | Sample size per year (n_y) | Chosen before drawing | At least 60 to 100 per year; equal sizes per year keep year intervals comparable | n_y and the random seed |
| 3 | Random draw | Seeded random selection from each year's frame | No hand-picking; replace a drawn item only if it is a frame error | Sample list with pull request numbers |
| 4 | Ladder grade per item | Pull request page, linked issue, review events, CI logs | Assign the highest rung fully met (see ladder) | Grade, evidence links, grader initials |
| 5 | Replay of rung 4 items | Rebuilt environment at the base commit | Test fails before the patch and passes after | Pass, fail or not rebuildable, with the log |
| 6 | Verified count per year (x_y) | Lines 4 and 5 | Rung 5 and above only | x_y |
| 7 | Rate and interval per year | x_y and n_y | p_y = x_y / n_y with a Wilson 95% interval | p_y, lower and upper bound |
| 8 | Extrapolated tasks per year | Lines 1 and 7 | N_y times p_y, and N_y times each bound | Point estimate and range |
| 9 | Window total | Sum of line 8 across years | Report the sum of points and the sum of bounds as a rough range | Total with an explicit note that the range is approximate |
| 10 | Second-grader check | A different person regrades a subset | Regrade at least 20 items; log disagreements | Agreement count and how disputes were settled |
The completeness ladder for each sampled change
Grade each sampled pull request on a ladder and count only the top rungs. A ladder keeps partial records visible instead of throwing them away, and it shows exactly which link is missing most often.
Review evidence has a precise meaning on most platforms. On GitHub a reviewer submits one of three decisions, comment, approve or request changes, and repository administrators can require approvals before merging. An approval with no comment is still a decision; a merge with no review event at all is not.
| Rung | Evidence required | Counts as complete? | Most common reason items stop here |
|---|---|---|---|
| 0 Not a task | Bot update, release bump, revert, vendored code or formatting-only change | No, and excluded from the frame if detectable up front | Automation accounts not filtered before sampling |
| 1 Change only | Merged code change with no traceable requirement | No | Work tracked in chat or meetings, never in a ticket |
| 2 Linked requirement | Rung 1 plus an issue or ticket written before the work | No | No test was added or changed with the fix |
| 3 Test evidence | Rung 2 plus a test added or changed in the same change | No | Merged without any review event |
| 4 Reviewed | Rung 3 plus a recorded review decision | Candidate, pending replay | Environment cannot be rebuilt at the base commit |
| 5 Replay verified | Rung 4 plus fail-before, pass-after confirmed by a second person | Yes | Not applicable |
| 6 Outcome evidenced | Rung 5 plus deployment, release or production record tied to the merge | Yes, reported separately | Deploys not linked to commits |
Can a separate reviewer rebuild the environment and verify the claimed result?
Only replay tells you whether a reviewed change is a usable task, so replay every rung 4 item in the sample rather than assuming it. The reviewer checks out the base commit, installs the dependencies that existed then, runs the new or changed tests without the patch, applies the patch and runs them again.
This is the expensive step, and public benchmark builders say so. The SWE-bench tutorial calls environment specification the most manual part of collection: each task needs a version and version-specific installation instructions, and the team may have to iterate between installation and validation before candidates install properly.
Record the outcome as pass, fail or not rebuildable, with the log. A not-rebuildable result is not a judgment on the engineering; it usually means pinned dependencies are gone, a private package registry was retired or the tests needed a service that no longer exists. Report those counts separately so a buyer can see how much of the history might come back with more environment work.
- Pin the toolchain and dependency versions from the lockfile at the base commit, not today's.
- Run the full changed test file, not only the named test, and keep both logs.
- Mark flaky results: a test that passes before the patch on any run disqualifies the item.
- Have a person who did not grade the item run the replay.
Worked scenario (illustrative): three years of a field service product
The scenario below is invented to show the arithmetic. A field service software company frames three years of merged pull requests, 1,600 per year after exclusions, and samples 70 per year. Graders and a separate replay reviewer find 8, 17 and 19 verified tasks. Continuous integration was only switched on midway through the first year, which is why that year is thin.
Across the 210 sampled items the ladder distribution was 49 not a task, 47 change only, 33 linked requirement, 14 test evidence without review, and 67 reviewed candidates, of which 44 replayed cleanly and 23 failed or could not be rebuilt. Twelve of the 44 also had a deployment record tied to the merge commit. All counts and percentages here are illustrative.
The all-years row pools the three samples, which works here because every year has the same frame and sample size; summing the three year ranges instead gives the rougher window range of about 636 to 1,520 that line 9 of the worksheet describes.
| Year (illustrative) | Frame and sample | Verified in sample | Rate and Wilson 95% interval | Extrapolated complete tasks |
|---|---|---|---|---|
| Year 1 | 1,600 merged; 70 sampled | 8 | 11% (6% to 21%) | about 183 (94 to 335) |
| Year 2 | 1,600 merged; 70 sampled | 17 | 24% (16% to 36%) | about 389 (252 to 568) |
| Year 3 | 1,600 merged; 70 sampled | 19 | 27% (18% to 39%) | about 434 (290 to 617) |
| All three years | 4,800 merged; 210 sampled | 44 | 21% (16% to 27%) | about 1,006 (768 to 1,294) |
Which assumption moves the count most, and how should uncertainty be shown?
The definition of test evidence moves the count more than any statistical choice. In the scenario, accepting any change that touches a test file would have pulled the 14 unreviewed items and many of the 23 replay failures into the total; a strict fail-before, pass-after rule keeps them out. State which rule you used in the first line of the report.
At a glance
Which assumption moves the count most, and how should uncertainty be shown
Show uncertainty as an interval for each year, not one rounded total. The NIST Engineering Statistics Handbook presents the Wilson method for a proportion, which behaves better than the simple normal approximation when the rate is small or the sample modest, both common here. Summing year intervals gives only a rough window range; say so rather than presenting it as exact.
- Linking rule: keyword links only, or manual sidebar links and ticket keys in titles too.
- Review rule: any review event, or only an approval or request-changes decision.
- Squash merges: whether the pull request, not the commit, is the unit for those years.
- Monorepos: whether one pull request touching several services counts once or per service.
- Replay: how many attempts a not-rebuildable item gets before it is recorded as such.
- Grader drift: the share of regraded items where two people disagreed, and how that was settled.
When to narrow the window instead of reporting one number
If one year's interval is wide and low, report that year separately or drop it, rather than averaging it into a healthier window. Process changes such as adopting CI, a new tracker or required reviews usually show up as a step in the ladder distribution, and the years after the step are the ones a buyer can actually evaluate.
Some changes reach rung 5 without a ticket: a pull request description written as a clear requirement, with a reproduction and a failing test, can serve the same role. Decide whether to accept that before sampling and apply it to every item. Changes to security fixes, customer-specific branches and code you do not own the rights to belong in the exclusion list, and secrets found during grading need to be rotated and scrubbed before any further review, a step covered in the secrets-scanning guide linked below.
- Stop and narrow if fewer than one in ten sampled items in a year reaches rung 4.
- Stop and get technical help if fewer than half of rung 4 items can be rebuilt at all.
- Bring in counsel if contributors, contractors or open source licenses make it unclear who can license which code.
Bringing an engineering history task count to a licensing conversation
A sampled, replay-checked count is the evidence a buyer asks for first, and it is also what tells a company whether its engineering history is worth preparing at all. SourceX helps software companies turn selected engineering records into licensed datasets, covering qualification, rights review, contracting, preparation, delivery and payment, and a task density estimate fits naturally into that qualification step.
On volume, the working guide is years of consistent history, typically two or more, with tens of thousands of records in one system, though rarer kinds of work can qualify with less; each dataset is assessed on its own. Qualification begins with a short assessment and no data upload, and no code or ticket leaves the company until it has approved the scope and terms and signed the agreement.
Frequently asked questions
Why not just count merged pull requests that reference an issue?
Because a reference proves only that two records are connected. Many referenced changes have no test that demonstrates the fix, no review decision, or cannot be rebuilt today. The reference count is a useful upper bound for planning, but the verified count after grading and replay is the number to report.
How large should the sample be?
Sixty to one hundred pull requests per year is a practical starting point for most teams, and equal sizes per year make the year intervals comparable. If an interval is too wide to support a decision, draw more from that year using the same seed procedure rather than starting over. The Wilson interval in the worksheet shows directly how much a larger sample would narrow the range.
Does a squash-merged history have fewer tasks?
Not necessarily. Squash merging collapses each pull request into one commit and loses the original commit times and authors, but the pull request, its review and its linked issue usually survive on the platform. Count pull requests as the unit for squash-merged years and note the lost commit detail as a limitation.
What if the issue tracker and the code host are different systems?
Join them on ticket keys found in pull request titles, branch names and descriptions, and record the join rule in the worksheet. Measure how often the join succeeds per year before sampling, because a year with broken key conventions will grade low on the ladder for reasons that have nothing to do with engineering quality.
Can automated tools grade the ladder for us?
They can pre-fill rungs 1 to 4 from platform data, such as linked issues, test file changes and review events. Replay and the second-grader check still need people, because the question is whether a stranger can verify the result, and that is exactly what an automated pre-fill cannot show.
Sources
- The SWE-bench collection code treats a pull request as valid only if it was merged and resolves at least one issue, and checks whether each instance has a test patch. Source
- SWE-bench detects linked issues through closing keywords and separates test hunks from code hunks by file path. Source
- SWE-bench writes instances with an issue and gold patch to one file and those that also have tests to a candidate file for evaluation. Source
- The SWE-bench collection tutorial describes versioning and installation configuration as the most manual step and keeps instances that solve at least one test after execution-based validation. Source
- GitHub links pull requests to issues through closing keywords only when the pull request targets the default branch; keywords in commit messages close issues without listing the pull request as linked. Source
- GitHub notes that squash merging loses information about when changes were originally made and who authored the squashed commits. Source
- GitHub reviewers submit comment, approve or request changes decisions, and administrators can require approvals before merging. Source
- When a GitHub Actions workflow references an environment, the environment appears in the repository's deployments. Source
- Git's first-parent option follows only the first parent of each merge commit, giving a mainline view of history. Source
- The NIST Engineering Statistics Handbook presents the Wilson method for confidence intervals on a proportion. Source
See if your company qualifies
A short company assessment. No data uploads are needed.