Skip to content

AI uses for records

Bug resolution customer outcome dataset: 785 fixes that did not hold

By SourceX Editorial · Updated

10 sections

Short answer

A bug resolution customer outcome dataset pairs each engineering closure with what the customer saw next: a reopen, a support case filed after the fix, or a new report naming the closed one. Public records show where that evidence hides. Of 785 CVE records calling an earlier fix incomplete, only 23 (2.9%) are named by the record they correct, so the join has to start from the later record.

Grader rubric

Bug resolution customer

This page covers bug resolution customer outcome dataset: 785 fixes that did not hold.Illustrative

Key takeaways

  • A closure is the engineering team's claim; the customer outcome has to be joined from records written later, often in another system.
  • Persistence shows up in four kinds of record: a reopen, a post-fix support case, a successor report and a regression report, and only the reopen sits on the original ticket.
  • In the CVE List, 785 follow-ups name the record whose fix was incomplete, while only 23 are named by their originals and 90.3% link back only in prose.
  • Publication dates are a poor clock: 27 follow-ups predate their originals, so outcome windows should start at the release that carried the fix.
  • Build the join backward from later records, store each link's provenance, and award held only when the window and link coverage both pass.

What is a bug resolution customer outcome dataset?

It is a set of engineering closures, each joined to what the customer experienced afterward: the ticket's own status history, support cases opened after the fix shipped, and any later report that names the closed one. The closure is the engineering team's claim that the work is finished. The outcome is evidence from the people who were supposed to stop seeing the problem.

Datasets cut from issue trackers usually stop at the closure. A pull request merges, a test turns green, the ticket moves to Done, and the row is labeled fixed. That label records a workflow step. Whether the customer's problem ended is a separate fact, and it sits in records written later, often by other people in another system.

This page is about those later records: which kinds exist, which way their links point, and how a buyer should join them to the closure. It counts one large public example, the CVE List, where 785 records state that an earlier fix was incomplete, and turns the result into an outcome rubric and a join method for private engineering and support histories.

Where does a problem that outlived its fix get recorded?

In four kinds of record, and only one of them sits on the original ticket. A reopen changes the closed record itself. A support case filed after the fix lives in the help desk and may or may not point at the engineering issue. A successor report is a new engineering record that names the old one, usually in its description. A regression report describes the fixed behavior breaking again, often with no reference to the earlier ticket at all.

The distinction matters because most extraction jobs walk forward: start from closed issues, follow their links, collect whatever hangs off them. That catches reopens, which are status events on the same record. It misses successors and late support cases unless someone went back and added a link to the closed ticket, and the public count below shows how seldom anyone does.

The table sets out the four record types, who writes each, and the direction its link usually runs.

Where does a problem that outlived its fix get recorded?
Record that shows persistenceWritten by, and whereWhich way the link runsWhat it can proveWhat it misses
Reopen of the closed ticketAn engineer or an automation rule, in the issue trackerNo link needed: same record, new status eventThe team itself judged the closure prematureProblems the team never reopened, and reopens made only to edit a field
Support case after the fix shippedA support agent, in the help deskCase to issue, when the agent links it or types the keyA customer on a fixed version still saw the symptomCustomers who gave up, and cases filed with no link or key
Successor reportOften a different team or an outside reporter, in the trackerNew record to old, usually in proseThe fix missed a code path, a configuration or a versionSuccessors that describe the symptom without naming the old record
Regression report after a later releaseMonitoring, QA or a customer, in the tracker or an error toolOften none; matched later by signature or stack traceBehavior that was fixed broke againWhether the first fix or a later change caused the break

Backward, nearly always, and in prose. In the CVE List snapshot of October 27, 2026, 785 published records say in their own description that an earlier CVE's fix was incomplete, naming 777 distinct earlier records in 830 pairs. Every follow-up names its original, since that sentence is how the count found them. Only 23 follow-ups (2.9 percent) are named anywhere in the current record of the original they correct.

Structured links are rarer still. The CVE record format includes a reference tag, related, for "a related (but not the same) vulnerability." Four of the 830 pairs use it to point from follow-up to original, and none uses it to point from original to follow-up. In 709 of the 785 follow-ups (90.3 percent), no reference URL or reference name contains any original's ID, in upper or lower case; the connection exists only as a sentence of English text. The other 76 also cite an original in a reference.

Ownership does not close the gap. In 559 follow-ups (71.2 percent), the organization that published the follow-up also published an original, yet in only 19 of them (3.4 percent) does an original name the successor. The newest records show the same habit: none of the 205 follow-ups published in 2025 or 2026 is named by its original so far.

For engineering history the consequence is mechanical. A crawler that starts at closed records and reads them would have reached 23 of these 785 successors. A crawler that starts at the later records and reads their text reaches all of them, because that is where the link was written. The table gives each count with the rule the script applied.

785 public follow-ups: which way do the links point?
What was countedCountShareRule the script applied
Published CVE records in the snapshot396,732BasecveMetadata.state is PUBLISHED at commit f034768 of CVEProject/cvelistV5
Follow-up records7850.2% of publishedOne sentence of the English CNA description holds the word incomplete, a fix word and another CVE ID not directly after see
Follow-ups named by their original232.9% of 785The follow-up's ID appears anywhere in an original's current JSON, references and ADP data included
Pairs joined by a reference tagged related4 backward, 0 forward0.5% and 0.0% of 830A CNA or ADP reference carries the tag related and its URL or name holds the other ID
Follow-ups whose references name no original70990.3% of 785No CNA or ADP reference URL or name contains any original's ID, matched in any case
Follow-ups from an original's own CNA55971.2% of 785Same assignerShortName as at least one original; 19 of these are named by an original
Originals with two or more follow-ups445.7% of 777Named by at least two follow-up records; the most is three
Two opposing bars for 785 CVE follow-up records: all 785 name the earlier record whose fix was incomplete, while only 23 are named by their earlier record; 709 follow-ups make the link only in description text and 76 also in a reference.
Link direction between 785 CVE follow-up records and the records they correct. Source: CVE List (CVEProject/cvelistV5), snapshot of October 27, 2026; counting rules in the methods note.

Three fixes that did not hold, and one record that pointed ahead

Apache's Log4j records are the familiar case, and one of the few linked in both directions. CVE-2021-45046, published December 14, 2021, opens: "It was found that the fix to address CVE-2021-44228 in Apache Log4j 2.15.0 was incomplete in certain non-default configurations." The current record for CVE-2021-44228 mentions CVE-2021-45046, and its description covers Log4j2 2.0-beta9 through 2.15.0, noting that 2.15.0 only disabled the behavior by default and that 2.16.0 removed it.

Rust's records show the more common pattern. CVE-2024-24576, published April 9, 2024, covered argument escaping when the standard library invoked Windows batch files. CVE-2024-43402, published September 4, 2024, says that fix was incomplete: before Rust 1.81.0 it could be bypassed when the batch file name ended in whitespace or periods, which Windows strips. The April record, as it stands today, does not mention the September one, so a dataset labeling the first fix from its own record would still call it a clean closure.

CPython's 2026 cookie records repeat the shape within a single quarter. CVE-2026-0672, published January 20, describes a patch that rejects control characters in http.cookies.Morsel names, values and parameters. CVE-2026-3644, published March 16, says the Morsel.update(), |= operator and unpickling paths were not patched. The January record does not name the March one.

The rare forward pointer looks like Apache HTTP Server's CVE-2021-41773, whose description ends: "The fix in Apache HTTP Server 2.4.50 was found to be incomplete, see CVE-2021-42013." One sentence turns a closed record into a route to its successor. The successor itself calls the 2.4.50 fix insufficient, a word the counting rule does not accept, which is one reason the 785 is a floor rather than a census.

Why publication dates make a poor clock for customer outcomes

Because the day a record is published is not the day the fix shipped or the day a customer noticed it failed. Measured between publication dates, the median follow-up record appeared 112.4 days after its original's record, and 219 of the 785 (27.9 percent) appeared more than a year later. Yet 27 follow-ups were published before the record they say was incompletely fixed, and 93 carry exactly the same timestamp as their original.

Private systems carry the same trap in a smaller form. An issue's created time, its resolved time, the merge of the fix, the release that carried it and the day a customer installed that release are five different clocks. An outcome window measured from the wrong one either closes before the customer could have met the fix or stays open long after the question was settled.

Start the window at the release or deployment that carried the fix, and treat the customer's first contact after installing that release as the earliest possible evidence of failure. Closures whose window has not passed on the export date stay unlabeled; the support outcome measurement lag page covers how to size that wait from a company's own follow-up timing.

Six outcome labels for a closed engineering ticket

Label each closure by the strongest later evidence found, and record where the join started. The rubric keeps the engineering claim apart from the customer-side outcome, and it keeps "nothing was found" apart from "nothing could have been found".

When several labels fit, apply them in this order: reopened in place, successor filed, customer still affected, window still open, outcome unknowable, held. A closure earns held only after its window passed and the link coverage check for its period passed, which makes held the hardest label to earn rather than the default.

The table defines each label, the evidence it needs and what it can teach:

Six outcome labels for a closed engineering ticket
LabelEarned whenJoin starts fromWhat a model can learn
HeldThe window, fixed before labeling and measured from the fix's release, passed with no reopen, successor or post-fix customer case, and coverage for the period was checkedThe closed ticket, then a search of every later recordWhat a fix that worked looked like, with a known observation time
Reopened in placeThe ticket's own history moves from a closed status back to an open one, with a comment or linked evidenceThe ticket's status historyWhich closure notes and patches came before a premature close
Successor filedA later engineering record names the closed one and says its fix missed a path, configuration or versionThe later record's text and link fieldsHow to read a new report as a sequel and write the missing test
Customer still affectedA support case citing the symptom arrives after the release, from an account already on a fixed versionThe help desk, matched by link, issue key or symptomThe gap between a passing test and what the customer saw
Window still openThe fix shipped too recently for its window to passRelease date against export dateNothing yet; held back until it matures
Outcome unknowableLater records exist but carry no link, key or signature that could connect themThe coverage audit for the periodNothing; counted so a buyer sees how much history could not be labeled

How to build the join backward, from the later record to the closed one

Start from every record created after a closure and work back, because that is where the connection gets written. The steps fit any pairing of tracker and help desk, such as Jira with Zendesk or GitHub Issues with Salesforce Service Cloud. Each step records how a link was found, so a buyer can weigh a link an agent typed above one inferred from similar wording.

Item five is the one most often skipped. The derived dataset should carry the forward links the source systems never got, so each closed ticket travels with its successors while the company's own records stay exactly as they were written.

Checklist · 6 items

  1. Collect every record created after the earliest closure in scope: support cases, new issues, reopen events, incident records and release notes, each with its event timestamps.
  2. Extract earlier identifiers from each record: link fields, issue keys matched against the tracker's key pattern, URLs, commit hashes, error signatures and quoted error messages.
  3. Resolve each identifier to a closed record and store the pair in a link table with its provenance: field link, text mention or inferred match, plus the reviewer who confirmed any inferred match.
  4. Classify each successor against its closed record: same defect not fully fixed, regression caused by the fix, separate defect with the same symptom, or no relation after review.
  5. Write the forward link into the dataset, never back into the source system.
  6. Report link coverage per period, meaning the share of later records holding any resolvable identifier. A period with weak coverage cannot produce held labels.

0 of 6 ticked. Tick each item as you go.

The strongest objection: most closures hold, and CVEs are not tickets

Both halves are true, and neither removes the problem. Follow-ups are uncommon: 785 of 396,732 published CVE records meet the rule, about 0.2 percent, and the rule undercounts because it skips words such as insufficient and bypass. A closure dataset will be mostly held labels, and its value sits in the minority that did not hold.

Security records also follow conventions of their own. The CVE Program's numbering rules leave the choice to the record keeper; rule 4.2.8 in version 4.0 states that CNAs "MAY determine that residual insecurity left by an incomplete Fix is a new Vulnerability and MAY assign a new CVE ID or MAY use an existing CVE ID." Where persistence surfaces is a record-keeping policy before it is an engineering fact.

Software companies face the same policy question, reopen the ticket or file a new one, and few write the answer down. That argues for the backward join rather than against it. When the policy is unknown, a dataset cannot assume persistence lands on the closed record. It also helps to remember who notices: in 226 of the 785 follow-ups, the successor came from a different organization than the original, the public analogue of a customer reporting what the engineering team believed it had finished.

How a buyer would use closure-outcome records, and what to request

Four uses justify the extra joining work. Closure-risk prediction learns which patches, test changes and closure notes precede a fix that fails. Successor detection trains a model to decide whether a new case or issue is the sequel to a closed one, the triage step the CVE corpus shows people rarely record. Test generation turns a successor report into the regression test the first fix lacked. Agent evaluation can score a coding agent's fix by whether a successor appeared, not only by tests that pass.

A request for this material should name the systems on both sides, the years, the observation window per product line, the outcome labels above and the link provenance field. Ask for the supplier's reopen-or-new-ticket rule for each period and a link coverage report, and exclude customer identities, credentials, and security defects that are not yet fixed and disclosed.

SourceX does not keep datasets on a shelf; its request page asks AI teams for scope, volume, language, format, timing and licensing needs, and sourcing starts from what the request describes. A closure-outcome request can also state its acceptance test, for example a reviewer re-deriving the labels for a sample of closures from the linked records alone.

Methods note: CVE List snapshot, counting rules and limits

Data: CVEProject/cvelistV5, cloned on October 27, 2026 at commit f0347685232c (committed 21:46 UTC that day). All 415,397 record files were read; 396,732 have state PUBLISHED. Scripts in the writer's scratch folder (cve_extract.py, cve_followups.py and cve_reftags_v2.py) read only the JSON files through the file system or git cat-file; no repository code was run. The schema tag text comes from CVEProject/cve-schema at commit ce5f5c8.

Follow-up rule: the text is the first English description in the CNA container, split into sentences at every period followed by whitespace. A record is a follow-up when one sentence contains the whole word incomplete (any case), one of the whole words fix, fixes, fixed, patch, patches, patched or remediation (a hyphen counts as a word boundary), and a CVE ID other than the record's own; only the first such sentence is used. Every other CVE ID in that sentence is an original, except an ID directly after the word see. One record was left with no original and was dropped and printed: CVE-2021-41773. Direction comes from this wording, not from dates.

Link rules: an original names its follow-up when the follow-up's ID appears anywhere in the original's current JSON text, escapes decoded, which is generous to forward linking. Reference checks cover CNA and ADP references by URL or name, matching IDs in any case, and the related tag is the schema's reference tag of that name. Same CNA compares assignerShortName. Gaps are exact differences between the datePublished timestamps of the follow-up and of its earliest-published original, with negative values kept. Shares are integer ratios rounded half up to one decimal, once; medians are taken over exact gaps and rounded once.

Not measured: follow-ups worded without incomplete (insufficient, bypass, variant, regression); links held only in vendor advisories outside the record; earlier versions of each record, since the snapshot shows current text; and anything about support systems, which no public corpus joins to these records. A few sentences that name several IDs include a related issue that was not itself incompletely fixed (CVE-2007-5045 is one); treating such IDs as originals can only raise the forward-link count. Saved outputs hold CVE IDs, dates and flags, with no names or description text.

Frequently asked questions

What is the difference between a closed bug and a resolved customer problem?

A closed bug is a status the engineering team set, usually when a fix merged or shipped. A resolved customer problem is an outcome: the people who reported it stopped seeing it on a fixed version over a stated window. A dataset needs both, recorded separately, because the first is easy to export and the second has to be joined from later records.

Why not just count reopened tickets?

A reopen captures only persistence that someone recorded on the same ticket. Successor reports, post-fix support cases and regressions are separate records, and their links usually point back to the closed ticket rather than forward from it. In the CVE List count, only 23 of 785 follow-ups were named by the record they corrected.

How long should the window after a fix be before labeling a closure as held?

Set it from your own history: measure how long successors and post-fix cases took to appear, counted from the release that carried each fix, and choose a window that covers most of them. Public CVE publication dates are too loose to borrow; their median gap was 112.4 days, but 27 follow-ups were published before their originals.

Can successor links be recovered from old data that has no link field?

Often, yes. Later records frequently cite an issue key, URL, commit hash or error signature in their text, and a script can resolve those to the closed record. Store each recovered link with its provenance, field link, text mention or inferred match, and report the coverage per period so a buyer knows where held labels can be trusted.

What should be removed before closure-outcome records are licensed?

Customer names and account identifiers, credentials and secrets pasted into tickets, and details of security defects that are not yet fixed and disclosed. The company also needs the right to license the records, which depends on its customer contracts and on whether third-party code or content appears in them; counsel reviews those questions for each dataset.

Sources

  • The CVE List repository snapshot (commit f034768, October 27, 2026) holds 415,397 record files, 396,732 published; the counts of follow-ups, forward links, reference links, CNAs and dates come from it. Source
  • CVE-2021-45046 (Apache, published December 14, 2021) says the fix for CVE-2021-44228 in Log4j 2.15.0 was incomplete in certain non-default configurations. Source
  • The current CVE-2021-44228 record names CVE-2021-45046 and describes Log4j2 2.0-beta9 through 2.15.0, with 2.15.0 disabling the behavior by default and 2.16.0 removing it. Source
  • CVE-2024-43402 (published September 4, 2024) says the fix for CVE-2024-24576 was incomplete and could be bypassed before Rust 1.81.0 with trailing whitespace or periods in a batch file name. Source
  • CVE-2024-24576 (published April 9, 2024) covers the Rust standard library's escaping of arguments to Windows batch files and does not name CVE-2024-43402. Source
  • CVE-2026-3644 (published March 16, 2026) says the fix for CVE-2026-0672 was incomplete: Morsel.update(), the |= operator and unpickling paths were not patched. Source
  • CVE-2026-0672 (published January 20, 2026) describes a patch rejecting control characters in http.cookies.Morsel and does not name CVE-2026-3644. Source
  • CVE-2021-41773's description ends by saying the fix in Apache HTTP Server 2.4.50 was found to be incomplete, see CVE-2021-42013. Source
  • CVE-2021-42013 says the fix for CVE-2021-41773 in Apache HTTP Server 2.4.50 was insufficient. Source
  • The CVE record format's reference tag related marks a reference for a related (but not the same) vulnerability. Source
  • CNA Operational Rules version 4.0, rule 4.2.8: CNAs may treat residual insecurity from an incomplete fix as a new vulnerability and may assign a new CVE ID or use an existing one. Source
  • SourceX's request page says datasets are sourced on request, not held in stock, and asks for scope, volume, language, format, timing and licensing. Source

Tell us what data you need

Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.

See if your company qualifies