AI uses for records
Reopened support case labels: grade the follow-up, not the flag
By SourceX Editorial · Updated
9 sections
Short answer
A reopen flag does not tell you whether the original support decision was wrong. In 469,656 first decisions from the public Eclipse and Mozilla bug trackers, 52.9% and 54.5% of reopened reports ended with their first label, while in Mozilla 60.2% of label changes happened with no reopen at all. Link each follow-up to the decision it questions, classify why it arrived, and version the label instead of overwriting it.
Agent run
Reopened support case labels
Key takeaways
- About half of reopened bug reports kept their first resolution: 52.9% in Eclipse and 54.5% in Mozilla, so a reopen count overstates reversed decisions.
- Labels also change silently: 12,681 of 21,082 changed or unresolved Mozilla decisions never passed through a reopen.
- No-action decisions are overturned most: 10.9% of Eclipse WORKSFORME closures changed, against 1.5% of FIXED, and 667 went from WORKSFORME to FIXED after a reopen.
- In Zendesk a solved ticket reopens on a new requester comment, including a thank-you, while a reply to a closed ticket spawns a separate follow-up ticket; collect both paths.
- Grade the original decision on what its record held at the time, version every label change, and report scores against both frozen and latest labels.
Does a reopen flag tell you the first decision was wrong?
Not by itself. A reopen says someone looked at the case again; it does not say the first decision was wrong, and plenty of decisions change without any reopen at all. To grade the original decision, a dataset has to link each follow-up to the decision it questions, record why the follow-up arrived, and keep both versions of the label.
The point matters because a reopened case is the most common shortcut to a negative label in support data: count the reopens, call them failures, train. Public bug-tracker histories show how far that shortcut drifts. In Eclipse, about half of the reopened reports ended with the resolution they started with, and in Mozilla most label changes never passed through a reopen.
Two live guides on this site cover neighboring questions: telling a closed ticket from a finished job, and how long to wait before labeling an outcome. Both ask when an outcome is mature. This page asks what a follow-up actually says about the decision that came before it, and how to re-grade that decision without corrupting a held-out set.
Two ways a helpdesk records a second look
Most systems record a second look either by reopening the original record or by opening a new one that points back to it, and a dataset needs both. In Zendesk, a requester's new comment on a solved ticket reopens it; the help center's own example is a requester who disagrees that the issue was resolved. Once a ticket is closed, any reply from any channel instead creates a separate follow-up ticket, which carries over most of the original's details and tags but not its comments, priority, type, assignee or organization.
Those two paths split one story across records. A count of reopened tickets misses every follow-up that arrived after closure, and a count of follow-up tickets misses every reopen inside the solved window. The comment path also catches courtesy: since any new requester comment on a solved ticket reopens it, a customer who writes back only to say thanks can produce a reopen even though nothing about the outcome changed.
Bug trackers formalize the same idea with more states. Bugzilla's built-in field help describes RESOLVED as a resolution awaiting verification by QA, from which a bug is either reopened or verified, and VERIFIED as QA agreeing that the appropriate resolution was taken. Its definition of WORKSFORME ends with a sentence any support team will recognize: if more information appears later, the bug can be reopened.
469,656 first decisions, and how often a later event changed them
For 4.0% of Eclipse decisions and 6.3% of Mozilla decisions, the final label differed from the first or the report was back open at the snapshot, and the reopen flag tracked those changes poorly. The data is the Eclipse and Mozilla Defect Tracking Dataset (Lamkanfi, Perez and Demeyer, MSR 2013), which stores every change to a report's status and resolution with a timestamp and a user id. Its JSON files hold 165,547 Eclipse and 394,878 Mozilla reports opened between 1994 and the start of 2013.
The counting rules, exactly as the script applies them at commit f508423: the first decision is the first non-empty value in a report's resolution history. A report counts as reopened when its status history holds a REOPENED entry at or after that moment. The final label is the resolution in the dataset's snapshot, where an empty value means unresolved. LATER and REMIND are left out of the headline figures because neither appears among either snapshot's current resolutions, so their changes mostly reflect retired codes rather than second decisions. Every report is processed and nothing is sampled, leaving 135,438 Eclipse and 334,218 Mozilla first decisions.
Three findings follow. Reopening is common but usually confirms rather than reverses: 10,799 Eclipse decisions (8.0%) and 18,444 Mozilla decisions (5.5%) were reopened, and 52.9% and 54.5% of those ended with the original label. Labels also change quietly: of the 21,082 Mozilla decisions whose label ended up different or unresolved, 12,681 (60.2%) never passed through a reopen, against 383 of 5,471 (7.0%) in Eclipse. And most reopenings come fast, with a median gap of 1.0 day in Eclipse and 0.9 days in Mozilla, although 5.8% in each project came more than 180 days later.
Who reopened matters as well. The person who filed the report did the reopening in 47.8% of Eclipse reopenings and 33.8% of Mozilla's; the rest came from other accounts, such as developers, triagers and QA. Verification was not final either: 639 Eclipse and 2,007 Mozilla decisions had been marked VERIFIED before they were reopened.
No-fix decisions are the ones follow-ups overturn
Decisions that declined to act were overturned far more often than fixes. In Eclipse, 10.9% of first WORKSFORME decisions, 9.4% of INVALID and 9.1% of WONTFIX ended with a different label or unresolved, against 1.5% of FIXED. Mozilla ranks its no-fix codes the same way, with WONTFIX at 12.4% and INVALID at 10.3%, though its FIXED decisions changed more often (5.8%), much of it through 3,236 quiet relabels from FIXED to WORKSFORME.
The most telling single flip is WORKSFORME to FIXED after a reopen: 667 times in Eclipse and 908 in Mozilla, a judgment that nothing could be reproduced followed by an actual fix. Support teams make the same call under other names, such as could not reproduce, no fault found, working as designed or user error. These are the closures a model learns to make when fast closing is rewarded, and they are the ones a follow-up is most likely to contradict.
Each standard first decision, its count, and the share whose final label differed or stayed unresolved at the snapshot:
| First decision (Bugzilla definition) | Eclipse decisions | Eclipse final label differs | Mozilla decisions | Mozilla final label differs |
|---|---|---|---|---|
| FIXED (a fix is checked in and tested) | 73,719 | 1.5% (1,129) | 118,409 | 5.8% (6,866) |
| DUPLICATE (duplicate of an existing bug) | 25,843 | 3.0% (780) | 90,760 | 3.9% (3,516) |
| WORKSFORME (attempts to reproduce failed) | 14,741 | 10.9% (1,610) | 46,835 | 7.7% (3,615) |
| WONTFIX (a bug that will never be fixed) | 10,431 | 9.1% (948) | 13,084 | 12.4% (1,622) |
| INVALID (the problem is not a bug) | 9,933 | 9.4% (931) | 37,518 | 10.3% (3,869) |
Six follow-up types and what each does to the original grade
Classify every follow-up before it touches the original label. The type decides whether the first decision is downgraded, confirmed, or left alone while a new case opens, and the evidence column says what a reviewer must see before choosing it.
Two rules sit underneath the table. The original decision is judged on what the record held when it was made, so a follow-up carrying genuinely new information leaves the first grade standing and starts a new decision. And a type that cannot be established from the record is coded unknown, never guessed; an unknown keeps the case out of both the success and the failure sets until someone settles it. The six types:
| Follow-up type | Evidence that identifies it | Effect on the original decision's grade | Example |
|---|---|---|---|
| Same problem persists after an action | Follow-up names the same order, asset or error, and the promised state is missing from the system of record | Downgrade to failed: the first resolution did not hold | A refund was confirmed, but the card statement shows no credit |
| No-action decision contradicted | A later fix or credit on the same issue, plus proof the deciding evidence was already in the record | Downgrade to failed (missed); if the evidence arrived later, keep the grade and tag new information | No fault found, then the same error log returns and a patch ships |
| Confirmation or courtesy reply | Comment carries thanks or confirmation, no new request, no state change | No change; counts as support for durability | A reply saying that worked reopens a solved ticket |
| New, unrelated request | Different product, order or issue key on the old thread or a follow-up ticket | Split into a new case; original grade unchanged | A shipping question arrives as a reply to last month's billing ticket |
| Changed circumstances | An event dated after the decision: a new order, a policy change, a change of mind | Original grade unchanged; record the context change | The customer cancels a plan they had asked to keep |
| Administrative relabel | Bulk change, merge or retired code, with no customer contact | Map the codes; no grade change | A retired defer code remapped in one pass |
One bug, two labels: a versioned record from Eclipse
Eclipse bug 10168 shows what a versioned label looks like and where the evidence runs out. Filed against the Java development tools on 25 February 2002, it reported a null pointer exception in the externalize strings dialog. Three days later user 8 resolved it WORKSFORME. On 25 March the reporter reopened it, and the next day user 8 assigned it and resolved it FIXED.
Under the rubric, the first label cannot simply be overwritten. The status history proves a link, because it is the same report reopened by the person who filed it, and it proves a reversal of direction, from no defect to a fix. What it cannot prove is attribution. The dataset holds no comment text, so a grader cannot tell whether the reporter added steps that made the crash reproducible, which would leave the WORKSFORME call reasonable on the evidence it had, or whether the original report already contained them. The case stays coded unknown until the thread is read.
The packet keeps each version instead of the final state alone:
| Date (UTC) | Event in the record | Label version | What a grader may conclude |
|---|---|---|---|
| 2002-02-25 | Report filed by user 25: NPE in externalize strings dialog (JDT) | None | Problem stated; nothing decided |
| 2002-02-28 | Resolved WORKSFORME by user 8 | v1: WORKSFORME | First decision: no defect reproduced |
| 2002-03-25 | Reopened by the reporter, 25 days after v1 | v1 contested | Linked follow-up from the person who raised it |
| 2002-03-26 | Assigned, then resolved FIXED by user 8 | v2: FIXED | Direction reversed; whether v1 was wrong needs the comment thread |
Freeze the label, version the change, report both scores
Treat every label in a held-out set as a dated claim. Store the label, the date it was assigned, the evidence it rested on and a version number. When a follow-up arrives, add a version with its follow-up type rather than editing the old one, then report model scores against both the frozen and the latest versions, and say how many held-out labels moved in between.
Freezing matters because evaluation sets age. In both bug trackers, 5.8% of reopenings came more than 180 days after the first decision, so a set frozen within its first few months keeps collecting late changes. A team that silently updates labels makes last quarter's score incomparable with this quarter's; a team that never updates them keeps grading models against decisions it already knows were wrong.
To test the rubric itself, hold out whole issue types or accounts rather than individual tickets, so that repeat contacts about one problem cannot land on both sides. Have two reviewers assign follow-up types to the same held-out sample and report agreement for each type. Expect contradicted no-action decisions and new-information tags to cause most disagreements, since both depend on what the record held on the day of the first decision.
Bug trackers are not helpdesks: where this evidence stops
The Eclipse and Mozilla histories show the mechanism; they are not a support benchmark. Their reporters were often developers, many reopenings came from QA, and the resolution vocabulary is engineering vocabulary. The data also stops at the start of 2013, before most current helpdesk automation. Customer support adds what this dataset lacks: courtesy replies, refunds and credits that reverse quietly in billing, and customers who never return to report a failed fix.
The strongest objection is that Mozilla's quiet relabels may be housekeeping rather than decisions, which would make the 60.2% figure overstate how often outcomes change without a reopen. Some of them surely are. The largest single flow, FIXED to WORKSFORME with no reopen (3,236 reports), looks like a convention for bugs that disappeared without a specific patch. That is why the rubric has an administrative-relabel type: a change in the label field is not automatically a change in the grade. The bug data shows that both kinds of change happen at scale; telling them apart takes the comment history, which this dataset leaves out.
Earlier research treats reopening as a process with several causes rather than a single failure signal. Zimmermann and colleagues at Microsoft Research studied reopened Windows bugs in 2012, beginning with a survey of 358 Microsoft employees about why bugs get reopened, and set out to characterize the overall reopen process rather than to predict it bug by bug.
Writing a reopened-case requirement a supplier can meet
Ask for the links and the versions, not just the flags. A workable requirement names the reopen events and follow-up ticket links you need, the window they should cover, the original decision with its timestamp and evidence, every later label with its own timestamp, and the comment text that lets a reviewer assign a follow-up type. It should also say how administrative relabels and merged tickets are marked.
Put the acceptance test in writing. For example: in a sample of 100 reopened or followed-up cases, every case links to its original decision, and at least 90% carry enough comment history to assign a type other than unknown. Those thresholds are illustrative.
Because SourceX finds datasets for buyers among U.S. companies on request rather than keeping an inventory, a reopened-case requirement becomes part of the request itself, next to volume, language, format and licensing terms. Nothing moves until the company holding the helpdesk archive has approved what is in scope, agreed the terms and signed. Whether a particular archive kept its follow-up links intact is checked supplier by supplier, and no result for any model is promised.
Frequently asked questions
Should a reopened support ticket be labeled a failure?
Not automatically. In the Eclipse and Mozilla bug histories, about half of reopened reports ended with their original resolution. Label the first decision a failure only when the follow-up shows the same problem persisted, or a no-action decision was contradicted by evidence that existed at the time.
What is the difference between a reopened ticket and a follow-up ticket?
In Zendesk, a requester's comment on a solved ticket reopens that ticket, while any reply to a closed ticket creates a new follow-up ticket that carries over most of the original's details. A dataset needs both paths to see every second look at a case.
How do you keep an evaluation set stable when labels change after a follow-up?
Version the labels. Keep the frozen label with its date and evidence, add later versions with a follow-up type, and report scores against both, together with the number of labels that moved.
Why are could-not-reproduce closures risky as training labels?
They are the decisions follow-ups overturn most. In Eclipse, 10.9% of first WORKSFORME resolutions ended with a different label or unresolved, against 1.5% of FIXED ones, and 667 went from WORKSFORME to FIXED after a reopen.
Can a label change without a reopen?
Yes. In Mozilla's history, 12,681 of the 21,082 decisions whose label ended up different or unresolved never passed through a REOPENED status. Some are housekeeping and some are real reversals, so each needs a follow-up type before it changes a grade.
Sources
- The Eclipse and Mozilla Defect Tracking Dataset (MSR 2013) provides each bug report's full history of status and resolution changes; counts in this article come from its v02 JSON files at commit f508423. Source
- Bugzilla's field help defines RESOLVED as awaiting QA verification, VERIFIED as QA agreeing with the resolution, and resolutions FIXED, INVALID, WONTFIX, DUPLICATE and WORKSFORME, noting that a WORKSFORME bug can be reopened if more information appears. Source
- Bugzilla documentation says status and resolution define the state of a bug from unconfirmed through fixed and confirmed by QA. Source
- Zendesk reopens a solved ticket when the requester responds with a new comment, for example when they disagree the issue was resolved. Source
- In Zendesk, any reply to a closed ticket creates a follow-up ticket that carries over most details and tags but not comments, priority, type, assignee, organization, skills or external ID. Source
- Zimmermann, Nagappan, Guo and Murphy (ICSE 2012) studied reopened bugs in Microsoft Windows, starting with a survey of 358 Microsoft employees, to characterize the overall reopen process. Source
Tell us what data you need
Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.