AI uses for records
Deployment stop decision dataset: what 90 release holds left behind
By SourceX Editorial · Updated
9 sections
Short answer
A deployment stop decision dataset treats each hold as a case: the candidate that was stopped, the evidence weighed, who had authority, the rule applied when nobody answered, and what shipped instead. OpenStack's public release record shows how often those parts go missing: across eight cycles, 89 of 622 deliverables were held, yet only 17 of 90 holds state a reason in words and none names its approver in git.
Agent run
Deployment stop decision
Key takeaways
- A hold is a decision with an alternative, not an absence of deployment; its record must name the stopped candidate, the evidence and what shipped next.
- In openstack/releases at commit 9da6d9c, 89 of 622 cycle-with-rc deliverables, 14.3 percent, were held at least once between 2023.1 and 2026.2, for 90 holds.
- 32 of the 90 holds carry nothing beyond a subject line, 17 state a reason in words, and only 9 link to a bug, review or commit through a URL or a Depends-On/Related-Bug footer.
- Approvals sit in the review system rather than git, and the rule for silence differed between generated hold requests and the written process.
- Held deliverables later shipped stable fixes more often, 60 of 77 against 136 of 479, which shows where defects cluster, not that holds failed.
What counts as a decision not to deploy?
A decision not to deploy is a recorded judgment, made at a named decision point, that one specific candidate should not ship yet, together with the alternative chosen instead: fix it and cut a new candidate, wait for a later window, or ship with the issue written down. An empty deploy log is not a hold. A hold has a candidate, a reason and a consequence, and each of the three can be checked later.
It sits between two neighbors that already have their own records. A rejected patch is a review decision about one change, the subject of our page on why a working patch was rejected. A rollback undoes something users already received, the subject of our page on reverted deployments. A hold happens after the change is merged and before anyone runs it in production, which is why its evidence is scattered across release files, test results and approval threads rather than kept in one ticket.
Risk frameworks already expect the decision to exist. NIST's AI Risk Management Framework lists under MANAGE 1.1 that "A determination is made as to whether the AI system achieves its intended purposes and stated objectives and whether its development or deployment should proceed," and it describes an initial go/no-go decision once the MAP function is complete. The framework asks for the decision; it does not say what record the decision should leave behind. That gap is what a dataset of stop decisions fills.
For an AI team the use is specific: training or evaluating an agent that recommends whether a release should go, and scoring it on cases where not yet was the right answer. Without hold cases in the data, every example teaches a model that merged, tested code ships.
Which seven fields make a release hold record usable?
Seven fields turn a hold into a case that a model can learn from and an auditor can rebuild: the identity of the stopped candidate, the trigger and its evidence, the reason in words, the decider and the authority they held, the rule that applies when nobody responds, the next candidate with what changed, and the artifact that finally shipped. Missing any one of them changes what the case can teach.
The decider and the silence rule are the easiest fields to overlook, and they matter most for training. A model that sees a stop without knowing who could order it, or whether anyone actively agreed, cannot tell a judgment from a default. The last column counts how many of OpenStack's 90 recent holds carry each field in the public release repository, using the script described in the methods note:
| Trace field | What it must hold | Where OpenStack keeps it | Present in the 90 holds |
|---|---|---|---|
| Candidate identity | The exact commit or artifact that was stopped and the one cut instead | Deliverable file: one version and commit hash per candidate | 90 of 90 |
| Trigger and evidence | The bug, test result or review that prompted the stop | Links and change lists in the request; bug tracker and code review hold the rest | 39 of 90 (9 with a URL or a Depends-On/Related-Bug footer) |
| Reason in words | Why the candidate was not good enough | Free text in the request's commit message | 17 of 90 |
| Decider and authority | Who could say no, and under which rule | Code review votes and an approval check job, outside git | 0 of 90 in git |
| Rule on silence | What happens if the decider does not answer | Process document, sometimes repeated in the request | 23 of 90 (20 merge, 3 abandon) |
| Next candidate and delta | The re-cut candidate and the changes between the two | Next version entry; a change list when one is pasted in | 90 of 90 (32 list changes) |
| What shipped | The artifact users received | The final release re-tags the last candidate | 90 of 90 |
What 90 OpenStack release holds put on the record
They put the decision on the record every time and the reasoning rarely. OpenStack, the open-source cloud platform, releases most of its services under a model it calls cycle-with-rc: each deliverable publishes one or more release candidates before a coordinated release, and on release day the release team re-tags "the commit used for the last RC," in the words of its process document. Every candidate after the first is therefore a dated decision that the previous candidate would not become the release.
We counted those decisions in the public openstack/releases repository at commit 9da6d9c of October 27, 2026, across the eight cycles from 2023.1 (Antelope, released March 22, 2023) to 2026.2 (Hibiscus, released September 30, 2026). Of 623 cycle-with-rc deliverable files, one per deliverable per cycle, 622 published at least one candidate. 89 of those, 14.3 percent, were held at least once: 88 cut a second candidate and one cut a third, 90 holds in all. All 622 final releases point at the same commit as their last candidate, so what shipped is never in doubt.
Why each hold happened is a different story. We read the commit that added each new candidate, after removing tool-generated sentences, process boilerplate, pasted change lists and links. 32 holds, 35.6 percent, carry nothing beyond a subject line naming the deliverable, the candidate and the cycle. 17, or 18.9 percent, state a reason in words, and only 7 of those also link to the bug, review or commit behind it.
The best of them read like the record a dataset needs. One names the fix for security advisory OSSA-2026-004. One explains that a library's minimum version "did not get bumped" as needed and a second candidate fixes it. One reports a blocker that stopped the previous release from uploading to the Python package index. Each tells a reader what was wrong, which artifact carried it and what changed. The full classification:
| What the hold's commit message carries | Holds | Share of 90 | Typical content |
|---|---|---|---|
| A reason in words | 17 | 18.9% | A security fix, a missed dependency bump, a failed package upload |
| Free text without a reason word | 8 | 8.9% | Missing release notes, or a note that this is the final candidate |
| A change list or link only | 27 | 30.0% | Commits between the two candidates, or a link to one change |
| A generated sentence only | 6 | 6.7% | A tool's standard line about picking up merged changes |
| The subject line only | 32 | 35.6% | Deliverable, candidate number and cycle, nothing else |
Who said no, and what happened when nobody answered?
The repository cannot say who said no. OpenStack's reviewer guide states that "Releases from someone other than the PTL or release liaison must be acknowledged by one of them with a +1 vote in gerrit," naming the two team contacts it lists for each deliverable, and a check-release-approval job verifies that approval when a request is submitted. Those votes and job results live in the code review system. The public git mirror carries branch refs only, no review notes, so none of the 90 holds names its approver in git, and the author of a request is not necessarily the person with authority to stop a release.
The rule for silence is recorded unevenly too. For last-minute candidates the process document says "Patches that get no feedback by the deadline should be abandoned," a sentence that entered the document in an edit dated October 10, 2019 and has not been removed since. Yet 20 generated hold requests in the 2023.1 to 2024.1 cycles told teams that requests with no response would be merged by a stated Friday, and 3 requests in 2026.2 repeat the abandon rule. A merged hold from those earlier cycles may record an explicit yes or nobody objecting in time, and the release record cannot tell which.
Other projects split the decision from its reason in the same way. Python's release schedule for version 3.13, PEP 719, records an added third release candidate and moves the final release from October 1 to October 7, 2024, in commit c778ab9, titled "PEP 719: Add 3.13.0rc3, postpone 3.13.0 final." The commit has no message body. The schedule proves a hold happened; anyone who wants the reason has to join it from somewhere else.
For a dataset, the decider field therefore has to come from the review system's export, stored as a role rather than a name, and the silence rule has to be stored with each decision as it stood on that date, not inferred from today's process page.
How do you score a hold when the failure never happened?
Score the evidence for the catch, not the catch itself, because the failure a hold prevents is never observed. A hold that removed a real defect and a hold that delayed a healthy candidate look identical in production: both ship clean. What differs is the chain from reason to fix to a test on the shipped artifact, so the outcome label should grade that chain and name the link that is missing.
Six labels cover the cases a release record produces. Confirmed catch demands the most and is rare in public data: none of OpenStack's 90 holds can earn it from the release repository alone, because the failing tests and bug reports sit in other systems. Declined hold is the label most datasets lack. OpenStack's process says that a -1 on a last-minute candidate "will mean that the PTL prefers to wait for a post-release stable update," a decision to ship and patch later that never reaches git, because the request is not merged.
A dataset that cannot link the supporting systems should say so in the label rather than upgrade a plausible catch to a confirmed one. The rubric, with what each label requires and what a model can safely learn from it:
| Outcome label | Evidence required | What a model may learn | Supportable from the OpenStack release repository alone? |
|---|---|---|---|
| Confirmed catch | Defect reproduced on the held candidate, fix in the next one, test passing on what shipped | That this kind of evidence justifies a stop | No: tests and bug reports live elsewhere |
| Plausible catch | A named defect or advisory linked to a fix in the next candidate, not reproduced | That a stated risk was acted on, not that it was real | Partly: 7 holds pair a reason word with a link |
| Process hold | A release-mechanics fault such as notes, packaging, metadata or upload | That shipping mechanics can stop a release | Yes, where the message says so |
| Hygiene hold | Merged changes picked up with no named defect | Little about risk, more about cadence | Yes, where the message says so |
| Declined hold | A stop that was proposed and refused, plus what happened after release | That shipping now and patching later was judged acceptable | No: refused requests never merge |
| Unscorable | No reason word, link or change list | Nothing; keep out of outcome labels | Yes: 44 of 90 holds |
Does a later stable patch prove a hold was wrong?
No, and reading it that way would teach a model the wrong lesson. In the seven cycles released at least 180 days before our snapshot, 60 of 77 held deliverables, 77.9 percent, later published a stable point release in the same cycle, against 136 of 479 deliverables that shipped their first candidate, 28.4 percent.
The tempting reading, that holds fail to stop defects, confuses where problems cluster with what holds achieve. Deliverables under active repair both trigger holds and ship patches; quiet deliverables do neither. The numbers fit that explanation at least as well as any other, and the release repository contains nothing that could separate the two, such as whether a later patch fixed the same issue that triggered the hold.
The same trap sits inside companies. A hold that ends with a green test suite and a closed ticket can still leave the originating problem in place if the fix treated a symptom. The record should tie any post-release fix back to the hold's stated reason, and the outcome label should change only when that tie is shown, not when a ticket closes.
Holds also cost something, and the dataset should keep the cost. OpenStack's process asks teams to avoid more than three release candidates "so we are not creating candidates that consumers are then trained to ignore." A model trained only on stops that turned out well learns to stop too often, so go decisions, declined holds and their outcomes belong in the same dataset as counterweights.
What should a buyer test before licensing hold records?
Test whether each case can be rebuilt by someone who was not in the room. Draw a random sample of holds with a published seed and a stated sort order, then check every sampled case against the list below before reading any summary statistic.
Then check the denominators. A hold rate needs the number of candidates that reached the decision point, not only those that stopped, and a dataset of stops alone cannot teach when to go. Ask how go decisions from the same releases are represented, how declined holds were recovered from the review system, and what the security scope removed: details of vulnerabilities not yet fixed, internal hostnames, customer names and personal data in approval threads. This acceptance test, written into the request, gives the supplier a target and the buyer a reason to refuse a delivery that misses it:
- The stopped and the re-cut candidates each resolve to a commit or artifact digest.
- The stated reason and its evidence carry timestamps earlier than the hold decision.
- The decider appears as a role, with the authority rule and its source cited.
- The silence rule in force on the decision date is stored with the case.
- Go decisions and declined holds are present, with counts for each release.
- Every outcome label comes from one fixed label set and names any missing evidence.
- Vulnerability details, hostnames and personal data are removed under a written scope.
Where do stop decisions live inside a software company?
In four places that rarely point at each other: the release or deployment ticket that names the candidate, the CI and test results that prompted the stop, the approval thread in code review or chat where someone wrote not yet, and the next release entry showing what actually shipped. Postmortems add the after-effects when a hold was declined and something broke; our pages on incident timelines and on uncertain root causes cover those records.
Joining the four is the real work. A company with years of releases may have made hundreds of stop decisions, but they form a dataset only if the approval system and the release log share identifiers that a script can match. That join, more than the number of releases, decides whether the archive is worth preparing. A team that kept release notes but deleted its review history can still supply candidates and outcomes, and should say plainly that deciders are missing.
SourceX sources datasets for AI buyers on request rather than holding them in stock, and nothing is delivered until the company supplying the records has approved the scope and signed. A buyer that needs release-hold cases should describe the decision points, the outcome labels and the acceptance test above when it submits a request; whether a given company's records can meet that test is settled only by reviewing them.
Methods note: OpenStack repository, counting rules, limits
Data: openstack/releases, cloned from its GitHub mirror on October 27, 2026 at commit 9da6d9cae346 (committed 09:46 UTC that day). Scope: series in doc/source/series_status.yaml with release ids 2023.1 to 2026.2, and deliverable files under deliverables/<series>/ whose release-model is cycle-with-rc. A candidate is a version matching X.Y.Z.0rcN; a hold is every candidate numbered 2 or higher. The commit that introduced a hold is the earliest non-merge commit, read with git log --full-history and each commit compared with its first parent, that adds that version line to that file; every hold matched exactly one such commit, 78 distinct commits for 90 holds.
Classification: after dropping Change-Id, Signed-off-by, Co-Authored-By and Generated-By lines, the script removes change-list lines (git log commands, lines that begin with a 7 to 40 character hexadecimal hash, a change log header), two generated sentences, the ack sentence, the merge-on-silence and abandon-on-silence sentences, link footers and URLs, and prints how many holds each removal touched. A hold has a link when its message carries a URL on a bug tracker, code review or code hosting site, or a Depends-On, Closes-Bug, Partial-Bug or Related-Bug footer. A reason word is a whole-word, case-insensitive match on one of 21 words (bug, bugs, fix, fixes, fixed, fixing, blocker, regression, regressions, CVE, security, critical, broken, breaks, failing, failure, failures, issue, issues, vulnerability, crash) in the remaining text or the subject. Shares are computed from integers and rounded once, half up.
The later-release comparison counts any non-candidate version above the final in the same file, for the seven series released at least 180 days before the snapshot. Not measured: review votes, abandoned requests, bug reports, test results, the content of changes between candidates, and anything about any company's internal releases. The Python example comes from python/peps at commit 35d53a8. The scripts and a per-hold output file, which holds no names or message text, are kept for recount.
Frequently asked questions
What is a deployment stop decision dataset?
It is a collection of cases in which a team decided that a specific release candidate should not ship yet. Each case records the stopped candidate, the evidence weighed, who decided under which rule, and what shipped instead. It lets a model be trained and tested on when not yet is the right call, not only on code that shipped.
How is a release hold different from a rollback?
A hold stops a candidate before users receive it; a rollback undoes a change that already reached production. Holds leave weaker traces because nothing failed in production to open an incident, so their evidence has to be joined from release files, test results and approval threads.
Can the value of a release hold be measured?
Not directly, because the prevented failure never happens. What can be graded is the evidence chain: a defect reproduced on the held candidate, a fix in the next candidate and a passing test on the shipped artifact. Holds without that chain belong under a plausible or unscorable label, not counted as saves.
Why should a dataset about not deploying include decisions to ship?
A model shown only stops learns to stop. Go decisions from the same releases, plus holds that were proposed and declined, supply the denominators and counterexamples that let a model tell a justified stop from reflexive caution.
What has to come out of release hold records before they are licensed?
Details of vulnerabilities that are not yet fixed or disclosed, internal hostnames and infrastructure identifiers, customer names, and personal data such as names in approval threads. Deciders can be kept as roles. The supplying company's security team and counsel set the written scope before anything is shared.
Sources
- OpenStack's release process: final cycle-with-rc releases re-tag the commit used for the last RC; teams should avoid more than 3 release candidates; last-minute candidate patches with no feedback by the deadline should be abandoned; a PTL -1 means waiting for a post-release stable update. Source
- The cycle-with-rc model: a single release at the end of the cycle with one or more release candidates close to the end of the cycle. Source
- Releases from someone other than the PTL or release liaison must be acknowledged by one of them with a +1 vote in Gerrit; a check-release-approval job checks that approval. Source
- Deliverable files for the 2023.1 to 2026.2 cycles: 622 cycle-with-rc deliverables with candidates, 89 held at least once, 90 holds, every final on its last candidate's commit (counted by the article's script). Source
- Release ids and initial release dates of the OpenStack series counted, from Antelope (2023.1, 2023-03-22) to Hibiscus (2026.2, 2026-09-30). Source
- Python 3.13 release schedule (PEP 719) lists candidate 3 and the final release on 2024-10-07. Source
- Commit c778ab9 to PEP 719, titled Add 3.13.0rc3, postpone 3.13.0 final, moved the final date from 2024-10-01 to 2024-10-07 and has no message body. Source
- NIST AI RMF 1.0, MANAGE 1.1: a determination whether the AI system achieves its intended purposes and whether development or deployment should proceed; an initial go/no-go decision after MAP. Source
Tell us what data you need
Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.