Skip to content

Privacy and preparation

Closed, settled or withdrawn: legal outcome label uncertainty

By SourceX Editorial · Reviewed by Noah Loul ·

9 sections

Short answer

Do not collapse them into one outcome field. Closed is a workflow state, settled and withdrawn are ways a claim ended, and whether anyone got relief or can refile is a separate question again. Label each matter with its end state, mechanism, benefit direction and preclusion, each taken from the order, agreement or forum record that proves it, and mark the label unknown or pending where no authoritative record settles it.

Grader rubric

Closed, settled or withdrawn

This page covers closed, settled or withdrawn: legal outcome label uncertainty.Illustrative

Key takeaways

  • Closed records that work stopped; it says nothing about whether a claim was settled, withdrawn, dismissed or decided.
  • Official statistics already split these outcomes: the EEOC counts a withdrawal with benefits as a merit resolution, while a withdrawal with nothing is an administrative closure.
  • A voluntary federal dismissal is without prejudice by default, so a dismissal label without preclusion recorded is incomplete.
  • Labels should come from forum records and executed agreements first, counsel's closing record second and system fields only after sampling checks.
  • Settlements still being performed, open appeal windows and partial resolutions need a pending or partial value and a maturity date, not a final label.
  • Build labels from nonprivileged process evidence that counsel has screened, and keep confidential settlement terms out of the dataset.

Treat closed, settled and withdrawn as answers to three different questions. Closed is a workflow state: the file or docket stopped moving. Settled and withdrawn describe how it ended: by an agreement between the parties, or by the claimant stepping back. A third question, whether anyone received relief and whether the claim can be brought again, needs its own field. Flattening all three into one outcome column produces labels that look complete and teach a model the wrong lesson.

Official statistics already keep these apart. The EEOC counts a charge withdrawn by the charging party who receives benefits through a separate agreement with the employer as a withdrawal with benefits, and it groups that category with negotiated settlements and conciliations as a merit resolution. A charging party who withdraws without receiving anything is counted as an administrative closure instead. The same word, withdrawn, lands on opposite sides of the favorable and unfavorable line.

Federal court data separates them too. The Federal Judicial Center's civil Integrated Data Base codebook, in the years where it splits dismissals more finely, gives voluntary dismissal, dismissal after settlement and other dismissals their own codes. Rule 41 of the Federal Rules of Civil Procedure then adds the preclusion question: a voluntary dismissal by notice or stipulation is without prejudice unless the notice or stipulation says otherwise, with an exception for a plaintiff who previously dismissed the same claim.

So the practical rule for a legal dataset is simple to state: record the end state, the mechanism, the benefit direction and the preclusive effect as separate fields, each with the document that proves it, and leave a field unknown when no authoritative record answers it.

Why does a closed matter status say so little about the result?

A closed status records that work stopped, not why. Matter management systems close files when a settlement is paid, when a client goes quiet, when a claim is transferred to other counsel, when an internal investigation ends without action, or simply during a year-end cleanup. Each of those is closed in the status field, and each means something different to a model that is learning to predict how disputes end.

Billing data does not fill the gap. The UTBMS litigation code set used on legal invoices describes phases and tasks of work: L100 covers case assessment, development and administration across the life of the case, L400 covers trial preparation and trial, and L500 covers work on appeal. A matter billed only under early-phase codes probably ended before trial, but the codes cannot tell you whether it ended in a settlement, a withdrawal or a dismissal on the merits.

Court statistics show the same caution. The National Center for State Courts' State Court Guide to Statistical Reporting tells courts not to report a case's manner of disposition until all parties and issues are resolved, and to leave cases placed on inactive status out of the manner-of-disposition counts unless they are reactivated. A dataset that labels a matter settled when one defendant settled and another is still litigating repeats exactly the error that guide is written to avoid.

The guide below is the asset: one row per outcome value, with the evidence that proves it and the evidence that only looks like proof. It is written for business legal records such as claims files, employment charges, contract disputes and collections matters. Counsel should adapt the definitions to the jurisdictions and forums in the reader's own records before any label is used.

The legal outcome definition guide
Outcome valueDefinitionEvidence that proves itNot enough on its ownCommon confusion
closed_adminWork stopped with no recorded resolution of the claimClosing memo stating the reason, or a forum's administrative closure noticeStatus field set to closedRead as a win because nothing further happened
settled_executedParties signed an agreement and its conditions were performedSigned agreement plus a dismissal entry or a performance confirmationEmail saying the matter settledAgreement signed but conditions still open
settled_pending_performanceAgreement signed, case dismissed with leave to reinstate or obligations still runningDismissal order naming a reinstatement date, or a schedule of open obligationsDocket entry reading dismissedTreated as final before the reinstatement date passes
withdrawn_with_benefitClaimant withdrew after receiving relief under a separate agreementWithdrawal record plus the agreement or a forum record of benefitsWithdrawal notice aloneGrouped with withdrawals that produced nothing
withdrawn_no_benefitClaimant withdrew and received nothing recordedWithdrawal record and a counsel note confirming no agreementAbsence of a settlement documentA confidential side agreement that was never filed
dismissed_without_prejudiceClaim ended but may be brought againNotice, stipulation or order stating without prejudice, or silent under a rule whose default is without prejudiceThe word dismissedLabeled as a defense win on the merits
dismissed_with_prejudiceClaim ended and cannot be brought againOrder or stipulation stating with prejudiceClosing memo summaryAssumed for every stipulated dismissal
adjudicated_for_claimant or adjudicated_for_respondentA judge, jury, arbitrator or agency decided the meritsJudgment, award or final decision with its dateBilling in the trial phaseVerdict later changed on appeal or post-trial motion
partialSome parties or claims resolved, others still openOrder or agreement naming the parties or claims it coversMatter-level statusRolled up to the matter as if fully resolved
unknownNo authoritative record answers the questionAn exclusion or review note explaining what is missingNot applicableFilled with the most common value to avoid blanks

Which record has the authority to set the outcome label?

The label should come from the most authoritative record available, and the dataset should say which one it used. In practice, a filed order or a signed agreement outranks anything a person typed into a matter system afterwards, and a matter system field outranks an inference from billing or email traffic.

At a glance

Which record has the authority to set the outcome label

  • Tier 1, forum records: a judgment, award, order of dismissal, agency closure notice or docket entry. These fix the date and the formal effect, including whether a dismissal was with or without prejudice.
  • Tier 2, executed agreements and their amendments: the signed settlement or release, any amendment, and proof that conditions were performed. Use the version in force at the label's as-of date, and keep the amendment chain.
  • Tier 3, counsel's closing record: a closing memo or disposition note written by the responsible lawyer. Useful for reasons and for matters that never reached a forum, but it is a summary of tiers 1 and 2, not a substitute.
  • Tier 4, system fields: matter status, outcome picklists and close dates in matter management or claims systems. Check them against tiers 1 to 3 on a sample before trusting them at scale.
  • Tier 5, inferred signals: billing phases, last activity dates and email. Use them only to flag records for review, never to set an outcome on their own.
  • Record conflicts rather than resolving them silently: when the matter system says settled and the docket shows a dismissal without prejudice with leave to reinstate, keep both values with their sources and let the as-of rule pick the label.

Uncertainty in legal outcomes is usually structural, not a labeling mistake: a settlement can still unravel, a dismissal can be reopened and a judgment can be appealed. The schema should carry that uncertainty in named fields so downstream users can filter on it instead of guessing.

One structural case deserves its own rule. Case procedures published by judges in the Northern District of Illinois note that, under Seventh Circuit decisions, a dismissal with prejudice generally leaves the court without jurisdiction to enforce a settlement, and they suggest dismissing without prejudice with leave to reinstate by a set date when settlement terms are still being performed. A matter in that window is settled in substance but not yet final in form, which is exactly what settled_pending_performance and a maturity date are for.

Recording legal outcome label uncertainty instead of hiding it
FieldAllowed values or formatRule
outcome_valueOne value from the definition guideSet only from tier 1 or tier 2 evidence; otherwise unknown or flagged
evidence_basisdocumented, counsel_summary, system_field, inferredName the tier actually used, not the best tier that might exist
as_of_dateDate the label describesInputs for any model task must stop before this date
maturity_dateDate after which the outcome can no longer change by ordinary meansReinstatement deadline, appeal period end or final performance date
preclusionwith_prejudice, without_prejudice, not_applicable, unknownTaken from the order or stipulation text, never assumed
benefit_directionclaimant_relief, no_relief, mixed, confidential, unknownConfidential means relief exists but terms are excluded
conflict_noteFree text with source referencesRequired whenever two tiers disagree

Worked example (illustrative): one discrimination charge, three systems, three answers

This example is illustrative. The matter, dates and counts are invented to show how the guide works and do not describe any real company or dataset.

A mid-sized employer's legal team holds 40 closed employment matters from one year. One of them began as a discrimination charge filed with the EEOC. The matter system shows status closed and outcome withdrawn. The billing history shows only early-phase work. The charge file contains a withdrawal request from the charging party, and the legal team's records contain a separation agreement signed two weeks earlier that provided benefits.

Read field by field, the answer is not withdrawn in the sense a model would learn from the picklist. The forum record is a withdrawal, the executed agreement shows relief, and the EEOC's own definitions would count this as a withdrawal with benefits, a merit resolution. The guide labels it withdrawn_with_benefit, evidence_basis documented, benefit_direction confidential because the agreement's terms stay out of the dataset, and preclusion not_applicable because no court was involved.

Across the 40 matters, a review of this kind might find, for illustration, that 12 picklist values change: some withdrawals become withdrawals with benefit, some settlements become partial because one claim stayed open, and several closed files become closed_admin or unknown because nothing authoritative explains them. The exception is just as instructive. One matter's agreement required staged performance over the following year, so its label at the dataset's as-of date is settled_pending_performance, and a model trained on it should not be told the dispute was finished.

What privilege and confidentiality leave out of the evidence

Outcome labels should be built from nonprivileged process evidence, and counsel should decide what that is before anyone extracts a field. The ABA's Model Rule 1.6 is broad: a lawyer's duty of confidentiality covers all information relating to the representation, not only privileged communications, and paragraph (c) asks lawyers to make reasonable efforts to prevent unauthorized disclosure. Many states adopt a version of the rule with their own variations, so the reader's own jurisdiction controls.

That breadth shapes the dataset. Counsel's strategy memos, settlement valuations and advice are out of scope for labeling. Filed orders and public docket entries are a different matter, though sealed filings are not. Settlement agreements often carry confidentiality clauses that bind the company as a party, so their terms usually appear only as a benefit_direction value of confidential, with the agreement itself excluded. Keep an exclusion record that lists each document type left out and the reason, so a reviewer can see why a field is blank.

Rights questions sit on top of confidentiality. Being able to read a closing memo inside the company is an access permission, not a right to license it, and removing names does not by itself create that right. Contracts with opposing parties, insurers and outside counsel may restrict use of matter records, and counsel reviews those terms against the specific records before anything leaves the company.

Exceptions that should keep a label open or send it to counsel

Some matters should not get a firm label at all, and the dataset is better for admitting it. These are the conditions where the right action is to hold the label open, narrow the dataset or ask counsel to decide.

  • Class or collective actions: Rule 41's voluntary dismissal provisions are expressly subject to Rule 23(e), so settlement of a certified class needs court approval, and the label waits for it.
  • Reinstatement or appeal windows still open: label settled_pending_performance or leave the merits label provisional until the maturity date passes.
  • Sealed filings or confidentiality orders: exclude the sealed material, and ask counsel whether even the existence of the outcome may be recorded.
  • Multi-party matters with staggered resolutions: label at the party or claim level, and do not roll a partial outcome up to the matter.
  • Prejudice unclear from the record: leave preclusion unknown rather than reading it from a summary; the default under Rule 41 depends on which paragraph applied and on any earlier dismissal of the same claim.
  • Matters transferred to other counsel or another forum: the closed status in the original file says nothing about how the claim ended elsewhere.
  • Records outside the U.S. federal system: state courts, agencies and foreign forums use their own disposition categories, so map them with counsel rather than by name.

Matter histories as licensed data, and what SourceX asks first

Closed matter histories are useful to AI developers when the outcome fields survive scrutiny, which is why the definitions above matter before any licensing discussion. For a legal department or claims operation, SourceX can take selected matter histories from sourcing and rights review through contracting, data preparation, delivery and payment. For legal records, that review starts with the questions in this guide: which documents set the label, which are privileged or confidential and which contracts limit use.

No matter record reaches SourceX before the company signs an agreement whose scope and terms it has approved. A company can learn whether it qualifies through a short assessment that requires no data upload, and counsel can review the specific matters before any record is considered.

Frequently asked questions

Is a settled case the same as a dismissed case?

Not as a label. Many settled federal cases end with a stipulated dismissal, so the docket shows a dismissal while the substance is an agreement. Record the mechanism (settlement) and the formal effect (dismissal with or without prejudice) as separate fields, each from its own document.

Does withdrawn mean the claimant lost?

No. The EEOC counts a withdrawal where the charging party receives benefits through a separate agreement as a merit resolution, alongside settlements. Only a withdrawal with no benefit recorded is an administrative closure. Your labels need the benefit direction to tell them apart.

Can matter management status fields be used as outcome labels?

Only after they are checked against forum records or executed agreements on a sample. Status picklists are often set for workflow reasons, such as closing files at year end, and rarely record preclusion or partial outcomes. Treat them as a starting point, not as ground truth.

What should happen to confidential settlement terms in a training dataset?

Keep them out. Record that relief exists with a value such as confidential and list the agreement in an exclusion record. Counsel decides whether even that fact may be recorded when a confidentiality clause or court order covers it.

When is a legal outcome final enough to label?

When its maturity date has passed: the reinstatement deadline, the appeal period or the last performance obligation, whichever applies. Before then, label it as pending and keep the date in the record so later users can refresh the label.

Sources

  • The EEOC defines withdrawal with benefits, negotiated settlements, administrative closures and merit resolutions as separate charge resolution categories, and counts withdrawals with benefits as merit resolutions. Source
  • The Federal Judicial Center's civil Integrated Data Base codebook gives voluntary dismissal, dismissal after settlement and other dismissals separate disposition codes in the periods where it splits them. Source
  • Federal Rule of Civil Procedure 41(a)(1) lets a plaintiff dismiss by notice or by a stipulation signed by all parties who have appeared, subject to Rule 23(e), and makes the dismissal without prejudice unless stated otherwise. Source
  • Northern District of Illinois case procedures state that, under Seventh Circuit decisions, dismissals with prejudice leave the court without jurisdiction to enforce a settlement, and describe dismissal without prejudice with leave to reinstate by a set date. Source
  • The State Court Guide to Statistical Reporting says manner of disposition is not reported until all parties and issues are resolved and excludes inactive cases unless reactivated. Source
  • The UTBMS litigation code set defines work phases such as L100 case assessment, development and administration, L400 trial preparation and trial, and L500 appeal. Source
  • ABA Model Rule 1.6 sets a lawyer's duty of confidentiality over information relating to the representation and requires reasonable efforts to prevent unauthorized disclosure. Source

See if your company qualifies

A short company assessment. No data uploads are needed.

See if your company qualifies