Systems and records
Sales forecast uncertainty labels: rep confidence is not a probability
By SourceX Editorial · Updated
10 sections
Short answer
Keep them in separate fields with separate provenance. A rep's confidence, a stage default or a commit category is a judgment recorded at a moment; a validated probability is an observed close rate for a defined cohort, measured afterwards against contract-backed outcomes and reported with an interval. Annotate who set each value, what evidence existed then and which later record confirmed the result, and compute validation only across many records.
Record timeline
Sales forecast uncertainty
Key takeaways
- CRM probability fields usually hold stage defaults or manual overrides, and neither has been checked against outcomes.
- Forecast categories such as commit are judgment buckets, not percentages, and owners can override the stage mapping.
- A validated probability is a calibration result for a cohort, with sampling uncertainty, never a value typed on one record.
- Outcome labels should come from signed agreements, amendments and what followed, not from the closed won status alone.
- Past confidence values exist only in field history, which standard Salesforce retention keeps for a limited period.
- Small segments should be labeled insufficient data rather than given a precise-looking rate.
How should revenue datasets distinguish rep confidence from validated probability?
Store them as different things with different provenance. A rep's confidence is a judgment entered at a point in time: a stage, an overridden percentage or a forecast category such as commit. A validated probability is a rate measured afterwards: of all opportunities that carried a given signal at a given point, how many closed under a defined outcome rule, with an interval that shows how uncertain the estimate is. A dataset that puts the first in a column called probability teaches a model that opinions are measurements.
CRM defaults make the confusion easy. In Salesforce, an opportunity's Probability is updated whenever its stage changes, and users with edit access can override it. In HubSpot, Deal probability is set from the win probability assigned to each pipeline stage, and a manual edit stops that automatic updating for the open stages. HubSpot's Weighted amount then multiplies the deal amount by that probability. None of these steps checks the percentage against what actually happened.
Sales forecast uncertainty labels fix this by recording, for every confidence value, who or what produced it, what evidence existed at that moment and which later record confirmed or contradicted the outcome. Validation is then something computed across many records, never a property typed into one.
What each forecast number in the CRM actually records
Each number answers a narrow question, and the dataset should say which. The table separates the common fields by who sets them and what they cannot tell you.
| CRM value | Who or what sets it | What it records | What it is not |
|---|---|---|---|
| Stage default probability | Admin, once per stage in pipeline settings | The organization's assumed win rate for a stage | A measured rate for this deal or segment |
| Overridden probability | Rep or manager with edit access | One person's judgment at the time of the edit | Evidence of buyer commitment |
| Forecast category | Stage mapping, or the owner overriding it | A judgment bucket such as pipeline, best case or commit | A probability; categories are not percentages |
| Manager forecast adjustment | Manager in the forecasting tool | A roll-up judgment that may not touch any deal record | A change to the underlying opportunity |
| Close date | Rep, often edited repeatedly | Expected timing as believed on the edit date | The date a contract was signed |
| Weighted value | Formula: amount multiplied by probability | Expected value under the stage assumption | A forecast that has been checked against results |
| Predictive score | A vendor or in-house model | Model output under its own training data | An independent label; using it as a target is circular |
What it takes for a close probability to count as validated
A probability is validated when, across a defined set of past opportunities, the stated values match observed close rates and the uncertainty of that match is reported. scikit-learn's documentation puts the idea plainly: for a well calibrated classifier, among samples given a predicted probability near 0.8, about 80 percent should actually belong to the positive class. The same check applies to percentages typed by people.
The standard tools are a reliability diagram and a proper score. A calibration curve bins the stated probabilities and plots each bin's average against the fraction of positives in it, with the usual trade-off that more bins need more data to keep each estimate stable. The Brier score, the mean squared difference between stated probabilities and outcomes, rewards calibration and discrimination together, which is why scikit-learn cautions that a lower Brier score does not by itself mean better calibration.
Both have sampling error. Work in Weather and Forecasting by Bradley, Schwartz and Hashino derived expressions for the sampling uncertainty of the Brier score and showed how to build confidence intervals for it. For revenue data, the practical consequence is that a segment with a few dozen closed opportunities rarely supports a validated percentage at all, and the dataset should say so instead of reporting a precise-looking number.
The uncertainty and evidence annotation framework
The framework below is the asset. Every opportunity snapshot gets these annotations; the validated rate fields are filled only by computation across cohorts, never by hand. It works with Salesforce, HubSpot or Dynamics records once their field names are mapped.
At a glance
The uncertainty and evidence annotation framework
Two of the annotations do most of the work. buyer_evidence_tier records what the customer had actually done by the snapshot date, such as approving a quote or signing an order form, so a commit call backed by a signature can be told apart from one backed by a friendly meeting. outcome_definition forces the team to decide, before anyone labels a record, what counts as a win: signature alone, signature that survives a cancellation window, or a first invoice paid. Different definitions produce different close rates from the same records, and a dataset that does not state its definition cannot be compared with another one. Keep the definition in the data card and apply it to every opportunity in the export, including the ones that embarrass the forecast.
| Annotation | Allowed values | Evidence required | Rule |
|---|---|---|---|
| confidence_source | stage_default, rep_override, manager_override, model_score | Field history showing who changed the value | If history is missing, use unknown, not stage_default |
| confidence_value_as_of | Value and snapshot date | Field history or forecast snapshot | Never backfill from the current record |
| buyer_evidence_tier | signed_order, approved_quote, mutual_plan, verbal, none | Document or activity record dated before the snapshot | Tier is set by the strongest dated document, not by stage |
| signer_authority | verified, unverified, not_applicable | Signature block, approval record or delegation policy | Unverified authority keeps signed_order provisional |
| outcome_definition | Written rule, such as signed and not canceled within a set period | Data card | One definition per dataset, stated before labeling |
| outcome_label | won_as_forecast, won_changed_terms, slipped, lost, no_decision, open_at_cutoff | Contract, amendment, order or delivery record | From documents after the snapshot; CRM status alone is not enough |
| calibration_cohort | Segment, stage, snapshot week and period | Cohort definition | Defined before computing rates |
| validated_rate | Observed close rate with interval, or insufficient_data | Computed across the cohort | Report insufficient_data below the minimum cohort size you set |
| attribution_limit | Free text | Reviewer note | Record known gaps: deleted deals, merged accounts, changed stage definitions |
When the signed contract disagrees with the opportunity record
A won opportunity is not an outcome until the contract and what followed agree with it. The CRM status records a rep closing the record; the outcome label should come from the signed agreement, its amendments and the order, delivery or invoice records that followed.
The contradictions are predictable, and each one changes the label.
- Closed won, but the signed order covers a shorter term, fewer units or a different product than the opportunity: label won_changed_terms and keep the opportunity's last values as the forecast, not as the outcome.
- Closed won, but the signature came from someone without authority under the customer's own approval rules, and the order was later re-signed: the outcome date is the valid signature, not the CRM close date.
- Closed won, then canceled within the period your outcome definition sets, or never provisioned: under most definitions this is lost or no_decision, whatever the CRM says.
- Closed lost, then a new opportunity for the same account and scope is won weeks later: link the two, or the first will be labeled lost when the buyer simply moved the deal.
- Close date moved repeatedly before a win: the forecast at each earlier snapshot was wrong on timing even though the deal eventually closed, which is the slipped label at those snapshots.
Keeping the as-of history so labels reflect what was known then
Labels must use what was known at the snapshot date, which means reconstructing past values from history records rather than reading today's opportunity. Salesforce keeps stage history for opportunities and records changes to tracked fields in OpportunityFieldHistory, but only for fields that have history tracking turned on.
Retention limits how far back that goes. Salesforce's help documentation says that without Field Audit Trail, field history is retained for up to 18 months, and up to 24 months through the API; with Field Audit Trail it is kept until deleted. A team that wants several years of confidence history has to export it before it ages out or rely on periodic forecast snapshots it saved itself.
Overrides need special care. Because a manual edit in HubSpot stops automatic stage-based probability updates, and Salesforce lets users with edit access override the stage value, the same probability field can mix system defaults and personal judgments within one export. The confidence_source annotation exists to separate them, and it can only be filled from history, not from the current value.
Worked example (illustrative): auditing a quarter of commit calls
The figures here are illustrative, invented to show the method; they do not describe any company.
A software company exports four quarters of opportunity history with snapshots taken in week 6 of each quarter. In the illustration, 180 opportunities were in the commit category at those snapshots. Applying an outcome definition of signed and not canceled within 60 days, 63 percent of them closed in the quarter, 21 percent slipped to a later quarter and 16 percent were lost or ended with no decision.
Splitting by confidence_source changes the picture. Commit calls that matched the stage mapping closed at an illustrative 67 percent; commit calls where the rep had overridden a lower-stage mapping closed at 48 percent, and the interval around that figure is wide because only 40 such opportunities exist. Opportunities with a signed_order tier at the snapshot closed far more often than those with verbal evidence, regardless of category.
The exception is the useful part. One segment, a new product line, has only 14 commit opportunities across the year. The framework records validated_rate as insufficient_data for that segment instead of borrowing the company-wide figure, and the data card says the stage definitions for that pipeline changed mid-year, which is itself an attribution_limit.
Where even a calibrated rate misleads
A rate that was well calibrated on last year's opportunities can still mislead. Calibration is a property of a cohort under the conditions that produced it, and revenue processes change those conditions often.
- Stage definitions or exit criteria change, so the same stage name no longer means the same thing across the period.
- Deleted or merged opportunities vanish from exports, which removes losses more often than wins and inflates rates.
- Rep turnover changes who is making the calls; a calibrated rate for a team says little about a new hire's overrides.
- Pricing, packaging or market shifts change base rates, so a rate computed before the shift is stale after it.
- Using a predictive score as both input and label makes any calibration result circular.
Conditions that call for a narrower dataset or a specialist
Stop, narrow the scope or bring in a revenue operations lead and a contracts or privacy reviewer when any of these apply. Customer contracts may restrict sharing their terms, so contract evidence often has to appear as a tier value rather than as the document. Rep-level confidence history is information about identifiable employees and their performance, which raises its own privacy and employment questions. Customer names and deal details usually need redaction or tokenizing.
Rights are a separate gate. Being able to export opportunities is an access permission, not a right to license them, and de-identifying account names does not change what customer contracts allow. Counsel reviews those terms against the specific records.
Opportunity histories as licensed data
Opportunity histories with intact field history, contract links and dated outcomes are exactly the records where this framework pays off, and they are the version of CRM data AI developers can evaluate against. Opportunity histories like these can become a licensed dataset through SourceX, with sourcing, rights review, contracting, preparation, delivery and payment all part of its process.
As a general guide, buyers look for consistent history over two or more years and large record counts in one system, though smaller sets can qualify when the work is rare. A short company assessment, with no data upload, shows whether a company qualifies.
Frequently asked questions
Is the opportunity probability in a CRM a real probability?
Usually not. It is either a stage default chosen by an administrator or a value someone typed in. It becomes a validated probability only when stated values are compared with actual outcomes across many past opportunities and the match holds within a reported interval.
What is the difference between forecast category and probability?
A forecast category such as commit or best case is a judgment bucket, often mapped from stage and sometimes overridden by the owner. Probability is a percentage, again often mapped from stage. Neither is evidence of buyer commitment, and they can disagree on the same deal.
How many deals do you need to calibrate sales forecasts?
There is no universal number, because the uncertainty depends on the close rate and on how finely you bin. Compute an interval for each cohort and report insufficient data where the interval is too wide to act on, rather than borrowing a company-wide figure.
Should closed won be used as the outcome label?
Only after checking it against the signed contract and what followed. Changed terms, invalid signatures, early cancellations and re-opened opportunities all make the CRM status a poor label on its own. Write the outcome rule down before labeling.
Why does field history retention matter for forecast datasets?
Because past confidence values exist only in history. Salesforce keeps standard field history for up to 18 months in the interface and 24 months through the API unless Field Audit Trail is on, so longer histories need exports or saved forecast snapshots.
Sources
- Salesforce updates an opportunity's Probability whenever its stage changes, and users with edit access to the field can override the value. Source
- Salesforce maps each opportunity stage to a forecast category, lets owners override the category on their opportunities and requires Closed/Lost stages to map to Omitted. Source
- HubSpot sets Deal probability from each stage's win probability, stops automatic updates after a manual edit, defines Weighted amount as amount multiplied by probability and lists forecast category options. Source
- Salesforce's OpportunityFieldHistory object represents the history of changes to the values in an opportunity's fields. Source
- Salesforce retains field history for up to 18 months, and up to 24 months through the API, unless Field Audit Trail is turned on. Source
- scikit-learn's probability calibration guide defines calibration, explains calibration curves and notes that a lower Brier loss does not necessarily mean a better calibrated model. Source
- Bradley, Schwartz and Hashino (2008) derive sampling uncertainty estimates and confidence intervals for the Brier score and Brier skill score. Source
See if your company qualifies
A short company assessment. No data uploads are needed.