AI uses for records
A budget variance evidence quality dataset labels claims, not tone
By SourceX Editorial · Reviewed by Noah Loul ·
9 sections
Short answer
A budget variance evidence quality dataset should label every causal claim, not the commentary as a whole: evidenced when a dated record reproduces the cause's size, plausible when the driver is named but unsized or unrecorded, speculative when nothing could test it. In 2,064 variance sentences from FinQA's annual report pages, 72.5 percent gave the cause no number, yet only 1.9 percent hedged with may, could or believe.
Value drivers
Budget variance evidence
Key takeaways
- Label each cause attached to each movement; one paragraph of commentary usually mixes evidenced, plausible and speculative claims.
- In 2,064 FinQA variance sentences the change carried a number 42.8 percent of the time and the cause 27.5 percent.
- Speculation hides in attribution hedges such as primarily (65.5 percent of sentences), not in may or believe (1.9 percent).
- Date every claim against its records: support that arrived after the writing is hindsight, and forecasts are graded as forecasts.
- A claim label grades the explanation only; treatment and authorization need separate fields from a qualified accountant.
What is a claim in variance commentary, and how is it labeled?
A claim is one cause attached to one movement. 'Freight rose because of expedited shipments' is a claim, and a paragraph of month-end commentary usually holds three or four of them. A budget variance evidence quality dataset should label each claim on its own, because commentary can be right about the variance and wrong, or merely hopeful, about why it happened.
Three labels cover most claims. Evidenced means the claim names a driver, sizes its contribution as an amount or a share, and points to a record that reproduces that size and existed when the commentary was written. Plausible means the driver is named and fits the records, but its contribution is unsized or the record that would size it is missing. Speculative means nothing in the packet could confirm or refute the claim: a motive, a market mood, a forecast or a condition nobody measured.
The variance is usually the solid part. Budget and actual both come from the ledger, so the gap is arithmetic. The explanation is a set of assertions the ledger does not contain, and that is where labels earn their keep. This page measures how public variance commentary phrases those assertions, then turns the result into a rubric, a casebook and a packet a buyer can test.
How 2,064 causal sentences in annual reports state their causes
Most causes in public variance commentary are named but not sized. FinQA, a public dataset that 11 finance professionals built from S&P 500 earnings reports covering 1999 to 2019, keeps the sentences printed around each table. Across its 2,789 distinct report pages from 137 companies, a fixed rule found 2,064 sentences that pair a change word, such as increased or declined, with a causal connector, such as due to or driven by.
The change was sized far more often than its cause. 883 of those sentences (42.8 percent) put a number on the movement, while 568 (27.5 percent) put one on the cause and 1,496 (72.5 percent) left the cause without any number. When the script looked for the cause's number in the table printed on the same page, it found it in 28 sentences, 1.4 percent; the movement's number appeared there in 101, or 4.9 percent.
A reading check kept the rule honest. In 50 sentences drawn with seed 717, 47 explained a change in a reported or projected figure, and the script's sized-or-unsized call for the cause matched a human reading in 42 of those 47. The counts by pattern follow; the final section states every pattern the script uses.
| Pattern in the sentence | Sentences | Share of 2,064 | What it tells a labeler |
|---|---|---|---|
| Cause given a number | 568 | 27.5% | A size exists to check; it still needs a record |
| Cause given no number | 1,496 | 72.5% | Plausible at best until a record sizes it |
| Attribution hedge: primarily, partially, mainly | 1,351 | 65.5% | Other causes exist and carry an unstated share |
| Epistemic hedge: may, could, believe | 39 | 1.9% | Rare; speculation seldom announces itself |
| Offsetting factor named | 530 | 25.7% | A net figure hides at least two claims |
| Market or macro cause: economy, rates, demand | 252 | 12.2% | 191 of these, 75.8 percent, carry no number |
| Forward-looking term: expect, will, anticipate | 101 | 4.9% | No record can exist yet; label it a forecast |
Why hedge words miss most speculative explanations
Searching commentary for may, might or we believe finds almost nothing: 39 of the 2,064 sentences. The words that carry the uncertainty are attribution hedges, present in 1,351 sentences or 65.5 percent. 'Primarily due to higher volume' concedes that other causes contributed without saying how much, and 937 sentences (45.4 percent) pair such a hedge with a cause that has no number.
Regulators and preparers read that word differently, which is the labeling problem in miniature. An SEC staff comment restated in Art Technology Group's 2008 response letter on EDGAR asked for specific figures in place of vague terms such as primarily. Intuit's 2006 response explained that it uses primarily to mean the cited reasons account for a substantial majority of the variation, and CSG Systems argued in 2008 that primarily and principally flag the most important drivers. Each reading is defensible, and none can be checked from the sentence alone.
So the label cannot be read off the tone. A crisp sentence that names one cause with no hedge claims the whole variance for it, which is strong if a record backs it and speculative if none does; 491 sentences (23.8 percent) name a cause with no number, no hedge and no offsetting factor. A hedged sentence whose driver is sized and recorded can be the best-evidenced line in the pack. Label each claim by what it sizes and what it cites.
The rubric: evidenced, plausible and speculative claims
Each claim gets one of the three labels and, where they apply, one of two flags: hindsight, for support that arrived after the writing, and contradicted, for a claim the records refute. The test column is what a reviewer checks; the wording column shows how such claims tend to read, and the quoted examples with figures are invented for illustration.
| Label or flag | Test the claim must pass | Typical wording | Reviewer action |
|---|---|---|---|
| Evidenced | Driver named, contribution sized, and a record in the packet dated on or before the commentary reproduces the size | License costs rose 9 percent; 40 added seats at the contract price explain 6 points (Illustrative) | Keep; link the record and the computation |
| Plausible | Driver named and moved in the stated direction, but the contribution is unsized or the sizing record is absent | Primarily due to higher volume | Keep; ask for the driver record or size the share |
| Speculative | No record in the packet could confirm or refute it | Reflecting softer market conditions | Keep as a negative example; never a training target |
| Hindsight flag | Sized and recorded, but the record is dated after the commentary | Confirmed by the credit note received the next month | Mark evidenced later; exclude from close-simulation tasks |
| Contradicted flag | A record in the packet shows the driver did not move, or moved the other way | Overtime drove the overrun, while payroll shows overtime hours flat (Illustrative) | Move to the counterexample set with the disproving record |
Who could have known it when the commentary was written?
Every claim needs a knowledge date: when the commentary was written, by whom, and which records existed at that moment. A claim supported only by a record that arrived later was not evidence for its writer, even if it proves true, and a model trained to draft close commentary should be graded only on what was knowable at the close.
Accounting already draws this line for financial statements. IAS 10 divides events after the reporting period into adjusting events, which provide evidence of conditions that existed at the end of the period, and non-adjusting events, which indicate conditions that arose afterwards. The same split works for variance claims: a supplier credit issued next month can confirm that a pricing dispute existed in the period, so the claim becomes evidenced later; a price cut announced next month cannot explain this month's gap at all.
The writer's position matters as much as the date, because position decides access. A budget owner may know about a slipped project before any record shows it, while an analyst writing from the ledger cannot. Auditing standards treat that kind of knowledge as a lead rather than proof: ISA 520 requires auditors to investigate significant unexpected differences by inquiring of management and obtaining appropriate audit evidence relevant to management's responses, and its guidance calls for further procedures when the explanation and its evidence are not adequate. Record the writer's role, and keep an unrecorded insider claim plausible until a record backs it.
Forward-looking claims need their own handling. In a seed-1 reading of 20 of the 101 sentences with expect, will or anticipate, 13 explained movements that had not happened yet; the rest used the words in a historical sense, as in an expected loss model. Nothing in a packet can evidence a forecast at writing time. Label them forecasts, attach the later outcome when it arrives, and grade them as predictions rather than explanations.
Casebook: eight claim patterns and the label each starts with
The casebook pairs eight common claim patterns with the label each starts with before any record is attached, and with the record that would move it. Counts are sentences from the FinQA audit, the examples paraphrase sentences seen in the validation sample, and the patterns overlap, so the counts do not add up to 2,064.
| Claim pattern, paraphrased from FinQA pages | Count in the 2,064 | Starting label | What would change it |
|---|---|---|---|
| Cause sized in the sentence: about $475 million lower sales as programs wound down | 568 | Plausible | Program revenue records that reproduce the amount make it evidenced |
| Hedged, unsized cause: net income rose primarily due to higher net revenue | 937 | Plausible | A driver record plus a size for the unnamed remainder |
| One cause, no hedge, no number: goodwill fell due to a tax contingency adjustment | 491 | Plausible, and it claims the whole gap | An adjustment entry equal to the change makes it evidenced |
| Driver measured, contribution not: average debt rose from $9.6 billion to $10.8 billion | Not separated by the rule | Plausible | A volume-times-rate bridge built from debt and rate records |
| Market or macro cause: demand fell as a result of economic recession | 252 | Speculative | A driver series in the packet, such as orders or prices, tied to the line |
| Offsetting factors: higher volume partially offset by lower prices | 530 | Two or more separate claims | A label for each side; the net size evidences neither |
| Forward-looking cause: operating profit expected to rise on program mix | 101 | Forecast | Later actuals, graded as a prediction |
| Hedged belief: may, could or we believe attached to a cause | 39 | Speculative | The same tests as any claim; the hedge is candor, not support |
Can an evidenced explanation sit on a wrong number?
Yes. A claim can be sized, recorded and timely while the variance it explains rests on a misposted accrual, a reclassification nobody approved or spending above an approver's limit. The label grades the explanation, not the accounting, so a dataset needs separate treatment and authorization fields, filled by a qualified accountant from the journal and approval records.
Arithmetic is the weakest test in the set. Shares that add up to the whole gap prove that someone allocated the gap, not that each share is right, and a residual folded quietly into the largest driver closes any bridge. The rubric therefore asks every claim to reproduce its own size from its own record and leaves whatever is unexplained as a named remainder.
Contradicted claims are the most useful byproduct. When a record shows the driver did not move, keep the claim with the disproving record in a counterexample set. The existing guide to evidence behind budget variance explanations describes how to keep rejected causes; the labels on this page decide which claims enter that set and which merely wait for a record.
What a labeled variance packet should hand a buyer
A packet that supports claim labels carries the commentary as issued, with its timestamp and the writer's role; the budget and actual lines; the driver records each claim could cite, such as unit counts, rates, headcount, contract terms and invoices, each with its own date; the claim segmentation; one label and any flag per claim; the reviewer's name, qualification and review date; and the separate treatment and authorization fields.
Buyers can test such a packet before training on it. Ask what share of claims are evidenced, plausible and speculative; how often two reviewers agree on a label; how many evidenced claims rest on records dated after the commentary; and whether contradicted claims were kept. A pack in which nearly every claim comes back evidenced has probably labeled the tone rather than the records.
Authorization and de-identification come first, with stable pseudonyms so vendor and employee references still join across records. SourceX keeps no finance datasets on the shelf; it looks for them at U.S. companies once a buyer describes the need, and a request can name the claim labels, the knowledge-date rule and a reviewer-agreement threshold as acceptance conditions.
How the FinQA audit counted, and what it cannot show
The audit reads the four JSON files in FinQA's dataset folder at commit 0f16e28 and runs no repository code. A page is one unique filename across train, dev, test and private_test, read in that order; no filename had conflicting records. A sentence is a string in the text before or after the table with at least four word tokens, counted once per page; the script printed 9,014 shorter strings and 1,197 page duplicates it dropped. A variance sentence contains one of 16 change terms (increase, decrease, decline with their -s and -d forms, rose, fell, grew, improved, improvement, reduction, reduced; higher and lower are excluded because they mostly modify nouns) followed by one of 17 connectors (due to, driven by, as a result of, result of, resulting from, attributable to, attributed to, caused by, because of, because, owing to, reflect, reflects, reflected, reflecting, stemming from, on account of), skipping a reflect form directly after 'to'. Text before the connector is the effect; text from it onward is the cause.
A quantity is a token beginning with a digit, other than a year from 1990 to 2030 without a dollar sign, a one- or two-digit footnote marker in parentheses, or a number after note, item, page, section or part. A cause or effect is checkable when one of its quantities equals a number in the page's table body by exact decimal value and is at least 10 or has a decimal point. Attribution hedges are primarily, principally, mainly, largely, partially, partly, predominantly, mostly and in part; epistemic hedges are may, might, could, believe and its forms, likely, possibly, probably, appear and seem and their forms; macro terms in the cause are economy and related words, recession, inflation, weather, market conditions, interest or exchange rates, currency, demand, words beginning competiti, pandemic and hurricanes; an offsetting factor is any word beginning offset; forward-looking terms are expect, expects, expected, anticipate, anticipates, anticipated, will, forecast, forecasts and projected. Percentages come from integer counts, rounded once, half up.
The limits are real. Annual report text explains year-over-year changes for public readers, not budget gaps for managers, and FinQA's pages were chosen for question answering, not sampled at random. The rule misses causes sized before the connector, counts a measured driver as a sized cause, and its table check is a lower bound because text and tables use different units. The audit shows how public commentary phrases causes; it does not show how often company commentary is right, which needs authorized company packets labeled by qualified accountants. The script, its output and the reading notes are kept with the analysis files; the only random step is the seed-717 sample.
Frequently asked questions
What is a budget variance evidence quality dataset?
It is a set of variance cases in which every causal claim in the commentary carries a label: evidenced, plausible or speculative, plus flags for hindsight and contradiction. Each case also holds the budget and actual lines, the dated driver records the claims could cite, and the reviewer who assigned the labels.
Is 'primarily due to' an acceptable variance explanation?
It is acceptable as a pointer, not as evidence. The phrase concedes other causes without sizing them, and preparers use it to mean different things. In a labeled dataset it starts as plausible and moves to evidenced only when a record sizes the named driver and the remainder is shown.
How do you label a cause that turned out true after the close?
Label it evidenced with a hindsight flag, and record the date of the confirming record. It is useful for grading explanations written later, but a task that simulates writing commentary at the close should treat the claim as plausible, because the writer could not have seen the proof.
Can a model learn to write evidenced commentary from public filings?
Filings can teach the phrasing and test a labeling rule, as the FinQA audit here does, but they rarely contain the driver records that would evidence a cause. Teaching a model to cite and size causes needs company packets that link commentary to unit counts, rates, contracts and invoices.
Who should assign the claim labels?
A reviewer with accounting qualifications, working from the packet's records, ideally with a second reviewer on a sample so agreement can be reported. Automated rules like the one used here can pre-sort claims by sizing and hedges, but they cannot judge whether a record really reproduces a claim.
Sources
- FinQA's public repository, at the commit read for this audit, holds the dataset files with the text before and after each table, the table and an id built from the original report name. Source
- The FinQA paper says 11 finance professionals built the dataset from publicly available earnings reports of S&P 500 companies from 1999 to 2019, collected in FinTabNet. Source
- ISA 520 requires auditors to investigate significant unexpected differences by inquiring of management and obtaining appropriate audit evidence relevant to management's responses, with further procedures when the explanation is not adequate. Source
- IAS 10 defines adjusting events as those that provide evidence of conditions existing at the end of the reporting period and non-adjusting events as those indicating conditions that arose after it. Source
- Art Technology Group's 2008 response letter on EDGAR restates an SEC staff comment asking for specific figures in place of vague terms such as primarily when describing changes. Source
- Intuit's 2006 response letter on EDGAR explains that it uses primarily to mean the cited reasons account for a substantial majority of a variation. Source
- CSG Systems' 2008 response letter on EDGAR argues that primarily and principally identify the most important drivers of a change. Source
Tell us what data you need
Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.