AI uses for records
Quarter close holdout: how to test whether a finance agent generalizes
By SourceX Editorial · Updated
9 sections
Short answer
To test whether a finance agent generalizes, train and tune it only on earlier quarters, then evaluate it on a later quarter that has fully closed and settled, sealed from everyone who builds the agent. Tag every task in that quarter that depends on a rule which changed after training, and score the agent on the ledger end state it produces, reported separately for unchanged and changed rules.
Evaluation split
Quarter close holdout
Key takeaways
- Random splits of finance records leak recurring vendors, accounts and month-end patterns into the test set and overstate accuracy.
- A holdout quarter is usable only after it has settled: late adjustments, reversals and reclassifications posted, and the period locked.
- Tag holdout tasks for policy drift, such as a new approval limit or a changed reimbursement rate, and report those tasks separately.
- Score the agent on the resulting ledger state and on whether it applied the rule in force on the transaction date, not on wording.
- Stop or narrow the test when the future quarter is dominated by a one-off event or holds too few tasks to tell results apart.
How do you use a future quarter to test finance agent generalization?
Split by time, not by transaction. Choose a training window of closed quarters, a tuning quarter after it for prompt and threshold choices, and a later holdout quarter that nobody building the agent may look at. Evaluate once on the holdout, report the result with its uncertainty, and only then decide whether the agent is ready for the next live quarter.
The holdout must be a quarter that is truly over. Its period should be locked, its late adjustments posted and its outcomes known, so every task has a verified answer. Within it, mark the tasks whose correct answer depends on a rule that changed after the training window ended. Those are the tasks that tell you whether the agent follows the rules in force or repeats what it saw.
This is a plan for measuring generalization, not a report of results. No benchmark figures for any agent are given here.
Why does a random split overstate a finance agent's accuracy?
Because finance records repeat. The same vendors bill every month, the same accruals are booked and reversed every close, and the same coding decisions recur for years. A random split puts near-copies of each test task into the training pool, so the agent can score well by remembering rather than reasoning.
The scikit-learn documentation states the general problem plainly: classical cross-validation methods such as KFold and ShuffleSplit assume samples are independent and identically distributed, which does not hold for data correlated in time, and time series models should be evaluated on future observations. Its TimeSeriesSplit tool trains on earlier folds and tests on the next one, so each training set is a superset of the one before.
Leakage of this kind is common, not exotic. Kapoor and Narayanan surveyed machine-learning-based science and found leakage affecting 329 papers across 17 fields, and set out a taxonomy of eight leakage types. Time also degrades models directly: Lazaridou and colleagues found language models performed up to 16% worse on text published up to two years after their training period ended, and larger models degraded the same way. A finance agent faces the same pressure each quarter as vendors, prices and policies move.
When is a quarter settled enough to hold out?
A quarter is ready for use as a holdout when its ledger periods are locked against new postings, post-close adjustments have been booked, and the outcomes your tasks depend on, such as payments clearing or accruals reversing, have happened. Before that, the correct answer for some tasks is still moving.
ERP period controls give a first signal. Microsoft's guidance for closing the general ledger in Dynamics 365 Finance suggests setting periods to On hold so no further posting occurs during period-end work, and warns against Permanently closed because a permanently closed period cannot be reopened. Many teams therefore leave a quarter On hold for weeks. Record the period status and the date it last changed in each task, and treat a period still open to posting as unsettled.
For public companies, reporting deadlines set a practical minimum. The SEC's Financial Reporting Manual gives the Form 10-Q deadline as 40 days after quarter end for accelerated and large accelerated filers and 45 days for non-accelerated filers. A quarter whose figures have not yet been filed is a poor holdout, because review-driven adjustments can still change the answers. Private companies should use their own close calendar and audit timing the same way.
The quarter-based holdout plan
The table below is the core of the plan. Fill in each row for your records and keep the completed table with the evaluation report, so anyone reading a result can see exactly which periods and entities it covers.
| Component | What it is | Rule | Record to keep |
|---|---|---|---|
| Training window | Consecutive closed quarters used for fine-tuning, few-shot examples and retrieval indexes | Ends before the tuning quarter begins; nothing later enters any index | First and last period, entities, task counts |
| Tuning quarter | One closed quarter used to choose prompts, thresholds and tools | Used as often as needed, never reported as the final result | Number of tuning runs and what changed between them |
| Settling gap | Time between quarter end and holdout use | Holdout starts only after periods are locked and late entries posted | Period status history and date of last posting |
| Sealed holdout quarter | The later quarter used once for the reported evaluation | Access limited to the evaluation owner; no examples copied into prompts | Access log and evaluation date |
| Unseen entity | A subsidiary or business unit absent from training | Tests transfer to a new chart of accounts or approval matrix | Entity mapping and chart of accounts version |
| Drift slice | Holdout tasks whose answer depends on a rule changed after training | Tagged before evaluation from the policy drift register | Task ids, rule changed, effective date |
| Refresh rule | When the holdout rolls forward | Each newly settled quarter becomes the next holdout; the old one may join training | Version of the plan and date rolled |
Which policy changes should the holdout quarter be tagged for?
Tag any task whose correct answer would differ under the rules of the training window. That separates two failures that look the same in an overall score: an agent that has not seen a vendor before, and an agent that applies last year's rule to this year's transaction.
Some changes come from outside the company. The IRS sets standard mileage rates by period, for example, and each edition of Publication 463 states the business rate for its tax year; an expense agent trained only on earlier years has never seen the newer rate applied. Others come from inside: a new approval limit, a revised capitalization threshold, a new chart of accounts segment or a change of revenue recognition policy. Build the register from policy documents and their effective dates, not from memory.
| Drift type | Illustrative example | How to tag a task | Correct agent behavior |
|---|---|---|---|
| External rate or table | A reimbursement rate published for a new year | Transaction date on or after the new rate's effective date | Applies the rate in force on the transaction date |
| Approval authority | A manager's spending limit lowered mid-year | Amount between the old and new limit | Routes for the extra approval instead of auto-approving |
| Accounting policy | A higher capitalization threshold adopted at year start | Asset purchases between the old and new threshold | Expenses or capitalizes under the new threshold |
| Chart of accounts | A cost center split into two | Postings to the retired cost center's successors | Codes to the new segment, never the retired one |
| Vendor or contract terms | New payment terms after a contract renewal | Invoices dated after the amendment | Uses the amended terms for due dates and discounts |
| Process change | Three-way match now required for a category | Invoices in the category without a receipt | Holds the invoice and asks for the receipt |
What should the evaluation measure on the future quarter?
Measure what the agent leaves behind in the ledger and whether it followed the rule in force, not whether its message reads well. Agent benchmarks such as tau-bench grade by comparing the final database state after the agent acts with an annotated goal state, and report pass^k, the chance an agent succeeds on all of k repeated attempts. Both ideas carry over directly to finance tasks, where a plausible summary can sit on top of a wrong journal entry.
Always compare against two baselines on the same holdout tasks. The first is the agent's own score on the tuning quarter: a large drop from tuning to holdout is the clearest sign that tuning fitted the earlier period. The second is the rule your team already follows, such as the existing coding rules or approval routing in the ERP, scored the same way. An agent that cannot beat the current rules on the drift slice has not yet earned more autonomy, however well it does on familiar vendors. Write both baselines into the plan before the holdout is opened, so nobody chooses the comparison after seeing the numbers.
- End-state accuracy: the journal entries, codings, matches or holds the agent produced equal the reviewed final state for the task.
- Drift-slice accuracy, reported separately from the unchanged-rule slice, with the gap between the two as the headline generalization signal.
- Unseen-entity accuracy, reported separately again, because a new chart of accounts is a different kind of novelty from a new rule.
- Reliability: repeat each task several times and report how often the agent succeeds on every attempt, since a close process cannot rely on an agent that is right only some of the time.
- Correct abstention: on tasks where the records are insufficient, the agent escalates or asks for the missing document instead of posting.
- Uncertainty: report counts of tasks per slice and an interval for each rate, resampling by vendor or entity rather than by line so repeated vendors do not shrink the interval falsely.
Illustrative worked example: an accrual and AP agent across two entities
This example is illustrative, not taken from a real company or evaluation, and uses counts and durations only. It shows how the plan is filled in, not what result to expect.
- Training window: eight closed quarters for entity A, covering invoice coding, accrual proposals and three-way match holds, about 9,000 tasks after removing duplicates.
- Tuning quarter: the next quarter for entity A, about 1,100 tasks, used for prompt and threshold changes over several rounds.
- Settling gap: the holdout quarter's periods move to On hold six weeks after quarter end and the last late accrual reversal posts in week seven, so the holdout is frozen in week eight.
- Sealed holdout: the following quarter for entity A, about 1,200 tasks, plus all of that quarter's tasks for entity B, about 400, which never appeared in training.
- Drift slice: the policy register shows a lowered approval limit for one department and a new reimbursement rate starting that year; 140 holdout tasks are tagged before evaluation.
- Readout: end-state accuracy, reliability and abstention reported for four slices (entity A unchanged rules, entity A drift slice, entity B, all), each with a count and an interval.
- Exception found: 60 of entity B's tasks relate to a one-time system migration; they are reported apart from the rest of entity B and excluded from the headline figure, with the reason written into the plan.
When is a single future quarter the wrong test?
Stop or narrow the test when the holdout quarter cannot support the conclusion you want to draw. A quarter dominated by an acquisition, a system migration or a restatement measures the agent on that event, not on ordinary work; report those tasks apart, or choose another quarter.
Narrow the claim when slices are small. If the drift slice holds a few dozen tasks, the interval will be wide, and the honest conclusion is that drift handling is untested rather than good or bad. Seasonal businesses need care too: a fourth-quarter holdout after training on the first three quarters of one year tests year-end processes the agent may never have seen.
Bring in specialist review when the drift involves accounting standards, tax rules or regulatory reporting. Whether an agent's treatment was correct under a new standard is a judgment for a qualified accountant, and the task's reviewed answer should record who made it.
Preparing quarter-stamped finance records for licensing
Records that support this kind of evaluation are worth more when every task carries its period, period status, entity, policy version and reviewed outcome. A company thinking about licensing finance history can check whether its ERP exports keep those fields across several years before any other work.
Qualification with SourceX begins with a short company assessment; no records are uploaded. As a rule of thumb, buyers look for years of consistent history, typically two or more, and tens of thousands of records in one system, with each deal assessed individually. Delivery happens only after the company approves the scope and terms and signs the agreement.
Frequently asked questions
What is a quarter close holdout in a finance dataset?
It is a later, fully closed quarter of finance records held back from training and tuning and used once to evaluate an AI agent. Because it comes after the training period, it tests whether the agent handles new vendors, amounts and rules rather than remembering earlier quarters.
How long after quarter end should the holdout be frozen?
Freeze it once its ledger periods are locked against posting and late adjustments and reversals have posted. For a public company that is after its quarterly report is filed; for others, after the internal close and any review adjustments. Record the freeze date in the plan.
What is policy drift in finance agent evaluation?
Policy drift is a change after the training window in a rule that decides the correct answer, such as an approval limit, a reimbursement rate or a capitalization threshold. Tagging tasks affected by drift and scoring them separately shows whether the agent applies current rules or repeats old ones.
Can the holdout quarter be added to training later?
Yes. Once a newer quarter has settled and becomes the next holdout, the previous one can join the training window. Keep a dated record of each roll-forward so results from different versions are not compared as if they used the same test.
Is one future quarter enough?
Often it is enough for a first readout, but not for every claim. If slices are small or the quarter was dominated by a one-off event, add another settled quarter or narrow the conclusion to what the tasks can support.
Sources
- scikit-learn notes that classical cross-validation assumes independent samples and that time series models should be evaluated on future observations; TimeSeriesSplit trains on earlier folds and tests on later ones. Source
- Kapoor and Narayanan found data leakage affecting 329 papers across 17 fields and described eight types of leakage. Source
- Lazaridou et al. found language models performed up to 16% worse on text published up to two years after training, and larger models degraded the same way. Source
- Microsoft suggests setting Dynamics 365 Finance ledger periods to On hold during period-end and warns that Permanently closed periods cannot be reopened. Source
- The SEC Financial Reporting Manual gives Form 10-Q deadlines of 40 days after quarter end for accelerated and large accelerated filers and 45 days for non-accelerated filers. Source
- IRS Publication 463 states the standard mileage rate for business use that applies to the tax year it covers, so the rate changes between years. Source
- tau-bench grades agents by comparing the final database state with an annotated goal state and reports pass^k reliability across repeated trials. Source
See if your company qualifies
A short company assessment. No data uploads are needed.