AI uses for records
Post-training instruction fidelity: testing requirement retention
By SourceX Editorial · Updated
10 sections
Short answer
To train and evaluate requirement retention, split each business task into atomic requirements traced to where they were stated and where they must appear in the final result. Grade each one on the end state, group failures into cohorts, select training examples for one cohort under a written rule, and compare against a same-size random selection on untouched held-out tasks, checking that previously passing tasks still pass.
Record timeline
Post training instruction
Key takeaways
- Instruction fidelity for agents is mostly retention: a requirement stated early or in a document must still hold in the final record.
- Trace each atomic requirement to its source, arrival step, binding scope and a verifier that checks the end state.
- Grade per requirement with a closed rubric, then report all-kept rate, average keep rate and keep rate by distance.
- Target one failure cohort, and compare the selected data with a same-size random control on a frozen held-out split.
- Count a gain only if tasks the base model already passed still pass, the way SWE-bench pairs FAIL_TO_PASS with PASS_TO_PASS tests.
How do you train and evaluate requirement retention through a whole task?
Treat every task as a list of atomic requirements, each traced to where it was stated and to the place in the final result where it must show up. Grade each requirement on the end state, group the failures into cohorts, select training examples that target one cohort under a written rule, and compare the trained model with a random selection of the same size on held-out tasks nobody used for selection.
Instruction fidelity in post-training usually means the model does what the prompt says. For an agent doing business work, the harder version is retention: a constraint given in the second message, or buried in an attached policy, still has to hold after a dozen tool calls, a correction and a handoff. The method below measures that directly and keeps the comparison honest, so a team can tell whether new data fixed the failure or only moved it.
The asset is a requirement tracing record and a grading rubric. Both are written so an evaluation lead can copy the fields into a task schema and an annotation guide without rewording.
Why single-turn instruction-following scores miss dropped requirements
Public instruction-following benchmarks test whether one response satisfies constraints stated in one prompt, which is necessary but not the same as carrying a requirement through a task. IFEval, from Google Research, checks verifiable instructions such as word counts, forbidden words or a required postscript, and its released input file holds 541 prompts. It reports prompt-level and instruction-level accuracy, each in a strict and a loose variant.
InFoBench breaks each instruction into yes-or-no questions and scores every decomposed question, and its evaluation prompt tells the judge that even a minor inaccuracy rules out a yes. FollowBench adds one constraint per level and reports three rates: all constraints satisfied, the average of individual constraints, and how many consecutive levels a model clears from level 1. Those ideas transfer well: decompose, grade each part, and track how performance falls as requirements pile up.
What they do not test is distance. Microsoft's Lost in Conversation release splits 600 instructions into shards revealed over several turns, which is closer to how customers actually state needs. Agent benchmarks such as tau-bench go further and judge the final database state: in its reward code, a run scores zero unless the hash of the resulting data matches the hash produced by replaying the expected actions, and any required output phrase is also checked. Business tasks need both properties at once: requirements that arrive late and in pieces, and grading on the record that was actually changed.
The requirement tracing record: one row per requirement, not per task
The record has one row per requirement and a header per task. Writing the requirement list before any model runs is what turns a vague transcript into a test, and it is also what lets the same task serve as training data later. The fields below are the minimum; the last three exist so failures can be grouped and the data can be traced back to its source.
| Field | What to record | Why it is needed |
|---|---|---|
| Task header | Task ID, version, business process, systems touched, the expected end state written as checkable statements, and the split (train, selection, held-out) | Fixes what success means before any run and keeps held-out tasks out of selection |
| Requirement ID and text | One atomic requirement per row, rewritten as a single checkable statement, plus the verbatim source wording | Atomic rows make partial credit and cohort counts possible |
| Source and arrival | Where it was stated (turn number, attached document, CRM field, policy page) and the step at which it became available | Lets you measure retention by arrival position and by distance to the step that uses it |
| Requirement type | Hard constraint, prohibition, preference, format or quantity rule, exception, or override of an earlier requirement | Different types fail differently; prohibitions and overrides are usually the weak spots |
| Binding scope | The steps, tool calls or output fields the requirement governs, including later handoffs | Separates kept-at-the-end from kept-throughout |
| Supersedes | The ID of any earlier requirement this one replaces, with the replacement wording | A correct agent applies the latest version; the grader must know which one that is |
| Verifier | State query, output check, tool-argument check, or a rubric question for a human or model judge, with the exact pass condition | Every requirement needs a decision rule someone else can apply |
| Cohort tags | Failure cohort labels such as late arrival, override, document-sourced, prohibition, cross-system handoff | Training selection and reporting are done by cohort |
| Lineage | Source system and record IDs, collection date, rights basis and de-identification status of any business records used | Lets a reviewer confirm the task may be used for training and rebuild it later |
Grading each requirement against the end state
Grade each requirement against the end state first and the transcript second: the stored record, the sent message or the tool arguments decide, and the conversation only explains. A model that says it will respect a constraint and then writes the wrong value has dropped the requirement.
Use a small closed set of grades so two reviewers reach the same answer. Report the share of tasks where every requirement was kept, the average keep rate across requirements, and the keep rate split by arrival position and by the number of steps between statement and use. Run each task several times; tau-bench reports pass^k, the chance that all k attempts at a task succeed, and the same idea exposes a model that keeps a requirement only some of the time.
| Grade | Definition | Evidence the grader needs | How it counts |
|---|---|---|---|
| Kept | The end state satisfies the requirement and no step violated it on the way | Verifier result on the final state plus a scan of tool calls in its binding scope | Pass |
| Kept after prompting | Satisfied only after the user or a reviewer repeated it | The turn where it was restated | Fail for retention; logged separately |
| Repaired | Violated at an intermediate step, then corrected before the end | The violating call and the correcting call | Pass on end state, flagged for side effects such as a sent message that cannot be recalled |
| Applied stale version | An earlier requirement was applied after it had been superseded | Supersedes link and the value written | Fail |
| Dropped | No sign the requirement was applied; the end state ignores it | Verifier result | Fail |
| Contradicted | The end state does the opposite, or a prohibited action was taken | Verifier result and the offending call | Fail, counted as severe |
| Correctly declined | The agent refused or escalated a requirement that conflicts with policy, and said so | Policy reference and the agent's explanation | Pass when the task marks the requirement as one to decline |
| Ungradable | The verifier cannot decide from the evidence captured | Note of what was missing | Excluded and fixed in the task, never counted as a pass |
Which failure cohort is the training data meant to fix?
Pick one cohort before collecting any training data, because a mixed fix cannot be attributed. Run the base model on the selection split, tag every failed requirement with its cohort, and choose the cohort that is both frequent and costly in the business process.
Typical cohorts in business tasks are listed below. Each needs a precise membership rule written in terms of the tracing record fields, so the held-out set can be split the same way.
- Late arrival: the requirement appears after the agent has already started acting, for example a delivery window given in the fourth turn.
- Override: a later message replaces an earlier instruction, and the stale version gets applied.
- Document-sourced: the requirement lives in an attached contract, policy or ticket field rather than in the chat.
- Prohibition: a do-not instruction, such as not contacting the customer by phone, that has to hold across every later step.
- Quantity and format carried into tool arguments: a count, unit, date format or reference number that must reach an API call unchanged.
- Cross-system handoff: a requirement accepted in one system that must survive into another, such as a note in a CRM that should shape the work order.
Selecting and formatting the training examples
Write the selection rule down before training and freeze the held-out split before selection starts. The rule names the cohort, the sources examples may come from, the filters applied and the number of examples; anything chosen by looking at held-out results is contamination.
Build a control set of the same size drawn at random from the same pool, and train both under the same budget in examples, tokens and steps. Without the matched control, an improvement may come from more data rather than from the right data.
For supervised fine-tuning, the target is a full trajectory in which every requirement is kept, with the requirement list available only as it would be in production. Hugging Face TRL computes loss on completion tokens by default for prompt-completion datasets, and offers an assistant-only loss for conversational data, so tool outputs and user turns are context rather than targets.
For preference training, minimal pairs work well: the chosen trajectory keeps every requirement and the rejected one is identical except that it drops or applies a stale version of one requirement. TRL's preference format pairs a prompt with chosen and rejected completions, which matches this design. Split train and held-out tasks by template, customer and time period, so a model cannot pass by recognizing a near copy.
Did the cohort improve, and did anything that already worked regress?
Report the target cohort's keep rate for three arms on the held-out split: base model, matched random control and targeted selection. Give a confidence interval resampled over tasks rather than requirements, since requirements in one task are not independent, and repeat runs to capture sampling variance.
At a glance
Did the cohort improve, and did anything that already worked regress
Then check regressions on tasks the base model already passed. SWE-bench encodes the same idea for code: each instance lists FAIL_TO_PASS tests the fix must make pass and PASS_TO_PASS tests that must still pass. Keep an equivalent set of previously passing business tasks, plus a general instruction-following suite such as IFEval, and treat any significant drop there as a failed experiment even if the cohort improved.
- Adopt when the targeted arm beats the matched control on the cohort, the interval excludes no change, and no previously passing set regresses beyond the tolerance agreed in advance.
- Narrow when the gain shows up only for some arrival positions or task types; split the cohort and rerun.
- Reject when the gain matches the random control, since that points to data volume, not selection.
- Investigate graders first when the improvement is concentrated in requirements judged by a model rather than a state check.
Worked example (illustrative): a service booking with four requirements
This example is illustrative; the task, requirements and grades are invented to show the record and rubric in use, not measured results. A customer books a repair through a chat agent that can read a CRM account and create a field service work order.
Requirement R1 arrives in turn 1: no visits before 10 a.m. R2 comes from a note on the CRM account: the gate code must go in the technician instructions. R3 arrives in turn 4 and supersedes the original service address. R4 arrives in turn 6: do not send text messages, email only.
| Requirement | Source and arrival | Verifier | Illustrative grade |
|---|---|---|---|
| R1 visit window after 10 a.m. | Chat, turn 1; used at step 7 | Work order start time is 10:00 or later | Kept |
| R2 gate code in instructions | CRM account note, available from step 2 | Technician instructions field contains the code | Dropped |
| R3 new service address | Chat, turn 4, supersedes the account address | Work order address equals the turn-4 address | Applied stale version |
| R4 email only | Chat, turn 6; binds the confirmation step | Confirmation sent by email, no SMS call in the log | Kept |
| Task result | Four requirements, two failed | All requirements kept is required to pass | Fail; cohorts document-sourced and override |
Requirements an agent should not carry to the end
Some requirements should be declined, and a tracing dataset that rewards blind retention teaches the wrong habit. A customer instruction that conflicts with the operator's policy, a request that exceeds the agent's permissions, or an instruction to skip a required verification step should be refused or escalated, and the record should mark it as a requirement to decline.
Superseded requirements are the other exception: the agent should forget the old version. Stop and bring in a domain reviewer when the verifier depends on professional judgment, such as whether a reply meets a regulatory standard, because a model judge will drift there. Stop as well when the tasks contain personal or confidential records you have not cleared for training.
Where do real business task records fit?
Requirement tracing works best on real tasks, because real customers state constraints late, contradict themselves and bury details in account notes, and synthetic tasks rarely capture that mix. Ticket threads, work orders and CRM histories that show the request, the changes and the final record are the raw material for both the training and the held-out splits.
Prepared task datasets can come through SourceX, which handles rights review, contracting and data preparation for companies licensing selected business records to AI developers. Buyers who need such tasks can have SourceX look for them among U.S. businesses on request, since it keeps no ready-made stock; the request spells out scope, volume, format, timing and the license needed.
Frequently asked questions
What is instruction fidelity in post-training?
It is how reliably a model follows the instructions it was given, measured after supervised fine-tuning or preference training. For agents it should include retention: whether a requirement stated early, or in an attached document, still holds in the final result after many steps.
How is requirement retention different from instruction following?
Instruction following benchmarks usually check one response against constraints in one prompt. Retention asks whether a requirement survives across turns, tool calls, overrides and handoffs, and it is graded on the end state of the systems the agent changed.
Can a model judge grade requirement retention?
Yes for requirements that need reading, such as tone or whether a summary mentions a fact, provided the judge answers one narrow yes-or-no question per requirement. Use state checks wherever the requirement maps to a stored value, and audit a sample of model judgments by hand.
How many repeated runs per task are needed?
Enough to see whether a requirement is kept consistently, not only once. Report the share of tasks where every attempt passed, as tau-bench does with pass^k, alongside the single-run rate, and choose the number of repeats before the experiment starts.
Should the requirement list be shown to the model during training?
Only in the form it would have in production. If the deployed agent sees raw chat and documents, train on raw chat and documents; an extracted requirement list in the prompt teaches a shortcut the model will not have later.
Sources
- IFEval evaluates verifiable instructions; its input data file holds 541 prompts and it reports prompt-level and instruction-level accuracy, strict and loose. Source
- InFoBench evaluates outputs on decomposed yes-or-no questions, and its judge instructions say even minor inaccuracies rule out a yes. Source
- FollowBench adds a single constraint per level and reports hard satisfaction rate, soft satisfaction rate and consistent satisfaction levels. Source
- Lost in Conversation provides 600 sharded instructions used to simulate multi-turn conversations. Source
- tau-bench sets the reward to zero unless the final data hash equals the hash after replaying the ground-truth actions, and checks required outputs. Source
- The tau-bench leaderboard reports Pass^1 through Pass^4. Source
- Each SWE-bench instance includes FAIL_TO_PASS and PASS_TO_PASS test lists. Source
- TRL's SFT trainer computes loss on completion tokens by default for prompt-completion datasets and supports assistant_only_loss for conversational data. Source
- TRL's preference dataset format pairs a prompt with chosen and rejected completions. Source
Tell us what data you need
Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.