AI uses for records
What context do AI agents need that business exports omit? A study protocol
By SourceX Editorial · Updated
10 sections
Short answer
AI agents likely need change history, decision rationale, policies in force, related records and outcomes of similar cases, which a standard export of a system's main records often leaves out. This page publishes a business task missing context study protocol: an export audit plus a task replay experiment that removes one context category at a time and measures the drop in success. It reports no results.
Agent run
Business task missing
Key takeaways
- This is a protocol page; it reports no results.
- Much agent-relevant context, such as change history, sits in separate history or audit endpoints, not the main record.
- Task replay scores an agent against the action a person took and the outcome that followed.
- Ablations remove one context category at a time, with length matching and randomized order to control for prompt effects.
- A fixed decision timestamp and a leakage audit keep future information out of the agent's context.
- Holding context is not the same as having permission to share or license it.
What context do AI agents need that business exports usually omit?
The likely candidates are change history, the reasons behind decisions, the policies people followed, related records in other systems, and how similar past cases turned out, because a standard export of a system's main records mostly shows current values. Whether those gaps actually change agent performance has to be measured. This page publishes a task replay and context ablation study protocol to measure it; it reports no results.
The study asks two linked questions. First, which categories of context does a standard export of a business system leave out, compared with what the system actually holds? Second, when an agent replays a real completed task, how much does removing each category change whether it reaches the action a person took and the outcome that followed?
Answering the first question is documentation work. Answering the second requires an experiment, and the experiment is the core of this protocol.
Where does missing context live in common systems?
Much of the context agents need is stored, but not in the main record. In many systems, the record you export shows the current state, while the path to that state lives in a separate history object or API endpoint that a basic export does not touch.
Three documented examples. Salesforce standard field history tracks changes only for fields an admin selects, up to 20 per object, and with Field Audit Trail off it keeps that history for up to 18 months, or up to 24 months through the API. Zendesk keeps a read-only audit history of every ticket update, each made of events such as field changes and comments, retrieved through a separate audits endpoint. Jira Cloud returns an issue's change history through its own paginated changelog endpoint. In each case, an export of the main object alone would not include the history.
The export audit in this study checks each participating system in the same way: list what the standard export contains, list what history, audit, comment and attachment endpoints hold, and code the difference into the context categories below.
The audit records the export method actually used, such as a report export, a bulk API job or a vendor backup, because the same system can produce very different files depending on the path. It also records retention limits from each vendor's documentation, since history that has aged out cannot be recovered for any study or dataset. Each finding cites the documentation page and the date it was checked.
| Context category | What it answers for an agent | Example location | Often outside a main-record export? |
|---|---|---|---|
| Change history | How did the record get to its current state? | Salesforce field history, Jira changelog | Yes, separate object or endpoint |
| Conversation and notes | What did people say while working the case? | Zendesk ticket comments within audits | Depends on export settings |
| Decision rationale | Why was this option chosen over others? | Free-text notes, approval comments | Often missing or unstructured |
| Policy and procedure | Which rule applied at the time? | Knowledge base articles, SOP documents | Yes, different system |
| Related records | What else was true about the customer or asset? | Linked accounts, orders, assets | Partly, needs joins |
| Outcomes of similar cases | What happened last time this came up? | Closed cases with resolution fields | Yes, needs retrieval across records |
What is task replay, and which tasks are in the study?
Task replay means taking a task a person completed in the past, rebuilding what was known at the moment the decision was made, and asking an agent to make the same decision. The person's recorded action, and the outcome that followed, become the reference the agent is scored against.
The population is completed tasks in participating companies' support, service, finance operations and project systems during a stated window. Tasks are sampled at random within strata by workflow type, so that common, easy tasks do not crowd out the rest. Eligible tasks have a clear decision point, a recorded action and an observable outcome. Tasks whose records cannot be de-identified for the study are excluded and counted.
Public agent benchmarks show the shape of this design. tau-bench pairs an agent with tools and domain policies in retail and airline settings and scores success by comparing the final database state with an annotated goal state. WorkArena uses common knowledge-work tasks on the ServiceNow platform, each with a validator. This study differs by using real, historical business tasks and by manipulating the context the agent receives rather than the task itself.
An illustrative task, invented to show the design and implying no result: a support agent at a software company decided to issue a goodwill credit on a ticket instead of escalating it to engineering. At the decision moment, the ticket showed the current priority and the customer's plan. The ticket's audit history showed that the priority had been lowered once already, an internal note recorded that a fix was scheduled, and the knowledge base held the credit policy. Replay asks whether an agent given only the current ticket reaches the same choice as one given the history, the note and the policy.
How does the context ablation work?
Each task is run under a full-context condition and under conditions that remove one category of context at a time. Because the same task appears in every condition, differences in success can be attributed to the removed context rather than to the task.
Removing context also shortens the input, and input length and position can change model behavior on their own. Research on long contexts found that models often use information at the start or end of the input better than information in the middle. The protocol therefore replaces removed content with length-matched neutral text and randomizes the order of context blocks, so that a drop in success reflects the missing information and not a shorter or reordered prompt.
| Condition | What the agent receives | Held constant | Question it answers |
|---|---|---|---|
| Full context | Main record, history, notes, policies, related records, similar past cases | Task, tools, model, instructions | Upper reference for this agent |
| Export only | Main record as a standard export would provide it | Length matched with neutral text | How far a plain export gets the agent |
| Minus history | Full context without change history | Length and block order randomized | Does the path to the current state matter? |
| Minus notes and rationale | Full context without free-text notes and comments | Length and block order randomized | Does the reasoning trail matter? |
| Minus policy | Full context without policy and procedure documents | Length and block order randomized | Does the agent need the rule in force at the time? |
| Minus related records | Full context without linked records from other objects | Length and block order randomized | Does cross-record context matter? |
| Minus similar cases | Full context without outcomes of comparable past cases | Length and block order randomized | Does precedent matter? |
How is leakage from the future prevented?
The biggest threat to a replay study is giving the agent information that did not exist when the person decided. A ticket's final status, a later comment that says the refund was approved, or a field changed after the fact would all hand the agent the answer.
The protocol fixes a decision timestamp for every task and rebuilds context from history as of that moment. Every context block is filtered by its own timestamp, similar past cases must have closed before the decision timestamp, and policy documents are taken at the version in force on that date where version history exists. A second analyst audits a random sample of rebuilt contexts for leakage before any agent runs, and the audit result is reported.
- Set the decision timestamp from the record of the action, not from task creation.
- Rebuild field values from history entries dated before the timestamp.
- Drop comments, attachments and linked records created after the timestamp.
- Restrict similar-case retrieval to cases closed before the timestamp.
- Use the policy version in force on the date, or exclude the policy block and record why.
- Audit a random sample of rebuilt contexts for leakage and publish the error rate found.
How is success measured?
Success has three parts, scored separately. Action match asks whether the agent chose the same action as the person, using a coded action set per workflow. State match asks whether the agent's proposed changes to the record match the changes the person made, field by field, in the spirit of benchmarks that compare final state with a goal state. Policy compliance asks whether the agent's action broke a written rule in force at the time.
A person's historical action is not always right, so the protocol also records the outcome that followed, such as whether the case reopened, the customer escalated, or a credit was reversed. Tasks with a bad outcome are analyzed separately, and agreement with the person on those tasks is not counted as success.
Action codes are assigned by two coders working independently from the agent's output, blind to the condition it ran under. Disagreements are adjudicated by a third coder, and chance-corrected agreement is reported for each workflow type. Where a workflow has a deterministic check, such as an expected field value, the check replaces human coding for that part of the score.
Agents are not deterministic, so each task runs several times per condition. Reliability is reported with a pass^k style measure, the share of tasks an agent completes correctly on every one of k runs, alongside the single-run success rate.
How are ablation drops tested against the full-context baseline?
The baseline is the full-context condition for the same agent on the same tasks. Each ablation is reported as a paired difference from that baseline, with a confidence interval from a bootstrap that resamples tasks within companies. Paired tests such as McNemar's test suit binary success on the same tasks; with six comparisons against baseline, the protocol applies a stated multiple-comparison correction.
Sample size is set in advance from the smallest difference worth detecting for the main comparison, export only versus full context. Results are reported per workflow type and pooled, and pooled results are weighted by the stratified design so that one large participant does not dominate.
Two agents built on different underlying models are run under every condition. A context category that matters for only one of them is reported as model-specific, not as a property of business work.
Which conclusions does the replay design not support?
The replay design would not show that any agent is ready for production, because replay removes the live back-and-forth with customers and colleagues that real work involves. It would not show which context matters for workflows outside the sampled types, and results from one set of models may not hold for later ones.
It also would not show that a company may share the context it holds. History, notes and linked records often contain personal data and third-party information, and access to records is not the same as permission to use them for a study or a license. Participating companies decide what to include under their own privacy and contract review.
Finally, the study measures the effect of context on task success, not the cost of supplying it. A category that helps a little but is expensive to extract and de-identify may not be worth adding, and that tradeoff is a separate decision for each team.
- Not supported: claims that a named product or model fails because of missing context.
- Not supported: a ranking of business systems by export quality.
- Supported: paired, dated estimates of how much each context category changes replayed task success for the agents and workflows tested.
What goes into the pre-registration for a context ablation?
Register the protocol before any agent runs, so the confirmatory comparisons are fixed and anything added later is labeled exploratory. Report every deviation with its date and reason.
At a glance
What goes into the pre-registration for a context ablation
- Research questions, the export audit procedure and the context category definitions.
- Population, workflow strata, sampling window and exclusion rules.
- Decision-timestamp rule, rebuild procedure and leakage audit plan.
- Ablation conditions, length-matching text and block-order randomization.
- Agents, model versions, tools, instructions and number of runs per task.
- Action codes, state-match fields, policy rules and outcome labels.
- Primary comparison, minimum difference worth detecting and resulting sample size.
- Interval method, paired tests and multiple-comparison correction.
- De-identification approach for task records and who performs it.
- Publication date and what will be released: tables, code and the context category dictionary.
How a business can use the protocol on its own records
A company can run the export audit on its own as a short exercise per system: compare what its standard export contains with what its history, audit and comment endpoints hold, and code the gap into the six categories. That tells it whether its records carry the context agents lack, which is the material question behind both internal agent projects and licensing.
When the audit shows a company's history, notes and outcomes are intact, SourceX offers one way to license selected records as AI datasets, handling rights review, contracting, preparation and delivery. Buyers generally favor two or more years of consistent history and tens of thousands of records in one system, and SourceX weighs each case individually. Finding out whether a company qualifies takes a short assessment and no uploaded records.
Why does removing one category at a time misread redundant context?
Because when two categories carry the same information, removing either one leaves the other to do the work, so both look worthless. Removal-only conditions also miss synergy, where a category helps only alongside another, such as a note that makes sense only with the change history that says when it was written.
Data valuation research handles the same problem with Shapley values, which divide credit fairly among data sources given a learning algorithm and a performance measure; Ghorbani and Zou's Data Shapley applies the idea to training data. The ablation above has five removal conditions, since notes and rationale come out as one block, so exact attribution is affordable in principle: every subset of five categories is only 32 arms per task. Most teams can afford less, and a 12-arm design catches the two failures that matter for a dataset specification.
The paired design reuses the protocol's conditions. Full context and the five removals stay as they are, the export-only condition serves as the minimal core, and five new arms each add one category to that core, under the same length matching and block-order randomization. When two categories both look redundant, one more arm removes the pair together, confirming they substitute for each other rather than both being inert.
| Design | Arms for five categories | What it detects | What it misses |
|---|---|---|---|
| Leave-one-out (the conditions above) | 6: full context plus five removals | Categories nothing else can replace | Redundant pairs and categories that help only together |
| Leave-one-in | 6: export only plus five single additions | Categories that help on their own | Categories that need a partner |
| Paired removal and addition | 12: full, export only, five removals, five additions | Essential, redundant, synergistic, inert and harmful verdicts | Interactions among three or more categories |
| Full factorial | 32: every subset of the five | Exact Shapley attribution and all interactions | Nothing within the five categories, at about 2.7 times the cost of the paired design |
How does the context verdict tree classify each category?
Each category gets one of five verdicts from two numbers: its removal effect, full-context success minus success without it, and its solo effect, success with export only plus the category minus export-only success. Both are paired differences on the same tasks, each counts only when its interval clears the smallest difference worth detecting fixed in the pre-registration, and verdicts are assigned per workflow type and per agent.
Harmful is not a hypothetical branch. Shi and colleagues built the GSM-IC dataset by adding irrelevant context to grade-school math problems, under a title that states the risk: large language models can be easily distracted by irrelevant context. Anthropic's report on contextual retrieval adds a practical caveat from the other direction: retrieving more chunks can help, but more information can distract the model, so there is a limit.
Three corruption arms test the harmful branch directly. A stale arm swaps in the policy version that preceded the one in force on the decision date; a conflicting arm inserts an earlier note or comment that contradicts the record; a distractor arm adds a plausible related record from an unrelated case. Their results show whether a dataset must carry version dates and case boundaries, not just the categories themselves.
| Verdict | Removal effect | Solo effect | What it means for a dataset specification |
|---|---|---|---|
| Essential | Success falls without it | Success rises with it alone | Require it for this workflow type |
| Redundant | No reliable change | Success rises with it alone | Require one of the substitutes, confirmed by the pair-removal arm |
| Synergistic | Success falls without it | No reliable change alone | Require it together with its partner category |
| Inert | No reliable change | No reliable change | Leave it out and save the preparation and tokens |
| Harmful | Success rises without it | Falls or does not change | Exclude it as supplied, or fix its quality first |
Which outcomes besides success should every condition record?
Failure severity and reviewer minutes, recorded beside action match, state match and policy compliance, because a category that barely moves success can still change how badly failures go or how long a person needs to check the work. Report both for every condition, including those whose success effect does not clear the minimum.
Read the protocol's pass^k measure per category as well. The gap from a single-run rate can be wide: tau-bench's leaderboard lists one tool-calling agent at 0.692 for pass^1 and 0.462 for pass^4 on retail tasks, and at 0.460 and 0.225 on airline tasks. A category that raises single-run success but not pass^k makes an agent luckier, not more dependable.
Severity and minutes are scored by people, so blinding extends to them: reviewers see outputs without the condition label, in random order and mixed across workflow types. The two added outcomes are:
- Failure severity on a four-level rubric: 0 harmless, 1 recoverable before anyone outside sees it, 2 costly to reverse, 3 irreversible or noncompliant, double-coded and adjudicated like the action codes.
- Reviewer minutes: the time a reviewer needs to verify a success or correct a failure, measured by a timer in the review tool, with the reviewer's experience recorded.
How do the verdicts become a minimum-context specification?
Each workflow type gets a minimum-context specification: the categories a dataset must carry, the ones that can substitute for each other, the ones that must travel together and the ones to leave out. It becomes the working definition of complete records.
An illustrative entry, showing the format and not a finding: for an invoice exception workflow, policy and change history required with version dates, notes or similar-case outcomes either-or, related records together with change history, nothing optional. Real entries carry the effect sizes, intervals and study version that produced them.
The specification also yields an acceptance test for delivered data: a sample of tasks rebuilt from the delivery should reproduce the full-context condition's success within the registered margin, a check a supplier can run before delivery and a buyer can rerun after it.
Publish every registered verdict, including inert and harmful ones, because a specification that lists only what helped invites buyers to overpay for context. That covers every arm, including those where no category moved success, each harmful verdict with the corruption arm that exposed it, and the workflow types where export only matched full context. The specification itself has five parts:
- Required: categories with an essential verdict, plus the version and timestamp rules the corruption arms showed were needed.
- Either-or: redundant pairs confirmed by the pair-removal arm, so the supplier can provide whichever is cheaper to prepare.
- Together: synergistic pairs, specified as linked fields rather than separate files.
- Optional: inert categories, left out by default to save preparation hours and tokens.
- Excluded as supplied: harmful categories, with the defect that made them harmful, such as missing version dates.
Sources for this section
- The Data Shapley repository (Ghorbani and Zou, ICML 2019) implements Shapley-based valuation to divide value fairly between training data points or sources given the learning algorithm and a performance measure. Source
- The GSM-IC repository releases grade-school math problems with irrelevant context added, from the paper Large Language Models Can Be Easily Distracted by Irrelevant Context (Shi et al., 2023). Source
- Anthropic's contextual retrieval report (September 19, 2024) notes that adding more retrieved chunks can help but more information can distract the model, so there is a limit. Source
- The tau-bench repository describes language agents given domain-specific API tools and policy guidelines, and its leaderboard lists a tool-calling agent at 0.692 pass^1 and 0.462 pass^4 on retail tasks and 0.460 and 0.225 on airline tasks. Source
Frequently asked questions
Has this study reported which context matters most?
No. This page publishes the protocol only: the export audit, the replay design, the ablation conditions and the analysis plan. Results will be published after the pre-registered runs, with every deviation disclosed.
What is a context ablation study?
It runs the same tasks repeatedly while removing one category of context at a time, such as change history or policies, and measures how much success drops compared with full context. Because the tasks stay the same, the difference points to the removed context.
Why not just give the agent everything?
Long inputs are not used evenly; research has found models often use information in the middle of a long context less well than information at the start or end. Knowing which context matters lets teams supply the right records rather than more of them.
Is the person's historical action the right answer?
Not always. The protocol records the outcome that followed each task and analyzes tasks with bad outcomes separately, so agreeing with a person who got it wrong is not counted as success.
Do standard exports include change history?
Often not by default. In Salesforce, Zendesk and Jira, change history lives in a separate history object or API endpoint from the main record, so an export of the main object alone leaves it out. Check your own system's export settings and documentation.
Sources
- tau-bench emulates conversations between an agent with domain tools and policies and a simulated user in retail and airline domains, scores success by comparing the final database state with an annotated goal state, and proposes pass^k to measure reliability across repeated trials. Source
- WorkArena is a benchmark of common knowledge-work tasks on the ServiceNow platform, such as filtering lists, filling forms, searching knowledge bases and using service catalogs, each with a validator. Source
- Lost in the Middle found that language models use long contexts unevenly: performance is often highest when relevant information is at the beginning or end of the input and drops when it is in the middle. Source
- Salesforce standard field history tracking records changes for a selected set of fields, up to 20 standard and custom fields per object. Source
- With Field Audit Trail off, Salesforce retains field history data for up to 18 months, and up to 24 months via the API. Source
- Zendesk ticket audits are a read-only history of all updates to a ticket; each audit is one update made of one or more events, such as field changes and comments, retrieved through a separate audits endpoint. Source
- Jira Cloud exposes an issue's change history through a separate paginated changelog endpoint, GET /rest/api/3/issue/{issueIdOrKey}/changelog. Source
- Preregistration separates hypothesis-generating (exploratory) research from hypothesis-testing (confirmatory) research, and planning the analysis in advance improves quality and transparency. Source
See if your company qualifies
A short company assessment. No data uploads are needed.