Skip to content

Privacy and preparation

How to measure which dataset fields help an agent finish a business task

By SourceX Editorial · Reviewed by Noah Loul ·

10 sections

Short answer

Measure which dataset fields help an agent finish a task with a pre-registered field ablation: run the same agent on the same held-out cases with the full record and with one field group removed, masked, shuffled or coarsened, keep budget and tools fixed, score verified end states, and report paired differences with clustered uncertainty. This page is the protocol and its field plan; it reports no results.

Agent run

Dataset feature ablation

This page covers how to measure which dataset fields help an agent finish a business task.Illustrative

Key takeaways

  • A field ablation reruns the same agent on the same held-out cases with one field group altered and scores verified end states, so the difference belongs to the field.
  • Choose the operator deliberately: dropping, masking, shuffling and coarsening each answer a different question, and coarsening is the arm that prices privacy.
  • Leakage from fields written after resolution is the most common reason an arm looks strong; audit field creation times before the first run.
  • Analyze within case: paired differences, standard errors clustered by customer or vendor, repeated trials and a correction for the number of arms.
  • This is a protocol with a reusable field plan; it reports no results, and any results belong in a separate, versioned report.

How do you measure which dataset fields help an agent finish the task?

Run a pre-registered field ablation: give the same agent the same held-out tasks under a full-record condition and under conditions where one named group of fields is removed, masked or coarsened, hold every other input and the budget fixed, and score each run on a verified business outcome. The field group's value is the paired difference in completion across the same tasks, reported with its uncertainty.

This page is a protocol, not a study. It sets out the question, the population of tasks, the ablation conditions, the metrics, the uncertainty analysis and the checks that rule out the usual false positives. No results are reported here, and every number in the worked case is a placeholder labeled illustrative.

The design borrows from three established practices. Permutation importance in machine learning breaks the link between one input and the target and watches the score fall. Agent benchmarks score the final state of a system rather than the wording of a reply. Statistical guidance for model evaluations treats each task as a sample and asks for standard errors, paired comparisons and a power check before anyone claims a difference.

Which business task, population and unit of analysis does the experiment commit to?

Fix the task before the fields. A field only helps relative to a job, so the research question names one task family, such as resolving a refund request from a Zendesk ticket export or matching a vendor invoice to a NetSuite purchase order, and the outcome that counts as finished.

At a glance

Which business task, population and unit of analysis does the experiment commit to

The unit of analysis is the task instance, not the agent step. Each instance is one historical case with a known, checkable end state: the refund issued for the right amount, the invoice posted against the right PO line. Cases without a verifiable end state stay out of the pool, because their contribution cannot be scored.

State the population and period in writing: which system, which business units, which date range, and which cases were excluded and why. Sample from that population at random or with a stated stratification, for example by case type or account size, and record the denominator at each exclusion step so a reader can see how many cases reached the experiment.

  • Research question: does removing field group G reduce verified completion of task T by at least the pre-registered margin?
  • Population: closed cases of task T in system S between two named dates, after documented exclusions.
  • Unit: one case, run under every condition, so comparisons are within case.
  • Primary outcome: verified completion against the recorded end state; secondary outcomes named in advance.
  • Comparison: full-record condition versus each single-group ablation, plus a minimal condition with only the fields the task cannot start without.

Remove, mask, shuffle or coarsen: picking the ablation operator for each field

The operator decides what the experiment can conclude, so choose it per field group and write the choice down. Dropping a column tests whether the agent needs the information at all. Masking keeps the key but replaces the value with a fixed token, which tests the content while keeping the record's shape. Shuffling a field across cases keeps its realistic values but breaks their link to the case, the same idea scikit-learn documents for permutation feature importance, where randomly shuffling one feature and measuring the score drop shows how much a model relies on it. Coarsening replaces a value with a less identifying version, such as a city instead of a street address, and is the operator that measures the privacy tradeoff directly.

Correlated fields defeat single-field ablations. The scikit-learn documentation warns that when two features are correlated and one is permuted, the model still reaches the information through the other, so both can look unimportant. In business records this is the norm: a ticket subject, its tags and its first message often say the same thing. Ablate correlated fields as one group and report the group.

Remove, mask, shuffle or coarsen: picking the ablation operator for each field
OperatorWhat it testsUse it whenMain pitfall
Drop the fieldWhether the agent needs this information at allThe field is optional in the source systemChanges record shape and prompt length, which can shift behavior on its own
Mask with a fixed tokenWhether the value matters, holding layout constantTools or prompts expect the key to existThe token itself can cue the agent that something was hidden
Shuffle across casesWhether the case-specific value matters, with realistic values keptThe field has a plausible distribution worth preservingCorrelated fields leak the same signal; group them first
Coarsen or generalizeHow much utility survives a less identifying versionThe field carries personal or commercially sensitive detailCoarsening rules chosen after seeing results invite tuning to the test
Add one group to a minimal recordThe marginal value of a group on top of the essentialsMany groups, few of which are expected to matterInteractions between groups are missed if only one is ever added

The field ablation plan with task and privacy tradeoffs

The plan is one row per field group, completed before any run. It records what the group contains, why it might help, which operator and condition test it, how exposed it is from a privacy or confidentiality standpoint, and the decision rule that will apply to the result. Writing the decision rule first is what stops a team from reading noise as a finding.

The privacy column is not a legal verdict. It ranks how much harm or obligation a field carries if it leaves the source system: direct identifiers, free text that may contain anything, financial details, internal notes about named staff. A field that is both high-exposure and low-utility is the easiest call; a field that is high on both is where the coarsened arm earns its cost.

The field ablation plan with task and privacy tradeoffs
Field group (example fields)Why it might help the agentAblation armPrivacy or confidentiality exposurePre-registered decision rule
Case narrative (subject, first message, latest reply)States the request the agent must act onMask; also a coarsened arm with named entities replacedHigh: free text may hold names, contact details, account numbersKeep the coarsened form if its loss is within the margin of the full form
Customer identity (name, email, phone)Lets the agent look up the accountDrop; replace lookups with a pseudonymous account keyHigh: direct identifiersExclude unless the pseudonymous key arm loses more than the margin
Account history (prior orders, open balances)Context for eligibility and limitsDrop; add-one to minimal recordMedium: commercial and financial detailKeep if removal costs more than the margin
Structured status fields (priority, category, SLA timer)Signals urgency and routingShuffle across cases as one groupLowKeep if shuffled arm loses more than the margin
Internal notes and agent commentsRecords what staff already checkedMask; separate arm dropping notes by named staffMedium to high: staff names, opinionsKeep only the arm that preserves the result with the least staff detail
Attachments (invoices, photos, PDFs)Evidence needed to verify a claimDrop; extracted-fields-only armVariable: depends on contentsPrefer extracted fields if the loss is within the margin
Policy and price tables at the time of the caseRules the agent must applyDrop; replace with current versionLow, but often confidential to the businessKeep the dated version if the current version arm loses more than the margin
Resolution fields (outcome code, refund amount)None at task time; these describe the answerMust be absent in every armVariesAny presence in the agent's input invalidates the run

Holding the comparison fair: equal budget, untouched holdout, outcome checked in the system

Every condition gets the same agent, model version, prompt template, tools, step limit, token limit and wall-clock limit. If removing a field shortens the prompt, the shorter prompt is part of the treatment, so record prompt length per run and report it beside the outcome. Change one thing per arm.

Split the cases before anything else. A development set is where the team writes prompts, fixes tool bugs and chooses coarsening rules; a holdout set is run once, under the frozen protocol, and nobody looks at it until then. Choosing field groups or decision margins after seeing holdout runs turns the experiment into a search, and the reported differences will not repeat.

Score the end state, not the transcript. The tau-bench benchmark compares the database state at the end of an episode with the annotated goal state, which removes a grader's reading of the conversation from the result. For tasks with several milestones, checkpoints in the style of TheAgentCompany give partial credit per verified milestone, with full completion reserved for runs that pass every checkpoint. Pre-register which of the two is the primary outcome.

Agents are stochastic, so run each case several times per condition. The tau-bench paper defines pass^k as the chance that all k independent trials of a task succeed, which separates a field that makes success reliable from one that only makes it possible. Fix k in advance and use the same seeds or sampling settings in every arm.

What else could explain an apparent improvement?

Most false findings in a field ablation come from leakage, not from the field. Kapoor and Narayanan's survey of machine-learning-based science documents data leakage as a widespread source of irreproducible results, and business records are full of fields written after the work was done. A closed reason, a final refund amount or a posted journal entry tells the agent the answer; if one survives in any arm, that arm will look strong for the wrong reason.

At a glance

What else could explain an apparent improvement

  • Answer leakage: a field recorded at or after resolution slips into the input. Check every field's creation time against the task's start time.
  • Proxy fields: removing a field does nothing because a correlated field carries the same signal. Ablate groups and report the correlation structure you assumed.
  • Prompt length and layout: the arm improves because the input got shorter, not because the field was noise. Report length and test a length-matched filler arm if the effect is large.
  • Grader dependence: the outcome check reads the same field the agent was given. The check must use the recorded end state, not the input record.
  • Version drift: model, tool or policy versions changed between arms run on different days. Run arms interleaved, not in sequence.
  • Selection after the fact: cases that failed under the full condition were dropped as noisy. Exclusions are fixed before any run and applied to all arms.
  • Run-to-run variance: a single trial per case makes chance differences look real. Repeat trials and analyze them as clustered within case.

Sample size, baseline and uncertainty that make a field ranking defensible

Treat the comparison as paired, because every case runs under every condition. Evan Miller's statistical guidance for language model evaluations recommends computing differences at the question level when two conditions share the same questions, which removes the variation between easy and hard cases from the error. Here the question is the case and the two conditions are full record and ablated record.

Cluster where cases are related. Many tickets from one customer, or many invoices from one vendor, are not independent samples; the same guidance recommends clustered standard errors when questions come in related groups and reports that they can be several times larger than naive ones. Name the clustering variable in the pre-registration.

Run a power analysis before collecting anything. Decide the smallest drop in verified completion that would change a data request, and estimate how many cases and trials are needed to detect it with acceptable error rates. If the available pool cannot support that, narrow the number of field groups rather than reporting differences the design could not detect.

Correct for the number of arms. Eight field groups means eight comparisons; without a stated correction such as Holm's method, one of them will look significant by chance. Report every arm, including the ones that showed nothing, with its paired difference, interval, number of cases, number of trials and the baseline completion rate of the full condition.

Pre-registration checklist before the first holdout run

At a glance

Pre-registration checklist before the first holdout run

Checklist · 10 items

  1. Task family, outcome definition and how the end state is verified in the system of record.
  2. Population, period, exclusions with counts at each step, and the sampling or stratification rule.
  3. Development and holdout split, with the date it was frozen and who holds the holdout.
  4. Field groups, their contents, the operator for each arm and the correlation assumptions behind the grouping.
  5. Fixed budget per run: model and version, prompt template, tools, step, token and time limits, trials per case.
  6. Primary and secondary metrics, including whether partial credit or pass^k is primary.
  7. Analysis plan: paired differences, clustering variable, interval method, multiple-comparison correction and the pre-set margin.
  8. Leakage audit: creation time of every field relative to task start, and the resolution fields that must be absent.
  9. Privacy exposure rating per field group and who assigned it.
  10. What will be published: all arms, denominators and intervals, with protocol and results reported separately.

0 of 10 ticked. Tick each item as you go.

Illustrative design: refund requests from a help desk export

This example is invented to show how the plan fills in; it is not a study and reports no results. A buyer wants to know which ticket fields a refund agent needs before requesting a licensed dataset. The task is to issue the correct refund, or correctly refuse it, in a sandbox copy of the order system, and the outcome is checked against the order record after the run.

The illustrative pool is 600 closed refund tickets from one year, stratified by product line, split into 200 for development and 400 for the holdout. Arms are the full record, five single-group ablations from the plan above, one coarsened-narrative arm and one minimal record. Each holdout case runs four times per arm, so pass^k can be reported for k up to four, and standard errors are clustered by customer account.

Before any run, the team finds the refund amount and a closed-reason code in the export and removes them from every arm; both were written after resolution. That single audit step is often the difference between a protocol that measures field utility and one that measures leakage.

Using a field ranking to scope a dataset request

A finished ablation gives a short, evidence-backed field list: groups that measurably help, groups that can be coarsened with no detectable loss, and groups that add exposure without benefit. That list is also a data minimization argument. Where the EU GDPR applies, Article 5(1)(c) requires personal data to be adequate, relevant and limited to what is necessary for the purpose; a measured ablation is one way to document why each personal field is in a dataset. This is general information, and counsel should review how the rule applies to a specific dataset.

Removing a field does not settle who may license the rest. Access to a system, the privacy basis for processing, ownership of the records and the contractual right to license them are separate questions, and de-identification alone does not create a right to license.

For teams that cannot run the experiment on their own records, the field plan doubles as a request specification. Datasets like this are sourced by SourceX from U.S. businesses on request rather than pulled from a catalog. Its request form collects scope, volume, language, format, timing and licensing needs, so a buyer can attach the field groups, coarsening rules and outcome definition the ablation will need.

Frequently asked questions

What should a field ablation plan with task and privacy tradeoffs include?

One row per field group listing its contents, why it might help, the ablation operator and arm that tests it, its privacy or confidentiality exposure, and the decision rule that will be applied to the result. Add the shared parts once: task and outcome definition, budget per run, holdout split, analysis plan and leakage audit.

Is field ablation the same as permutation feature importance?

They share the core idea of breaking one input and measuring the loss, and shuffling is one ablation operator. Field ablation for agents differs because the outcome is a verified business end state after many tool calls, the agent is rerun rather than a fixed model rescored, and the arms can also drop, mask or coarsen fields to measure privacy tradeoffs.

What alternative explanation could account for an apparent improvement?

The usual culprits are answer leakage from fields written after resolution, correlated fields carrying the same signal, changes in prompt length, a grader that reads the ablated field, version drift between arms and run-to-run variance. Each has a specific check in the pre-registration, and an improvement should only be reported after all of them pass.

How many cases and trials does a field ablation need?

It depends on the smallest difference that would change a decision, the baseline completion rate, how related the cases are and how many arms are compared. Run a power analysis on development-set variance before the holdout; if the pool is too small, test fewer field groups rather than report differences the design cannot detect.

Why does this page report no results?

A result needs collected runs on a defined population with the uncertainty analysis above. This page publishes the method so a team can run it on its own tasks and records; any results would be published separately, with the protocol version they followed.

Sources

  • scikit-learn's user guide describes permutation feature importance as shuffling one feature to measure the score drop, and warns that correlated features can make important features look unimportant. Source
  • The tau-bench paper evaluates agents by comparing the final database state with the annotated goal state and defines pass^k as the chance that all k trials of a task succeed. Source
  • TheAgentCompany splits tasks into weighted checkpoints checked by programmatic evaluators, awarding partial credit and full completion only when every checkpoint passes. Source
  • Kapoor and Narayanan survey and classify data leakage as a cause of reproducibility failures in machine-learning-based science. Source
  • Miller's statistical guidance for model evaluations recommends paired question-level differences, clustered standard errors for related questions and power analysis before an evaluation. Source
  • GDPR Article 5(1)(c) sets the data minimization principle: personal data must be adequate, relevant and limited to what is necessary for the purposes of processing. Source

Tell us what data you need

Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.

See if your company qualifies