Skip to content

Privacy and preparation

Data preparation context loss study: pin every lost signal to a stage

By SourceX Editorial · Reviewed by Noah Loul ·

10 sections

Short answer

A data preparation context loss study traces the same approved records through every stage, from the source system view to export, normalization, de-identification and delivery, then has blinded reviewers rebuild one business task from each stage's copy. Paired differences between adjacent copies pin each loss to the step that caused it, reported alongside residual privacy risk and repair effort. No results are published yet; this page publishes the design.

Checklist

Data preparation context

This page covers data preparation context loss study: pin every lost signal to a stage.Illustrative

Key takeaways

  • Capture the same approved records at five points, source view, export, normalization, de-identification and delivery, so every difference belongs to one stage.
  • Catalog each change first, then test whether it mattered by having blinded reviewers rebuild one business task from each copy.
  • Report reconstruction accuracy, residual identifiers and repair effort together; any one alone rewards the wrong pipeline.
  • Tool defaults cause silent losses: Presidio's hash operator has used a random salt by default since version 2.2.361, which breaks links unless a consistent salt is set.
  • Normalization can improve a dataset, and the control map should credit the stage that does.
  • This is a study design and a buyer's request template; no results are published.

What does a data preparation context loss study measure?

It measures how much of the context a business task depends on survives each preparation stage, and it pins every loss to the stage that caused it. The same approved records are captured five times, from the view inside the source system through export, normalization, de-identification and delivery, and reviewers who cannot see the stage label try to rebuild one task from each copy.

That design answers a question the narrower tests leave open. A redaction utility experiment compares redaction levels on a model's score; a missing-context ablation removes categories of context from an agent's input; a migration preservation audit checks what survived a cutover. A preparation loss study follows one sample through the whole chain, so a buyer can see whether the export route, the cleanup script or the redaction step removed a signal, and whether any step added one.

This page publishes the study design and the instruments needed to run it. No results are published yet. Every value in the tables and the figure is a definition or an illustration, not a measurement.

Five copies of the same records: the stage ladder

Capture every sampled record at five fixed points, with the same record identifiers, so that any difference between adjacent copies belongs to one stage. The ladder below is the minimum; a pipeline with extra steps, such as translation or summarization, adds a rung for each.

Two documented behaviors show why export and normalization deserve their own snapshots. Zendesk's API reference describes ticket audits as a read-only history of every update to a ticket, one audit per update, served from an endpoint separate from the ticket record, so an export built from ticket records alone can arrive without the sequence of changes. The pandas documentation lists the strings its CSV reader treats as missing by default, among them NA, N/A, NULL, null and None, so a status field that legitimately reads None, or a region code NA, becomes an empty value unless the script turns that default off.

Five copies of the same records: the stage ladder
StageWhat happensContext most at riskSnapshot rule
S0 Source viewAn authorized reviewer reads the record inside the owner's systemNone yet: this is the reference copyA read-only render kept inside the owner's environment; it never leaves
S1 ExportRecords leave through an API, report or bulk exportHistories, attachments and links held in separate objectsRaw files as exported, with the method, filter and timestamp
S2 NormalizationScripts parse, type, deduplicate and harmonize fieldsLiteral values read as missing, cut text, merged or split threads, time zonesOutput of the exact script version that will run in production
S3 De-identificationIdentifiers are removed, replaced, masked or tokenizedRoles, links between records, dates and meaning in free textOutput of the configured operators and detector versions
S4 Delivery packageFiles are assembled, documented and split for the buyerRecord order, keys across files and field definitionsThe exact package a buyer would receive

Catalog every context change before scoring anything

Before anyone scores anything, compare each pair of adjacent copies field by field and record every change in a loss catalog. The catalog says what changed; the reconstruction test says whether the change mattered for the task. Keeping the two apart stops a team from calling every visible edit a loss, or every invisible one harmless.

De-identification produces the subtlest entries, because good methods are built to keep some structure and drop the rest. MIMIC-III, a de-identified clinical database maintained at MIT, documents its date shifting precisely: time of day, day of the week and seasonality survive, while the year, the day of the month and information across patients do not, so two patients shown in the ICU on the same shifted date cannot be assumed to have been there together. The business equivalent is a per-customer date shift that hides the fact that a burst of tickets opened on the same morning came from one outage.

Tool defaults create many of the rest. Microsoft's Presidio documentation says its default anonymization operator replaces each detected entity with its entity type, such as PHONE_NUMBER, so every person in a thread can become the same placeholder and the question of who did what disappears. The same documentation notes that from version 2.2.361 its hash operator uses a random salt by default, so one value receives different hashes across entities and calls unless a consistent salt is supplied. An upgrade can therefore break links between records without any change to the pipeline's own code.

Catalog every context change before scoring anything
Change typeWhat it looks like in business recordsStage where it usually entersHow to detect it
Lost linkA reply, attachment or approval no longer points to its parent recordS1 export or S3 tokenizationCount joins per record in each copy and flag any drop
Truncated textA long note or email body is cut at a field or cell limitS1 export or S2 parsingCompare character counts per field across copies
Shifted or collapsed dateEvent times move, lose their time zone or keep only the yearS2 normalization or S3 date shiftingRecompute intervals, order and weekday between events
Altered roleDistinct people become one placeholder, or an import account becomes the authorS1 import or S3 replacementCount distinct actors per case in each copy
Flattened historyOnly the latest status survives; the sequence of changes is goneS1 exportCompare event counts per record with the source view
Silent nullA literal value such as None or NA is read as missingS2 parsingDiff value distributions before and after parsing
ImprovementDuplicate threads merged, status codes harmonized, time zones made explicitS2 normalizationReviewer accuracy rises between adjacent copies

Blinded reconstruction: can a reviewer rebuild the task from this copy?

Score each copy by whether a qualified reviewer, who does not know which stage it came from, can rebuild the business task the records describe. Write the task as a fixed set of questions whose answers come from the source view, so every copy is graded against the same key.

For a support case the key might ask six things: what the customer reported, who owned the case at each step, which policy or entitlement applied, what action resolved it, how long each step took, and whether the resolution held. Two rules from Anthropic's guidance on agent evaluations carry over directly: a task is well made when two domain experts, judging an answer separately, would reach the same pass or fail verdict, and a task with several components should earn partial credit. Pilot the key on source-view copies until two reviewers agree, before any transformed copy is scored.

Allocate copies so that no reviewer sees the same case twice. With five stages, a Latin-square rotation gives each reviewer every stage across different cases, which takes memory out of the comparison. Render every copy in the same neutral viewer so formatting does not reveal the stage, and add one control: the source view converted to the delivery format with nothing else changed. If the control scores below the source view, part of the measured loss is presentation, not content.

Human review is the right instrument here and the expensive one. Anthropic describes human graders as the gold standard for quality and as slow and costly, and OpenAI's GDPval benchmark, whose tasks were built and vetted by professionals averaging more than 14 years of experience, uses head-to-head comparison by human experts as its primary metric. A preparation loss study borrows that discipline at small scale: fewer cases graded carefully beat many cases graded loosely.

  • Actor: who acted at each step, scored per step.
  • Sequence: the order of events, scored correct, partly correct or wrong.
  • Rule: the policy, contract term or entitlement that applied.
  • Decision: what was decided and why, as far as the records show.
  • Timing: the interval between key events, within a stated tolerance.
  • Outcome: whether the resolution held, such as a reopened case or a reversed credit.

The transformation-stage evidence matrix: accuracy, privacy and repair together

Report three outcomes for every stage transition at once, because each one alone rewards the wrong pipeline. Accuracy alone favors leaving identifiers in, privacy alone favors deleting everything, and repair effort alone favors doing nothing. The matrix below is the reporting template, one row per transition, and it is the study's main deliverable.

Privacy is measured on the transformed copy itself. Presidio's maintainers warn that automated detection may miss sensitive information and that additional protections should be used, so the privacy column counts residual identifiers found by a second, independent detector and by a human audit sample, not the first tool's own report. NIST's SDNist package applies the same principle to de-identified tables: it scores a de-identified dataset against its original counterpart and reports utility and privacy together.

Repair effort is the cost of putting lost context back without breaking privacy: reviewer minutes per case to restore it from retained lineage, and whether repair is possible at all once identifiers are gone. A loss that a key table can restore in minutes is a configuration problem; a loss that only the source system can restore is a design problem.

The transformation-stage evidence matrix: accuracy, privacy and repair together
TransitionAccuracy measurePrivacy measureRepair measureAttribution rule
S0 to S1 exportChange in rubric score, paired by caseIdentifiers newly exposed in exported fieldsMinutes per case to restore missing history from the sourceLoss belongs to the export route or its filter
S1 to S2 normalizationChange in rubric score, paired by caseIdentifiers moved into free text by parsingMinutes per case to fix from the raw exportLoss belongs to the script version that ran
S2 to S3 de-identificationChange in rubric score, paired by caseResidual identifiers found by a second detector and a human auditWhether a consistent token or key restores the linkLoss belongs to an operator, detector or setting
S3 to S4 packagingChange in rubric score, paired by caseIdentifiers recombined across delivered filesMinutes per case to rejoin files from documented keysLoss belongs to packaging and documentation
S0 to format controlChange in rubric score with content unchangedNone: content is unchangedNone: nothing to repairPresentation effect, reported beside every row

Attributing a loss to one stage without fooling yourself

Attribute a loss to a stage only when the paired difference between adjacent copies of the same cases exceeds a margin fixed in advance and its interval excludes zero. Report the difference per case first, then summarize; an average across stages hides which step did the damage.

Cluster uncertainty by the unit that makes cases alike, such as a customer account or a source system, and report the number of cases, reviewers and clusters behind every interval. Fix the margin that counts as a meaningful loss before scoring starts, in the same preregistration that names the task, the sample and the reviewers.

Expect a small study to find large losses and miss small ones. Anthropic suggests that 20 to 50 tasks drawn from real failures are a good start for an agent evaluation suite; a preparation pilot of similar size can expose a stage that destroys a signal outright, while a subtle drop needs more cases or a plain statement that the study could not detect it. Publish null and negative results with the same detail as losses.

  • Reviewer learning: rotate stage order so later copies are not always graded by reviewers with more practice.
  • Format cues: subtract the format-only control so presentation is not counted as lost content.
  • Answer-key drift: freeze the key before scoring and log every change with its reason.
  • Sample selection: draw cases from the approved scope by a recorded rule, including dull cases, not only rich-looking ones.

The preparation control map: preserved, lost or improved

The study ends in a control map: a grid of task signals against preparation stages in which each cell reads preserved, lost, improved or untested under a rule fixed before scoring. It is the page a preparation team keeps open when it changes a script or a redaction setting, because it shows which signals each stage is responsible for protecting.

Improvement is a real outcome, not a rounding error. Normalization that merges duplicate threads or makes time zones explicit can make a task easier to rebuild than the source view was, and the map should credit the stage that did it. The figure shows the layout with illustrative values only: five signals traced through four stages after the source view.

The preparation control map: preserved, lost or improved
VerdictRule, fixed before scoringWhat the preparation team does next
PreservedPaired difference and its interval sit inside the equivalence marginLock the stage configuration and record its version
LostDrop larger than the margin, with an interval excluding zeroChange the operator, script or export route, then rerun that stage only
ImprovedGain larger than the margin, with an interval excluding zeroDocument the step so later pipeline changes keep it
UntestedToo few cases, or the signal absent from the sampleSay so in the delivery notes and do not imply preservation
A preparation control map tracing five task signals, links, dates, roles, history and outcome, from the source view through export, normalization, de-identification and delivery, with each stage marked preserved, lost or improved.
The preparation control map. Illustrative: verdicts are examples of the layout, not measured results.

Approvals and preregistration before the first record moves

Run the study only on records the owner has approved for this purpose, with rights review, privacy review and disclosure approval recorded before any copy is made. The source-view copy is the most sensitive artifact in the study, so it stays inside the owner's environment and is read only by authorized reviewers.

Preregister the question, the task key, the sampling rule, the stages, the margins and the reviewers, with dates and versions. Preserve the denominators: how many records were eligible, sampled and excluded, and why. Decide in advance what will be published, including results that make a pipeline look bad.

Keep the empirical report separate from its component protocols. The redaction ladder, the context ablation and the lineage manifest each have their own designs; a preparation loss study reuses them as instruments and reports only what the end-to-end trace adds. Until a study like this is run on approved records, it remains a research commission, and any figure attached to it is illustrative.

What a buyer should request when a dataset arrives transformed

Ask the supplier to show, on a small approved sample, what each stage did to the context your task needs. The request is concrete: the stage ladder used, the loss catalog for the sample, a reconstruction comparison of the source view against the delivery package, and the operator and detector versions behind de-identification.

SourceX sources datasets for AI buyers on request rather than holding them in stock: buyers share scope, volume, language, format, timing and licensing needs, and the intended use and acceptance test belong in the same request. Because SourceX handles de-identification either itself or through its processing partner, a buyer can ask which operators and settings a delivery will use. A requirement written as a task, such as the six reconstruction questions for a support case, tells a supplier which signals must survive preparation better than a field list does. Whether any dataset can meet it depends on the records, the rights review and the preparation, and is assessed case by case.

  • Name the task and the questions a reviewer must answer from the delivered records.
  • State which signals must survive: actors, sequence, rule, timing and outcome.
  • Ask which de-identification operators and versions will run, and how links are kept.
  • Ask for a reconstruction check on a small sample before full delivery.
  • Write the acceptance test into the request, with the margin that counts as a loss.

Methods note: the context-loss instruments and what they omit

The stage ladder, loss catalog, reconstruction rubric, evidence matrix and control map are editorial instruments written for this article on 2026-10-26. Their supporting evidence comes from Zendesk's ticket audits API reference, the pandas CSV parsing documentation, the MIMIC-III date-shifting documentation, Microsoft Presidio's README and anonymizer documentation, NIST's SDNist package, Anthropic's guidance on agent evaluations and OpenAI's GDPval announcement, each checked on 2026-10-26.

Not measured: the size of any stage's loss in any real pipeline, reviewer agreement, repair times and residual identifier counts. No SourceX pipeline or customer records were used. The figure's cell values are illustrative.

Frequently asked questions

What is context loss in data preparation?

Context loss is any change during preparation that removes information a task depends on, such as a broken link between a reply and its ticket, a shortened note, a shifted date or distinct people collapsed into one placeholder. It is judged against a task, not against the raw file: a change that no task needs is not a loss.

Does de-identification always reduce a dataset's usefulness?

No. Methods that replace identifiers consistently can keep links and roles intact, and MIMIC-III's date shifting keeps time of day, weekday and season while removing the year. Usefulness falls when an operator collapses distinct people into one placeholder or breaks links, which is why the study scores each operator setting separately.

How many cases does a preparation loss study need?

Enough to detect the smallest loss that would change a decision, fixed before scoring starts. A pilot of a few dozen cases, the range Anthropic suggests for starting an agent evaluation suite, can expose a stage that destroys a signal outright but will miss subtle drops. Report what the sample could and could not detect.

Can normalization make a dataset better?

Yes. Merging duplicate threads, harmonizing status codes and making time zones explicit can make a task easier to rebuild than the source view was. The control map marks those cells as improved so later pipeline changes keep the step.

Who should see the source-view copy?

Only reviewers the record owner has authorized, inside the owner's environment and under an approved scope. The source view supplies the answer key, so it is the most sensitive copy in the study and should never be exported.

Are context-loss rates by preparation stage available yet?

No. This page is a study design, and no SourceX results are published. Any future results would be published only after the study runs on approved records, with methods, denominators and negative findings included.

Sources

  • Zendesk's API reference describes ticket audits as a read-only history of all updates to a ticket, with each audit representing a single update, listed through a per-ticket audits endpoint. Source
  • The pandas CSV reader treats strings including NA, N/A, NULL, null, None, nan and the empty string as missing by default; keep_default_na=False overrides the defaults, and dtype str with suitable na_values preserves values. Source
  • MIMIC-III's date shifting preserves time of day, day of the week and seasonality, and removes the year, the day of the month and information across patients. Source
  • Presidio's README says its automated detection may not find all sensitive information and that additional systems and protections should be employed. Source
  • Presidio's anonymizer documentation lists replace, redact, hash, mask, encrypt and keep operators; replace is the default and substitutes the entity type, and from version 2.2.361 the hash operator uses a random salt by default unless a consistent salt is provided. Source
  • NIST's SDNist provides benchmark data and evaluation metrics for de-identified data generators, evaluating a de-identified dataset against its original counterpart and reporting utility and privacy. Source
  • Anthropic's guidance on agent evaluations says a good task is one where two domain experts would independently reach the same verdict, recommends partial credit for multi-component tasks, describes human graders as gold standard but slow and expensive, and calls 20-50 tasks drawn from real failures a great start. Source
  • OpenAI's GDPval uses head-to-head comparison by human experts as its primary metric, with tasks built and vetted by professionals averaging more than 14 years of experience. Source

Tell us what data you need

Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.

See if your company qualifies