Detailed test protocol

01 / SourceX De-identification Benchmark (SX-TXT-1.0)

SourceX De-identification Benchmark (SX-TXT-1.0)

The downloadable suite contains 1,200 synthetic extracted-text records, 7,960 planted sensitive spans, 12 entity classes and 1,600 non-sensitive utility probes. It includes an authenticated Tonic Textual SDK runner, exact and overlap scoring, class/source/challenge breakdowns, and scorer regression tests. Its scope is narrower than the complete operational review below.

Download the runnable benchmark kit · Inspect the corpus manifest · Run status

02 / Permission boundaries for real supplier data

Permission boundaries for real supplier data

Initial supplier review uses the completed inventory and company identity, confidentially, with approved recipients. It does not involve actual records. Actual records or samples may go only to recipients and within limits approved in the Master cover or signed Confirmation. Full processing starts after the Confirmation is signed and SourceX instructs the parties to proceed, unless a signed deal record expressly allows earlier preparation. Separately approved early access or samples stay within their approved permissions.

Pre-Closing evaluation assesses quality, completeness, rights and suitability, including limited technical tests. It does not authorize AI training, fine-tuning, model improvement or unrelated commercial use. Commercial use depends on the contractual Closing conditions and permitted purpose. Required de-identification must be completed and checked before commercial use or any sharing that requires it.

Supplier submissions must match the signed inventory and reflect complete, readable business records. They must not be created or altered with AI tools in anticipation of delivery. The synthetic public benchmark is a separate test artifact. Its use does not authorize experiments on a supplier’s real records.

03 / 1. Define the question before the run

1. Define the question before the run

Evaluate two separate claims: whether the selected system finds sensitive information, and whether the configured end-to-end route produces a coherent, usable and appropriately controlled release. A Textual SDK run establishes neither access to Tonic’s full Entity Replacement workflow nor its published performance.

Pre-register the intended deployment, recipient use, languages, modalities and threat model. Treat public workforce rosters, professional profiles and other reasonably available information as potential linkage sources. The rights review remains separate from technical performance.

04 / 2. Freeze a reproducible run record

2. Freeze a reproducible run record

Record the corpus revision and hashes, input manifest, tool and model versions, entity taxonomy, custom rules, thresholds, synthesis settings, seed or determinism policy, provider configuration, extraction/rendering stack, scoring-code commit and environment. Record any external model endpoints and retention arrangements. Keep credentials and reversible mappings out of the published record.

Confirm in writing which Tonic service SourceX can use. If only SDK detection and basic synthesis are available, label the system accordingly. Do not present that run as Textual Graph Synthesis.

05 / 3. Use two complementary evaluation sets

3. Use two complementary evaluation sets

Public reference set

Use the public PrivacyBench corpus and the matching scoring code . Pin exact revisions that reproduce the v2.5 answer key and metric definitions. Preserve native inputs and the required output schema. Verify applicable data and code licenses and provide attribution.

Report the SourceX route as its own row alongside attributed vendor references. If its preprocessing, taxonomy, workflow or judge differs, explain why the comparison is limited rather than describing it as equivalent.

Representative SourceX holdout

Use supplier-authorized data within an approved processing environment. Stratify by source system, file type, language, size, entity type and risk. Include email headers, aliases, nicknames, same-name people, shared domains, rare context, attachments and unsupported inputs. Keep a dedicated challenge set for secrets and high-risk identifiers; report it separately from natural prevalence.

Separate tuning material from final evaluation. After remediation, use a fresh holdout sample. Reviewers should label the originals independently of model output where practical; use a second reviewer and adjudication for ambiguous and critical cases. Record disagreements and the final rubric.

06 / 4. Report component measures

4. Report component measures

Report micro and macro aggregation explicitly. Keep benchmark versions and measurement units separate. For PrivacyBench v2.5, use its exact combined-score definition; do not substitute an F1 or a custom transformation index. Do not combine these dimensions into an unvalidated SourceX headline score.

07 / 5. Put uncertainty next to the number

5. Put uncertainty next to the number

Specify sample sizes, selection probabilities, label counts and confidence-interval methods. Resample at the appropriate cluster level when messages or entities are correlated. Show per-class results even when support is small. Avoid treating all repeated mentions of one person as independent evidence.

Zero observed misses does not establish zero leakage. Report the sample and a suitable upper confidence bound under stated assumptions. A review of selected examples is not an estimate of whole-corpus accuracy.

08 / 6. Set release criteria in advance

6. Set release criteria in advance

The dataset schedule must contain numeric thresholds, units, strata, the sampling plan, reviewer responsibilities and escalation rules before the final run. This protocol does not invent universal thresholds or imply they have been agreed.

• Any observed critical identifier or secret that violates the agreed policy blocks the affected release until remediated and re-evaluated.
• Every original approved input must reconcile to an output, documented exclusion or tracked failure. Unprocessed material cannot disappear from the denominator.
• Required formats and entity classes must be tested. Unsupported content stays quarantined or is excluded under the agreed rule.
• Human review must address contextual linkage and confidentiality in addition to detector output. Findings and decisions remain auditable.
• Release requires a named decision-maker, approved output manifest, resolved exceptions and the required contractual conditions.

09 / 7. Separate the processing boundary from delivery

7. Separate the processing boundary from delivery

Before Closing, full identifiable records are limited to authorized preparation recipients. Other evaluation access is limited to the inventory or specifically approved samples. The AI training delivery contains only the selected output under the agreed treatment; original source records and reversible mappings remain outside that delivery. Check that permissions, required treatment and use authorization are recorded separately.

Deal-specific handling details are documented in the signed records and review evidence. Declined material is handled under the contractual deletion deadline, including samples, raw exports, temporary processing files and de-identified review copies. Any retained legal or routine recordkeeping copies remain restricted and unused. Optional attestation needs its own agreed scope and completed evidence.

10 / 8. Publish the result with its practical limits

8. Publish the result with its practical limits

Publish the date, system identity, frozen configuration, data revision, sample design, scoring rules, component results, uncertainty, failure analysis and limitations. Include a comparison only where the configurations and scoring are sufficiently aligned. Obtain independent review before describing a study as independent.

The authenticated run record, not this protocol, is the source of any SourceX performance result. Publish the run status and its review status with the result.

11 / Research references

Research references

The evaluation checks detection, replacement quality and corpus integrity using the SourceX scoring definitions above. Official references include Tonic PrivacyBench and Tonic’s licensing tenets . These references provide context; they do not attest to SourceX’s implementation.