Deal economics
Automated extraction and data margin: a net-of-quality savings test
By SourceX Editorial · Reviewed by Noah Loul ·
10 sections
Short answer
Automated extraction improves data margins only when the labor it saves exceeds the quality costs it adds: exceptions routed to people, a larger blind audit, errors that escape to the buyer and context the extractor drops. In this article's illustrative model, a 485-hour headline saving per 10,000 records nets 330 hours, and falls to 115 when context loss, audit bias and weak transfer combine.
Value drivers
Automated extraction data
Key takeaways
- Headline automation savings count only the extraction pass; net savings also charge exceptions, audit, escaped errors and context repair.
- In the illustrative model, 485 headline hours per 10,000 records net 330, about 68 percent of the claim.
- Context loss is the term to watch: repairable losses cost hours, while undetected ones surface later as rejected deliveries.
- Audit by blind re-extraction rather than by reviewing machine output, because reviewers anchored on a suggestion miss its errors.
- Ask for field-level error and context-retention rates on held-out records, by record type, beside the hours saved.
Can automated extraction improve data margins without reducing quality?
Yes, when the labor it removes is larger than the quality costs it adds, and when those costs are measured rather than assumed. Automated extraction, meaning software that pulls fields, links and text out of business records into a dataset, cuts keying time sharply. It also routes some records to people, needs an audit, lets some errors through and can drop context that a buyer's task depends on.
The margin question is the net. A tool vendor's or operator's headline usually compares the machine pass with fully manual extraction and stops there. This article builds the rest of the ledger: an editable waterfall from headline to net savings, a field-by-field view of where automation loses context, a blind audit design that measures the loss, and a stress test that shows how fast the net shrinks. All model values are illustrative, and none are SourceX figures.
Why labor savings are only half the automated extraction ledger
The savings side is real and documented, and so is the quality side, which is why both belong in one ledger. Keying work is shrinking: the Bureau of Labor Statistics projects employment of data entry keyers to fall from about 131,800 in 2025 to about 98,200 in 2035, a decline of 25.5 percent, and it lists the occupation's 2025 median annual wage at $41,340.
Quality does not follow speed automatically. In a field experiment with 758 Boston Consulting Group consultants, Dell'Acqua and colleagues found that on tasks inside GPT-4's capability frontier, consultants using the model completed 12.2 percent more tasks and finished 25.1 percent faster. On a task designed to sit just outside that frontier, consultants using it were 19 percentage points less likely to reach a correct solution than those working without it.
Extraction has the same shape. Fixed-layout fields such as invoice numbers and amounts sit well inside the frontier; the reason a refund was approved, buried in the middle of a long ticket thread, may not. A margin case that averages the two hides exactly the records where quality falls.
The net-of-quality waterfall: from headline hours to net savings
Net savings equal manual hours minus automated hours, with every quality term charged on both sides. In this illustration, per 10,000 records, a headline saving of 485 hours becomes 330 once exceptions, a larger audit, escaped errors and context repair are counted, about 68 percent of the claim.
The inputs, all illustrative: manual extraction at 3 minutes a record; a machine pass with monitoring at 15 hours per 10,000 records; 5 percent of records routed to people at 6 minutes each; an audit of 160 records for manual work and 400 for automated work at 3.75 minutes each; escaped errors on 1.0 percent of manual and 1.6 percent of automated records at 30 minutes each to fix at acceptance; and context repair on 1 percent of manual and 4 percent of automated records at 12 minutes each.
To turn hours into margin, assume preparation labor equals 20 points of a deal's contract value under manual extraction. Saving 330 of 580 hours then lifts deal margin by about 11.4 points, where the headline 485 would promise about 16.7. The waterfall, in hours per 10,000 records:
| Line | Manual hours | Automated hours | Savings after this line | What replaces the assumption |
|---|---|---|---|---|
| 1. Extraction pass | 500 | 15 | 485 | Time logs and machine run logs per batch |
| 2. Exceptions routed to people | 0 | 50 | 435 | Exception rate and minutes per exception, by record type |
| 3. Audit sample | 10 | 25 | 420 | Audit plan: sample size and blind re-extraction time |
| 4. Escaped errors fixed at acceptance | 50 | 80 | 390 | Buyer-found defects traced to extraction, with fix time |
| 5. Context repaired after detection | 20 | 80 | 330 | Context-retention audit results and repair tickets |
| Total | 580 | 250 | 330 net | Reconcile to preparation labor in the deal ledger |
Which fields can a machine extract safely, and which lose context?
Fields with a fixed format and a checkable answer are safe to automate; context that lives in sequence, links or free text is where automated extraction loses quality. Route by field family rather than by record, because a single record usually holds both kinds.
Position inside a record matters too. The Lost in the Middle study found that language models used information best when it appeared at the beginning or end of a long input and significantly worse when it sat in the middle, even for models built for long contexts. A commitment recorded halfway down a 40-message support thread is that case exactly, so audits should be stratified by record length.
Context also includes where a record came from and what it may be used for. When the Data Provenance Initiative audited 1,858 fine-tuning datasets, it found license omission of over 70 percent and error rates of over 50 percent on widely used dataset hosting sites: the metadata that decides whether a dataset can be used was missing or wrong where datasets were redistributed. Extraction pipelines face the same risk with source system, date range and rights fields. The routing matrix:
| Field family | Typical automated failure | Check that catches it | Default route |
|---|---|---|---|
| Identifiers, amounts and codes | Misread characters or a wrong field mapping | Format rules, checksums and totals that must reconcile | Automate with deterministic checks |
| Dates and status history | Current value kept, history dropped, time zones flattened | Replay events and compare with the source's current state | Automate with a replay check |
| Links across systems | Ticket, order and refund no longer joined | Join rates and orphan counts against source totals | Automate, then reconcile counts |
| Free-text rationale and commitments | Reason or promise omitted, paraphrased or invented | Blind human re-extraction on a stratified sample | Human-verified |
| Attachments and images | Attachment skipped or its content left unlinked | Attachment counts per record against the source | Route to people when decisive |
| Personal identifiers for de-identification | Names inside free text missed | Sampled human review after the automated pass | Automated pass plus human sample |
How do you audit automated extraction without trusting the machine?
Re-extract a random, stratified sample by hand without showing the reviewer the machine's output, then compare field by field. Checking the machine's answer instead of producing an independent one measures agreement with the machine, not accuracy.
Anchoring on a suggestion is a documented failure. In a Radiology study by Dratsch and colleagues, 27 radiologists read 50 mammograms with assessments from what they believed was an AI system; incorrect suggestions impaired performance at every experience level, and inexperienced readers were the most susceptible. An extraction reviewer shown a confident machine value faces the same pull.
Size the audit like an acceptance plan. The NIST/SEMATECH e-Handbook frames lot acceptance sampling around an acceptable quality level the producer should meet and a lot tolerance percent defective, a defect level the consumer would find unacceptable, with the producer's and consumer's risks of a wrong decision set in advance. For extraction, the buyer's acceptance test supplies the tolerance. If a model grades free-text fields, calibrate it against people: Anthropic's guide to agent evaluations distinguishes code-based, model-based and human graders, and describes a team that moved to model graders while keeping periodic human calibration. The audit steps:
- Draw the sample at random within strata: record type, record length band and field family.
- Have a reviewer who has not seen the machine output extract each sampled record, then adjudicate disagreements with a second reviewer.
- Score field error rate, escaped-error rate at acceptance and context retention, meaning the share of required links, sequence, rationale and attachments still present.
- Run the same audit on a manual baseline batch so the comparison is like for like.
- Report each rate with its count and interval, and hold automated release for any stratum whose upper bound passes the buyer's tolerance.
Stress the savings: four ways the quality tax grows
The net shrinks fastest when context loss rises, because every lost element costs repair time if someone catches it and a rejected delivery if no one does. In the stress test below, three plausible changes together cut the illustrative net saving from 330 hours to 115, about a quarter of the headline.
Vendor accuracy is the input most likely to be wrong. When Scale AI researchers built GSM1k, a fresh set of problems matched to the GSM8k benchmark, leading models showed accuracy drops of up to 8 percent, and several model families showed evidence of systematic overfitting. An extractor tuned on public forms can lose accuracy on a company's own templates in the same way, so the exception rate should be measured on held-out company records, not taken from a benchmark.
Break-even sits further out than a skeptic might expect but closer than a pitch implies: with the other baseline inputs held, automation stops saving hours only when about one record in five needs context repaired. Undetected context loss leaves no repair time to count, only buyer rejections, so the true break-even is lower than the arithmetic shows. The four stresses:
| Scenario | What changed | Automated hours | Net savings | Share of 485 headline |
|---|---|---|---|---|
| Baseline | Inputs as in the waterfall | 250 | 330 | 68 percent |
| Free-text-heavy records | Context repair on 10 percent of records | 370 | 210 | 43 percent |
| Audit reviews machine output | Escaped errors rise to 2.5 percent | 295 | 285 | 59 percent |
| Accuracy does not transfer | Exceptions rise to 10 percent | 300 | 280 | 58 percent |
| All three together | The three changes combined | 465 | 115 | 24 percent |
Shortcut metrics that hide the quality cost
Four common metrics make automation look better than the ledger allows, and each was considered and rejected for this framework. Each has a replacement that keeps the quality cost visible.
- Overall accuracy: averages easy fields with hard ones, so a high headline figure can coexist with most free-text commitments missing. Replace it with error and retention rates by field family.
- Records processed per hour: measures the machine pass alone and rewards skipping exceptions. Replace it with accepted records per labor hour, every line included.
- Reviewer agreement with the machine: inflated when reviewers see the machine output first. Replace it with blind re-extraction error rates.
- Vendor benchmark accuracy: measured on someone else's documents. Replace it with exception and error rates on a held-out sample of the company's own records.
What could make the automation margin case wrong
Five developments could move the net in either direction, and an investor should know which one a company's case leans on. Some would strengthen the case for automation; others would hide its cost.
- Models could close the gap. If extracting free-text rationale moves inside the capability frontier, the context and escaped-error lines shrink and the net approaches the headline; the frontier experiment describes GPT-4 in 2023, not current systems.
- Buyers could stop catching context loss. Acceptance tests that check formats but not links or rationale let lost context pass, which flatters margin today and moves the cost into the buyer's training or evaluation results later.
- The manual pool could shrink. The BLS projection for data entry keyers points to fewer people doing keying work, which raises the cost of keeping a manual baseline and the value of audit skills.
- Audits could drift. Reviewers who see machine output, or model graders without human calibration, report agreement as accuracy and hide the escaped-error line.
- Provenance could fail silently. As the Data Provenance audit shows, metadata errors can be widespread without anyone noticing, and a dataset with clean fields but wrong source or rights fields is not licensable.
Diligence questions on automated extraction, SourceX included
Ask for the full ledger by record type, with the quality terms measured on held-out records, and apply the same questions to SourceX.
The ledger applies to SourceX because preparation is part of its job: it helps companies make licensed AI datasets out of selected business records, and preparing and delivering that data are among the steps it runs. De-identification is done by SourceX itself or handed to its processing partner, and a large delivery either remains in the seller's own storage or travels on encrypted drives.
SourceX publishes no automation rates, preparation hours or margins, and investors get its company and commercial metrics from the company directly. For its Rule 506(c) raise from accredited investors, materials come on request through the investor page, which is also where to ask how audit samples are drawn.
- What share of preparation hours goes to the machine pass, exceptions, audit, escaped-error fixes and context repair, per 10,000 records and by record type?
- How is the audit drawn, and do reviewers see the machine output before they extract?
- What are field error and context-retention rates by field family, with counts and intervals?
- How many defects did buyers find at acceptance that traced back to extraction, and how long did each take to fix?
- Which accuracy figures come from public benchmarks, and which from held-out company records?
- Which field families stay human-verified, and what evidence would justify automating them?
Methods note: the net-of-quality model and its blind spots
The waterfall, routing matrix, audit design and stress test were built as editorial frameworks for this article, dated 2026-10-26. The public evidence comes from the BLS employment projections, the Dell'Acqua field experiment, the Lost in the Middle study, the Data Provenance Initiative audit, the Radiology study by Dratsch and colleagues, the NIST/SEMATECH e-Handbook, Anthropic's guide to agent evaluations and the GSM1k paper.
All model inputs are illustrative and stated in the waterfall section: hours per 10,000 records; exception, escape and context-loss rates; audit sizes and repair times; and a 20-point labor share for the margin translation. Not measured: real exception, error or context-loss rates for any extraction tool, at SourceX or elsewhere; the cost of undetected context loss; tool licensing; and how buyers' acceptance tests treat missing context.
Frequently asked questions
How much does automated extraction save on data preparation?
It depends on what is counted. In this article's illustrative model, the machine pass saves 485 of 500 extraction hours per 10,000 records, but after exceptions, a larger audit, escaped errors and context repair the net is 330 hours. Measure each line on a company's own records before relying on any figure.
What is context loss in automated extraction?
It is information a buyer's task needs that the extracted dataset no longer carries: the link between a ticket and its refund, the order of status changes, the reason behind a decision or an attachment. Every field can be individually correct while the record has lost the context that made it useful.
Why should extraction audits be blind?
Reviewers who see a confident machine value tend to accept it, a pattern documented as automation bias in clinical reading studies. Independent re-extraction followed by adjudication measures accuracy; reviewing the machine output only measures agreement with it.
Does automated extraction lower data quality?
Not necessarily. It tends to do well on fixed-format fields and to lose quality on free text, sequences, cross-system links and attachments. Routing by field family, with blind audits on the weaker families, keeps most of the savings without hiding the quality cost.
Does SourceX publish its automation rates or margins?
No. SourceX has not published automation rates, preparation hours or margins and shares company and commercial metrics with investors directly. Investor materials are available on request at sourcex.si/investors, where questions about audit design can be raised; any offering is open only to verified accredited investors.
Sources
- BLS employment projections list data entry keyers at about 131,800 jobs in 2025 and about 98,200 projected for 2035, a 25.5 percent decline, with a 2025 median annual wage of $41,340. Source
- Dell'Acqua and colleagues' field experiment with 758 BCG consultants found GPT-4 users completed 12.2 percent more tasks, 25.1 percent faster, inside the frontier, and were 19 percentage points less likely to be correct on a task outside it. Source
- Lost in the Middle found language model performance is often highest when relevant information is at the beginning or end of the input and degrades significantly when it is in the middle of long contexts. Source
- The Data Provenance Initiative audit of 1,858 datasets found license omission of 70 percent or more and error rates of 50 percent or more on widely used dataset hosting sites. Source
- Dratsch and colleagues (Radiology, 2023): 27 radiologists reading 50 mammograms with purported AI suggestions; incorrect suggestions impaired performance across experience levels, with inexperienced readers most susceptible. Source
- The NIST/SEMATECH e-Handbook defines the acceptable quality level and the lot tolerance percent defective used to design lot acceptance sampling plans, with producer's and consumer's risks. Source
- Anthropic's guide to agent evaluations describes code-based, model-based and human graders and an example team that moved to model graders with periodic human calibration. Source
- The GSM1k paper (current arXiv version) reports accuracy drops of up to 8 percent for leading models on fresh problems matched to GSM8k, with several model families showing evidence of systematic overfitting. Source
Request SourceX investor materials
SourceX shares company and commercial metrics with investors directly. Any offering is made only to verified accredited investors.