AI uses for records
Post-training dataset format conversion without losing meaning
By SourceX Editorial · Updated
9 sections
Short answer
To convert business records into training examples without losing meaning, write a transformation contract first: the target format, a field map with lineage, the facts that must survive, and rules for missing values, codes, time and text normalization. Run fidelity checks on every example, then show on held-out real cases that the converted set fixes the targeted failure without regressions.
License terms
Post training dataset format
Key takeaways
- Most meaning is lost in status codes, units, time offsets, negations and missing references, not in the main text.
- A transformation contract names each target field's sources, rule and invariants, with a check that enforces each clause.
- Store neutral role and content messages and let the trainer apply the model's chat template once.
- Run fidelity checks on the full set and send every failure to someone who knows the source system.
- Judge the conversion on the failure cohort it was meant to fix, against a matched baseline, with a regression set.
How do you convert business records into training examples without losing meaning?
Write a transformation contract before converting anything: for every field in the target example, name the source fields it comes from, the rule applied and the facts that must survive the rule. Then run source fidelity checks on every converted example, and prove the converted set fixes the model failure that motivated it on held-out real tasks without breaking what already worked.
Format conversion looks like plumbing, so it rarely gets a specification. Yet most of the meaning in a business record sits in things a converter can silently drop: a status code that distinguishes waiting on the customer from waiting on engineering, a unit, a time zone, a negation in a quoted email, the attachment the reply refers to. A post-training dataset that loses these still looks fluent, which is why the loss goes unnoticed until the model is wrong in production.
This page covers the conversion step itself. Choosing which records become training pairs, and rebuilding the context an answer was written in, are covered in the guide to turning business records into instruction-response pairs; delivery shape is covered in the chat fine-tuning format guide.
Where does meaning usually get lost between record and example?
Meaning is lost at the joins and the normalizations, not in the obvious text fields. The common failure points are predictable enough to check for by name.
- Status and category codes collapsed into fewer labels, so pending and on hold become the same thing.
- Numbers stripped of units or currency fields, or rounded during a type cast.
- Timestamps converted without their offset, which reorders events across time zones.
- Quoted earlier messages merged into the latest reply, so the model learns to repeat history as if it were the answer.
- Negations and conditions lost when long notes are truncated to fit a length limit.
- References to attachments, linked records or prior tickets kept as text while the referenced content is gone.
- Compatibility text normalization that rewrites characters, superscripts and ligatures the source used deliberately.
- Chat-template control tokens applied twice, or the wrong template applied for the target model.
- Near-duplicate records landing on both sides of the train and evaluation split.
The transformation contract, clause by clause
The contract is a short, versioned document kept next to the conversion code; reviewers sign it before the first full run. Each row below is a clause, with an illustrative entry and the check that enforces it. Version 1.0, dated 2026-10-18.
| Clause | What it records | Illustrative entry | Check that enforces it |
|---|---|---|---|
| Target format and version | The record shape the trainer expects | Conversational prompt-completion: a prompt list of role and content messages and a completion list | Schema validation on every example; reject unknown keys |
| Field map with lineage | For each target field, the source dataset, field and transformation type | completion.content from ticket.public_reply, direct; prompt from subject, description and prior turns, indirect | Every target value traces to at least one declared source field |
| Invariants | Facts that must be identical before and after | Order numbers, quantities with units, dates, product codes, negations | Extract invariants from source and example and compare exactly |
| Missing value semantics | Which source strings mean missing, and how missing is shown in the example | Empty, N/A and a single dash mean missing; shown as not recorded, never as zero | No missing marker converted into a real value |
| Code and category maps | Every allowed source code and its target wording | pending means waiting on customer; hold means waiting on internal team | Map is total over the source categories; unmapped codes fail |
| Time handling | Offsets, the as-of moment and ordering rules | All times kept with explicit offset; context cut at the moment the reply was sent | No context item later than the as-of time; ordering preserved |
| Text normalization | Which Unicode form and cleanup steps are allowed | NFC only; signatures and quoted history removed by declared rules | Diff of removed spans is logged and sampled for review |
| Template rendering point | Whether the chat template is applied at conversion or at training | Neutral messages stored; template applied once by the trainer | No model-specific control tokens in stored examples |
| Split rule | How examples are assigned to train and evaluation | By account and by month; all turns of one thread in one split | No thread, account or near-duplicate text on both sides |
Which target format should each record pattern become?
Pick the format from the behavior you want to train, then check the record actually contains the evidence for it. The TRL library documents the common shapes: standard (plain strings) and conversational (lists of messages with a role and content), each in language modeling, prompt-only, prompt-completion and preference variants with chosen and rejected completions.
Prompt-completion suits records with a clear request and an accepted answer, such as a ticket and the reply that closed it. Full conversational messages suit multi-turn threads where earlier turns shape the answer. Preference pairs need two candidate outputs and a recorded judgment between them, for example a draft and the edited version a supervisor approved; without that judgment, do not invent a rejected answer from the record.
Store examples as neutral role and content messages and let the training stack render them. The Hugging Face chat templating guide notes that different models use different control tokens, even models fine-tuned from the same base, and that formatting with the wrong tokens can hurt performance badly. It also warns that adding special tokens again after applying a chat template can duplicate them. Conversion code that bakes in one template ties the dataset to one model and invites both errors.
Source fidelity checks to run on every converted example
Run these automatically on the full set, and send a stratified sample of passes and every failure to a reviewer who knows the source system. A check that only runs on a sample will miss the rare record types where conversion breaks.
At a glance
Source fidelity checks to run on every converted example
- Lineage trace: every target field lists its source fields and transformation, in the spirit of OpenLineage's column lineage facet, which maps each output field to its input fields and labels transformations as direct or indirect, including whether a transformation masks the data.
- Invariant equality: identifiers, quantities, units, dates and codes extracted from the source match those in the example exactly.
- Negation and condition check: sentences containing not, unless, except or only if in the source still contain them in the example.
- Missing-value check: values the source treats as missing are not rendered as real values. The Frictionless Table Schema treats this as declared metadata, a missingValues list of strings that should be read as missing, which is the right model for the contract.
- Category coverage: every source code seen in the data appears in the map; Table Schema's categories property is a usable way to declare the allowed set.
- Time order: no context item is later than the as-of time, and offsets are explicit. RFC 9557, which updated RFC 3339 in 2024, uses Z for a time whose UTC value is known but whose local offset is not, keeps +00:00 for UTC as the intended reference and discourages the older -00:00 form, so preserve that distinction rather than guessing an offset.
- Normalization diff: the Unicode standard warns that compatibility normalization (NFKC and NFKD) can remove formatting distinctions such as superscripts and ligatures, so log any change beyond NFC.
- Truncation flag: any example cut to fit a length limit is marked, and the cut never removes the part the completion depends on.
- Split leakage: exact and near-duplicate text is checked across splits. Lee and colleagues found that train and test overlap affects over 4% of the validation set of standard datasets and leads researchers to overestimate model accuracy.
Proving the conversion fixed the failure it was meant to fix
A conversion is justified by a model failure, so evaluate it against that failure. Start by naming the failure cohort, for example escalation replies that promise a callback the team never offers, and build a held-out set of real cases from that cohort that no converted example came from.
Then compare like with like. Train one model on the converted examples and another on the same number of examples from your existing data, with the same base model, settings and training budget. Score both on the failure cohort and on a regression set of tasks the model already handled. A fix that improves the cohort while degrading other tasks is a trade-off for a person to approve, not a success.
Small numbers of examples can move behavior, which is why fidelity matters. The LIMA authors fine-tuned a large model on 1,000 carefully curated prompts and responses and argued that alignment mostly teaches the format and style of interaction. If a few hundred converted examples carry a systematic distortion, such as every pending ticket rendered as resolved, the model can learn that distortion just as efficiently.
- Evidence to keep: the failure cohort definition, the selection rule for converted records, the matched baseline, the untouched evaluation set, cohort and regression results, and the contract version used.
- Lineage to keep: for every example, the source record keys and the contract version, so a bad example can be traced and removed without regenerating everything.
Worked example: a help desk ticket that changed meaning (illustrative)
This example is illustrative. The ticket, codes and counts are invented to show how the contract and checks catch a meaning change; they are not from any SourceX supplier.
A team converts closed tickets from a help desk system such as Zendesk into prompt-completion examples to fix a failure: the model tells customers their issue is with engineering when it is actually waiting on them. The first converter maps the source status field by keyword. Pending, which in this team's configuration means waiting on the customer, and on-hold, which means waiting on an internal team, both become the target label in progress.
- Source: status on-hold, internal note says engineering is reproducing the bug, public reply says the team will update the customer when engineering responds.
- First conversion: prompt includes subject, description and status in progress; completion is the public reply. The status nuance the model needed has gone.
- Check that catches it: category coverage fails because the map sends two source codes to one target label without a declared merge.
- Second check that catches it: an invariant extracted from the source (who the ticket is waiting on) does not appear in the converted prompt.
- Fix: the contract maps pending to waiting on customer and on-hold to waiting on internal team, and adds who-is-waiting as an invariant. Internal notes stay out of the prompt because the customer never saw them.
- Evaluation: on a held-out set of 200 illustrative tickets from the failure cohort, the team compares the corrected set against a matched baseline and checks a regression set of routine replies before accepting it.
Records that should not be converted yet
Some records fail the contract for reasons the converter cannot fix, and forcing them through is how meaning gets lost. Hold back records whose answer depends on an attachment, screenshot or linked record that was not exported. Hold back threads where you cannot tell which reply the business accepted as correct. Hold back categories whose codes were redefined during the period, until the contract maps each era separately.
Stop the conversion and escalate when the fidelity checks fail on a meaningful share of a record type, when the cohort result does not improve against the matched baseline, or when an existing capability regresses beyond what the team agreed to accept. At that point the evidence usually points to the source data or the failure definition, not to the converter, and a post-training lead and someone who runs the source system should look together.
Rights are a separate gate. The ability to export tickets is an access question; whether customer messages may be used is a privacy question; whether the company may license records and derived examples for training is a contract question. Converting records into a new format does not change any of those answers, and de-identification alone does not create a right to license.
Requesting records with a conversion contract attached
A supplier can prepare records far better when the request names the target format, the invariants and the split rule up front. Attaching the transformation contract to a data request turns vague requirements into fields a supplier can check against its own system.
SourceX works from the same kind of request: it keeps no datasets on the shelf and looks for each one at U.S. businesses after the buyer has named scope, volume, language, format, timing and licensing needs, so a conversion contract can travel with the request itself. No record leaves the supplier until that company has signed off on the scope and terms, and de-identification is handled by SourceX through approved processing routes.
Frequently asked questions
What is a transformation contract for training data?
It is a short, versioned specification of how source records become training examples: the target format, the field map with lineage, the facts that must survive conversion, missing value and category rules, time handling, normalization and the split rule. Each clause has a check that enforces it on every example.
Should I store examples with the chat template already applied?
Usually not. Store neutral role and content messages and let the trainer apply the target model's template once. Baking in one template ties the dataset to one model family and risks duplicated special tokens.
How do I know if converted examples actually helped?
Compare a model trained on the converted examples with one trained on the same number of baseline examples under identical settings. Score both on held-out real cases from the failure cohort and on a regression set of tasks the model already handled.
Is Unicode normalization safe for business records?
Canonical normalization (NFC) is generally safe for text. Compatibility normalization (NFKC) can rewrite characters such as superscripts and ligatures, which may carry meaning in part numbers, measurements or formulas, so log any change it makes.
What should happen to records that reference missing attachments?
Hold them back or mark them, rather than converting the text alone. If the accepted answer depends on content the model will never see, the example teaches the model to answer without the evidence.
Sources
- Hugging Face TRL documents standard and conversational dataset formats, with prompt-only, prompt-completion and preference variants. Source
- The Transformers chat templating guide explains that models use different control tokens, that wrong tokens hurt performance and that re-adding special tokens after templating duplicates them. Source
- OpenLineage's column lineage facet maps each output field to the input fields used to compute it, with DIRECT or INDIRECT transformation types and a masking flag. Source
- The Frictionless Table Schema defines missingValues and a categories property for declaring missing markers and allowed values. Source
- RFC 9557 updates RFC 3339 so that Z marks a time whose UTC value is known but whose local offset is unknown, keeps +00:00 for UTC as the reference, and discourages -00:00. Source
- Unicode Standard Annex 15 explains that compatibility normalization can remove formatting distinctions such as ligatures and superscripts. Source
- Lee and colleagues found train-test overlap affecting over 4% of the validation set of standard datasets, which leads researchers to overestimate model accuracy. Source
- The LIMA paper fine-tuned a large model on 1,000 curated prompts and responses and argued alignment mainly teaches format and style. Source
Tell us what data you need
Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.