AI uses for records
Fine-tuning evidence citation behavior: find the record, then answer
By SourceX Editorial · Updated
9 sections
Short answer
Make the record a precondition of a correct answer. Each training item pairs a business question with a pool holding the supporting record and realistic distractors, such as a superseded policy or another customer's ticket, and some items omit the record entirely. The target first selects and quotes the supporting span, then answers, or says the record is missing. Judge the result on untouched held-out tasks.
Agent run
Fine tuning evidence
Key takeaways
- Name one citation failure cohort before writing items; each needs its own distractors and metric.
- Distractors from the same systems as the oracle, such as superseded policies, teach the behavior; random documents do not.
- Some items must omit the supporting record so the model learns to say it is missing.
- Train the citation format your system will consume and verify every quoted span and offset first.
- Judge the pack against a matched-budget baseline on untouched held-out tasks, with regression checks on unrelated skills.
How can training tasks make a model find the supporting record before it answers?
Build items where answering from memory or from the nearest-looking document scores zero. That means three things in the data: the question can only be answered from one specific record, the context also contains records that look right but are not, and the target response shows the selection and the quoted evidence before the answer. A model trained on clean question and answer pairs learns to answer; a model trained on these items learns to look first.
Recent research recipes share this shape. RAFT, from the Gorilla project, generates question, answer and document triplets in which the documents are the relevant chunk plus distractor chunks, with settings for the number of distractors and the share of items that include the relevant document at all. Its target format asks the model to copy supporting sentences between quote markers and to end with a tagged final answer. Self-RAG trains a model to emit reflection tokens that decide whether to retrieve, whether a passage is relevant and whether the output is supported.
The rest of this guide turns that shape into a pack you can build from business records: start from a named failure, write items that target it, and keep an evaluation the training never touched.
Which citation failure are you training away?
Pick one failure cohort before writing a single item, because each one needs different distractors and a different metric. Mixing them makes it impossible to tell later which examples did the work. The table lists the cohorts that show up most often when models answer from business records.
| Failure cohort | What it looks like | Training signal that targets it | Metric that should move |
|---|---|---|---|
| Answers from memory | Correct-sounding answer that no record in context supports | Items whose answer contradicts common practice but matches the record | Share of claims supported by the cited record |
| Cites the wrong record | Citation points to a similar record that does not entail the claim | Near-duplicate distractors from the same customer or product | Citation precision |
| Cites a stale version | Quotes a superseded policy or price list | Superseded versions placed beside the current one, with effective dates | Accuracy on version-sensitive questions |
| Over-cites | Lists every passage, most of them unnecessary | Targets that cite only the minimal supporting span | Citation precision and citation count |
| Answers when nothing supports it | Confident answer with no supporting record in context | Items without the record whose target says it is missing | Correct abstention rate |
| Refuses when the record is there | Says it cannot find information that is present | Items with the record buried among long distractors | Answer rate on answerable items |
What should the source grounded response training pack include?
The pack is a set of items plus the documents that make them checkable. Every field below exists so that an evaluator, human or automated, can verify the target without asking the author what was meant. This is version 1.0 of the specification, dated 2026-10-18.
At a glance
What should the source grounded response training pack include
| Field | Purpose | Illustrative value | Check before use |
|---|---|---|---|
| item_id and cohort | Ties the item to the failure it targets | gr-0187, stale_version | Cohort is one of the declared cohorts |
| question | The request as a user would write it | Can this order still be returned for a refund? | Answerable only from a record in the pool |
| record_pool | Every document shown to the model | Five records: order, two policies, two tickets | All IDs resolve to stored, de-identified records |
| oracle_ids | The record or records that support the answer | returns-policy v4 | Empty only when record_present is false |
| distractor_ids and type | Why each distractor is hard | returns-policy v3 (superseded) | Each has a stated type |
| record_present | Whether the oracle is in the pool | true | Share of false items is set per cohort |
| as_of_date | The date the question is asked | 2026-02-11 | Oracle is in effect on this date |
| target_evidence | Quoted span with document ID and offsets | returns-policy v4, characters 412 to 538 | Span text matches the stored record exactly |
| target_answer | The answer, or a statement that support is missing | No: the 30-day window closed on 9 February | Every claim is entailed by target_evidence |
| split and lineage | Keeps training and evaluation apart | train; source tickets from one customer group | No record or customer appears in both splits |
Should the target quote inline or return structured citations?
Choose the format your production system will consume, then train exactly that format. Inline quotes are easy for a reviewer to read and survive any interface; structured citations with document IDs and character offsets are easier to verify automatically and to render as links.
RAFT's generation prompt asks for copied sentences between begin and end quote markers and a final line that starts with an answer tag. Claude's citations feature shows what a structured version can look like: for plain-text documents each citation carries the cited text, a zero-indexed document index, the document title and a character range with a zero-indexed start and an exclusive end. If your target uses offsets, follow the same conventions throughout the pack and verify every offset against the stored record before training, because one misaligned offset teaches the model that citations are approximate.
Self-RAG takes a third route: special tokens in the output mark retrieval decisions, passage relevance and whether the generation is fully supported, partially supported or unsupported. That suits teams that control the decoding loop; teams fine-tuning through a hosted API usually stay with text quotes or structured fields.
How hard should the distractors be, and how often should the record be missing?
Distractors are where the behavior is learned, so draw them from the same systems as the oracle. Random documents teach nothing; a model learns to look carefully only when the wrong answer is one record away. In business records the hardest distractors are usually the previous version of the same policy, a ticket from the same customer about a different order, the same clause in a sibling contract, and a knowledge-base article that describes the general rule while the record shows the exception.
At a glance
How hard should the distractors be, and how often should the record be missing
Items without the supporting record teach the model to say so. RAFT exposes the share of items that include the relevant document as a setting rather than a fixed value, which is the right framing: treat it as a variable you test against your held-out set, not a constant you copy. Track correct abstention and answer rate together, because pushing one up can push the other down.
- Same-entity distractor: another record about the same customer, asset or vendor.
- Version distractor: the superseded policy, rate card or template, with its effective dates visible.
- Sibling distractor: the same document type for a different account, site or contract.
- General-rule distractor: the standard procedure when the oracle records an approved exception.
- Missing-record item: the pool holds only distractors, and the target names what would be needed.
Did the selected examples fix the failure without breaking anything else?
Answer that with a controlled comparison on tasks the training never saw. Fix a selection rule for which items enter the pack, train a baseline with the same number of examples or tokens drawn without that rule, and evaluate both on an untouched held-out set split by cohort. If the selected pack only wins with a larger budget, you have measured volume, not selection.
For citation quality, the ALCE benchmark from Princeton offers a reusable method. Its evaluation code scores citation recall per sentence by asking a natural language inference model whether the cited passages jointly entail the sentence, and citation precision by checking whether each cited passage was needed, flagging citations whose removal leaves the sentence still supported. The benchmark scores fluency, correctness and citation quality across three datasets. For human review, the Attributable to Identified Sources framework asks whether a generic hearer would affirm that, according to the source, the statement is true.
Then check what else moved. Run the existing evaluation suites for tasks that do not involve records, such as drafting and arithmetic over provided tables, and compare per cohort, not only the overall score. A citation fine-tune that improves grounded answers but makes the model refuse ordinary requests has traded one failure for another.
| Measurement | How it is computed | What it tells you |
|---|---|---|
| Citation recall | Share of answer sentences entailed by their cited passages | Whether claims are backed by what is cited |
| Citation precision | Share of citations needed to support the sentence | Whether the model pads citations |
| Answer accuracy by cohort | Correct answers on held-out items for each failure cohort | Whether the targeted failure improved |
| Correct abstention | Share of missing-record items where the model says support is missing | Whether it stops guessing |
| Regression suite | Existing evaluations unrelated to records | Whether another capability got worse |
Worked example (illustrative): a returns-policy assistant at a distributor
This example is invented to show the method; it reports no results. A distributor's support assistant answers return questions and often quotes the 45-day window from an older policy instead of the current 30-day one. The cohort is stale_version.
The team writes items from closed return tickets whose final outcome is recorded. Each pool holds the order record, the current returns policy, its superseded predecessor and two tickets from the same customer. In a share of items the current policy is withheld and the target says the effective policy for that date is not in the provided records. Held-out items come from a customer group that contributed nothing to training, and the as_of_date of every held-out item falls after the last training item, so the evaluation also tests a later period.
Before training, a reviewer samples items and confirms that each quoted span exists verbatim at the stated offsets and that the answer follows from the quote alone. After training, the team reports stale-version accuracy, citation recall and precision, abstention and the regression suite for both the selected pack and the matched baseline, each with a count of evaluated items so readers can judge uncertainty.
When is find-before-answer the wrong thing to train?
Not every task should route through a record. Arithmetic over figures already in the prompt, drafting tone, and general knowledge questions gain nothing from a retrieval step and can get slower or more evasive. Train the behavior only for questions whose correct answer depends on a record that can change.
Stop and narrow the scope when records conflict and nobody has written down which source wins, such as a contract clause against a later email approval. A model cannot learn a precedence rule the organization has not settled. Bring in a domain reviewer before training on regulated content, and keep personal data out of quoted spans unless the agreement and the de-identification plan allow it.
Getting business records you are allowed to train on
The pack needs records with stable IDs, versions and dates, and the right to use them for training, not just for retrieval at inference time; those are different permissions. Records accessible through a system login are not thereby licensed for training. Records obtained through SourceX are sourced from U.S. businesses for a specific request, with the buyer stating scope, volume, language, format, timing and licensing needs at the outset; each supplying company approves the scope and terms and signs before delivery, so the permitted uses for grounded-response training are written into the agreement.
What should an evidence grounded answer dataset request say?
It should name the repeat failure the data must fix, what the model sees, which records count as evidence, how each claim links to that evidence, and the test a delivery must pass. A request for "grounded question and answer pairs" leaves the supplier to guess all five, and the guesses rarely match your retriever or your grader.
Start from failures you already have. Anthropic's guidance on agent evaluations (January 9, 2026) calls 20 to 50 simple tasks drawn from real failures a great start, defines a good task as one two domain experts would independently pass or fail the same way, and recommends a reference solution for each task. OpenAI's GDPval evaluation uses the same shape for professional work: each task pairs a request, often with reference files, with a deliverable from an experienced professional, and model output is graded in blinded comparisons by experts in that occupation.
The missing-evidence line matters because scoring shapes behavior. OpenAI's September 2025 paper on hallucinations argues that standard training and evaluation procedures reward guessing over acknowledging uncertainty, so a delivery judged only on correct answers teaches confident guessing. On sourcex.si, these lines fit the request form's scope, volume, language, format, timing and licensing fields, with the acceptance test written into the same request; datasets are sourced on request, not held in stock.
| Request line | What to write | What it rules out | Acceptance check on delivery |
|---|---|---|---|
| Repeat failure | One named failure with 20 to 50 real failing cases, each with the wrong answer and the record that should have been used | A generic grounded set that targets nothing you measured | Every delivered item is tagged to the named failure |
| Task inputs | The question as users phrase it, plus the record types, source systems and date range the model sees at inference | Items built from documents your retriever never returns | Every pool record is a listed type inside the stated date range |
| Evidence boundary | Which records count as evidence, which never do (drafts, chat hearsay, general FAQ text) and the precedence rule when two conflict | Targets that quote an unofficial or superseded source | Every cited record is an allowed type and in effect on the item's date |
| Support unit | Each answer claim linked to a record ID and the exact quoted span; no written reasoning required | Rationales nobody can check against a record | Quoted text matches the stored record at the stated location |
| Missing-evidence share | The share of items with no supporting record, and the exact statement the target makes instead | Data that only ever rewards answering | A guess on a missing-record item fails, scored apart from accuracy |
| Reference answer and pass rule | A reference answer per item and a pass rule two domain reviewers would apply the same way | Subjective quality grades | A second reviewer agrees on a stated sample before acceptance |
| Intended use and split | Training, evaluation or both, with held-out items separated by account and by date | Training records leaking into the evaluation | No record or account appears in both splits |
Why ask for source-linked support instead of reasoning traces?
Because a quoted span can be checked against the stored record by anyone, while a written chain of reasoning cannot be checked against whatever actually produced the answer. A request that asks annotators or models to show their reasoning buys text that looks like evidence and is not.
Anthropic's alignment research (April 3, 2025) measured the gap. When reasoning models were slipped a hint about the answer, Claude 3.7 Sonnet mentioned the hint 25 percent of the time on average and DeepSeek R1 39 percent. In settings with an exploitable reward hack, the models admitted using it less than 2 percent of the time in most scenarios, and more outcome-based training raised faithfulness only to 28 and 20 percent on two evaluations before leveling off. Rationales a person writes after answering carry a milder version of the risk: they describe a tidy path, not necessarily the one taken. The January 2026 evaluation guidance makes the general point that it is often better to grade what an agent produced, not the path it took.
So strike reasoning traces from the request and ask for decisions a reviewer can confirm:
- Claim-to-span links: every sentence of the target answer tied to a record ID and an exact quote.
- Rejection reasons: for each look-alike record the annotator set aside, a reason from your list, such as superseded, other account or general rule, which record metadata can confirm.
- Precedence note: the named rule applied when records conflict, chosen from rules your team has written down.
- Missing-evidence statement: the record type that would have answered when nothing in the pool does.
Before you send it: could citation tooling be the real cause?
Sometimes, and it is cheaper to find out before buying records. Many citation failures that look like model behavior come from text extraction, chunking, citation parsing or the grader, and each can be tested on the same failing cases with no new data.
Check the grader first: Anthropic's January 2026 guidance warns that you won't know whether graders work unless you read the transcripts and grades from many trials. Then check the citation mechanism. Claude's citations documentation says that because the API extracts the cited text itself, its citations always point to valid locations in the provided documents, and that in Anthropic's evaluations the feature was significantly more likely to cite the most relevant quotes than purely prompt-based approaches. If a structured citation feature clears the failure, the request you were about to send is unnecessary.
If failures survive these tests and persist with the correct record placed in the prompt, the gap is model behavior and the checklist applies.
| Symptom in failing answers | Tooling cause to rule out | Quick test | Fix that needs no new records |
|---|---|---|---|
| Right quote, wrong location | Offsets computed on text that was later normalized by PDF extraction, HTML stripping or whitespace cleanup | Recompute offsets on the exact text the model received | Store and cite the post-extraction text; version the extractor |
| Cites a passage the record does not contain | Citations written as free text and parsed from prose | Rerun the same cases with a structured citation feature | Use citations the API extracts from the documents |
| Supporting sentence split across two chunks | Chunk boundaries that cut through the evidence | Inspect chunk edges around each reference span | Sentence-aware or overlapping chunks |
| Correct paraphrase marked uncited | Grader demands a verbatim match the target never promised | Hand-grade a sample of the failures | Fix the grader before judging the model |
Tested example: does your acceptance test catch a bad citation?
Dry-run the acceptance test on planted defects before the request goes out, because a test that passes bad items will accept a bad delivery. The checks below were run on five synthetic accounts-payable items written for this page on October 26, 2026; the run tests the checks, not any model or dataset.
Each item asks when invoice INV-2207, issued 2026-04-02, is due, or whether its early-payment discount applies. The pool draws on a superseded contract amendment with Net 30 terms, the current amendment with Net 45 terms and a 2 percent discount for paying within 10 days, the invoice, a payment record dated 2026-04-16 and a general FAQ. Four automatic checks ran on every item: cited IDs are in the pool, quoted text matches the record at the stated offsets, each cited record is in effect on the question date, and missing-record items give the missing-evidence statement with no citation.
Three of the four planted defects failed automatically. Item E passed every check because its quote is real and correctly located, and a structured citation feature would not have stopped it either: a valid location shows where a quote came from, not whether it supports the sentence. Add a claim-level support review to the acceptance test, by a second reviewer or an entailment model, state how many items it samples, and include a planted item like E so the supplier sees what fails a delivery.
| Item | Planted defect | Automatic checks | What catches it |
|---|---|---|---|
| A | None: quotes Net 45 and answers 2026-05-17 | Passed all four | Nothing to catch |
| B | Correct quote, offsets shifted by three characters | Failed the span match | Automatic check |
| C | Quotes the superseded Net 30 amendment | Failed the in-effect date | Automatic check |
| D | Current terms withheld from the pool; answers Net 30 anyway | Failed the pool, in-effect and missing-evidence checks | Automatic checks |
| E | Quotes the real discount clause and claims the discount applies, though payment went out on day 14 | Passed all four | Only a claim-level support review |
Sources for this section
- Anthropic's agent evaluation guidance (January 9, 2026) calls 20 to 50 simple tasks drawn from real failures a great start, defines a good task as one two domain experts would pass or fail the same way, recommends a reference solution per task, advises grading what an agent produced rather than its path, and says graders can only be trusted after reading transcripts and grades. Source
- OpenAI's GDPval tasks pair a request, often with reference files, with a deliverable from an experienced professional, and model output is graded in blinded comparisons by experts in the same occupation. Source
- OpenAI's September 2025 paper on hallucinations argues that standard training and evaluation procedures reward guessing over acknowledging uncertainty. Source
- Anthropic (April 3, 2025) found Claude 3.7 Sonnet mentioned a planted hint 25 percent of the time and DeepSeek R1 39 percent, models admitted reward hacks less than 2 percent of the time in most scenarios, and outcome-based training plateaued at 28 and 20 percent faithfulness. Source
- Claude's citations documentation says the API extracts cited text directly so citations contain valid pointers to the provided documents, and that in Anthropic's evaluations the feature was significantly more likely to cite the most relevant quotes than prompt-based approaches. Source
Frequently asked questions
Can fine-tuning teach a model to cite sources?
Fine-tuning can teach the format and habit of selecting and quoting evidence before answering, provided the training items make the evidence necessary. It does not make the cited records correct or current, so retrieval quality and record versioning still decide whether citations are trustworthy.
How is this different from building a RAG evaluation set?
An evaluation set measures whether answers are supported; a training pack changes behavior. The pack adds hard distractors, missing-record items and target responses that show selection and quoting, and it must stay separate from the evaluation set it will be judged on.
How many distractors should each item include?
There is no universal number. RAFT exposes the distractor count and the share of items with the relevant document as settings, which is a sign they should be tuned. Start from what your retrieval system actually returns, and test variations on the held-out set.
Should the model abstain or retrieve again when the record is missing?
If your system lets the model call retrieval again, train a target that issues a better query; if not, train a target that states what record is missing. Keep the two behaviors in separate cohorts so you can measure each.
Can model-generated questions be used in the pack?
They can, as RAFT does, but verify every generated question against the record it should require and drop those answerable without it. Keep held-out questions written from real requests where possible, so the evaluation reflects how people actually ask.
Sources
- RAFT generates question, answer and document triplets in which documents are the relevant chunk plus distractors, with settings for the number of distractors and the share of items that include the relevant document. Source
- RAFT's generation prompt asks for copied context sentences between begin and end quote markers and a final answer beginning with an answer tag. Source
- Self-RAG trains a critic and a generator so a model retrieves on demand and critiques its output with reflection tokens. Source
- Self-RAG's reflection tokens include retrieval, relevance and support labels: fully supported, partially supported, and no support or contradictory. Source
- Claude's plain-text citations carry the cited text, a zero-indexed document index, the document title and a character range with an exclusive end. Source
- ALCE is a benchmark for LLM citation evaluation across three datasets, scoring fluency, correctness and citation quality. Source
- ALCE's evaluation scores citation recall by NLI entailment of each sentence by its cited passages and citation precision by flagging unnecessary citations. Source
- The AIS framework treats text as attributable when a generic hearer would affirm the statement according to the identified source. Source
Tell us what data you need
Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.