Skip to content

AI uses for records

RAG source citation location datasets: scoring the line or region cited

By SourceX Editorial · Updated

10 sections

Short answer

To evaluate whether a citation points to the supporting line or image region, store a gold anchor for every claim in a pinned document version: a quoted span with its prefix and suffix, a page and box, or a table cell. Then score the cited location on four tests: it resolves, it overlaps a gold anchor, it is no wider than needed, and the cited content alone supports the claim.

Retrieval

RAG source citation location

This page covers RAG source citation location datasets: scoring the line or region cited.Illustrative

Key takeaways

  • A citation can name the right document and still fail the reader; score where it points, not only what it names.
  • Pick the anchor by evidence form: quotes with context for prose, offsets for stable text, page fractions for scanned regions, cell keys for tables.
  • Store every region with its coordinate frame; most box mismatches are frame errors, not model errors.
  • Report resolves, hits, tight, sufficient, necessary and current as separate scores.
  • Aggregates need joint anchors, and absence claims belong in abstention tests, not location metrics.

What does a location-level check catch that a document-level check misses?

It catches citations that name the right document but send the reader to the wrong place in it. A document-level score marks a citation correct when support exists somewhere in the cited file, which hides the failure a user actually meets: they click, land on page one of a forty-page policy, and cannot find the sentence the answer relied on.

Most published citation metrics work at the passage level. ALCE, the Princeton benchmark for citation quality, uses an NLI model to ask whether the passages a sentence cites, joined together, entail that sentence (citation recall), and whether each cited passage is needed (citation precision). The AIS framework from Google researchers frames the underlying test as whether a reader would accept the statement 'According to P, s' for a source P. Both assume P is already the right excerpt. A location dataset tests that assumption.

The failures a location check surfaces are specific and common in business documents: the right file but a superseded version, the right page but the wrong table row, a highlight covering the whole page, a quote stitched from two paragraphs, and a box drawn on a page image rendered at a different size than the one being scored.

Which anchor type fits each kind of evidence?

Choose the anchor by the form the evidence takes in the delivered file, not by what is easiest to annotate. The W3C Web Annotation Data Model already defines most of the selectors you need, which saves inventing a format and lets annotation tools read the gold set directly.

Two rules from that model are worth adopting as written. Text selectors are recorded after normalization, so the normalization routine is part of the dataset. And when one target carries several selectors, they should select the same content, with consumers picking one; a refinement chain (page, then a box on that page) is how you express nesting.

Which anchor type fits each kind of evidence?
Evidence formAnchor to storeDefined byBreaks when
Born-digital prose (policies, manuals, knowledge base articles)Exact quote plus a short prefix and suffixW3C TextQuoteSelectorThe quoted text is edited, or the same wording repeats without distinguishing context
Extracted plain text with stable normalizationStart and end character positions, end excludedW3C TextPositionSelectorExtraction or normalization changes and offsets shift silently
Live web pagesText directive with start text, optional end text and contextWICG text fragments draftThe page is rewritten; store the quote as well so drift is detectable
Scanned pages, slides and page imagesPage index plus a rectangle as x, y, width, heightMedia Fragments xywh via FragmentSelector; IIIF Image API regionThe box is scored in a different pixel frame than it was drawn in
Charts, diagrams and irregular regionsPolygon or other SVG shape on the page imageW3C SvgSelectorAnnotators trace outlines inconsistently; set a tolerance before labeling
Tables in PDFs or spreadsheetsRow key, column header and the cell value as a quoteProject convention layered on a quote or region anchorMerged cells, wrapped headers or re-sorted rows
Several places needed togetherA set of anchors, all requiredProject convention; scored jointlyAnnotators disagree on which pieces are strictly necessary

The location anchored citation evaluation specification

Each record ties one atomic claim to one or more gold anchors in a frozen corpus, with enough metadata to resolve the anchor mechanically and to explain disagreements later. The fields below are the minimum; the example column is illustrative.

At a glance

The location anchored citation evaluation specification

Keep a manifest of document versions and content hashes alongside the records, and store every coordinate with the frame it was measured in. Nothing in the record should depend on a viewer's zoom level or on which PDF library rendered the page.

Annotate in two passes. In the first, the annotator reads the claim, finds every place in the pinned version that supports it, and records the smallest sufficient unit for each. In the second, a different annotator sees only the anchors, not the claim author's reasoning, and confirms that each one resolves and supports the claim on its own. Disagreements on minimal_extent are the most useful output of this pass: they show where the tightness rule needs a written example before scoring starts.

The location anchored citation evaluation specification
FieldWhat it holdsValidation ruleExample (illustrative)
claim_id and claim_textOne atomic statement the answer must makeOne fact per claim; no conjunctions joining two factsC-0412: Coverage lasts 24 months from installation
doc_refDocument id, version label and content hashHash must match the corpus manifestwarranty-policy v4, hash recorded in manifest
anchor_typequote, position, region, cell or setMust match the evidence form tablecell
text_anchorExact quote, prefix, suffix and normalization idQuote must occur at the stated offsets after normalization24 months, prefix Residential units:, suffix from installation
region_anchorPage index, shape and the coordinate frameFrame names units, origin and page size usedpage 4, x 0.12 y 0.58 w 0.40 h 0.04 as page fractions
gold_alternatesOther anchors that support the claim equally wellEach alternate passes the same support testSummary table on page 1 repeats the term
minimal_extentSmallest unit that supports the claimAnnotator states sentence, row, cell or regionSingle table row
max_extentWidest citation still counted as preciseSet per anchor type before labelingUp to the enclosing table, not the page
support_verdictWhether the gold anchor alone supports the claimfull, partial or none, with a reasonfull
provenanceAnnotator ids, second-pass result and adjudication noteTwo independent passes on a fixed share of recordsA1 and A2 agree; no adjudication needed

How should a cited location be scored against the gold anchor?

Score each citation on separate tests and report them separately; a single blended number hides whether the system cites loosely, cites the wrong version, or cites nothing usable. Decide thresholds before scoring and publish them with the results.

Break every score down by anchor type as well. A system that is precise on born-digital prose and loose on scanned pages has a parsing or rendering problem that an overall average will hide, and the fix belongs to a different team than a prompt change would.

  • Resolves: the cited anchor can be found in the pinned version. A quote that no longer matches or a page index past the end of the document fails here.
  • Hits: the cited location overlaps at least one gold anchor. For text, measure character overlap; for regions, measure intersection over union in a shared frame. The cut-off is a team choice, not a standard.
  • Tight: the cited extent is no wider than max_extent. A whole-page highlight that contains the right row hits but should not pass.
  • Sufficient: the cited content alone supports the claim when a reviewer or an entailment model reads only that content, the AIS 'according to' test applied to the location rather than the document.
  • Necessary: when several locations are cited, each one is needed. ALCE's precision check flags a citation as irrelevant when it does not support the sentence on its own and the remaining citations still entail the sentence without it; the same idea flags padding.
  • Current: the cited version is the one the answer should rely on for that date. A perfect anchor into a withdrawn policy is a failure.

Why region anchors drift between annotation and scoring

Most region-scoring errors come from coordinate frames, not from the model. A box drawn on a page rendered at one resolution, then compared with a box from a parser that reports PDF points, will miss even when both people pointed at the same line.

DocLayNet shows how easily this happens in practice: its README says the layout boxes, stored in COCO format, refer to the page PNGs, which were resized to 1025 by 1025 pixels, and not to the original PDFs. The IIIF Image API offers a way out with its pct: region form, which expresses x, y, width and height as percentages of the image's reported size. Storing regions as page fractions, with the page size recorded once, removes most of the drift.

Rotation, cropping during scanning and multi-column reflow cause the remaining cases. Record the page size and rotation from the same file the model reads, and re-run anchor resolution whenever the extraction pipeline changes.

Worked example (illustrative): three citations to one warranty policy

An assistant answers a homeowner's question about warranty length and cites three places. The policy, version labels, page numbers and claim are invented for illustration; the scoring logic is the point.

Worked example (illustrative): three citations to one warranty policy
Cited locationResolves and hitsTight and sufficientVerdict
Page 4 of v4, whole page highlightedYes; the page contains the gold rowNot tight: the page exceeds the table max_extentLocated but loose; counts as a precision failure
Row 3 of the coverage table on page 4 of v3Resolves in v3; hits the analogous rowTight and sufficient for v3, but v4 is currentVersion failure, even though the location is exact
Box on a scanned addendum, drawn at 150 dpi, scored at 300 dpiFails before normalization; overlaps after converting both to page fractionsTight once normalized; the addendum alone supports the claimPass after frame repair; the original miss was a data error
No citation for a second claim about transfer rulesNot applicableClaim has a gold anchor on page 6Missing citation; scored as a recall failure

Is the cited passage current, controlling and visible to the person asking?

Only when location, version, effective date and access all check out; a precise anchor answers just the first of these. Add three labels to each gold anchor so the dataset can test the others: the date range the source governs, whether it is the controlling source or a summary of one, and which roles may see it.

The W3C model's TimeState, with a sourceDate stating when the target should be interpreted, is a ready-made place to pin the version a citation refers to. For access, store the permission group on the document record and score a citation into a document the asking user cannot open as a failure even if the text is right. Report these failures in their own column, because the fix is in retrieval and permissions, not in citation formatting.

When no single line can support the claim

Some claims are true only across locations, and a few have no location at all. Treat them as their own record types rather than forcing a single anchor.

Aggregates and comparisons (a total across invoice lines, a difference between two plan tiers) use a set anchor in which every member is required; score the set jointly, the way ALCE joins cited passages before testing entailment. Absence claims, such as a policy not mentioning flood damage, cannot be anchored: test whether the system scopes or abstains, and keep them out of location metrics. Duplicated wording is the third case. When a quote matches in several places, the W3C model says the selection should be treated as matching all of them, so list each occurrence as an acceptable alternate or add context to the prefix and suffix.

Signals that location scores are not yet trustworthy

Stop and fix the data, or narrow the evaluation, when the score moves for reasons unrelated to the model. Three signals matter most: anchor resolution falls after a parser upgrade, two annotators disagree on minimal_extent for more than a small share of records, or an automatic sufficiency judge disagrees with human review on a sample.

Bring in a domain reviewer when the claims are legal, financial or safety statements, where 'supports' depends on reading a clause in context. Bring in the source owner when versions conflict and nobody can say which one governed on the answer date. Scanned handwriting, stamps and heavily redacted pages usually need region anchors drawn by hand and checked twice; if that review is not affordable, exclude those pages and say so in the dataset card.

Sourcing documents with stable anchors for a citation location set

Real business documents make the hard cases appear on their own: superseded versions, repeated boilerplate, tables split across pages and scanned attachments. Ask for originals with their version history and page images together with any text layer, so anchors can be drawn and re-resolved against the same files.

Teams after a citation-location set built on such files can request one from SourceX: it tracks down the records at U.S. businesses only when asked, holds no stock, and takes them through rights review, contracting, preparation and delivery. The request-data form is where scope, volume, language, format, timing and licensing needs go.

Frequently asked questions

What is a RAG source citation location dataset?

It is an evaluation set in which each claim is paired with the exact place in a pinned document that supports it: a quoted span, a character range, a page region or a table cell. It lets you score whether a system's citation sends the reader to supporting content, not only to the right file.

How is this different from a question-answer-citation triple set?

Triple sets record the evidence behind each answer so retrieval and answer correctness can be scored. A location set adds the rules for resolving and comparing anchors, tightness limits, coordinate frames and version checks, so the citation itself can be graded as a pointer.

Should text anchors use quotes or character offsets?

Store both. The W3C Web Annotation model defines a quote selector with prefix and suffix and a position selector with start and end offsets; quotes survive small edits, offsets are exact, and a mismatch between them tells you the text changed.

How do you score a bounding box citation?

Convert the cited box and the gold box to the same frame, ideally fractions of the page, then measure intersection over union against a threshold you fixed in advance. Also check that the box is not much larger than the gold region, or whole-page boxes will pass.

Can an LLM judge decide whether a cited location supports a claim?

It can triage, but validate it against human labels on a sample first. ALCE uses an NLI model for this test, and any automatic judge should be reported alongside its measured agreement with reviewers.

Sources

  • The W3C Web Annotation Data Model defines text quote, text position, SVG and fragment selectors, selector refinement, TimeState sourceDate, and normalization and multiple-match rules for text selectors. Source
  • The WICG text fragments draft defines the :~:text= directive with optional prefix, start text, end text and suffix for linking to a passage in a web page. Source
  • The IIIF Image API region parameter selects part of an image by x, y, width and height in pixels or as percentages of the reported image size. Source
  • ALCE evaluates citation quality with citation recall (cited passages jointly entail the sentence) and citation precision (each cited passage is needed), using an NLI model. Source
  • The AIS framework (Attributable to Identified Sources) tests whether a statement can be affirmed as 'According to P, s' for a provided source P. Source
  • DocLayNet provides human-annotated layout bounding boxes for 80,863 pages in COCO format; the boxes refer to PNG page images resized to 1025 x 1025 pixels, not the original PDFs. Source

Tell us what data you need

Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.

See if your company qualifies