Skip to content

AI uses for records

Document lineage retrieval experiment: 73 spec questions, five rankers

By SourceX Editorial · Updated

10 sections

Short answer

A document lineage retrieval experiment holds the corpus, questions and ranker fixed and changes only the lineage a system may read: status, supersession links in both directions and dates. In a public pilot on 73 lineage questions over XMPP specifications, text-only ranking returned the governing document for 10 of 40 answerable questions, choosing the newest result found 7, and following supersession links found 29.

Provenance trail

Document lineage retrieval

This page covers document lineage retrieval experiment: 73 spec questions, five rankers.Illustrative

Key takeaways

  • Score trust on two failures at once: silent authority errors, which lineage should cut, and needless flags, which it must not inflate.
  • Dates mislead: in 13 of 69 XEP supersession relations, the older document was revised after its replacement, and in 11 of them the older document is retired.
  • Lineage is often one-sided: only 31 of 69 relations are recorded on both documents, and 16 point to a successor that was never approved.
  • In the pilot, text-only ranking found the governing XEP for 10 of 40 questions, recency-first 7, forward links 29 and full lineage 33.
  • The answer-level arm is designed with family-level splits and stop rules; its results are not published.

What does a document lineage retrieval experiment test?

A document lineage retrieval experiment tests whether giving a retrieval system the history of each document, its approval status, which documents replaced it and which it replaced, reduces answers drawn from versions that no longer govern. Everything else is held fixed: the corpus, the questions, the ranking function and the scoring. Only the lineage the system may read changes from arm to arm.

Trust needs an operational meaning for the experiment to be scored. Here a retrieval result deserves trust when it is the governing version for the question, or when it carries an explicit flag that no governing version is available in the index. A confident answer from a retired document fails, and so does a refusal when the governing document was there to be found. That pair of failures, silent authority errors and needless flags, is what lineage metadata has to reduce together, or at least trade at a rate set in advance.

The hypothesis is directional and testable: with lineage metadata read in both directions, silent authority errors fall against text-only and recency-first retrieval, without needless flags rising by more than a preset margin. The rest of this page builds the test, runs the part that can be run reproducibly on public data, and publishes the design for the part that cannot.

Why dates are not lineage: four traps in a public standards archive

Because a document's last edit often records its retirement, not its relevance. The XMPP Standards Foundation keeps its extension protocols, the XEPs, in a public repository where each specification's header carries a status, supersedes and superseded-by fields and a dated revision history. At commit 4132c53, read on 28 October 2026, the 505 specifications that are not humorous record 69 supersession relations between two XEPs, and those relations show four distinct ways a date can mislead a ranker.

First, retirement creates revisions. In 13 of the 69 relations, the older specification was revised after its successor; in 11 of them the older one is retired, while XEP-0054 and XEP-0115 are still approved. XEP-0016, Privacy Lists, is Deprecated and points to XEP-0191, Blocking Command, as its replacement; its last revision is dated 2017-05-20, while the replacement's last revision is dated 2015-03-12. A ranker that prefers the newest document puts the retired one first.

Second, a newer document can claim to replace one that still governs. In 16 relations, involving 15 distinct successors, the successor is still Experimental, Deferred or Proposed, never approved, and in 8 of those the older specification is itself approved. XEP-0390, Entity Capabilities 2.0, is Deferred and lists XEP-0115 in its supersedes field, yet XEP-0115 carries no superseded-by entry and still holds the status Draft, which brings in the third trap. XEP-0001, the process document, says the status once called Draft was renamed Stable because the old name led to widespread confusion about stability; 80 of the 505 files still use the old label, so a model reading the word literally would treat approved standards as unfinished.

Fourth, lineage is often recorded on one side only. Of the 69 relations, 31 appear on both documents, 24 only on the older one and 14 only on the newer one. The text does not fill the gap reliably: among the 65 question documents that are not approved, 18 contain successor wording such as superseded by or in favor of in their abstract or body, yet on reading XEP-0071 names no successor, and text can disagree with metadata. XEP-0073's note says it was obsoleted by XEP-0211 and XEP-0212, while its header names XEP-0242 and XEP-0243.

The knowledge authority map for versioned documents

Each lineage trap has a signal that settles it and a behavior that keeps the answer trustworthy. The map below is the labeling scheme for the experiment and, later, the acceptance test for licensed archives; every row comes from a case in the XEP corpus, and every row has an enterprise twin, from a retired expense policy revised to add a deprecation banner to a draft standard operating procedure that announces it replaces the current one:

The knowledge authority map for versioned documents
Lineage trapSignal that settles itWhere the signal livesTrustworthy behavior
Retired document revised after its successorStatus at the question date, not the last edit dateStatus field and its change historyRank by status first; dates only break ties
Successor never approvedThe successor's own approval statusSuccessor's status fieldKeep the older approved document as governing
Relation recorded on one side onlyA link read from either documentSupersedes and superseded-by fields on bothResolve lineage in both directions before answering
Status label renamedThe process document's vocabulary and its change dateProcess or policy registerMap old labels to current meaning before ranking
Governing document outside the indexA successor reference that resolves to no indexed fileSuperseded-by field naming an external standardFlag the gap and name the external document
Text and metadata disagreeThe approval record, not the prose noteHeader fields and approval minutesPrefer the governed field; report the conflict

Design sheet: what is fixed before the first query runs

The design fixes seven decisions before any query runs, so the result cannot be tuned toward a preferred arm. The public pilot below applies the same sheet with the choices noted in the right-hand column; a run on licensed company archives would keep the structure and replace the corpus. The sheet:

Design sheet: what is fixed before the first query runs
DecisionWhat is fixedPilot setting
HypothesisLineage read in both directions cuts silent authority errors without raising needless flags beyond a set marginDirectional; no margin tested in the pilot
Unit and held-out setLineage families, not single questions, so related documents never sit on both sides of a split44 families; no parameter was tuned, so all 73 questions are reported
QuestionsOne per older document in each relation, worded by two fixed rulesOlder title with the word legacy removed; successor title as a check
ArmsSame ranker and top five, differing only in the lineage each may readText only, newest of top five, approved status only, forward links, full lineage
Gold labelGoverning version from status and links, or an expected flag40 questions with a governing XEP, 33 without one
MetricsCorrect document or correct flag; silent authority error; off-family result; needless flagCounted per arm and per gold type
Stop rulesCoverage, harm, futility and integrity rules set in advanceWritten for the answer-level run below

Pilot results: five rankers on 73 lineage questions

Lineage changed the outcome far more than recency did. On the 40 questions whose governing document is in the corpus, a BM25 ranker reading only text put the governing XEP first 10 times. Choosing the newest of its top five results found it 7 times. Following the retrieved document's superseded-by links found it 29 times, and resolving lineage in both directions with status found it 33 times.

The 33 questions with no governing document in the corpus separate arms that can say so from arms that cannot. Text-only and recency-first ranking never flag, so they fail all 33, returning a document in the question's own lineage that does not govern 27 and 16 times respectively. A status filter that abstains when no approved document is in the top five flagged correctly 26 times. Forward links alone flagged only when a successor pointed outside the corpus, 9 times, and ended on a document in the question's lineage that does not govern in 20 cases.

Recency also widened the search in the wrong direction: picking the newest of the top five returned a document outside the question's lineage, neither an ancestor nor a descendant, for 42 of the 73 questions, against 12 for text-only ranking. Wording mattered less than lineage. With the successor's title as the query, text-only ranking found the governing document for 10 of the same 40 questions and recency-first for 13, while full lineage raised needless flags from 5 to 14, a cost worth watching in any real run. The counts by arm, for older-title questions:

Pilot results: five rankers on 73 lineage questions
ArmGoverning XEP first (of 40)Correct flag (of 33)Silent authority errors (of 73)Needless flags
Text only100510
Newest of top five70240
Approved status only, abstain if none202616
Follow superseded-by links299250
Full lineage, both directions333305
A matrix of five retrieval arms against outcomes on 73 XEP lineage questions: governing document found for 10, 7, 20, 29 and 33 of 40 answerable questions; correct flags for 0, 0, 26, 9 and 33 of 33; silent authority errors 51, 24, 1, 25 and 0.
Public pilot on XMPP Extension Protocols at commit 4132c53. Full lineage is an upper bound by construction; no language model was run.

What the pilot cannot show, and the strongest objection

The pilot sizes the problem; it does not prove that lineage makes answers more trustworthy. The gold labels come from the same status fields and links the lineage arms read, so full lineage is an upper bound by construction, and its perfect score on unanswerable questions says only that it applied its own rules. What is not circular is the failure of the arms that ignore lineage: the text-only and recency-first errors are measured against the foundation's own records, which neither arm consulted.

The strongest objection is that relevance, not lineage, is the real problem: if a ranker cannot find the right family, metadata cannot rescue it. The off-family counts support part of that view, since recency-first ranking lost 42 questions to documents outside the lineage. But text-only ranking stayed within the lineage far more often, 61 of 73 times, and still returned a document that does not govern in 51 of them. Relevant text can be outdated, unauthorized or insufficient, and on this corpus that, not topic drift, is the larger failure.

Scale and setting limit the result. Seventy-three questions from one standards body cannot estimate an effect for company archives, where policies change with less ceremony and lineage fields are rarer. The ranker is lexical BM25 with no embeddings, the questions are titles rather than natural requests, and no language model or person saw any result. Those gaps are what the answer-level arm is for.

The answer-level arm: designed, results not yet published

The arm that matters to buyers, a language model answering from retrieved documents with and without lineage fields shown, has not been run, and no result is published here. Its design reuses the pilot's families, splits them at random into a development half and a held-out half, freezes prompts and model versions on the development half, and reports only the held-out half.

At a glance

The answer-level arm: designed, results not yet published

It adds three measures to the retrieval metrics: whether the answer cites the governing version, whether it abstains or flags when none is available, and whether it states a retired rule without a caveat. A reliance study can then show participants answers from both arms and record how often they accept a wrong one; trust that is calibrated means accepting correct answers and questioning flagged ones. Paired results per question go to an exact McNemar test on the discordant pairs. The rules that stop or void the run:

  • Coverage gate: if fewer than half of the supersession relations in the development families are recorded on both documents, stop and repair lineage first. Across the whole XEP corpus only 31 of 69 relations are recorded on both documents, under half, but no development split was drawn for the pilot, so the gate has not been applied to one.
  • Harm rule: stop if the lineage arm's needless flags exceed 10 percent of answerable development questions, since an assistant that refuses too often loses users who then stop checking.
  • Futility rule: stop if, on the development half, the lineage arm removes fewer than five silent authority errors net of any it adds.
  • Integrity rule: any change to the prompt, the model or the scoring after held-out results are seen voids the run, which must restart on a fresh split.

An evidence-grounding test for lineage-aware answers

An answer drawn from a versioned archive passes only when each check below holds. The checks follow the authority map, and a buyer can apply them to a sample of answers before accepting a dataset or a system:

  • The cited document's status on the question date is approved or in force, read from the status field and its history, not inferred from the last edit.
  • Every successor the document lists, and every document that lists it, has been checked; a claimed successor without approval does not retire an approved document.
  • Retired status labels are mapped to the current vocabulary before the answer relies on them.
  • When the governing document is outside the index, the answer says so and names it rather than answering from the retired one.
  • When header fields and prose notes disagree, the answer follows the governed field and reports the conflict.
  • When no version governs, the answer flags the gap; a confident answer from a retired document is scored as a silent authority error.

What to request when licensing versioned document archives

Ask for the history that makes these checks possible, not just the current files. A useful archive includes every version of each document with its effective and retirement dates, status changes as dated events, supersession recorded on both the old and the new document, the approval record for each status change, and the status vocabulary with the date of any renaming. Deletion or archiving notices matter as much as new versions.

A supplier can measure its own lineage coverage before sharing anything: the share of relations recorded on both sides, the share of retired documents revised after their successor, and the number of successors that never reached approval. Those three figures predict how far a buyer's lineage arm can get.

SourceX licenses business data directly from companies and sources each dataset on request rather than from stock. A buyer who needs versioned policies, procedures or knowledge articles can describe the archive, the lineage fields above and this grounding test as the acceptance test in a data request, and SourceX follows up about potential sources.

Methods note: XEP snapshot, ranking rules and counting rules

Data: every xep-NNNN.xml file in github.com/xsf/xeps at commit 4132c531487fe9450ae90b905b8df950d0d478b2 (26 October 2026), cloned on 28 October 2026. Three scripts, xep_parse.py, xep_lineage_pilot.py and xep_lineage_pilot_v2.py, which reruns the pilot unchanged and adds the two sensitivity runs below, use the Python standard library, run no repository code and have no randomness. Header fields are read with regular expressions from the header element; body text is everything after it, with example blocks and tags removed and the repository's entity definitions expanded one level; only footnotes inside entity references were dropped, and other note text in the body was kept. 518 XEPs were parsed and the 13 of type Humorous excluded, leaving 505.

Rules as applied: a relation is a pair recorded in an older document's superseded-by field or a newer document's supersedes field; a reference counts as a XEP when it starts with XEP- and four digits, with trailing text such as in part kept as a qualifier; other references are external. Approved means status Final, Draft or Active. The governing set for a question is the approved successors, followed transitively, that have no approved successor themselves; if none, the question document when approved; if not, approved predecessors with no approved successor; otherwise none, labeled governed outside the corpus when any successor is external. A family, as counted in the 44, is a connected component of in-corpus relations, with a document whose successors are all external counted on its own. Results are scored against the question document's lineage, not that family: off-family means neither ancestor nor descendant of the question document, and a result that is the question document, an ancestor or a descendant but does not govern is a silent authority error. Revision dates are the latest parseable date in the header. Tokens are lowercase letter and digit runs of two or more characters, minus 26 stop words (a, an, and, are, as, at, be, by, for, from, in, into, is, it, of, on, or, that, the, this, to, with, via, using, xmpp, xep); BM25 is a Python implementation written for the pilot, not Lucene itself: it uses Lucene-style idf, ln(1 + (N - n + 0.5) / (n + 0.5)), with k1 = 1.2 and b = 0.75, Lucene's defaults, over title, abstract and body, with ties broken by XEP number. Its index holds 503 XEPs, because XEP-0491, XEP-0492 and XEP-0503 give the same unexpanded entity reference as their header number and only the last is kept; keyed by file number, with all 505, every published count is the same. Results depend on these settings: with the classic Robertson-Spärck Jones idf, ln((N - n + 0.5) / (n + 0.5)), the five arms find the governing XEP for 8, 7, 17, 23 and 26 of the 40 questions, against 10, 7, 20, 29 and 33, and keep the same order. The recency arm takes the newest of the top five, ties by rank; the status arm takes the first approved document in the top five or flags; forward links follow superseded-by to the highest-scoring in-corpus successor until none remains, flagging when only external successors remain; full lineage returns the best-scoring member of the retrieved document's governing set or flags. Percentages in the outputs are exact fractions rounded once, half up.

Not measured: dense or hybrid retrieval, natural-language questions, any language model, any human judgment, and any company archive. After a first run, two rules were corrected before the results above were recorded: the status arm was given abstention, and the governing rule was given the approved-predecessor step; the pilot is therefore not preregistered. The authority map, design sheet, stop rules and grounding test were drawn up for this page on 28 October 2026.

Frequently asked questions

What is document lineage in retrieval?

The recorded history of a document: its versions, its approval status over time, the documents it replaced and the documents that replaced it. A lineage-aware retriever reads those fields to decide which version governs a question on a given date, instead of trusting text similarity or the latest edit.

Does a recency boost fix outdated documents in RAG?

Not reliably. In the XEP pilot, choosing the newest of the top five results found the governing specification for 7 of 40 questions, fewer than text-only ranking, because retirement itself creates new revisions and because recency pulls in unrelated recent documents.

What is a silent authority error?

An answer or top result drawn from a document that no longer governs, given without any flag. It is the failure lineage metadata is meant to prevent; in the pilot, text-only ranking produced 51 of them across 73 questions.

Can lineage be inferred from document text alone?

Only sometimes. Of 65 retired or unapproved question documents in the XEP corpus, 18 contain successor wording in their abstract or body, though on reading XEP-0071 names no successor, and at least one note disagrees with its own header. Structured status and supersession fields are the dependable source.

How large should a document lineage test set be?

Large enough that families, not questions, can be split into development and held-out halves with dozens of families in each. The 44-family public pilot is enough to size the problem, not to measure a small effect on answers.

Sources

  • The XEPs repository at commit 4132c53 (26 October 2026): 505 non-humorous specifications, their status, supersession fields and revision dates, and the lineage retrieval pilot computed from them for this article. Source
  • XEP-0001 defines the XEP statuses, says the status now called Stable was previously called Draft and was renamed because of widespread confusion about stability, and describes superseded XEPs moving to Deprecated and then Obsolete. Source
  • The XEP document type definition gives each header a status, supersedes and superseded-by elements and one or more revisions. Source
  • XEP-0016, Privacy Lists, has status Deprecated, names XEP-0191 as its successor and was last revised on 2017-05-20. Source
  • XEP-0191, Blocking Command, lists XEP-0016 in its supersedes field and was last revised on 2015-03-12. Source
  • XEP-0390, Entity Capabilities 2.0, has status Deferred and lists XEP-0115 in its supersedes field. Source
  • XEP-0073's note says it was obsoleted by XEP-0211 and XEP-0212, while its superseded-by field names XEP-0242 and XEP-0243. Source
  • Lucene's BM25Similarity documentation gives default parameters k1 = 1.2 and b = 0.75. Source
  • SourceX's data request page says it licenses business data directly from companies, follows up about potential sources, and sources datasets on request rather than holding them in stock. Source

Tell us what data you need

Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.

See if your company qualifies