Skip to content

AI uses for records

Retrieval chunk boundary evaluation: a protocol for rules cut off from their conditions

By SourceX Editorial · Updated

10 sections

Short answer

To test retrieval when a chunk boundary separates a rule from its condition, annotate each rule and its governing condition as character spans, stratify pairs by distance, and measure how often the top k results contain both spans under each chunking setting, then check whether answers state the condition. This is a protocol; no results are reported.

Evaluation split

Retrieval chunk boundary

This page covers retrieval chunk boundary evaluation: a protocol for rules cut off from their conditions.Illustrative

Key takeaways

  • The failure to measure is a retrieved rule whose limiting condition stayed in another chunk, which similarity scores do not reveal.
  • Annotate rule and condition as character spans so scores do not depend on how any chunker split the text.
  • Stratify pairs by the distance between rule and condition, and keep same-sentence pairs as a floor that should not move.
  • Compare every configuration with one pre-declared baseline using paired, document-clustered differences.
  • Score answer utility separately: does the answer apply the condition, abstain when it cannot, and cite the limit?
  • This page publishes the method only; it reports no measured results.

How do you test retrieval when a chunk boundary separates a rule from its condition?

Treat each rule and the condition that limits it as two annotated spans, then measure how often a retriever returns both spans together in the top results under each chunking setting you are considering. Pair that with a task check: does the answer state the condition, or does it give the bare rule as if it applied to everyone?

What follows is a measurement design for this failure rather than a report of one. It sets out the question, the test population, the chunking conditions to compare, the metrics, how to handle uncertainty and a pre-registration checklist. No results are reported here, and nothing below should be read as a measured effect of any chunker.

The reason to measure this specific failure is that similarity-based retrieval can look healthy while it fails. A chunk that says refunds are issued within the stated window will match a refund question well. If the sentence that says the window does not apply to reseller orders sits in the next chunk, the retriever has returned relevant text and the system will still give a wrong answer to a reseller customer.

Where splitters put boundaries, and why conditions fall on the wrong side

Common splitters decide boundaries from characters and length, not from meaning. LangChain's RecursiveCharacterTextSplitter tries a list of separators in order, by default paragraph breaks, then line breaks, then spaces, then individual characters, and the base TextSplitter defaults to a chunk size of 4,000 characters with 200 characters of overlap. Chroma's evaluation package ships a recursive token chunker whose default separators also include periods, question marks and exclamation points.

At a glance

Where splitters put boundaries, and why conditions fall on the wrong side

Structure-aware chunkers move the boundary to document structure instead. Unstructured's chunk-by-title strategy starts a new chunk when a new section title appears, even when the next element would still fit, and splits oversized elements at spaces or new lines under a hard maximum size. That protects section edges but does nothing for a condition stated in a different section.

Anthropic's write-up on contextual retrieval names the underlying problem directly: chunk size, chunk boundary and chunk overlap are design choices that can affect retrieval performance, and a chunk read on its own can lose the context that makes it usable. Its remedy is to prepend chunk-specific explanatory context to each chunk before embedding and indexing, which is one of the conditions this protocol can compare.

  • Exception clause in the next paragraph: the rule ends a paragraph and the paragraph that begins Unless or Except carries the limit.
  • Cross-reference: the rule says subject to section 4.2, and section 4.2 sits several chunks away.
  • Lead-in list: a sentence such as The following orders are excluded is followed by a list that a length-based split cuts in half.
  • Table and footnote: a value in a table cell is qualified by a footnote below the table or on the next page.
  • Definition elsewhere: the rule applies to eligible accounts, and eligible is defined in a glossary at the end of the document.

Research question, hypotheses and the comparison you commit to

Write the primary question so that it can only be answered by the measurement you plan: across a fixed corpus, fixed embedding model and fixed k, does the proportion of rule and condition pairs fully retrieved in the top k differ between chunking configurations? State the secondary question separately: does the share of answers that correctly state the governing condition differ between the same configurations?

Commit in advance to one baseline. A sensible baseline is the configuration you run in production today, because the decision you are making is whether to change it. Every other configuration is compared with that baseline on the same questions, never with each other in a post hoc ranking.

State hypotheses as directions, not numbers. For example, a structure-aware configuration retrieves more complete pairs than the baseline in the next-paragraph and lead-in list strata, and no configuration changes completeness in the same-sentence stratum, which serves as a check that the harness is not producing differences where none should exist.

The chunking sensitivity and task utility experiment, stratum by stratum

The test population is rule and condition pairs drawn from documents your system will actually serve: policies, contracts, product terms, procedures and knowledge base articles. Sample documents first, then annotate every conditional rule inside a sampled document, so that the denominator is all pairs in the sample rather than the pairs an annotator found interesting.

Stratify by how far the condition sits from the rule, because distance is what chunking changes. The table below is the asset: each stratum, how to recognize it, what counts as a complete retrieval and a control that guards against a misleading score. Counts per stratum are set during the pilot; none are given here.

The chunking sensitivity and task utility experiment, stratum by stratum
StratumHow the pair is identifiedComplete retrieval meansControl or hard negative
Same sentenceRule and condition share one sentenceOne retrieved chunk contains the whole sentence spanActs as a floor; differences here point to a harness problem
Adjacent sentenceCondition in the sentence after the ruleBoth spans appear in the top k, in one chunk or twoA paraphrased question that avoids the rule's wording
Next paragraphCondition opens the following paragraph, often with Unless or ExceptBoth spans appear in the top kThe same rule from an older version of the document
Lead-in listA sentence introduces a list of exclusions or qualifying casesLead-in sentence and the relevant list item both retrievedA list item from a different list with similar wording
Table and footnoteA cell value is qualified by a marker and a noteCell row and footnote text both retrievedA table with the same column headers and no footnote
Cross-referenceRule names another section that limits itRule and the referenced section's governing sentence retrievedA section with the same number in a different document
Remote definitionA defined term in the rule is defined elsewhereRule and the definition both retrievedThe same term defined differently in another policy
No conditionRule with no qualifying text anywhereRule retrieved; no condition expectedChecks that answers do not invent a condition

Which chunking settings belong in the grid, and which stay fixed

Vary one family of settings at a time and hold everything else fixed: the corpus snapshot, the parser output, the embedding model, the retriever, k and the answering model's prompt. If the embedding model changes between runs, a difference in completeness cannot be attributed to chunking.

A practical grid has a small number of levels per factor. Crossing every size with every overlap and every strategy multiplies the runs and the comparisons, which widens the correction for multiple testing; decide in the pre-registration which cells matter for the decision and run those.

Which chunking settings belong in the grid, and which stay fixed
FactorLevels to considerWhat it isolatesHold fixed
Chunk lengthProduction length, one shorter, one longerWhether longer chunks keep pairs together at the cost of precisionSeparator list and overlap
OverlapNone, production overlap, a larger overlapWhether overlap rescues conditions just past the boundaryChunk length
Boundary ruleLength-based recursive split, sentence-aware split, title or header-aware splitWhether boundaries that follow structure separate fewer pairsTarget length
Context added to chunksNone, section path prefix, generated chunk contextWhether added context lets a split condition be found anywayBoundary rule and length
Retrieval unitReturn the chunk, or return the chunk plus its neighbors or parent sectionWhether expansion at query time fixes splits made at index timeIndex-time chunking

Scoring completeness separately from answer quality

Score retrieval and the answer as two layers, because a correct answer can come from the model's prior and a complete retrieval can still be ignored. The retrieval layer is computed from character offsets; the answer layer needs a reviewer or a validated grader.

Chroma's chunking evaluation framework is a useful model for the retrieval layer: it stores each reference excerpt as a start and end character range and computes recall, precision and intersection over union from the overlap between retrieved chunk ranges and reference ranges, so the score does not depend on how text was split. Ragas defines context recall as the share of claims in a reference that the retrieved context supports, and context precision as a rank-weighted mean of precision at each position; both are useful secondary measures but neither knows that two specific spans must arrive together.

Scoring completeness separately from answer quality
MetricDefinitionDenominatorWhat it catches
Pair completeness at kBoth the rule span and the condition span are covered by the top k chunksAll annotated pairs in the stratumThe split itself; the primary outcome
Condition-only missRule span retrieved, condition span not retrievedPairs whose rule span was retrievedThe dangerous case, where the answer looks grounded
Span recall, precision and IoUCharacter overlap between retrieved ranges and reference rangesReference characters, retrieved characters, or their unionWhether gains come from returning much more text
Condition statedThe answer applies or names the governing condition correctlyQuestions whose correct answer depends on the conditionTask utility, the outcome the business cares about
Correct abstentionThe system declines or asks when the condition cannot be checkedQuestions where the user's facts do not settle the conditionConfident answers given without the facts needed
Citation supportEvery cited passage supports the sentence that cites itCited sentences in answersAnswers that cite the rule but not the limit

How many pairs, and how to report the uncertainty

Analyze the comparison as paired, because every configuration answers the same questions. Compute the per-question difference between a configuration and the baseline, then report the mean difference with a confidence interval rather than two separate scores side by side. Evan Miller's paper on adding error bars to evaluations recommends exactly this kind of question-level paired analysis, and clustered standard errors when questions come in related groups.

Pairs from the same document are related, since one document's formatting habits affect all of its pairs, so cluster by document. Set the sample size from a pilot: run the baseline and one alternative on a small annotated batch, estimate the variance of the paired difference, and choose the number of pairs needed to detect the smallest difference that would change your decision. Write that smallest difference down before the main run.

Report every stratum with its count, including strata where nothing changed. A pooled completeness score can improve while the next-paragraph stratum, the one that motivated the study, gets worse.

Pre-registration checklist before the first run

At a glance

Pre-registration checklist before the first run

Checklist · 10 items

  1. Question and decision: the primary and secondary questions, and the production change each possible result would trigger.
  2. Population: document types, sources, date range and the corpus snapshot identifier.
  3. Sampling: how documents were drawn, and the rule that every conditional rule in a sampled document is annotated.
  4. Annotation guide: what counts as a rule, a condition and a governing definition; span boundaries in characters; a double-annotated subset with agreement reported.
  5. Strata and controls: the stratum list, the hard negatives and the no-condition controls.
  6. Configurations: the baseline, each alternative and every setting held fixed, with library names and versions.
  7. Metrics: the primary outcome, secondary outcomes and how answers are graded, with the grader validated against human labels on a held-out set.
  8. Analysis: paired differences, clustering unit, interval method, the smallest difference that matters and the correction for multiple comparisons.
  9. Exclusions: what may be dropped after the run (for example a parser failure) and how each exclusion is logged.
  10. Reporting: the result tables to publish whatever the outcome, including null results.

0 of 10 ticked. Tick each item as you go.

What a clean result would and would not prove

A configuration that wins this protocol has shown that it keeps rules and their conditions together more often on your corpus, with your embedding model and your k. It has not shown that it is better at every retrieval task, and a change in parser, document mix or model can undo the gain, so the evaluation should run again when any of those change.

Some splits do not matter. If the condition is repeated in the rule's own chunk, carried in chunk metadata such as a section path, or enforced outside retrieval by a permission check, separating the original sentences causes no wrong answer, and those pairs belong in the report as a separate category rather than as failures.

The protocol also has blind spots. It measures conditions an annotator could find in the text; it cannot detect a condition that lives only in an unwritten practice or in a system rule, and annotators will miss some written ones, which is why the double-annotated subset and its agreement figure belong in the write-up.

Getting documents where rules really carry conditions

Synthetic policies rarely reproduce the messy ways businesses write exceptions, so the most informative test corpora are real operating documents: versioned policies, contract templates with schedules, product terms and procedures with footnotes and cross-references.

For teams that need such documents licensed rather than scraped, SourceX can be asked to find them: a request names the document types, volume, language, format, timing and license, and SourceX approaches U.S. businesses that hold matching material instead of selling from stock. It then runs sourcing, rights review, contracting, preparation and delivery, and the supplying company approves scope and terms and signs before anything is handed over.

Frequently asked questions

Does more chunk overlap fix split conditions?

Sometimes, for conditions that sit just past a boundary, but overlap cannot help when the condition is in another section or a glossary. It also returns more duplicated text, which can lower precision. Test overlap as its own factor so you can see which strata it actually helps.

Is semantic or structure-aware chunking always better for rules with exceptions?

Not necessarily. Header-aware splitting keeps sections intact but can still separate a rule from a condition in another section, and semantic splitters vary with the embedding model. Only a stratified comparison on your own documents answers it.

Can I use Ragas context recall as the main outcome?

It is a reasonable secondary measure, since it scores how many reference claims the retrieved context supports. It does not require the rule and its condition to arrive together, so a primary outcome built on annotated span pairs is closer to the failure this protocol targets.

How do I build ground truth for rule and condition pairs?

Have annotators mark character spans for each conditional rule and its governing condition in sampled documents, following a written guide. Double-annotate a subset, resolve disagreements with a second reviewer and report agreement, because the spans are what every retrieval score is computed against.

Should I report results for one chunk size only?

Report every configuration you ran, every stratum and the counts behind each figure, including null results. Publishing only the winning setting hides how sensitive the result was to choices that other teams will make differently.

Sources

  • Anthropic's contextual retrieval post says chunk size, chunk boundary and chunk overlap can affect retrieval performance, and describes prepending chunk-specific explanatory context to each chunk before embedding and BM25 indexing. Source
  • Chroma's chunking evaluation package scores retrieval with recall, precision and IoU computed over character ranges of reference excerpts, and its recursive token chunker splits on paragraph, line and sentence punctuation separators by default. Source
  • Ragas defines context recall as the number of reference claims supported by the retrieved context divided by the total number of reference claims. Source
  • Ragas defines context precision as the mean of precision at k over the ranked retrieved contexts, weighted by whether each item is relevant. Source
  • LangChain's RecursiveCharacterTextSplitter tries separators in order, by default paragraph breaks, line breaks, spaces and then characters. Source
  • LangChain's base TextSplitter defaults to a chunk size of 4,000 and a chunk overlap of 200. Source
  • Unstructured's chunk-by-title strategy starts a new chunk at each new section title and treats max characters as a hard limit, splitting oversized elements at spaces or new lines. Source
  • Miller's Adding Error Bars to Evals recommends clustered standard errors for grouped questions and analyzing question-level paired differences when comparing two systems. Source

Tell us what data you need

Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.

See if your company qualifies