Privacy and preparation
Legal research source freshness dataset: when a cited source changed after the memo
By SourceX Editorial · Reviewed by Noah Loul ·
9 sections
Short answer
Treat the memo as correct or incorrect as of its own date, and record the later change separately. A legal research source freshness dataset stores the memo's as-of date, the exact version of each cited source, the later event that changed it and when that event took effect, then gives two labels: whether the memo was right when written, and whether it can still be relied on today.
Redaction
Legal research source
Key takeaways
- A memo is dated work product; a later amendment makes it stale, not wrong as of its date.
- Record the version each citation pointed to, because an unversioned link resolves to today's text.
- Separate the publication date of a change from its effective date and any compliance date.
- Label as-of correctness and current reliance separately so a model learns to flag staleness instead of rewriting history.
- Check every later event against an official edition, not only a convenience copy.
- Exclude privileged and client-confidential memos unless counsel clears a specific, documented scope.
How should a legal research task handle a source updated after the memo was written?
It should keep the memo's conclusion fixed to its date, identify which cited sources have changed since, decide whether each change touches the conclusion, and say plainly what must be refreshed before anyone relies on the memo now. The memo is not rewritten and not marked wrong; it is marked stale where the change is material.
That is also the behavior a legal research model needs to learn. ABA Formal Opinion 512 warns that uncritical reliance on content from a generative AI tool can produce inaccurate legal advice or misleading statements to courts, and calls for independent verification or review of the output. At least one federal court has told self-represented litigants that using AI does not relieve them of responsibility for the accuracy of every authority cited. A tool that quotes a memo's old reading of a since-amended regulation as current law fails exactly this test.
A source freshness dataset trains and tests that skill. Each example pairs a dated memo with a dated record of what happened to its sources afterward, and the expected answer separates two questions the profession already keeps apart: was the analysis right on the day it was written, and is it safe to use today?
Why a citation link silently points at today's text
Most memo archives store citations as text or as links, and neither says which version of the source the author read. An unversioned link is worse than no link, because it quietly resolves to the current text. The Akoma Ntoso naming convention for legal documents makes the point explicit: a reference that omits the version identifier for a versioned act resolves to the version in force at the moment of lookup, not when the referencing document was written. Versions are identified by an Expression date.
Official tools now make point-in-time reading possible for some sources. The eCFR lets a reader view federal regulations as they stood at past dates back to January 2017. The U.S. Code site shows a currency date above each section and lists later laws as pending updates. Those features only help if the dataset records which date the memo author was reading from.
So the first job is not labeling. It is reconstructing, for each citation, the version in force on the memo's as-of date, and storing that version identifier next to the citation text. Without it, a reviewer comparing the memo with the source today will see a mismatch and cannot tell whether the author erred or the law moved.
Which version of the source does the memo actually rely on?
Use the date the research was last current, not the date the file was saved or the matter closed. Memos are often finalized days or weeks after the research, and edited drafts can carry later save dates than the analysis they contain.
At a glance
Which version of the source does the memo actually rely on
Then pin each citation to a version from an official or authoritative edition. The eCFR describes itself as authoritative but unofficial and tells legal researchers to verify against the official CFR, the daily Federal Register and the List of CFR Sections Affected. A dataset that checks freshness only against a convenience copy inherits its gaps, so the record should name which edition confirmed the version.
Statutes need the same care. The U.S. Code site states the public law through which the Code is current, and a section can be affected by a newer law that has not yet been folded into the text. A memo author who read a section during that gap read text that was already out of date, which is a different situation from a change that came later. Recording the currency date the author saw keeps those two cases apart.
- Memo as-of date: the date the author last verified the research, taken from the memo itself or the research log, with its evidence noted.
- Cited version: an Expression date, point-in-time date, release point or reporter citation that fixes the text the author read.
- Edition checked: the official or authoritative edition used to confirm that version, and the date it was checked.
- Pinpoint: the section, subsection or page the conclusion depends on, because a change elsewhere in the same source may be irrelevant.
- Unresolvable flag: set when the version cannot be reconstructed; those examples stay out of the scored set.
Which kinds of later change matter, and which are noise?
A change matters when it alters the text or authority the memo's conclusion rests on as of a date the reader cares about. Many later events do not. The dataset should classify the event type and its timing, then let a reviewer decide materiality against the pinpoint.
Timing has more than one date. Federal Register rules state when they take effect, and some set a separate, later compliance date; one 2023 rule, for example, was published in late November, took effect a month later and set a compliance date well over a year after that. A memo written between publication and the effective date may be correct about current law and still need a forward-looking warning.
Some changes are formal. Supreme Court slip opinions carry a notice that they are subject to formal revision before the preliminary print of the United States Reports. A memo quoting a slip opinion may cite text or page references that later change without any change in the holding. That is a citation update, not a stale conclusion, and the label should say so.
- Amended or repealed provision: compare the pinpoint text before and after; material if the conclusion depended on the changed words.
- Published but not yet effective rule: record publication, effective and compliance dates; label depends on the date of reliance.
- Later decision affecting a cited case: record the court, date and the treatment as reported by the court itself; route to counsel for weight.
- Withdrawn or replaced agency guidance: record the withdrawal notice and date from the issuing agency.
- Formal revision only, such as slip opinion pagination: update the citation, mark the conclusion unaffected.
- Renumbering without substantive change: map old to new section, mark unaffected.
The source freshness benchmark brief
The brief below is the record each example carries. One row in the dataset is one cited authority in one memo, so a memo with eight citations produces eight rows sharing a memo identifier. Adapt names to your matter system, but keep the split between what was true on the memo date and what is true on the check date.
| Field | What it records | Allowed values or format | Who fills it |
|---|---|---|---|
| memo_id, memo_as_of_date | Stable identifier and the date the research was last current | Opaque ID; YYYY-MM-DD with evidence note | Legal operations from the research log |
| authority_ref, pinpoint | Citation as written and the exact part the conclusion uses | Citation text; section or page | Reviewing lawyer |
| version_cited, edition_checked | The version in force on the as-of date and where it was confirmed | Expression or point-in-time date; edition name and check date | Research librarian or paralegal |
| later_event_type, event_dates | What changed after the memo and when it was published, effective and required | amendment, repeal, new rule, later decision, withdrawal, formal only, renumbering; dates | Research librarian |
| materiality | Whether the change touches the pinpoint the conclusion relies on | material, not material, needs counsel | Reviewing lawyer |
| label_as_of | Was the memo's use of this source right on its own date | correct, incorrect, unverifiable | Reviewing lawyer, second reviewer on disagreement |
| label_current | Can the memo's use of this source be relied on at the check date | rely, refresh before use, do not rely | Reviewing lawyer |
| check_date, screen_record | When freshness was checked and the privilege and confidentiality screen result | YYYY-MM-DD; cleared scope ID or excluded | Counsel |
Scoring the model: flag the change without rewriting the memo
The model's job on each example is to say whether the cited source changed after the as-of date, whether the change is material to the memo's conclusion, and what the reader should do now. Score those three outputs separately, because a model that detects every change but calls them all material is as unhelpful as one that misses them.
At a glance
Scoring the model: flag the change without rewriting the memo
Two errors deserve their own counts. The first is retroactive blame: calling a memo incorrect because the law changed after it was written. The second is silent currency: answering with the memo's old reading as though it were current. Both appear in real use, and a benchmark that blends them into one accuracy number hides which one a system is prone to.
Keep the as-of label out of the model's input. The model sees the memo text, its as-of date and the later-event record or access to versioned sources, and must produce the current-reliance answer. A separate task can ask for the as-of judgment, scored against the reviewing lawyer's label.
Report results by later_event_type as well as overall. A system can be strong on repealed provisions, which leave an obvious trace, and weak on rules published but not yet effective, where the right answer depends on the date of reliance. A breakdown by event type shows which kind of staleness a tool misses, and it tells the people preparing the next batch which event types to add more examples of.
- Change detection: did the model find each cited source that changed after the as-of date?
- Materiality: did it match the reviewer's material or not-material call for the pinpoint?
- Action: did it recommend rely, refresh or do not rely in line with label_current?
- Retroactive blame count: memos called wrong solely because of later events.
- Silent currency count: old readings presented as current law.
Illustrative case: a regulation amended four months after the memo
This case is illustrative and does not describe a real matter. A compliance memo dated in March concludes that a client's practice falls below a reporting threshold in a federal regulation. The memo cites the regulation section and quotes the threshold. In July, a final rule amending that section is published; it takes effect in September and lowers the threshold so the practice would now be covered.
The brief for the regulation citation records the March as-of date, the point-in-time version in force in March confirmed against the official edition, the amendment as later_event_type with its July publication and September effective dates, and materiality as material because the pinpoint is the threshold itself. label_as_of is correct. label_current is do not rely after September and refresh before use between July and September.
Two other citations in the same memo, a definitions section that was only renumbered and a case whose slip opinion page references later changed, get not-material labels and citation updates. The illustrative scoring outcome: a model that answers the current question with the March threshold earns a silent currency error; a model that calls the March memo wrong earns a retroactive blame error; the expected answer flags one material change and two formal ones.
Where to stop: privilege, retroactivity and unofficial-only evidence
Some examples should not enter the dataset, and some need a lawyer before they get a label. Freshness work is detailed enough that a team can lose sight of the screen that comes first: whether the memo may be used at all.
At a glance
Where to stop: privilege, retroactivity and unofficial-only evidence
Client memos are usually privileged or confidential. Use memos written for internal training, public-law research written without client facts, or memos counsel has cleared for a documented scope with client identifiers and facts removed. Rights to the text, privilege and client confidentiality are separate questions; removing names does not by itself create a right to license the memo.
- Stop if counsel has not cleared the memo's scope, or the screen record is missing.
- Ask counsel when a later change is retroactive or applies to pending matters, because the as-of label itself may then be contested.
- Ask counsel when a later court decision's effect on a cited case is not stated by the court, rather than inferring treatment from a secondary service.
- Drop the example when the cited version can be confirmed only from an unofficial copy with no authoritative edition available.
- Drop the example when the as-of date cannot be established from the memo or research log.
- Narrow the scope when memos mix jurisdictions and the later events were checked for only one.
Bringing a memo archive to a licensing review
A memo archive with as-of dates, versioned citations and a counsel screen already in place is far easier to assess than one that needs those reconstructed. A firm or legal department weighing a license of training memos or public-law research can have SourceX run rights review, contracting, preparation and delivery for a selected scope, which the firm approves and signs before delivery. Whether the organization qualifies is checked first through a short assessment, with no documents uploaded.
Bring the screen record and the freshness fields to that conversation rather than raw memos. They show what can be offered, which part is excluded and why, and they let reviewers see quickly whether the archive supports a freshness task or only general legal drafting.
Frequently asked questions
Is a legal memo wrong if the statute it cites was amended later?
Not as of its own date. The memo is judged against the law in force when the research was last current. It becomes stale for present reliance if the amendment touches the provision its conclusion depends on, and should be refreshed before anyone acts on it.
How do I find the version of a federal regulation that applied on a past date?
The eCFR offers a point-in-time view of federal regulations back to January 2017. It describes itself as unofficial, so confirm the text against the official CFR, the Federal Register and the List of CFR Sections Affected before recording it as the cited version.
Which date should a memo record use as its as-of date?
Use the date the author last verified the research, taken from the memo or the research log. File save dates and matter close dates often come later and can make a correct memo look stale.
Can a changed slip opinion make a memo stale?
Usually not. Supreme Court slip opinions are subject to formal revision before the preliminary print, so page references and minor wording can change without any change in the holding. Record a citation update and mark the conclusion unaffected unless the substance changed.
Can client memos be used in a legal research freshness dataset?
Only within a scope counsel has cleared, because client memos are usually privileged or confidential. Training memos and public-law research without client facts are safer starting points, and removing names does not by itself create a right to license the text.
Sources
- In the Akoma Ntoso naming convention, a reference without a version identifier resolves to the version in force at lookup, and versions carry Expression dates. Source
- The eCFR point-in-time system shows federal regulations as they stood at past dates back to January 2017. Source
- The eCFR is not an official legal edition; legal researchers should verify against the official CFR, the Federal Register and the List of CFR Sections Affected. Source
- The U.S. Code site shows a currency date for each section and lists later laws as pending updates. Source
- Supreme Court slip opinions are subject to formal revision before publication in the preliminary print of the United States Reports. Source
- A Federal Register final rule can carry an effective date and a separate, later compliance date after its publication date. Source
- ABA Formal Opinion 512 warns that uncritical reliance on generative AI output can lead to inaccurate legal advice and calls for independent verification or review. Source
- A federal district court notice states that using AI does not relieve a party of responsibility for the accuracy of cited legal authorities. Source
See if your company qualifies
A short company assessment. No data uploads are needed.