Skip to content

AI uses for records

Retrieval and stale knowledge articles: when the most clicked is outdated

By SourceX Editorial · Updated

9 sections

Short answer

Retrieval should rank the current, approved article that governs the answer for that user, even when an outdated article has far more clicks, and should decline when no current source exists. To test it, build an evaluation set that labels each query's controlling article version, the superseded articles that still look relevant, the as-of date, the user context and when to abstain.

Retrieval

Retrieval stale knowledge

This page covers retrieval and stale knowledge articles: when the most clicked is outdated.Illustrative

Key takeaways

  • Clicks, votes and promotion reward an article's age and past rank, not whether it is still correct.
  • Use the last content edit and the publishing state of the indexed version, not a general updated timestamp.
  • Label topic and authority separately; most stale-article failures are on topic and not authoritative.
  • Every label needs an as-of date and a user context, since the controlling article can differ by plan or region.
  • Faithfulness to retrieved context does not prove currency; an answer can be faithful to a superseded article.
  • A content owner, not an annotator, decides which source governs when it is not obvious.

How should retrieval handle a popular help article that is outdated?

Retrieval should rank the controlling source first: the current, approved version of the article that governs the answer for this user, even when an older article has far more clicks. To test that, build an evaluation set that labels, for each query, the controlling article version, the superseded articles that still look relevant, the date the label holds for and when the system should decline to answer.

The hard part is that every popularity signal a help center collects rewards age. An article published years ago has had years to gather clicks, upvotes, inbound links from macros and tickets, and a place in the top results that earns it more clicks. When the product changes and a new article is written, the old one keeps winning on every signal except being right.

Similarity search does not fix this. An outdated article and its replacement usually share most of their vocabulary, so embeddings rank them close together and popularity features break the tie in the wrong direction. The evaluation has to know which one controls, and that knowledge does not exist in the text.

Why clicks, votes and links keep pointing at the old article

Clicks measure what people were shown and chose, not what was correct. Joachims, Swaminathan and Schnabel showed that position bias in search rankings strongly influences how many clicks a result receives, which is why they argued for propensity-weighted learning instead of training rankers on raw clicks. An article that ranked first for years has clicks that partly reflect its rank.

Help-center platforms record other signals with similar problems. Zendesk's Help Center API exposes vote_sum, the sum of upvotes and downvotes, and a promoted flag on each article; both are set by people at some past moment and neither expires when the product changes. Votes on an old article may be votes for an answer that was right at the time.

So treat popularity as a feature to test against, not a label to learn from. The dataset brief below deliberately includes queries where the most clicked article is the wrong answer, because those are the queries where a retriever that leans on popularity will fail.

Which date actually tells you an article is current?

Use the date of the last content change to the version you index, and the publishing state of that version, rather than any general updated timestamp. Platforms keep several dates and statuses, and they do not mean the same thing.

Even the right date only says when the text last changed. An article edited last week can still describe a feature that was retired yesterday, and an article untouched for years can be perfectly current for a stable workflow. That is why the brief joins article dates to the dates when the underlying product, policy or price list changed, and treats an article as suspect when its last content edit is older than the last change to the thing it describes.

Which date actually tells you an article is current?
SignalWhere it appearsWhat it meansTrap when used for freshness
edited_atZendesk Help Center articleLast change to the title or body in the displayed localeUnedited text can still be wrong after a product change
updated_atZendesk Help Center articleChanges on non-edit events too, such as a new commentMakes a stale article look recently maintained
outdatedZendesk Help Center articleDeprecated; the docs say it is always false on the source articleReads as a freshness flag but carries no signal
PublishStatus Draft, Online, ArchivedSalesforce Knowledge article versionsWhich version is published, in progress or retiredUnfiltered queries in recent API versions can return all three
vote_sum and promotedZendesk Help Center articleReader votes and an editor's promotionPopularity and placement, not currency
Policy or release effective dateRelease notes, policy register, change logWhen the underlying rule changedOften lives outside the help center and must be joined in

The freshness-labeled retrieval dataset brief

The brief is the asset: a specification a team can hand to annotators and engineers to build a freshness-labeled retrieval evaluation from a real help center. Each row is a field every query record must carry. Keep the brief versioned, because the as-of date makes every label perishable.

At a glance

The freshness-labeled retrieval dataset brief

Sample queries on purpose rather than at random. Random samples from search logs are dominated by head queries whose top article is both popular and current, so they show little. Over-sample three groups: queries whose most clicked article was edited before the last relevant product or policy change; queries where two articles with high text similarity have different publish dates; and queries asked by users whose plan, region or product version differs from the majority. Report results for each group separately, alongside a random sample that shows ordinary performance.

The freshness-labeled retrieval dataset brief
FieldWhat it holdsHow it is filledWhy it matters
Query ID and textA real or realistic user questionSampled from search logs or tickets, de-identifiedAnchors every other label
Query intentTask, account state or policy the user needsAnnotator, from the ticket or search sessionSeparates how-to from entitlement questions
As-of dateThe date the labels are true forSet per release of the datasetLets the same query have different correct answers over time
User contextPlan, region, role or product version that changes the answerFrom account attributes, generalizedControls which article governs for this user
Controlling sourceArticle ID, version and locale that governs the answerAnnotator plus content owner sign-offThe target the retriever must rank first
Superseded candidatesOlder or archived articles that look relevant, with the reason each is supersededMined from version history, click logs and duplicatesHard negatives that popularity will favor
Required evidence setEvery passage needed for a complete answerAnnotator, checked against the controlling versionPrevents passing on one partial chunk
Expected answer factsShort atomic statements the answer must containAnnotator from the controlling sourceScores the answer, not only the ranking
Abstain flagWhether no answer should be givenAnnotator, with the reason (no current source, restricted, conflicting)Rewards declining over quoting a stale article
Label provenanceWho labeled, who approved, which policy versionWorkflow metadataMakes disputed labels traceable and refreshable

Labeling topic and authority as two separate judgments

Write two relevance judgments for every query-article pair: is it on topic, and is it the authority for this user on this date. Most failures of interest score high on the first and zero on the second, and a single blended grade hides them.

The standard qrels format used by trec_eval already supports this in practice. Each line is a query ID, an ignored iteration field, a document ID and a relevance value, where relevance is a non-negative integer or -1 for unjudged. Keep a topical qrels file and an authority qrels file over the same document IDs, with the document ID including the version, and run both through the same evaluation.

A workable scheme, which teams should adapt, is a topical grade of 0 to 2 and an authority grade where 2 means controlling and current, 1 means consistent with the controlling source but not itself authoritative, and 0 means superseded, archived or wrong for this user. The gap between a retriever's topical score and its authority score is the number that tells you whether it is ranking by subject or by correctness.

Does the retrieved evidence support this answer for this user on this date?

Only when three things hold: the cited passages come from the controlling version, that version is visible to this user, and the answer's facts match it as of the dataset date. Groundedness metrics check a different thing. Ragas faithfulness, for example, counts how many claims in a response are supported by the retrieved context, so an answer copied accurately from a stale article scores as fully faithful.

At a glance

Does the retrieved evidence support this answer for this user on this date

Score a run with separate checks that map to the brief:

  • Authority at rank: the controlling version appears in the top results, measured on the authority qrels.
  • Superseded intrusion: whether any superseded candidate ranks above the controlling version, counted per query.
  • Evidence completeness: share of the required evidence set retrieved.
  • Answer currency: share of expected answer facts present, and any fact that matches only a superseded article flagged as a stale claim.
  • Visibility: no cited article is restricted from this user's context.
  • Abstention: on queries flagged abstain, the system declines or routes to a person rather than answering.

Worked example (illustrative): two help articles on resetting two-step login

Illustrative only; the product, articles and counts are constructed. A software company's help center has two articles on resetting two-step login for a team member. Article A was written before an admin console redesign and has far more clicks and upvotes; it tells admins to use a settings page that no longer exists for most customers. Article B was published after the redesign, has a handful of clicks and describes the new console.

For the query "reset two-step login for a user on my team", the brief labels Article B as controlling for accounts on the current console, Article A as superseded with the reason "console redesign", and lists two required passages from B: the admin path and the note that the user must re-enroll. A popularity-weighted retriever ranks A first; a pure embedding retriever ranks them close together with A slightly ahead because its wording matches the query better.

The exception is what makes the case useful. Some customers on a legacy plan still have the old console, and for them Article A is controlling. The same query therefore carries two labeled user contexts with different controlling sources. A retriever that simply demotes old articles fails the legacy context, which is exactly the error a blanket recency boost introduces.

The run is scored per context. In the current-console context, a stale answer from A is faithful to its context yet fails answer currency. In the legacy context, citing B fails authority. Neither failure shows up in a single topical relevance score.

The fix the case points to is not a recency boost but a context-aware filter on publishing state and applicability, with the controlling version decided by the content owner. The evaluation set tells you whether that filter works without breaking the minority context.

Where freshness labels break down and need an owner's decision

Freshness labels need a named content owner whenever the controlling source is not obvious from the help center itself. Annotators can find versions and dates; they cannot decide which policy is in force, and guessing produces labels that look precise and are wrong.

FreshQA, the benchmark introduced with FreshLLMs, groups questions that need fast-changing knowledge and questions with false premises, and its authors commit to updating it at regular intervals; a private freshness set needs the same maintenance plan, or it decays into a test of last year's product.

  • No trustworthy content date: the help center was migrated and every article shows the migration date. Recover dates from the old system or the release log before labeling.
  • Owners disagree about which article governs. Record both, mark the query abstain, and ask the policy owner to decide before the next release.
  • Regulated or contractual content, such as refund rights, data retention or security commitments. Have the responsible owner approve the controlling source and keep the approval with the label.
  • Translations lag the source article. Label each locale separately, since a current English article does not make its translation current.
  • Restricted articles for internal agents or specific plans. Label visibility explicitly and test that restricted content never reaches users outside it.
  • Archived versions appear in the index because an export or query did not filter by publishing state. Salesforce's Knowledge developer guide notes that from API version 47.0 queries without a PublishStatus filter can return draft, published and archived articles to users with the right permissions.

Getting help-center histories with versions and dates intact

Freshness evaluations are only as good as the history behind them: article versions with their publish and archive dates, the release or policy changes that superseded them, and search or ticket data showing what users actually asked and clicked. Public help centers show only the current version, which is why version history from the operating company matters.

Help-center histories with every article version and archive date intact are something a retrieval team can ask SourceX to find; with no stock to draw on, SourceX looks among U.S. businesses for a knowledge base that matches the systems, languages, volume, format, timing and license terms the request sets out. The company that owns the help center agrees the scope and terms and signs before any version history is handed over.

Frequently asked questions

Should click data be used as relevance labels for a help-center retriever?

Only with care. Clicks are shaped by position, so raw clicks reward whatever already ranked first, often an old article. Use clicks to find candidate queries and hard negatives, and label authority separately with a content owner.

Is removing outdated articles from the index enough?

It helps, but it is not a test. Archived versions can come back through exports and unfiltered queries, and some old articles are still controlling for legacy plans or regions. An evaluation with labeled superseded candidates shows whether the retriever handles both cases.

Which Zendesk timestamp should define article freshness?

Prefer edited_at, which tracks changes to the title or body, over updated_at, which also changes on events such as comments. Neither proves the content is current, so join in release or policy change dates.

Does a faithful answer mean a correct answer?

No. Faithfulness metrics check whether claims are supported by the retrieved context. If that context is a superseded article, the answer can be fully faithful and still wrong for the user today.

How often should a freshness-labeled evaluation set be refreshed?

Whenever the product, policies or help center change enough to move controlling sources, and on a fixed schedule otherwise. Each release of the set should carry its as-of date so results from different releases are never compared as if they measured the same thing.

Sources

  • Zendesk's Help Center Articles API documents edited_at (last title or body edit in the locale), updated_at (also changes on events such as comments), vote_sum, promoted and the deprecated outdated property. Source
  • Salesforce Knowledge article versions move through Draft, Online (published) and Archived publishing statuses. Source
  • Salesforce's Knowledge developer guide says that from API version 47.0, queries without a PublishStatus filter return draft, published and archived articles to users with the relevant permissions. Source
  • Joachims, Swaminathan and Schnabel show that position bias strongly influences clicks and propose propensity-weighted learning to rank from biased click feedback. Source
  • FreshLLMs introduces FreshQA, a benchmark of questions needing fast-changing knowledge and questions with false premises, which its authors commit to updating at regular intervals. Source
  • trec_eval's standard qrels format is a tuple of query ID, ignored iteration field, document ID and relevance, where relevance is a non-negative integer below 128 or -1 for unjudged. Source
  • Ragas faithfulness measures the share of claims in a response that can be supported by the retrieved context. Source

Tell us what data you need

Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.

See if your company qualifies