Skip to content

Systems and records

Support knowledge conflict dataset: when the help center and runbook disagree

By SourceX Editorial · Updated

9 sections

Short answer

When the help center and an internal runbook disagree, the agent should follow the source the organization made responsible for that specific claim, checked against the request date and the customer's plan, region and product version, then file a discrepancy so the losing article is fixed. A useful support knowledge conflict dataset records both versions, the rule applied, approvals, what was promised and whether the resolution held.

System export

Support knowledge conflict

This page covers support knowledge conflict dataset: when the help center and runbook disagree.Illustrative

Key takeaways

  • Authority, not recency or audience, decides which source controls a disputed claim.
  • Classify the conflict first: policy terms, product steps, security steps and approvals follow different rules.
  • Capture both articles as they stood on the case date; today's export can invent or hide conflicts.
  • Record what the customer was promised separately from what was done and from the correction filed.
  • Label outcomes only after an observation window; a solved status is not proof the answer held.
  • Screen runbook text for credentials, internal paths and security steps before any case leaves support.

How should a support agent resolve conflicting help center and internal runbook guidance?

Resolve the conflict by authority, not by recency or audience. The agent identifies which source owns the specific claim in dispute, checks that its version applied on the date of the customer's question and to that customer's plan, region and product version, acts on it, and files a discrepancy so the losing article is corrected. If neither source clearly controls, the agent escalates to the policy owner before promising anything.

A support knowledge conflict dataset captures that whole path as one case: both article versions as they stood, the disputed sentences, the rule that decided which controlled, any approval, what the customer was told, the correction, and whether the problem stayed solved. A ticket marked solved is not the outcome; the outcome is what happened afterward.

This matters for AI because retrieval systems surface both documents. Research on large language models classifies conflicts among retrieved context passages as their own category, inter-context conflict, separate from conflicts between context and what the model learned in training. Cases where a person resolved such a conflict correctly, with the reason recorded, are scarce and useful evidence.

Why newest wins and internal wins are both unsafe defaults

Each shortcut breaks on a common case, so the packet should record the authority rule rather than a default. Newest wins fails when a runbook page was edited for a typo after the help center was updated for a policy change, because the edit date says nothing about which text carries the current policy.

Internal wins fails just as often. Runbooks collect workarounds written during incidents, and the Knowledge-Centered Service practices guide describes work-in-progress articles as ones that should exist only while the issue is being worked on. A workaround page left behind after the fix is internal and recent-looking, and wrong.

Public wins fails the other way. Help centers lag behind policy changes, translations lag behind source articles, and a public article may describe the standard plan while the customer holds an enterprise agreement with different terms. The question is always which source the organization made responsible for this claim, for this customer, on this date.

Authority rules by type of conflict

Classify the conflict first, because the controlling source depends on what kind of claim is in dispute. The table is the decision rule the packet applies; the policy owner can change a row, but the change should be written down before cases are labeled.

Authority rules by type of conflict
Conflict typeUsually controlsEvidence to confirmAgent action
Policy term such as an eligibility window or limitThe policy owner's current published policy versionPolicy version and effective date; date of the customer's requestAct on the policy; if the public article promised more, escalate for an exception decision
Product steps that differ by releaseThe article matching the customer's product versionRelease notes or version field on each articleGive the steps for the customer's version and flag the other article
Security or identity verification stepThe stricter internal procedureOwner of the security control and the runbook's validation stateNever relax a verification step because a public article omits it
Who may approve an exceptionThe internal approval matrixNamed role in the runbook; approval record on the ticketRoute to the approver; do not promise the outcome first
Plan, region or segment specific termsThe article whose audience matches the customerArticle audience or channel settings; customer plan and regionAnswer from the matching article and note the mismatch
Workaround versus permanent fixThe article tied to the current, validated resolutionArticle state, linked incident status and archive dateUse the validated fix; ask for the stale workaround to be archived
Neither source validated or ownedNo one yetMissing owner, draft state or contradicting approvalsEscalate to the policy owner and record the case as unresolved

Where the version, audience and status evidence lives

The evidence needed to apply those rules usually exists already in the knowledge tools, but it has to be captured as it stood on the date of the case. Exporting today's article text and pairing it with a ticket from last year produces a conflict that never existed, or hides one that did.

Agent-side evidence matters as much as article metadata. Many helpdesks record which articles an agent linked or inserted into a reply, and that record shows which source the agent actually relied on, which may not be the one the final reply quotes. Where the tool keeps it, add the linked article and version to the case alongside the two snapshots.

Where the version, audience and status evidence lives
ToolField or featureWhat it tells youCaveat
Zendesk help centerArticle view permissions by user segment, including an agents and admins optionWhether an article was internal or customer facingKnowledge admins see all content regardless of restrictions
Zendesk Help Center APIedited_at, updated_at, draft, user_segment_idedited_at reflects title or body edits in a locale; user_segment_id null means visible to everyoneupdated_at is a broader update time; compare content changes on edited_at, which tracks title or body edits
Salesforce KnowledgeChannel visibility: internal app, customer, partner, public knowledge baseWhich audiences could see the article versionChannel fields are hidden from layouts until an admin adds them
Salesforce Knowledge versionsPublishStatus and IsLatestVersion on article versionsWhich version was online and which are archivedSeeing draft or archived versions needs specific user permissions
Confluence runbooksPage version history and version compareWhat the runbook said on a given date and what changedRetention rules or deleted versions can leave gaps
KCS-style article stateConfidence (work in progress, not validated, validated, archived) and audienceWhether the organization trusted the article for this audienceOnly as reliable as the team's discipline in updating state

The authority and discrepancy resolution task packet

Each case is one customer question where two sources disagreed, with enough evidence that a reviewer could reach the same decision without the label. The fields below are a packet a customer operations team can fill as is; example values are illustrative.

At a glance

The authority and discrepancy resolution task packet

  • Include cases where the agent chose wrongly and a later contact or correction exposed it; they teach as much as correct cases.
  • Include cases with no conflict but two similar-looking articles, so a model learns not to invent disagreements.
  • Keep one case per customer question, even if the same pair of articles conflicted on many tickets, and record the count separately so frequent conflicts do not dominate the pack.
  • Version the pack itself, with a date and a change log, because the authority table and observation windows will be revised as the policy owner learns from it.
The authority and discrepancy resolution task packet
FieldIllustrative valueSource recordRule
case_id and ticket_refKC-0117 / ticket 88412HelpdeskTicket text de-identified before inclusion
customer_contextAnnual plan, EU region, product version 7CRM and account settings at the timeCaptured as of the ticket date, not today
source_a_snapshotHelp center article, version edited on day 0, audience everyoneKnowledge tool version historyStore the text and version metadata, not a live link
source_b_snapshotRunbook page, version 14, state validatedWiki historySame rule as source A
conflict_spanRestore within 30 days versus restore within 14 daysBoth snapshotsQuote only the contradicting sentences
conflict_type and rule_appliedPolicy term; current policy version controlsAuthority tableOne rule per case; cite the row
approvalTeam lead approved a one-time exceptionTicket audit trailApprover role and timestamp; blank if none
agent_commitmentWorkspace restored; customer told the current windowAgent replyRecord what was promised, separate from what was done
correctionDiscrepancy ticket opened; help center fixed in a new versionKnowledge feedback queue and article historyLink both the request and the published fix
outcome_labelResolved and held through the observation windowReopens, follow-up contacts, related ticketsAssign only after the window closes

Worked case (illustrative): two different restore windows for a deleted workspace

This case is invented for illustration; the product, numbers and dates are not drawn from any real company. It walks through the starting state, the linked evidence, the decision and the outcome check.

Starting state: a customer deletes a workspace by mistake and asks for it back 20 days later. The public help center article says deleted workspaces can be restored within 30 days. The internal restore runbook says 14 days, and the agent sees both.

Evidence: the runbook's history shows the 14-day figure was added when the data retention policy changed, and the policy owner's document carries the same effective date. The help center article's last content edit predates that change. Both articles are marked validated, so state alone does not decide it.

Decision: the conflict is a policy term, so the current policy version controls and the help center is stale. Because the customer relied on a public statement, the agent does not refuse on the spot; the runbook's approval matrix sends the request to a team lead, who approves a one-time restore. The agent opens a discrepancy request against the help center article.

Outcome check: the restore succeeded, the article was corrected in a new version within the week, and the customer did not reopen the ticket or contact support about the workspace during a 30-day observation window. The paired counterexample belongs in the same packet: a runbook page created during an outage tells agents to disable a sync setting, while the help center gives the permanent fix. The incident is closed, so the runbook page should have been archived; the help center controls, and the case teaches a model that recency and internal status do not settle authority.

Evidence that a restore, refund or fix still held weeks after closure

Only evidence collected after closure can show that a resolution held. A solved status records that an agent closed the ticket; it does not record whether the answer was right, whether the customer came back, or whether the conflicting article went on misleading other customers.

Set an observation window per conflict type before labeling, and label each case with one outcome: held, reopened, new contact on the same issue, contradicted by a later policy decision, or not yet mature. Count a case as held only when the window has fully passed with none of the negative signals, and keep immature cases out of training and evaluation splits rather than guessing.

Score the tasks built from these cases on separate skills: detecting that two sources conflict, naming the controlling source and the rule, drafting a reply that matches the controlling source without overpromising, and escalating when no source has authority. Treat a confident answer on an unresolved case as an error and an escalation as the correct output.

Screening runbooks and customer text before the cases leave support

Runbooks are written for insiders and often hold material that should never leave the company. Before any case is shared, a privacy reviewer and the runbook owner should screen both the knowledge text and the ticket text.

  • Remove credentials, internal hostnames, admin console paths, security bypass steps and fraud detection thresholds from runbook snapshots, or drop the case if the conflict cannot be shown without them.
  • De-identify customer names, emails, account identifiers and free text, keeping plan, region and product version as categories so the authority rule still works.
  • Exclude content the company did not author, such as vendor documentation pasted into a runbook, unless the company holds the right to share it.
  • Check customer contracts for confidentiality or data-use terms that restrict reuse of support conversations, and keep enterprise-specific terms out of the packet unless counsel clears them.
  • Record who screened each case and what was removed, so a buyer can see that redaction did not erase the conflict itself.

When a conflict-resolution history becomes a licensing candidate

Support histories where people resolved conflicting guidance, approved exceptions and corrected the knowledge base are operational records that public datasets rarely contain. Support organizations that decide to license this kind of history can rely on SourceX for sourcing, rights review, contracting, preparation and delivery to AI developers.

Qualification is checked through a brief assessment with no tickets or articles uploaded. Scope and terms are approved and the agreement signed before anything is delivered, so a screened sample of conflict cases is best treated as input to that scoping conversation rather than something to send early.

How do real decision logs record which version won and who decided?

Unevenly, even in teams that publish their decision records in the open. Five public decision logs, read at pinned commits on 2026-10-27, hold 176 records: Mozilla's accounts platform (52), Home Assistant's architecture repository (22), two GOV.UK platform logs (38 and 24) and The National Archives' Transfer Digital Records service (40). Of these, 114 carry a status field; The National Archives' records have none and mark a replaced decision only with a sentence near the top. Only the Mozilla log has a field for deciders, filled on 50 of its 52 records, which leaves 126 of the 176 with no field naming who made the call; 8 of Home Assistant's 22 records link the issue or discussion where the decision was reached instead.

Eleven records are marked as replaced, reverted or deprecated. Eight point to the record that replaced them, two point to a discussion outside the log, and one states that no superseding record exists. Only 3 of the 8 pairs link correctly in both directions, and 2 of the 8 winning records never left proposed or pending: Mozilla's ADR-0014 says ADR-0017 superseded it, while ADR-0017 is still marked proposed. Counting rules are in the reviewer notes; totals per log:

How do real decision logs record which version won and who decided?
Decision log (commit)RecordsStatus fieldDecider fieldMarked replaced, reverted or deprecated (pairs linked both ways)
Mozilla accounts, mozilla/fxa (18310cd)5230502 (0)
Home Assistant, home-assistant/architecture (5438568)222203 (1)
GOV.UK, alphagov/govuk-aws (81ee9f6)383802 (1)
GOV.UK, alphagov/govuk-infrastructure (792a223)242401 (0)
The National Archives, tdr-dev-documentation (476d278)40003 (1)

Which copy governs when the same policy sits in two places?

The copy whose container says it governs; when no container says so, neither copy can be called current. GOV.UK's platform engineering team is folding what remains of its govuk-aws repository into govuk-infrastructure. The older repository's README opens with a Deprecated heading, tells readers to use govuk-infrastructure for anything new and says its decision records are kept for posterity. Inside it, 29 of the 38 records still read Accepted.

One of those records, a policy on non-GOV.UK domains dated 10 October 2022, was recorded again in the new log on 6 May 2025 with reworded but near-identical rules. Both copies read Accepted and neither mentions the other. The older copy keeps a consequences section naming three Signon domains that broke the policy at the time; the newer copy has no consequences section. A retriever that indexes both folders without the README finds two current-looking versions of one policy with different dates and different follow-up text.

That adds an authority rule the conflict-type table above does not cover: a deprecation notice on the container, whether a repository, a wiki space or a retired help center section, outranks the status written inside each record, and a later record replaces an earlier one only if it names it. Record-level marking fails more quietly. In The National Archives log, the two records on making the GOV.UK Notify staging service live point at each other, yet neither link matches a file that exists: the newer record's file name carries a typo, and its link back carries a different one.

How would a buyer use conflict-resolution records in training or evaluation?

As cases whose label is the recorded resolution, not the later date. A buyer shows a model both records plus any container notice and asks which one governs and on what evidence; the gold answer cites the marker, such as a status line, a link from the replaced record or a deprecation banner. Pairs where the marking is missing or broken stay in the set, labeled unresolved like the unowned sources in the authority table, because the right output is to flag the conflict for its owner rather than pick the newer copy. Retrieval mechanics, such as filtering superseded versions at query time, belong to the separate guide on evaluating RAG over versioned, outdated and conflicting documents.

A support team can run the census checks on its own archive before sharing a sample, and a buyer can ask for the totals in the shape of the table above, with two-way working links and recorded deciders as the figures to read first. Each conflict case should hold:

  • the losing and winning article IDs, with the version of each that was live on the date of the resolution;
  • where the resolution is marked: a status field, an archive state, a banner on the space or a note in the article body;
  • the result of a scripted check that the link from loser to winner and the link back both open the intended article;
  • who resolved it, as a role, or a pointer to the approval or discussion record when the article names no one;
  • an unresolved flag for pairs where neither copy is marked, kept in the set instead of being dropped.

Sources for this section

  • Mozilla's accounts platform decision log (52 records at commit 18310cd) uses a template with an optional Deciders field; ADR-0014 is marked superseded by ADR-0017, whose own status is proposed. Source
  • Home Assistant records architecture decisions as ADRs after discussion (22 records at commit 5438568); ADR-0002 is superseded by ADR-0020, and two installation-method records are marked reverted by discussions. Source
  • The govuk-aws decision log (38 records at commit 81ee9f6, 29 marked Accepted) includes the Non-GOV.UK domain policy dated 10 October 2022, with a consequences section naming Signon domains in violation. Source
  • The govuk-aws README marks the repository deprecated, tells readers to use govuk-infrastructure for anything new and says its decision records are kept for posterity. Source
  • The govuk-infrastructure decision log (24 records at commit 792a223) records the Non-GOV.UK domain policy again on 6 May 2025 as Accepted, without a consequences section or a reference to the earlier copy. Source
  • The National Archives' Transfer Digital Records decision log (40 records at commit 476d278) has no status fields and marks replaced decisions with a superseded-by sentence; in one pair neither link matches an existing file name. Source

Frequently asked questions

Should agents follow the internal runbook or the public help center when they disagree?

Neither by default. The agent should follow whichever source the organization made responsible for the specific claim, checked against the date, plan, region and product version of the request, and escalate when ownership is unclear.

What should an authority and discrepancy resolution task packet include?

Snapshots of both sources as they stood on the case date, the contradicting sentences, the conflict type and rule applied, any approval, what the customer was promised, the correction made, and a mature outcome label. Each field should cite the record it came from.

Which records show what the agent decided and whether it worked?

The ticket audit trail proves what the agent decided and who approved it, the knowledge tool's version history proves the correction, and reopen and follow-up contact records over an agreed window prove whether the resolution held.

How do you know which article version a customer or agent actually saw?

Use the knowledge tool's version history and content edit timestamps to reconstruct the text as of the ticket date. A current export cannot show this, so capture version metadata with every snapshot.

Is a closed ticket a good label for a correct resolution?

No. Closure records an agent action, not an outcome. Wait for the observation window to pass and label from reopenings, repeat contacts and later policy decisions.

Sources

  • Knowledge Conflicts for LLMs: A Survey (EMNLP 2024) classifies context-memory, inter-context and intra-memory conflicts. Source
  • The KCS v6 Practices Guide defines article state by confidence (work in progress, not validated, validated, archived) and audience, and says work-in-progress articles should exist only while the issue is being worked on. Source
  • Zendesk lets admins restrict article viewing to user segments, including an agents and admins segment for internal articles. Source
  • Zendesk's Help Center Articles API documents edited_at, updated_at, draft and user_segment_id, with null user_segment_id meaning visible to everyone. Source
  • Salesforce Knowledge channel fields (internal app, customer, partner, public knowledge base) must be added to the page layout to be set. Source
  • Salesforce documents filtering KnowledgeArticleVersion on PublishStatus and IsLatestVersion, and permission requirements for draft and archived versions. Source
  • Confluence Cloud page history lets users select two versions and compare what changed. Source

See if your company qualifies

A short company assessment. No data uploads are needed.

See if your company qualifies