SOP, playbook and knowledge base datasets for AI training

An SOP dataset is a collection of a company's written procedures, such as standard operating procedures, checklists, playbooks, runbooks and knowledge-base articles, with page history, owners and the links between pages. SourceX sources these from established businesses' Confluence, Notion and SharePoint spaces and, where partners hold them, pairs each SOP with execution histories showing how employees actually carried it out, something the written steps cannot show. Each dataset is de-identified and licensed for an agreed permitted use.

Request this dataset Email the data team

Dataset manifest

Sourced to your spec

What it is: Written procedures with page history, ownership and links to execution records
Typical systems: Confluence, Notion, SharePoint, Guru, ServiceNow Knowledge, Zendesk Guide
Typical history: Varies by partner; page histories and execution logs can span years
Modality: Wiki pages and documents with structure; execution records as structured events
Delivery formats: Pages as Markdown or HTML with JSONL metadata and history
Preparation: Names and customer details de-identified; credentials and internal URLs removed; links kept as IDs
Licensing: Scoped per knowledge space; permitted use and derivatives agreed per license
Availability: Sourced to your spec; execution pairing depends on the partner's systems

Request this dataset

What a delivery contains

Fields vary by source system and are fixed per order. A typical delivery includes:

Field	Type	What it holds
doc_id	string	Pseudonymous, stable page ID that every link, revision and execution record refers to.
doc_type	enum	SOP, checklist, playbook, runbook, policy, how-to or troubleshooting article, from the space's own labels or agreed classification.
space	object	The team space, its page tree and the parent page, which show where the procedure sits.
steps	array	Ordered steps with sub-steps, decision branches, roles and the systems each step touches.
body	string	Full page text as Markdown with headings, tables and callouts intact; macros are resolved or marked.
links	array	Links to other pages, templates and forms in the delivery, by stable ID.
ownership	object	Owner role, approver role, review cycle and the date of the last review.
revisions	array	Page history with timestamps, editor roles, change summaries and the steps that changed.
usage	object	View counts, search terms that led to the page, feedback votes and comments, where the platform records them.
executions	array	Records of the procedure being carried out — steps done, skipped or reordered, exceptions, handoffs and outcome.
execution_link	object	How each execution was tied to this page and SOP revision, with a confidence flag for inferred matches.
status	enum	Current, under review, deprecated or archived, plus the page that replaced it.

Example record

{
  "doc_id": "sop_3a7f21",
  "doc_type": "sop",
  "title": "Vendor bank detail change requests",
  "space": { "id": "sp_ap_ops", "parent": "Accounts payable controls" },
  "ownership": { "owner_role": "ap_manager", "approver_role": "controller",
                 "review_cycle_months": 12, "last_reviewed": "2023-09-01" },
  "steps": [
    { "n": 1, "text": "Log the request in the AP queue and attach it as received.", "system": "ticketing" },
    { "n": 2, "text": "Call the vendor back on the number held in the vendor master, not the one in the request.", "role": "ap_specialist" },
    { "n": 3, "text": "If the vendor is paid more than [THRESHOLD] a month, get a second approver.", "branch": true },
    { "n": 4, "text": "Update bank details in the ERP and hold payments for 5 business days.", "system": "erp" }
  ],
  "revisions": [
    { "v": 7, "t": "2022-11-14T10:20:00Z", "editor_role": "ap_manager",
      "summary": "Added callback step after a near-miss" },
    { "v": 9, "t": "2023-09-01T08:05:12Z", "editor_role": "controller",
      "summary": "Hold period raised from 3 to 5 days" }
  ],
  "usage": { "views_90d": 212, "feedback": { "helpful": 14, "not_helpful": 3 } },
  "executions": [
    { "exec_id": "ex_91c4", "sop_version": 9, "outcome": "completed",
      "steps": [
        { "n": 1, "t": "2024-01-08T09:12:00Z", "status": "done" },
        { "n": 2, "t": "2024-01-08T09:40:00Z", "status": "done", "note": "Vendor confirmed new account" },
        { "n": 3, "status": "not_applicable", "reason": "below threshold" },
        { "n": 4, "t": "2024-01-08T10:02:00Z", "status": "done" }
      ] },
    { "exec_id": "ex_a207", "sop_version": 9, "outcome": "escalated_to_fraud_review",
      "steps": [
        { "n": 1, "t": "2024-02-19T13:58:00Z", "status": "done" },
        { "n": 2, "t": "2024-02-19T14:15:00Z", "status": "exception",
          "note": "Number on file not answering; request sent from a lookalike email domain" }
      ] },
    { "exec_id": "ex_c5e0", "sop_version": 9, "outcome": "completed", "qa_flag": "steps_out_of_order",
      "step_order": [1, 4, 2], "note": "ERP updated before the callback; payment hold applied" }
  ],
  "execution_link": { "method": "ticket_field", "confidence": "explicit" },
  "status": "current"
}

Synthetic record for illustration. Field names, structure and format are agreed per order.

What AI teams use it for

Ground agents in company procedure

Retrieval over real SOPs gives an agent the company's own rules, thresholds and exceptions, the parts a general model cannot know.

Imitate how experienced staff run the process

Executions by experienced staff become worked examples of the procedure, including the cases where they stopped and escalated instead of finishing.

Turn procedures into agent workflows

Step lists with roles, systems and branches map onto tool calls and approvals, and execution records show where an automated version would need human review.

Detect stale or conflicting documentation

Revision history plus execution data shows when practice drifted from the written procedure, and pages that contradict each other can be found and labeled.

Evaluate procedure adherence

Executions graded against the SOP version in force at the time make test items for whether an agent follows procedure, deviates where it should and escalates on time.

Use-case guides: Enterprise and computer-use agents, Document AI, enterprise search and RAG, Customer support agents, Legal AI, Healthcare administration AI

What makes this data valuable

Execution pairing

Execution records linked to the exact SOP revision make every step checkable against what happened.

Revision history

Dated changes, often after incidents or audits, show why a procedure reads the way it does.

Explicit structure

Numbered steps, roles, branches and named systems are easier to learn from and to grade against.

Ownership and review

Owners and review dates separate maintained procedures from abandoned pages.

Exceptions on record

Escalations, skipped steps and workarounds capture judgment the SOP leaves implicit.

Usage feedback

Views, search terms and feedback votes show which pages people actually relied on.

What written procedures leave out

Written procedures leave out how the work is actually done. An SOP is a company's statement of how a task should be done, written at one point in time by someone who may no longer do the work. Some steps are followed exactly. Others are skipped once people learn they rarely matter, or are done in a different order because a system makes that easier, and some exceptions were never written down because the experienced staff just know what to do. None of that appears on the page.

An agent trained only on written procedures inherits that gap. It follows steps that people have quietly dropped, misses the checks they added, and has no example of the judgment calls the procedure leaves open. Execution histories close the gap. Linked to the SOP revision in force at the time, they show how often each step actually happens and where variations cluster. With outcomes attached, they also separate harmless shortcuts from deviations that lead to rework, escalation or loss, a distinction no procedure document records. Revision history adds the causal thread, since procedures are often rewritten after an incident or audit, and executions before and after the change show whether the new step held.

How procedures are matched to their executions

Procedures are matched to executions either through a reference recorded by the system that ran the work or by inference, because execution records rarely name the procedure they follow. Explicit pairing is possible when the process runs through a system that references the SOP: a ticket type tied to a runbook, a checklist template generated from the procedure, or a form with one field per step. Elsewhere, executions are matched by process type, timing and the systems involved, and the match carries a confidence level so you can filter on it.

Scoping therefore begins with the process rather than the page count. SourceX looks for partners whose procedures cover the domain you need and whose systems log enough of each run to show what was done. The manifest states which procedures are paired, how the pairing was made, how many years of executions exist and how much of each process the logs capture. Procedures that cannot be paired can still be licensed for grounding and retrieval, and they are labeled as such.

What to check before licensing

Ask how each execution record was matched to an SOP and to the revision in force at the time. An explicit reference in a ticket or form is stronger than a match inferred from timing or keywords.
Measure staleness on the sample by counting pages that have an owner, a review in the last cycle, and no later page that replaces them.
Check how many procedures have execution records at all, and how many executions each one has. A handful of paired SOPs is a different asset from a fully paired library.
Confirm the partner owns the content. Knowledge bases often hold vendor manuals, regulatory text, franchise or licensed operating manuals and consultant-written methods.
Review de-identification in screenshots and embedded images, customer examples, internal URLs, hostnames and credentials pasted into troubleshooting articles.
Check how macros, page includes and embedded diagrams were exported. Content can render empty or be duplicated when a wiki is flattened to text.
Ask whether drafts, personal spaces and archived spaces are in scope, and how they are flagged.

How licensing works through SourceX

1
Define
Send the domain, modality, volume, format, timeline and permitted use you need.
2
Source
SourceX identifies businesses that hold matching data and are open to licensing it.
3
Qualify
Fit, rights and quality are checked, and you review samples before committing.
4
License
Scope, permitted use, exclusivity, price and obligations are agreed in writing.
5
Deliver
Approved data is prepared, de-identified where required and transferred securely.

Questions buyers ask

Why pair an SOP with execution histories?

Because an SOP records how a process is meant to run, and execution histories record how it actually ran. A procedure paired with thousands of real executions shows the process as practiced, including the shortcuts, the exceptions that come up and what experienced staff did about them. For training and evaluating agents, that pairing is far more valuable than the written procedure alone, so SourceX pairs the two wherever a partner holds both.

Where do execution records come from?

From the systems the procedure runs through. Tickets, checklists and task tools record which steps were completed and when; ERP, CRM and ITSM audit logs record actions taken in each system; QA reviews record whether the work met the standard. These are system records reconstructed from business software, not screen recordings. Coverage depends on how much of the process happens in systems that log it.

Which platforms can knowledge bases be exported from?

Most wiki and knowledge-base platforms support export, including Confluence, Notion, SharePoint, Guru, ServiceNow Knowledge, Zendesk Guide and Document360, as well as procedures kept as Google Docs or Word files. Export routes differ in what they keep: some carry page hierarchy, labels, attachments and revision history, flat PDF or HTML exports often keep only the current text, and some platforms do not expose page history through their APIs at all. The method is agreed per dataset to keep as much history as the platform allows.

How do you handle outdated or contradictory pages?

They are labeled rather than silently removed, because stale pages are part of what agents face in production. Revision dates, owners, review cycles and replacement links show which page was in force at which time, and execution records show when practice moved away from the text. You can scope the delivery to current pages only, or keep superseded versions for training models to detect staleness.

Can I license an SOP library without execution data?

Yes. Procedures, checklists and playbooks alone are useful for retrieval, grounding and workflow design, and many partners keep knowledge bases with no matching execution records. Paired data is the stronger asset for training and evaluating agents, so say in the dataset description of your request whether pairing is required or only preferred, and the minimum coverage you would accept.

Can procedures from regulated industries be included?

Sometimes. Procedures in finance, healthcare, insurance and manufacturing often reference regulations, controls and audit findings, and some partners treat them as confidential. Internal control procedures can also reveal how fraud checks work, so they may be generalized or excluded. The partner decides which procedures can be released, and the manifest notes any exclusions.

Enterprise workflow and task execution histories
Linked task trajectories from request to outcome, across every tool the work touched
Enterprise document archives
A company's working files with folders, versions and sharing metadata
IT service management and incident histories
Incidents, problems, changes and requests with work notes, CI links and outcomes
Customer support ticket datasets
Resolved support cases with full threads, internal notes and outcomes
Human feedback and QA-scored work
Work items with scores, verdicts and corrections from the people who reviewed them

Evaluating this data for procurement?

Diligence packets are prepared per dataset. Rights, privacy processing and quality differ between datasets.

Request dataset diligence

Tell us what your models need

Send your spec — domain, volume, format, timeline and permitted use — and SourceX will match it against partner data and come back with what can be licensed.