Skip to content

AI uses for records

Business diagram reasoning dataset: turn flowcharts into agent tasks

By SourceX Editorial · Updated

9 sections

Short answer

A business diagram reasoning dataset pairs each flowchart with a real case, the records needed to apply it and a verified outcome. Each task names the diagram version and the exact node, edge and image region the answer depends on, the linked records, the action an agent should take and the later record that shows what happened, so a model is graded on its decision path rather than a plausible final message.

Agent run

Business diagram reasoning

This page covers business diagram reasoning dataset: turn flowcharts into agent tasks.Illustrative

Key takeaways

  • A flowchart is a policy; a task adds one real case, its evidence and a verified outcome.
  • Every decision on the path needs a region on the page image and a record field that answers it.
  • A status field shows someone decided; a later business record shows the decision was right.
  • Paired counterexamples that differ in one deciding fact test whether a model follows the diagram.
  • Label cases where practice deviates from the diagram instead of training on them as policy.
  • Split by diagram family and company so a model cannot memorize a diagram.

How can business flowcharts become grounded agent action tasks?

A flowchart becomes an agent task when you attach a case to it. On its own, a process diagram is a policy: it says what should happen in general. A task adds a starting state for one real case, the records that answer each decision on the path, the action the agent must take and a later record that shows the action was right.

Public research has made progress on the reading part. FlowVQA, presented at ACL 2024 Findings, contains 2,272 flowchart images and 22,413 question-answer pairs testing information localization, decision-making and logical progression; its flowcharts were generated as Mermaid scripts from process articles on sites such as WikiHow and Instructables and from code. The Follow the Flow authors found that vision-language models hallucinate nonexistent connections and decision paths, and they defined flowchart attribution as identifying the path in a diagram that grounds a model's response.

Business flowcharts add what those benchmarks do not have: the records the diagram governs. An agent that reads a returns policy diagram correctly but checks the wrong delivery date still issues the wrong refund decision. The task pack below is built so each step on the path points to evidence a grader can check.

Where business process diagrams live and what each format gives you

The format decides how much of the diagram you can extract as structure and how much you must recover from pixels. A BPMN file from a process engine already names every gateway and flow; a scanned procedure binder gives you an image and nothing else.

In BPMN 2.0.2, conditions sit on the sequence flows that leave a gateway, and an exclusive or inclusive gateway can name a default flow as an attribute. That maps cleanly to an agent task: each condition is a question the case records must answer, and the default flow is the action when none of the conditions holds. In Mermaid, decision nodes are drawn as rhombus or diamond shapes and edges can carry text labels, which gives the same structure in a lighter text syntax.

Where business process diagrams live and what each format gives you
Source formatWhat is machine-readableWhat you must still extractMain risk
BPMN XML from a process engine or modelerGateways, sequence flows, conditions, lanesWhich records answer each conditionExecutable model differs from what staff actually do
Mermaid or other diagram-as-code textNodes, edges, edge labels, shapesVersion history and the cases it applied toLabels written as shorthand only the author understands
Visio, Lucidchart or draw.io exportsShapes and connectors, often with textWhich connector belongs to which decisionConnectors that touch visually but are not attached
Slides or PDFs in a policy binderText layer at bestNode boxes, arrows, reading order, footnotesArrows lost in rendering; footnotes that override the path
Scans or phone photos of printed chartsNothing until OCR and layout detection runEvery node, edge and label from pixelsLow OCR confidence on small labels and handwritten edits

What should the diagram grounded workflow task pack include?

Each task is one row that a grader can replay without asking the author anything. Store the fields below for every task, and keep the diagram, case records and outcome record as separate files referenced by ID so the same diagram can serve many cases.

The example values in the table are illustrative. Field names are suggestions; what matters is that every decision on the path has an evidence pointer and every outcome has a record.

What should the diagram grounded workflow task pack include?
FieldWhat it holdsIllustrative valueHow it is verified
Diagram referenceFile, version and effective datesReturns policy chart v3, effective from MarchVersion matches the date of the case
RegionPage and bounding box of each node and edge usedPage 2, box around the damage decisionBox drawn on the page image, not on a re-render
Node pathOrdered node and edge IDs from start to actionStart, window check, damage check, issue RMAEvery edge exists in the diagram
Starting stateWhat the agent knows at the first stepOrder closed, customer reports a cracked itemTaken from the case record at that time
Linked evidenceRecord and field that answers each decisionCarrier delivery scan date; photo attachmentField value checked against source system
Expected actionWhat the agent should do and with which valuesCreate RMA with reason code damaged in transitMatches the path and the records
Outcome recordLater record showing what happenedWarehouse receipt with inspection noteDated after the action; outcome field filled
CounterexampleA paired case that takes another branchSame item, delivered outside the windowDiffers only in the deciding fact
OCR and layout qualityConfidence and any manual correctionsTwo labels corrected by a reviewerCorrection log kept with the task
Provenance and split keyReal or illustrative, company and diagram familyReal case, company A, returns familyUsed to keep families on one side of the split

Which image, table or attachment holds the evidence the answer depends on?

Point to the smallest region that contains the deciding evidence, in the file the grader will open. For diagrams that means the node or edge on the page image; for case evidence it means the field in the record or the region in an attachment, such as the date in a delivery photo or the row in a rate table.

At a glance

Which image, table or attachment holds the evidence the answer depends on

Use coordinates a tool can check. The hOCR specification defines a bounding box by its upper-left and lower-right corners, in pixels measured from the top-left corner of the document image, and its x_wconf property carries an OCR-engine-specific confidence where higher values mean higher confidence. Because that confidence is engine-specific, record which engine produced it and do not compare values across engines.

Layout detection decides whether a footnote, legend or side table is read with the diagram. DocLayNet, a human-annotated layout dataset of 80,863 pages from financial, scientific, patent, tender, legal and manual documents with 11 class labels, ships predefined train, validation and test sets chosen to avoid leaking unique layout styles across sets. The DocVQA authors found that models lag far behind humans on document questions that require understanding structure. Both point to the same practice: keep layout and region information with the task, not only the extracted text.

  • Source file: the original export, not a screenshot of it.
  • Page and region: page number and bounding box for every node, edge and attachment region used.
  • Layout: which blocks belong to the diagram, the legend and any side table.
  • Table headers: the header row for any table a decision reads, so a value is not read from the wrong column.
  • Footnotes: notes that change a branch, such as a regional exception.
  • Version: the diagram version and its effective dates.
  • OCR confidence: engine, confidence per label and every manual correction.
  • Cross-file links: IDs that join the diagram to the case, the attachment and the outcome record.

Status fields versus proof: which record shows the agent chose correctly?

The proof is a record created by the business process after the decision, not the status field the agent or a person set at the time. A ticket marked approved shows that someone decided; it does not show the decision was right. The return receipt, the credit memo, the closed claim or the absence of a reopened case does.

Separate three things in every task: the decision the diagram requires given the evidence, the action taken in the system, and the outcome that followed. A model can be graded on the first two immediately. The third needs time to mature, so record when the outcome record was checked and how long after the action it appeared.

Status fields versus proof: which record shows the agent chose correctly?
DecisionWeak evidenceProof recordWhen it matures
Approve a returnTicket status set to approved in ZendeskWarehouse receipt and inspection note in the ERPWhen the item arrives and is inspected
Escalate a service jobAgent note saying escalatedSupervisor assignment and the follow-up visit in ServiceTitanWhen the next visit is closed
Reject a vendor invoiceException code set in the AP queueVendor credit note or corrected invoice in NetSuiteWhen the vendor responds
Route a change requestJira issue moved to a review columnApproval record and the deployed changeWhen the change ships or is withdrawn

Worked case: a returns flowchart applied to one order

This case is illustrative; the company, records and durations are invented to show the structure. A retailer's returns policy diagram, version 3, has a start node, a decision asking whether the claim falls inside the return window, a decision asking whether the item arrived damaged, and actions for issuing a return authorization, opening a carrier claim and escalating to a supervisor, which is the default flow.

Starting state: a customer writes 12 days after delivery that a lamp arrived cracked and attaches a photo. Linked evidence: the order record, the carrier delivery scan date and the photo. The window decision reads the policy footnote on page 2, which sets a 30-day window for this product class, and the scan date, so the case is inside the window. The damage decision reads the photo and the customer message. Expected action: issue a return authorization with the reason code for damage in transit and open a carrier claim.

Outcome: a warehouse receipt dated 9 days after the authorization carries an inspection note confirming the crack, and the carrier claim closes as accepted. The task stores the node path, the region boxes for both decisions and the footnote, the three evidence records and the two outcome records.

Counterexample: the same lamp, the same photo, but the delivery scan is 41 days old. The window condition fails, the damage branch is never reached and the default flow sends the case to a supervisor. A model that writes a friendly approval message because the photo shows damage fails this task even though the message reads well. Pairing the two cases tests whether the model follows the diagram or the most persuasive evidence.

When the diagram and actual practice disagree

Label the disagreement instead of picking a winner silently. Real records often show staff taking a path the diagram does not allow, because the diagram is out of date, because an unwritten exception exists, or because people made mistakes.

Keep three labels: follows the diagram, follows a documented exception, and deviates without a documented reason. Train or grade against the diagram only for the first two. Cases in the third group are useful evaluation material for detecting non-compliance, but they should not teach an agent that the deviation is the policy.

  • Check the diagram version against the case date before labeling a deviation.
  • Look for footnotes, legends and side tables that add exceptions the boxes do not show.
  • Ask the process owner about unwritten rules when one deviation repeats across many cases.
  • Stop and narrow the pack to cases with a matching diagram version when version history is missing.
  • Send cases involving regulated decisions, such as credit or employment, to a domain reviewer before they enter a benchmark.

How to score a model on path, region, action and outcome

Score each layer separately so a failure tells you where the model went wrong. A single pass or fail on the final message hides whether the model misread the diagram, cited the wrong evidence or chose the wrong action.

Split by diagram family and by company, not by task. If the same diagram appears in training and evaluation with different cases, a model can learn the diagram rather than learn to read diagrams.

How to score a model on path, region, action and outcome
LayerWhat is comparedPass condition
PathModel's node and edge sequence against the gold pathSame sequence to the action node
RegionCited region against the gold region for each decisionCited box overlaps the gold box for every decision
EvidenceRecord and field the model used for each decisionSame record and field, or an agreed equivalent
ActionSystem action and its valuesSame action type and values as expected
Outcome agreementExpected action against the matured outcome recordOutcome record confirms the action

Requesting process diagrams together with the records they govern

The value of this kind of dataset comes from the joins: the diagram version in force, the case records, the attachments and the outcome records, linked by stable IDs. A request that asks for diagrams alone produces a reading benchmark; a request that asks for linked cases produces an action benchmark.

Requests to SourceX for linked diagram and case records are filled by sourcing from U.S. businesses at the time of the request, with scope, volume, format, timing and licensing needs captured on the form. Since SourceX also manages rights review, preparation and delivery with the business supplying the records, linkage fields and outcome records can be named in the scope before any preparation starts.

Frequently asked questions

What is a business diagram reasoning dataset?

It is a set of tasks in which a model must read a business process diagram and apply it to a real case. Each task links the diagram regions, the case records and a later outcome record. It tests whether a model can follow a documented process with evidence, not just describe a chart.

Can synthetic flowcharts replace real ones?

They help with reading skills. FlowVQA, for example, generated its flowcharts as Mermaid scripts from public process articles and code. Synthetic charts do not come with the case records, outdated versions, footnote exceptions and practice deviations that make business tasks hard, so evaluation should include real diagrams with linked cases.

Do I need BPMN files, or are diagram images enough?

Images are enough if you extract nodes, edges and labels and keep region coordinates for each. BPMN or diagram-as-code files save that work and remove ambiguity about which arrow connects to which decision. When both exist, keep both and check that they match.

How should OCR errors in diagram labels be handled?

Record the OCR engine, the confidence per label and every manual correction with the task. Correct labels that change a decision before the task is used, and keep the uncorrected text if you also want to test robustness to OCR noise. Never compare confidence values across different OCR engines.

How should diagram tasks be split between training and evaluation?

Split by diagram family and by company so the same diagram never appears on both sides. Keep paired counterexamples on the same side as their original case. Record the split key in each task so the split can be checked later.

Sources

  • The BPMN 2.0.2 specification places condition expressions on sequence flows leaving gateways and defines a default flow as an attribute of exclusive and inclusive gateways. Source
  • FlowVQA contains 2,272 flowchart images and 22,413 question-answer pairs covering information localization, decision-making and logical progression, with flowcharts generated as Mermaid scripts from WikiHow, Instructables and code. Source
  • The Follow the Flow authors report that vision-language models hallucinate nonexistent connections and decision paths in flowcharts and define flowchart attribution as identifying the path that grounds a response. Source
  • The DocVQA authors report that models lag far behind humans on document questions that require understanding document structure. Source
  • DocLayNet is a human-annotated layout dataset of 80,863 pages with 11 class labels and predefined splits chosen to avoid leaking unique layout styles across sets. Source
  • Mermaid flowchart syntax draws decisions as rhombus or diamond nodes and allows text labels on links between nodes. Source
  • The hOCR 1.2 specification defines bbox by upper-left and lower-right corners in pixels from the image's top-left, and x_wconf as an engine-specific confidence where higher means more confident. Source

Tell us what data you need

Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.

See if your company qualifies