AI uses for records
Business diagram reasoning dataset: turn flowcharts into agent tasks
By SourceX Editorial · Updated
9 sections
Short answer
A business diagram reasoning dataset pairs each flowchart with a real case, the records needed to apply it and a verified outcome. Each task names the diagram version and the exact node, edge and image region the answer depends on, the linked records, the action an agent should take and the later record that shows what happened, so a model is graded on its decision path rather than a plausible final message.
Agent run
Business diagram reasoning
Key takeaways
- A flowchart is a policy; a task adds one real case, its evidence and a verified outcome.
- Every decision on the path needs a region on the page image and a record field that answers it.
- A status field shows someone decided; a later business record shows the decision was right.
- Paired counterexamples that differ in one deciding fact test whether a model follows the diagram.
- Label cases where practice deviates from the diagram instead of training on them as policy.
- Split by diagram family and company so a model cannot memorize a diagram.
How can business flowcharts become grounded agent action tasks?
A flowchart becomes an agent task when you attach a case to it. On its own, a process diagram is a policy: it says what should happen in general. A task adds a starting state for one real case, the records that answer each decision on the path, the action the agent must take and a later record that shows the action was right.
Public research has made progress on the reading part. FlowVQA, presented at ACL 2024 Findings, contains 2,272 flowchart images and 22,413 question-answer pairs testing information localization, decision-making and logical progression; its flowcharts were generated as Mermaid scripts from process articles on sites such as WikiHow and Instructables and from code. The Follow the Flow authors found that vision-language models hallucinate nonexistent connections and decision paths, and they defined flowchart attribution as identifying the path in a diagram that grounds a model's response.
Business flowcharts add what those benchmarks do not have: the records the diagram governs. An agent that reads a returns policy diagram correctly but checks the wrong delivery date still issues the wrong refund decision. The task pack below is built so each step on the path points to evidence a grader can check.
Where business process diagrams live and what each format gives you
The format decides how much of the diagram you can extract as structure and how much you must recover from pixels. A BPMN file from a process engine already names every gateway and flow; a scanned procedure binder gives you an image and nothing else.
In BPMN 2.0.2, conditions sit on the sequence flows that leave a gateway, and an exclusive or inclusive gateway can name a default flow as an attribute. That maps cleanly to an agent task: each condition is a question the case records must answer, and the default flow is the action when none of the conditions holds. In Mermaid, decision nodes are drawn as rhombus or diamond shapes and edges can carry text labels, which gives the same structure in a lighter text syntax.
| Source format | What is machine-readable | What you must still extract | Main risk |
|---|---|---|---|
| BPMN XML from a process engine or modeler | Gateways, sequence flows, conditions, lanes | Which records answer each condition | Executable model differs from what staff actually do |
| Mermaid or other diagram-as-code text | Nodes, edges, edge labels, shapes | Version history and the cases it applied to | Labels written as shorthand only the author understands |
| Visio, Lucidchart or draw.io exports | Shapes and connectors, often with text | Which connector belongs to which decision | Connectors that touch visually but are not attached |
| Slides or PDFs in a policy binder | Text layer at best | Node boxes, arrows, reading order, footnotes | Arrows lost in rendering; footnotes that override the path |
| Scans or phone photos of printed charts | Nothing until OCR and layout detection run | Every node, edge and label from pixels | Low OCR confidence on small labels and handwritten edits |
What should the diagram grounded workflow task pack include?
Each task is one row that a grader can replay without asking the author anything. Store the fields below for every task, and keep the diagram, case records and outcome record as separate files referenced by ID so the same diagram can serve many cases.
The example values in the table are illustrative. Field names are suggestions; what matters is that every decision on the path has an evidence pointer and every outcome has a record.
| Field | What it holds | Illustrative value | How it is verified |
|---|---|---|---|
| Diagram reference | File, version and effective dates | Returns policy chart v3, effective from March | Version matches the date of the case |
| Region | Page and bounding box of each node and edge used | Page 2, box around the damage decision | Box drawn on the page image, not on a re-render |
| Node path | Ordered node and edge IDs from start to action | Start, window check, damage check, issue RMA | Every edge exists in the diagram |
| Starting state | What the agent knows at the first step | Order closed, customer reports a cracked item | Taken from the case record at that time |
| Linked evidence | Record and field that answers each decision | Carrier delivery scan date; photo attachment | Field value checked against source system |
| Expected action | What the agent should do and with which values | Create RMA with reason code damaged in transit | Matches the path and the records |
| Outcome record | Later record showing what happened | Warehouse receipt with inspection note | Dated after the action; outcome field filled |
| Counterexample | A paired case that takes another branch | Same item, delivered outside the window | Differs only in the deciding fact |
| OCR and layout quality | Confidence and any manual corrections | Two labels corrected by a reviewer | Correction log kept with the task |
| Provenance and split key | Real or illustrative, company and diagram family | Real case, company A, returns family | Used to keep families on one side of the split |
Which image, table or attachment holds the evidence the answer depends on?
Point to the smallest region that contains the deciding evidence, in the file the grader will open. For diagrams that means the node or edge on the page image; for case evidence it means the field in the record or the region in an attachment, such as the date in a delivery photo or the row in a rate table.
At a glance
Which image, table or attachment holds the evidence the answer depends on
Use coordinates a tool can check. The hOCR specification defines a bounding box by its upper-left and lower-right corners, in pixels measured from the top-left corner of the document image, and its x_wconf property carries an OCR-engine-specific confidence where higher values mean higher confidence. Because that confidence is engine-specific, record which engine produced it and do not compare values across engines.
Layout detection decides whether a footnote, legend or side table is read with the diagram. DocLayNet, a human-annotated layout dataset of 80,863 pages from financial, scientific, patent, tender, legal and manual documents with 11 class labels, ships predefined train, validation and test sets chosen to avoid leaking unique layout styles across sets. The DocVQA authors found that models lag far behind humans on document questions that require understanding structure. Both point to the same practice: keep layout and region information with the task, not only the extracted text.
- Source file: the original export, not a screenshot of it.
- Page and region: page number and bounding box for every node, edge and attachment region used.
- Layout: which blocks belong to the diagram, the legend and any side table.
- Table headers: the header row for any table a decision reads, so a value is not read from the wrong column.
- Footnotes: notes that change a branch, such as a regional exception.
- Version: the diagram version and its effective dates.
- OCR confidence: engine, confidence per label and every manual correction.
- Cross-file links: IDs that join the diagram to the case, the attachment and the outcome record.
Status fields versus proof: which record shows the agent chose correctly?
The proof is a record created by the business process after the decision, not the status field the agent or a person set at the time. A ticket marked approved shows that someone decided; it does not show the decision was right. The return receipt, the credit memo, the closed claim or the absence of a reopened case does.
Separate three things in every task: the decision the diagram requires given the evidence, the action taken in the system, and the outcome that followed. A model can be graded on the first two immediately. The third needs time to mature, so record when the outcome record was checked and how long after the action it appeared.
| Decision | Weak evidence | Proof record | When it matures |
|---|---|---|---|
| Approve a return | Ticket status set to approved in Zendesk | Warehouse receipt and inspection note in the ERP | When the item arrives and is inspected |
| Escalate a service job | Agent note saying escalated | Supervisor assignment and the follow-up visit in ServiceTitan | When the next visit is closed |
| Reject a vendor invoice | Exception code set in the AP queue | Vendor credit note or corrected invoice in NetSuite | When the vendor responds |
| Route a change request | Jira issue moved to a review column | Approval record and the deployed change | When the change ships or is withdrawn |
Worked case: a returns flowchart applied to one order
This case is illustrative; the company, records and durations are invented to show the structure. A retailer's returns policy diagram, version 3, has a start node, a decision asking whether the claim falls inside the return window, a decision asking whether the item arrived damaged, and actions for issuing a return authorization, opening a carrier claim and escalating to a supervisor, which is the default flow.
Starting state: a customer writes 12 days after delivery that a lamp arrived cracked and attaches a photo. Linked evidence: the order record, the carrier delivery scan date and the photo. The window decision reads the policy footnote on page 2, which sets a 30-day window for this product class, and the scan date, so the case is inside the window. The damage decision reads the photo and the customer message. Expected action: issue a return authorization with the reason code for damage in transit and open a carrier claim.
Outcome: a warehouse receipt dated 9 days after the authorization carries an inspection note confirming the crack, and the carrier claim closes as accepted. The task stores the node path, the region boxes for both decisions and the footnote, the three evidence records and the two outcome records.
Counterexample: the same lamp, the same photo, but the delivery scan is 41 days old. The window condition fails, the damage branch is never reached and the default flow sends the case to a supervisor. A model that writes a friendly approval message because the photo shows damage fails this task even though the message reads well. Pairing the two cases tests whether the model follows the diagram or the most persuasive evidence.
When the diagram and actual practice disagree
Label the disagreement instead of picking a winner silently. Real records often show staff taking a path the diagram does not allow, because the diagram is out of date, because an unwritten exception exists, or because people made mistakes.
Keep three labels: follows the diagram, follows a documented exception, and deviates without a documented reason. Train or grade against the diagram only for the first two. Cases in the third group are useful evaluation material for detecting non-compliance, but they should not teach an agent that the deviation is the policy.
- Check the diagram version against the case date before labeling a deviation.
- Look for footnotes, legends and side tables that add exceptions the boxes do not show.
- Ask the process owner about unwritten rules when one deviation repeats across many cases.
- Stop and narrow the pack to cases with a matching diagram version when version history is missing.
- Send cases involving regulated decisions, such as credit or employment, to a domain reviewer before they enter a benchmark.
How to score a model on path, region, action and outcome
Score each layer separately so a failure tells you where the model went wrong. A single pass or fail on the final message hides whether the model misread the diagram, cited the wrong evidence or chose the wrong action.
Split by diagram family and by company, not by task. If the same diagram appears in training and evaluation with different cases, a model can learn the diagram rather than learn to read diagrams.
| Layer | What is compared | Pass condition |
|---|---|---|
| Path | Model's node and edge sequence against the gold path | Same sequence to the action node |
| Region | Cited region against the gold region for each decision | Cited box overlaps the gold box for every decision |
| Evidence | Record and field the model used for each decision | Same record and field, or an agreed equivalent |
| Action | System action and its values | Same action type and values as expected |
| Outcome agreement | Expected action against the matured outcome record | Outcome record confirms the action |
Requesting process diagrams together with the records they govern
The value of this kind of dataset comes from the joins: the diagram version in force, the case records, the attachments and the outcome records, linked by stable IDs. A request that asks for diagrams alone produces a reading benchmark; a request that asks for linked cases produces an action benchmark.
Requests to SourceX for linked diagram and case records are filled by sourcing from U.S. businesses at the time of the request, with scope, volume, format, timing and licensing needs captured on the form. Since SourceX also manages rights review, preparation and delivery with the business supplying the records, linkage fields and outcome records can be named in the scope before any preparation starts.
Frequently asked questions
What is a business diagram reasoning dataset?
It is a set of tasks in which a model must read a business process diagram and apply it to a real case. Each task links the diagram regions, the case records and a later outcome record. It tests whether a model can follow a documented process with evidence, not just describe a chart.
Can synthetic flowcharts replace real ones?
They help with reading skills. FlowVQA, for example, generated its flowcharts as Mermaid scripts from public process articles and code. Synthetic charts do not come with the case records, outdated versions, footnote exceptions and practice deviations that make business tasks hard, so evaluation should include real diagrams with linked cases.
Do I need BPMN files, or are diagram images enough?
Images are enough if you extract nodes, edges and labels and keep region coordinates for each. BPMN or diagram-as-code files save that work and remove ambiguity about which arrow connects to which decision. When both exist, keep both and check that they match.
How should OCR errors in diagram labels be handled?
Record the OCR engine, the confidence per label and every manual correction with the task. Correct labels that change a decision before the task is used, and keep the uncorrected text if you also want to test robustness to OCR noise. Never compare confidence values across different OCR engines.
How should diagram tasks be split between training and evaluation?
Split by diagram family and by company so the same diagram never appears on both sides. Keep paired counterexamples on the same side as their original case. Record the split key in each task so the split can be checked later.
Sources
- The BPMN 2.0.2 specification places condition expressions on sequence flows leaving gateways and defines a default flow as an attribute of exclusive and inclusive gateways. Source
- FlowVQA contains 2,272 flowchart images and 22,413 question-answer pairs covering information localization, decision-making and logical progression, with flowcharts generated as Mermaid scripts from WikiHow, Instructables and code. Source
- The Follow the Flow authors report that vision-language models hallucinate nonexistent connections and decision paths in flowcharts and define flowchart attribution as identifying the path that grounds a response. Source
- The DocVQA authors report that models lag far behind humans on document questions that require understanding document structure. Source
- DocLayNet is a human-annotated layout dataset of 80,863 pages with 11 class labels and predefined splits chosen to avoid leaking unique layout styles across sets. Source
- Mermaid flowchart syntax draws decisions as rhombus or diamond nodes and allows text labels on links between nodes. Source
- The hOCR 1.2 specification defines bbox by upper-left and lower-right corners in pixels from the image's top-left, and x_wconf as an engine-specific confidence where higher means more confident. Source
Tell us what data you need
Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.