Leadership and readiness
Portfolio AI data utility pilot: one workflow across five companies
By SourceX Editorial · Reviewed by Noah Loul ·
10 sections
Short answer
A portfolio AI data utility pilot should compare one narrowly defined workflow, such as warranty claims or invoice exceptions, across several companies using a single shared scope: the same start and end events, the same time window, the same outcome codes and the same review sample. It measures whether each company's records are complete, linked and licensable, after confirming which entity controls them. It does not rank companies or predict model gains.
License terms
Portfolio AI data utility
Key takeaways
- Compare one workflow with one written definition across companies; otherwise the pilot measures definitions, not records.
- Trace authority per operating company before scoring: controlling entity, original and inherited terms, and any shared systems.
- Normalize outcomes through a crosswalk while keeping each raw export untouched and auditable.
- Score record fitness, gate status and effort, and never claim model gains or deal value from a records pilot.
- End with a decision per company: proceed to assessment, fix first, or wait on a specific legal answer.
What should a cross-company workflow pilot actually compare?
It should compare the fitness of each company's records for one workflow, not the companies themselves. The question a sponsor or operating partner can answer in a few weeks is narrow: if five portfolio companies all run, say, warranty claims, which of them hold a complete, linkable and rights-clear history of that work, and how much effort would each need before a buyer could use it.
That framing keeps the pilot useful even when the answer is uneven. A company whose claims history is thin is not a failed company; it is a company whose records are not ready yet. The pilot output is a matched scorecard plus a decision per company: include, fix first, or leave out.
The trap is letting each company define the workflow its own way. One counts a claim when the customer calls, another when a technician opens a case, a third when finance books a credit. Without one shared definition, any comparison measures the differences in definitions rather than the records.
Which workflow is narrow enough to match across five companies?
Pick a workflow with a clear trigger, a recorded decision and a checkable outcome that every company in the group performs. If you cannot write the start event and the end event in one sentence each, the workflow is too broad to match.
The examples below are illustrative. They show the kind of test to apply, not a ranking of which workflow is best for any particular group.
Two further filters narrow the choice. The workflow should run at enough volume in every company that a random sample of a few dozen cases per company is a small fraction of the whole, and it should leave a decision trail in the system rather than in email or a person's memory. A workflow that is mostly decided in hallway conversations will score poorly everywhere and teach the group nothing about its records.
Avoid workflows where the records are mostly about individual people in sensitive roles, such as employee discipline or medical leave, for a first pilot. They can be valuable, but the privacy and consent work would dominate the pilot and hide the record-quality questions the group is trying to answer.
| Candidate workflow (illustrative) | Start event | End event and outcome | Match risk to check |
|---|---|---|---|
| Warranty or return claims | Claim record created in the service system | Claim approved, denied or partially credited, with reason | Some companies log claims in a CRM case, others in the ERP |
| Accounts payable invoice exceptions | Invoice fails a match rule in the AP tool | Exception cleared, with resolution type | Match tolerances differ by company and change over time |
| Field service dispatch to completion | Work order created | Job closed with resolution and any revisit | Revisits may be new orders in one system and reopened orders in another |
| Support escalation to engineering | Ticket escalated (for example, Zendesk to Jira link) | Fix shipped or ticket closed as not reproducible | Link fields between help desk and issue tracker are often missing |
| Quote to signed order | Quote issued in the CRM | Order signed, lost or expired | Pricing content may be competitively sensitive across companies |
After an acquisition, which company can say yes for each record source?
Usually the operating company that collected the records and holds the customer, vendor and employee contracts, not the fund or the holding company. Ownership of shares does not by itself transfer the right to use a subsidiary's records for a new purpose, so authority has to be traced record source by record source.
Trace three things for every company in the pilot. First, which legal entity is the contracting party and data controller for the workflow's records today. Second, what the customer terms and privacy notices said when the records were collected, including terms inherited from an acquired business before close. Third, whether any system is still shared with a former parent under a transition services agreement, in which case the records may sit on infrastructure the company does not control.
Changing the terms after the fact does not solve an inherited gap. Federal Trade Commission staff wrote in February 2024 that a company adopting more permissive data practices, such as using data for AI training, and informing users only through a surreptitious, retroactive change to its terms or privacy policy may be acting unfairly or deceptively. For records about people in the EU, the GDPR's Article 6(4) asks whether a new purpose is compatible with the original one, weighing the link between the purposes, the context of collection, the nature of the data, the possible consequences and the safeguards in place.
Treat the authority trace as a gate, not a score. A company whose authority question is open stays in the pilot only as a design reference until counsel answers it.
What goes in the matched scope and cross-company evaluation brief?
The brief is one shared document, signed off by each company's workflow owner, that fixes every definition before anyone pulls a record. Fill it once for the group, then attach one completed column per company. The rows below are the minimum; a team can use them as is.
Version the brief and date every change. If a company discovers mid-pilot that its claims system was migrated partway through the window, record the change, apply it to every company's window where relevant, and note which samples were drawn under which version. A comparison is only as fair as the moment its definitions froze.
Keep the group copy free of company records. The brief holds definitions, counts and decisions; the records and the review samples stay inside each company's own environment, reviewed by people that company authorizes.
| Brief field | What to write | Why it must match | Who confirms |
|---|---|---|---|
| Workflow definition | Start event, end event and the decision in between, in one sentence each | Different triggers produce different case counts for the same work | Group operating lead with each workflow owner |
| Time window | Same calendar window for all companies, plus any system migration dates inside it | History before a migration may be partial or re-keyed | Each company's system owner |
| Unit of record | One case equals one claim, invoice, order or ticket, with its unique ID field named | Merged or split cases distort completeness rates | Each company's system owner |
| Common outcome codes | A crosswalk from each company's status values to four to eight shared outcomes | Raw statuses are not comparable across tools | Workflow owners, reviewed by the pilot lead |
| Linked evidence | Which notes, attachments, line items and follow-up records must link by ID | Linkage is what turns a log into a usable example | Each company's system owner |
| Authority and consent | Controlling entity, governing customer terms, inherited terms, shared systems | A record without clear authority cannot enter any dataset | Each company's counsel |
| Exclusions | Fields and record types removed before review, such as payment data or health details | Exclusions must be equal or the comparison is skewed | Privacy lead |
| Review sample | Sample size per company, how it is drawn and who reviews it | Equal, randomly drawn samples keep scores comparable | Pilot lead |
| Information barrier | Which fields the group team may see, and which stay inside each company | Prevents sharing of pricing or customer terms between companies that compete | Group counsel |
| Decision rule | What score or gate result leads to include, fix first or leave out | Agreeing the rule first stops post hoc ranking | Sponsor and operating partner |
How do you keep the comparison fair when every company runs different software?
Normalize the outcome, not the source data. Keep each company's raw export untouched, then build a crosswalk table that maps its native statuses and reason codes to the shared outcomes in the brief. A Salesforce Service Cloud case closed as Resolved, a Zendesk ticket marked Solved and an internal tool's code 40 may all map to the same shared outcome, but the raw values stay in the record so a reviewer can audit the mapping.
Document the mapping the way a dataset would be documented for a buyer. Datasheets for Datasets, a widely cited proposal by Gebru and colleagues, suggests that every dataset be accompanied by a datasheet recording its motivation, composition, collection process and recommended uses. The MLCommons Croissant responsible-AI vocabulary offers machine-readable fields for the same idea, including rai:dataCollection for how records were gathered and rai:dataLimitations for known limits on generalization. Writing one per company as part of the pilot makes the later licensing step faster.
Equal sampling matters as much as equal definitions. Draw the review sample at random from the same window in every company, and stratify by outcome so that rare outcomes, such as partial credits or reopened jobs, are not missing from a small company's sample.
Which measurements belong on the cross-company scorecard?
Measure record fitness, gate status and effort; do not measure revenue potential or model accuracy. The NIST AI Risk Management Framework's MEASURE function calls for documenting the test sets, metrics and details about the tools used during testing and evaluation, and its playbook says such documentation supports repeatability and consistency. Apply the same discipline here: every metric below needs a written definition and the query or script that produced it.
Report the scorecard as one column per company with the gate result at the top. A company that fails the authority gate shows its quality metrics greyed out or omitted, so nobody reads a strong completeness figure as permission to proceed. Resist adding a weighted total; a single number invites a ranking the pilot was not designed to support.
| Metric | Definition | How to compute | What it cannot tell you |
|---|---|---|---|
| Outcome completeness | Share of cases in the window with a non-empty shared outcome | Count of cases with mapped outcome divided by all cases | Whether the recorded outcome was correct |
| Evidence linkage | Share of sampled cases whose notes, attachments and line items open from the case ID | Reviewer check on the random sample | Whether the linked content is usable after redaction |
| History depth | Months of continuous history without a migration gap | Earliest to latest case date, minus gaps longer than one month | Whether older cases follow today's process |
| Reviewer agreement | Share of sampled cases where two reviewers assign the same shared outcome | Double-review a subset and compare | Whether the shared outcome codes are the right ones |
| Exclusion load | Share of sampled cases needing redaction or removal | Reviewer tally against the brief's exclusion list | The legal adequacy of the redaction |
| Authority gate | Pass, open or fail from the authority trace | Counsel's written answer per company | Anything about record quality |
| Preparation effort | Estimated person-days to export, map and review the full window | Workflow owner estimate, checked against pilot time spent | The value of the finished dataset |
How long should the pilot run, and who does the work?
Plan for a short, fixed timebox with named people in each company, because a cross-company pilot stalls when it depends on goodwill. A workable shape is a design phase in which the brief is agreed and signed, a pull-and-sample phase run inside each company, and a review phase in which the group sees only definitions, counts and decisions.
Each company needs three roles: a workflow owner who knows how the work is really done, a system owner who can export and explain the fields, and a reviewer who reads the sample. The group needs a pilot lead who owns the brief and the crosswalk, and access to counsel for the authority gate. If a company cannot name all three people, record that as its first finding; it predicts how hard a full preparation would be.
Budget the effort in person-days, not weeks of elapsed time, and record actual time spent per company. The gap between estimated and actual effort is one of the most useful outputs, because it tells the sponsor which companies' records will be expensive to prepare at full scale.
Worked example: warranty claims at five companies (illustrative)
This example is hypothetical and its figures are illustrative counts and percentages, not results from any real portfolio. A sponsor holds five industrial products companies, A to E, that all handle warranty claims. The brief defines a claim as starting when a claim record is created and ending when it is approved, denied or partially credited, over the same 36-month window, with six shared outcome codes.
Companies A and B run claims as cases in the same CRM and pass every gate; their illustrative outcome completeness is above 90 percent and linkage of photos and parts lines is high. Company C logs claims as ERP credit memos with free-text reasons, so its completeness looks high but reviewer agreement on the shared outcome is low; its decision is fix first, starting with the reason-code crosswalk.
Company D is the exception. It was carved out of a larger group two years ago and its claims system still runs under a transition services agreement on the former parent's tenant. The records may be perfectly good, but the authority gate is open until counsel confirms who controls that tenant's data, so D stays out of the comparison. Company E has an EU subsidiary whose claims include consumer contact details; the pilot keeps the EU records out of the sample until an Article 6(4) compatibility review is done, and scores only the U.S. records.
The group output is three decisions, not a league table: A and B proceed to a licensing assessment, C fixes its outcome codes and reruns the sample in a quarter, and D and E wait on specific legal answers.
When does the pilot need to pause or go to counsel?
Pause the moment the comparison would move competitively sensitive information between companies, or when any company's authority to use its records is unresolved. Portfolio companies that sell into the same market can be competitors, and group-level comparison of pricing, margins or customer terms raises information-sharing questions. When the Justice Department withdrew three health care antitrust policy statements in February 2023, it described them as overly permissive on certain subjects, such as information sharing; older safe-harbor assumptions about exchanging data are not a substitute for advice.
Bring in specialist review in these specific cases:
- The workflow's records carry prices, discounts or customer-specific terms and two or more companies compete for the same customers.
- An acquired company's pre-close terms are missing, conflicting or promised customers their data would not be shared.
- Any system is shared with a former parent, a co-investor's company or a vendor under a transition services agreement.
- Records include health, financial account, biometric or children's information, or come from EU or UK data subjects.
- A company's leadership has not agreed to the pilot and is being asked to export records on the sponsor's instruction alone.
From a matched pilot to a licensing assessment
A completed brief and scorecard are the inputs a licensing conversation needs, because they already answer which records exist, who controls them and what preparation remains. That is also the ground SourceX covers for an individual company, from qualification and rights review through contracting, preparation, delivery and payment of a licensed dataset built from its selected records.
Each operating company would take its own first step through a short company assessment, which needs no data upload. Nothing would be delivered until that company approves the scope and terms and signs the agreement; the group-level pilot never substitutes for each company's own decision.
Frequently asked questions
What is a portfolio AI data utility pilot?
It is a short, structured test of whether several portfolio companies hold usable, rights-clear records for one shared workflow. It compares record completeness, linkage, history and authority under one definition. It does not build a model or estimate deal value.
How many companies should a cross-company data pilot include?
Enough to make the shared definitions honest, usually three to five companies that genuinely run the same workflow. More companies add mapping work faster than they add insight. Companies that do not run the workflow should be left out rather than forced into the scope.
Can the fund decide to license a portfolio company's records?
Usually the operating company that collected the records and signed the customer contracts holds the authority, and its own board and counsel decide. The fund can sponsor and coordinate a pilot, but each company's records, terms and consents have to be traced separately.
Should the pilot measure model accuracy?
No. A data utility pilot measures whether records are complete, linked, reviewable and licensable. Model results depend on the buyer's task and training setup, and claiming them from a records pilot would overstate what was tested.
What if one company uses completely different software?
Keep its raw export as it is and map its statuses to the shared outcome codes in a crosswalk table. If reviewers cannot agree on the mapping for a meaningful share of the sample, mark the company as fix first rather than forcing a score.
Sources
- The NIST AI RMF MEASURE function (2.1) calls for documenting the test sets, metrics and details about the tools used during test, evaluation, verification and validation. Source
- The NIST AI RMF Playbook says documenting measurement approaches, test sets and metrics enables repeatability and consistency. Source
- Datasheets for Datasets proposes that every dataset be accompanied by a datasheet documenting its motivation, composition, collection process and recommended uses. Source
- The Croissant responsible-AI specification defines fields such as rai:dataCollection and rai:dataLimitations for documenting how data was collected and its generalization limits. Source
- FTC staff wrote in February 2024 that adopting more permissive data practices, such as AI training, through a surreptitious, retroactive change to terms or a privacy policy may be unfair or deceptive. Source
- GDPR Article 6(4) lists factors for judging whether further processing is compatible with the original purpose, including the link between purposes, the context of collection, the nature of the data, possible consequences and safeguards. Source
- The Justice Department withdrew three antitrust health care policy statements on February 3, 2023, describing them as overly permissive on certain subjects such as information sharing. Source
See if your company qualifies
A short company assessment. No data uploads are needed.