Data preparation tools for inventorying business records before an AI licensing review

What are data preparation tools, and which ones help with a licensing inventory?#

Data preparation tools are software that extract, inspect, clean, describe and reshape records so another person or system can use them. For a company getting ready for an AI licensing review, the useful subset is narrower: tools that help you show what systems you run, which record categories each one holds, how far back the history goes, how many records there are, which fields are sensitive, and how a clean copy would be produced.

That is a different job from building analytics dashboards or training pipelines. A licensing reviewer is not asking whether your data is ready for a model. They are asking what exists, who controls it, what it contains and whether the company may license it. Tools help with the first three questions. The fourth is a legal and commercial decision.

This guide sorts data preparation tools by class, not by brand. Product names appear only as examples of a class, described from their own documentation or official repositories. Nothing here is a product test or a ranking.

What should a record inventory contain before a licensing review?#

A reviewer-ready inventory is a table with one row per system and record category, plus enough metadata to judge scope without opening the records. The NIST Privacy Framework uses the same building blocks in its Inventory and Mapping category: systems that process data are inventoried, their owners or operators are identified, and the categories of individuals, data actions and data elements are recorded.

Keep it metadata-only at this stage. Counts, date ranges, field names and sample-free descriptions are enough to start a conversation, and they avoid copying sensitive content into yet another spreadsheet.

• 1System and instance: for example Salesforce production org, Zendesk Support, Jira Cloud site, NetSuite account, ServiceTitan tenant.
• 2Record category: cases, tickets and comments, work orders, invoices, call notes, issue histories.
• 3Earliest and latest record date, and whether history was migrated in from an older system.
• 4Approximate record counts per category and total storage size.
• 5Business owner and system administrator (two different people in most companies).
• 6People the records describe: customers, employees, patients, contractors, third parties named in free text.
• 7Field list with free-text and attachment fields marked, plus known sensitive fields.
• 8Export path: native admin export, bulk API, database replica or vendor request, and what that path leaves out.
• 9Contract and terms notes: the vendor agreement, customer contracts and privacy notices that may limit reuse (to be read by counsel, not by a tool).

Data preparation tools compared by class#

Each class of data preparation tools does one part of the inventory well. The table shows the job each class performs, examples of the class, the artifact you can hand a reviewer, and what the class leaves undone. Most mid-size companies need two or three classes, not all six.

When are native exports and bulk APIs enough?#

For a first inventory, the vendor's own export or API is usually the right starting point because it reflects the system as the vendor defines it and needs no new software. The catch is that every native export has documented gaps, and those gaps belong in the inventory.

Salesforce's Data Export service produces a zip archive of CSV files, and Salesforce Help states that formula and roll-up summary fields are excluded and Recycle Bin data is not included. Zendesk's export documentation says data exports are not enabled by default, that JSON export is recommended for accounts with more than 200,000 tickets, and that CSV exports do not include ticket comments. For a support history, comments are often the most valuable part, so the format choice changes what a reviewer sees.

Bulk APIs fill the gap when you need filtered or very large pulls. Salesforce documents Bulk API 2.0 as a REST-based API that processes large data operations, including queries, asynchronously. The value for an inventory is repeatability: a saved job definition shows exactly which objects, fields and date range were pulled, which is easier to review than a one-off file.

• 1Record the export path, run date and running user for each system.
• 2Write down what the documented export excludes (derived fields, deleted records, comments, attachments).
• 3Keep the job definition or export settings next to the output, not in someone's inbox.

When do ETL/ELT tools, catalogs and profilers earn their setup cost?#

They earn it when records live in more than a handful of systems, when the same inventory must be refreshed, or when a reviewer needs to see how a cleaned copy was derived from the source. If you already run a warehouse, these tools are often in place and the inventory is mostly a matter of documenting what they hold.

ETL/ELT and integration tools move data. Airbyte's repository describes it as open-source data movement for ELT pipelines, from APIs, databases and files to warehouses, lakes and AI applications. dbt handles the transformation step (its main branch now holds the dbt v2.0 rewrite, with v1 development on a 1.latest branch): its repository explains that analysts write select statements, dbt turns them into tables and views in the warehouse, and testing checks the quality of those transformations. Together they give you something reviewers value, a written record of how each output table was built.

Catalogs describe data rather than move it. OpenMetadata's repository says it connects technical metadata, lineage, ownership, usage, policies, glossaries and classifications into one metadata graph. That maps well to the inventory's owner and system columns. The limit is that a catalog stores what people enter: an owner field shows who is responsible for a table, not who holds the rights to license it.

Profilers summarize content. The fg-data-profiling package (renamed from ydata-profiling) generates an extended analysis of a dataframe that can be exported as HTML or JSON. For an inventory, a profile answers questions like how complete the date field is, how many rows have empty descriptions, and whether free-text fields carry most of the content. Profiles of raw data can expose values, so handle report files with the same care as the data they describe.

What can PII and DLP scanners tell you, and what can they not?#

PII and DLP scanners tell you where personal or regulated values probably are, field by field and in free text. They do not tell you whether the company is allowed to use or license those records, and they do not find everything.

Presidio, an open-source SDK now maintained by the community-governed Data Privacy Stack organization (it began as a Microsoft project), provides identification and anonymization modules for entities such as credit card numbers, names, locations and US phone numbers, using named entity recognition, regular expressions, rule logic and checksums. Its README warns that, because it relies on automated detection, it may not find all sensitive information and that additional protections should be used. Google Cloud Sensitive Data Protection documents a similar split: discovery builds data profiles that classify data by infoType and sensitivity, while inspection reports each finding with its location, likely infoType and a likelihood rating.

For an inventory, run scanners to populate the sensitive-fields column and to estimate how much free text needs redaction later. Record the scanner, its configuration and the date. Treat a clean scan as evidence that a configured detector found nothing, not as proof that a field contains no personal data, especially in multilingual text or scanned documents. For a deeper comparison of detection tools, see the PII detection tools guide linked below.

What do data preparation tools leave undone?#

They leave undone every question that depends on documents and decisions rather than on the data itself: ownership, consent, contract rights and buyer fit. A licensing review has to answer these separately, and the inventory should have a place to record the answers.

Four questions that are easy to blur stay distinct. Access is whether your team can technically read and export the records. Privacy permission is whether notices, consents and law allow the intended use of personal data. Ownership is whether the company, a customer or a vendor holds the records or rights in them. Contractual licensing rights are whether customer contracts, vendor terms and confidentiality obligations allow the records to be licensed to a third party. A tool can prove the first. Counsel reviews the other three against the specific facts.

De-identification does not settle them either. Removing names from a support history can lower privacy risk, but it does not create a right to license records that a customer contract restricts. Buyer fit is a further question: whether the record categories, history depth and volume match what an AI developer would actually use. That is a market judgment, not a data preparation output.

How do you choose data preparation tools without vendor benchmarks?#

Choose by the artifact you need to hand a reviewer, then by where the tool runs and what it copies. Feature checklists and speed claims matter less here than whether the output is reproducible and whether the tool keeps sensitive records inside your control.

• 1Output a reviewer can read: a file, report or job definition that someone outside the data team can check.
• 2Reproducibility: saved configuration, code or operation history so the same inventory can be regenerated next quarter.
• 3Where processing happens: local or inside your own cloud versus a vendor's hosted service. OpenRefine, for example, runs from a browser on your own computer.
• 4What it copies: tools that replicate whole systems create new sensitive copies that also need access controls and deletion plans.
• 5Coverage of your actual systems: check the official connector or export documentation for each system on your list rather than a marketing page.
• 6Free-text and attachment handling: support histories, call notes and job notes carry most of the value and most of the risk.
• 7Skills already on staff: a team fluent in SQL gets more from dbt than from a new visual tool; a solo operations lead may get further with native exports and OpenRefine.
• 8Licensing and cost model of the tool itself, read from its own license file or terms.

A one-page inventory build sequence#

This sequence produces a metadata-only inventory and one documented sample export path per system. It is written for a mid-size company with a small data or IT team, and it can be done before anyone outside the company sees a record.

• 11. List systems from finance and IT records: subscriptions, vendor contracts and SSO app lists usually surface every system that holds records.
• 22. Name a business owner and an administrator for each system and write both into the inventory.
• 33. For each system, list record categories and pull counts and earliest and latest dates using native reports or a single API query.
• 44. Read the vendor's official export documentation and record the export path and what it excludes.
• 55. Run a profiler on a small internal extract of each major category to capture field completeness and where free text sits.
• 66. Run a PII or DLP scanner on the same extract and record entity types found, scanner configuration and date.
• 77. If records span many systems, record how a warehouse or pipeline would join them, or register the datasets in a catalog with owners and lineage.
• 88. Add contract and notice references per system for counsel: vendor terms, key customer agreements, privacy notices, acquisition documents.
• 99. Leave the rights, consent and licensing columns blank until counsel and leadership fill them in.
• 1010. Version the inventory, store extracts and profile reports under the same access controls as the source systems, and delete working copies you no longer need.

How this applies if you are considering licensing company records#

If the inventory shows years of consistent history in one system, it is worth checking whether the records could be licensed. SourceX runs sourcing and qualification, rights review, contracting, data preparation, delivery and payment for businesses licensing operational data to AI developers. As a general guide, buyers look for years of consistent history (typically 2 or more years) and tens of thousands of records or more in one system, assessed deal by deal.

The first step is a short company assessment, and no data uploads are needed to see if a company qualifies. The company approves exactly what is released, once per deal, and under SourceX's standing offer terms de-identification is performed by SourceX directly or through its processing partner. Your inventory work shortens that review; it does not need to be finished before you ask.