AI data market
Is web scraping for AI training legal? Why private business data is different
By SourceX Editorial · Reviewed by Noah Loul ·
Short answer
Web scraping for AI training is legally contested in the US: outcomes depend on how content was accessed, which terms applied and whether the copying is fair use. Private business data is different. Support tickets, CRM histories and job records sit behind logins and contracts, so the lawful route to AI training is a license the company approves.
Key takeaways
- No single US statute settles AI scraping; disputes usually combine contract, computer-access, copyright and privacy claims.
- Courts now treat how material was obtained as its own question, separate from whether training on it is a permitted use.
- Records inside a company's own systems are not public web content; reaching them takes credentials, an export and the company's permission.
- Site owners, infrastructure providers and software vendors are restricting automated collection for AI, which shrinks the scrapable supply.
- A license gives an AI developer what scraping cannot: a named source, written permitted use, a privacy record and a dated approval.
What does US law say about web scraping for AI training?#
Web scraping for AI training is neither clearly legal nor clearly illegal in the United States, because no single statute addresses it and disputes are decided on their facts. Plaintiffs usually stack several claims against one crawler, and each claim asks a different question.
That is why scraping lawsuits rarely turn on whether a page was visible to the public. They turn on what terms the scraper accepted, whether it got past a login or a technical block, what it copied and what the copies were used for.
| Legal theory | Question it asks | Where it bites hardest |
|---|---|---|
| Contract and terms of use | Did the scraper agree to terms that ban automated collection or AI use? | Sites that require an account or clickthrough first |
| Computer-access laws, such as the federal Computer Fraud and Abuse Act | Did the scraper get into a system it was not permitted to enter? | Content behind logins, paywalls or technical blocks |
| Copyright | Was original expression copied, and is the use fair? | Books, articles, images and code |
| Privacy and biometric laws | Was personal information collected or used without a lawful basis? | Profiles, photos and contact details |
| EU text and data mining rules | Did the rightholder reserve mining rights in a machine-readable way? | Content made available to users in the EU |
Why do courts care how the data was obtained?#
Courts increasingly ask how training material was acquired as a question separate from how it was used. In Bartz v. Anthropic, a federal judge held in June 2025 that training on lawfully purchased books was fair use because it was exceedingly transformative, but that downloading and keeping pirated copies in a central library was not excused. That is a district-court ruling, not binding precedent, and other judges have weighed the same factors differently: in Kadrey v. Meta, Judge Chhabria ruled against the authors because they had not produced enough evidence of market harm.
The US Copyright Office drew a similar line in its May 2025 pre-publication report on generative AI training, which is not binding on courts. It said commercial use of vast troves of copyrighted works to produce competing content goes beyond established fair use boundaries, especially where it is accomplished through illegal access, and it favored letting voluntary licensing markets keep developing.
In late September 2026, the Third Circuit affirmed, in Thomson Reuters v. Ross, that copying Westlaw headnotes to train a legal research tool was not fair use; LawNext reported it as the first federal appellate decision on fair use in AI training. Ross involved a non-generative tool, so commentators expect generative AI training questions to be settled in other cases. For an operating company, the practical lesson is that how training material was obtained, and under what permission, is now examined on its own, and that question may apply to suppliers as well as AI developers.
- 01Your work records
- 02Personal details removed
- 03You approve the release
- 04Licensed to AI developers
Public web content vs private business records#
Private business records differ from public web content in the one feature scraping depends on: no outsider can reach them. Support tickets, CRM histories, dispatch logs, NCRs and code reviews live in authenticated systems such as Zendesk, Salesforce, ServiceTitan, an ERP or GitHub, under the company's account and the vendor's contract.
That changes every row in the table above. There is no public page to crawl, so any collection would need credentials, an export or an API call, and each of those runs through the company's permission and its vendor's terms.
| Question | Public web content | Records in internal systems |
|---|---|---|
| Who can reach it? | Anyone with a browser or a crawler | Authorized users with company credentials |
| Who can grant permission? | Often unclear; site owners, authors and users may all have claims | The company, subject to its contracts and privacy obligations |
| Which controls apply? | Robots.txt, terms of use, crawler blocking | Logins, role permissions, vendor terms, confidentiality agreements |
| What documentation exists? | Usually a URL and a crawl date | System of record, date range, export log and approval trail |
| Is it already in public corpora? | Often, which lowers uniqueness | Generally not, because it was never published |
| How does it reach AI training? | Crawling, often contested | A license the company negotiates and approves |
The public web is closing to AI crawlers#
Public web data is getting harder to collect as site owners, infrastructure providers and software vendors restrict automated access for AI. The Data Provenance Initiative's Consent in Crisis audit of 14,000 web domains found that within a single year, 28 percent or more of the most actively maintained critical sources in the C4 corpus became fully restricted through robots.txt.
Infrastructure followed. In July 2025, Cloudflare said it would block AI crawlers by default on the websites it hosts and announced a pay-per-crawl service that lets sites be paid when AI bots crawl them. Epoch AI researchers project that, if current trends continue, language models will fully use the stock of public human-generated text sometime between 2026 and 2032; that is a forecast with wide uncertainty, not a settled fact.
Business software vendors are tightening bulk extraction too. A Hunton Andrews Kurth alert reported that Slack's API terms, effective May 29, 2025, prohibit bulk exporting API data and using it in large language models, and ENR reported in November 2025 that Procore's terms bar marketplace partners from bulk-downloading platform data for commercial purposes, including LLM training. Both rules target third-party apps and partners, not a customer's own records.
Can AI companies scrape my website or help center?#
AI companies can technically crawl any page your website serves without a login, including marketing pages, product documentation and help center articles. Whether a crawler honors your signals, and whether collecting your content was lawful, are separate questions that depend on your terms, the crawler's conduct and the jurisdictions involved.
What you control is the boundary. A short review with your CTO and general counsel usually covers these points.
- List every public surface: marketing site, docs, help center, community forum, status page and public repositories.
- Decide which surfaces you want AI assistants to read and cite, and which give away paid expertise.
- Set robots.txt rules and CDN-level crawler controls to match, and revisit them when new crawlers appear.
- Update website terms of use to state your position on automated collection and AI training.
- Confirm that ticket histories, internal wikis and customer files are not exposed through shared links, open storage buckets or indexed attachments.
- Write down what you decided and when, so you can show your position later.
Why internal records reach AI only through licensing#
Internal records reach AI developers through licensing because the company holding them is the only party that can authorize access, define the permitted use and prepare the data. A license sets the scope, excluded record types, how personal and confidential details are removed, security terms and what happens to the data when the term ends.
Buyers increasingly need that paper trail. California AB 2013, for example, requires developers of generative AI systems to post training-data documentation stating, among other things, whether datasets were purchased or licensed and whether they include copyrighted material or personal information.
Even large transfers of internal business data run through formal channels. SiliconANGLE reported in August 2026 that Google agreed to buy Spirit Airlines' internal business data in a bankruptcy court auction, subject to a court hearing. Court records cited in reporting describe emails, Microsoft Teams messages, OneDrive files and SharePoint items: records no public crawler could reach.
| What a buyer must show | Scraped public content | Licensed internal records |
|---|---|---|
| Where the records came from | A crawl log, if one was kept | Named systems, entities and date ranges |
| Who authorized the use | Often no one explicitly | A signed license from the company |
| Which uses are permitted | Unclear or disputed | Written scope, such as training or evaluation |
| How personal data was handled | Rarely documented | A privacy record of what was removed and how |
| Whether release was approved | No approval step exists | A dated release authorization |
Illustrative: a field service software vendor draws the line#
Illustrative: a fictional field service management software vendor learns that AI crawlers have been reading its public help center. A board member asks two questions: is the company being harmed, and could its records be worth licensing instead?
The CTO maps two zones. The public zone holds the marketing site, product documentation and help center articles. The private zone holds years of Zendesk tickets linked to Jira issues, GitHub pull requests and release notes, plus Salesforce renewal notes and Slack threads where support escalated bugs to engineering.
General counsel recommends different treatment for each. The public zone gets a crawler policy and updated website terms. The private zone gets a metadata-only fit check covering system names, date coverage and record families, with no files shared; customer contracts are reviewed for confidentiality clauses, and Slack threads naming customers are flagged for removal.
The board approves a rights review of the linked ticket-to-fix records only. Public content is now governed by policy, and the private records that no crawler ever touched move forward on the company's terms.
How SourceX approaches scraped versus licensed data#
SourceX works only with records a company holds in its own systems and chooses to license, and its own rights in a deidentified dataset are set out in the signed supplier agreement. Each package follows the SourceX five-step transaction, Supply, Rights, Preparation, Approval and Delivery, and the supplier approves every step.
Nothing is shared during the initial assessment, which collects metadata rather than files. For each package that proceeds, SourceX documents a SourceX Evidence Packet covering provenance, licensing rights, permitted use, the privacy record and release authorization. Data is licensed, not sold outright, so the company keeps ownership.
Frequently asked questions
Does robots.txt legally stop AI crawlers?
In the US, robots.txt is mainly a technical signal whose legal weight depends on the claim and the facts. In the EU, the text and data mining exception in Directive 2019/790 does not cover works whose rightholder expressly reserved mining rights, for online content by machine-readable means, so a clear reservation carries more weight there.
Can our software vendors train AI on the data we store with them?
That depends on each vendor's terms, especially how they define customer data and any AI or machine learning clauses, so review order forms and data processing agreements. FTC staff warned in February 2024 that adopting more permissive practices, such as AI training, through a quiet retroactive change to terms or privacy policies may be unfair or deceptive.
If our public content was already scraped, can we still license our data?
For internal records, usually yes, because scrapers could not reach them. Public pages may already sit in training corpora, which lowers their uniqueness and makes them easier to reproduce. Buyers tend to focus on records that were never published, such as resolved tickets linked to fixes, job histories and quality decisions with outcomes.
Is licensing internal records to an AI developer legal?
Whether a specific license is permissible depends on customer agreements, vendor terms, employee notices, confidentiality obligations and privacy laws that may apply, such as the CCPA or GDPR. These questions are assessed deal by deal with counsel. Removing personal and confidential details before delivery is a standard part of preparation.
Do EU rules matter to a US company licensing data?
They can. The EU AI Act requires providers of general-purpose AI models to keep a policy for complying with EU copyright law, including honoring text and data mining opt-outs. Buyers that place models on the EU market may ask suppliers for provenance and rights documentation even when the supplier is a US company with US records.
Sources
- In a June 23, 2025 order in Bartz v. Anthropic, Judge William Alsup held that using lawfully purchased books to train Anthropic's models was fair use because it was exceedingly transformative, but that downloading and retaining pirated copies in a central library was not excused and would proceed to trial. Source
- The Copyright Office's Part 3 pre-publication report (May 9, 2025) states that making commercial use of vast troves of copyrighted works to produce expressive content that competes with them in existing markets, especially where this is accomplished through illegal access, goes beyond established fair use boundaries, and recommends letting voluntary licensing markets for AI training keep developing. Source
- In late September 2026, a Third Circuit panel affirmed that the Westlaw headnotes are copyrightable and that ROSS's copying of them to train a legal-research AI was not fair use; LawNext reported it as the first federal appellate decision on fair use in AI training. Source
- Judge Chhabria's Kadrey ruling turned on market harm: the authors had not produced enough evidence that Meta's copying harmed the market for their books. Source
- The Ross case involved a non-generative AI tool, and commentators noted that the ruling left questions of transformativeness and market effect in generative-AI training to other courts. Source
- Consent in Crisis (Data Provenance Initiative, July 2024) audited 14,000 web domains and found that within a single year, 28%+ of the most actively maintained critical sources in C4 became fully restricted from use through robots.txt. Source
- On July 1, 2025, Cloudflare said it would block AI crawlers by default on websites it hosts and announced a pay-per-crawl service through which sites can be compensated when an AI bot wants to scrape their content. Source
- Epoch AI projects that language models will fully utilize the stock of public human-generated text between 2026 and 2032, or even earlier if models are intensely overtrained. Source
- According to a Hunton Andrews Kurth client alert, the Slack API Terms of Service, effective May 29, 2025, prohibit bulk exporting of data accessible through Slack's APIs, creating persistent copies or archives, and using such data in large language models. Source
- ENR reported in November 2025 that Procore's terms of service say marketplace partners cannot bulk-download data from its platform for commercial purposes, including training large language models. Source
- California AB 2013 requires developers of generative AI systems to post training-data documentation stating whether the datasets were purchased or licensed and whether they include copyrighted material or personal information. Source
- Article 53(1)(c) of the EU AI Act requires providers of general-purpose AI models to put in place a policy to comply with EU copyright law that identifies and complies with text-and-data-mining rights reservations. Source
- Article 4 of Directive (EU) 2019/790 permits text and data mining of lawfully accessible works unless the rightholder has expressly reserved that use in an appropriate manner, such as machine-readable means for content made publicly available online. Source
- FTC staff warned on February 13, 2024 that a company adopting more permissive data practices, such as using consumers' data for AI training, and disclosing it only through a surreptitious, retroactive change to its terms of service or privacy policy may be engaging in unfair or deceptive practices. Source
- SiliconANGLE reported on August 17, 2026 that Google agreed to buy Spirit Airlines' internal business data in a bankruptcy auction, according to filings in the U.S. Bankruptcy Court for the Southern District of New York, with a hearing scheduled to consider the deal. Source
- Court records cited in reporting describe the Spirit Airlines data as emails, Microsoft Teams messages, OneDrive files and SharePoint items. Source
Related resources
See if your company qualifies
A short company assessment. No data uploads are needed.