Legal Web Scraping for Business Intelligence in 2026: What You Can (and Cannot) Do

The web scraping market is widely cited at over $1 billion in 2025 with double-digit annual growth (industry analyst estimates; figures vary by source and have not been independently verified). Yet the conversation most businesses are having is still stuck in 2018: "Is scraping legal?" The better question is: what specifically can you scrape, under which conditions, and with what documentation in place?

The rules have changed substantially since 2021. Three major court decisions, two landmark GDPR enforcement actions, and a pending DMCA lawsuit against one of the SEO industry's most widely used tools have redrawn the lines. Knowing where those lines sit is the difference between a defensible business intelligence operation and a cease-and-desist letter, or worse, a €240,000 regulatory fine and an order to delete your entire database.

I have built scraping pipelines for clients across e-commerce, B2B sales, and competitive research. This article covers the actual legal framework as it stands in mid-2026, with concrete examples of what you can and cannot collect, and a practical checklist to keep your operation on the right side of regulators on both sides of the Atlantic.

How Did Web Scraping Law Change After 2021? Three Decisions That Defined the Current Rules

Three cases established the current framework for legal web scraping business intelligence.

Van Buren v. United States (2021, U.S. Supreme Court, 6-3): The Court ruled that the Computer Fraud and Abuse Act's "exceeds authorized access" clause applies only when you access areas that are completely off-limits. Using permitted access for the wrong purpose does not violate the CFAA. This ruling set the ceiling on how broadly the CFAA can reach scraping operations and underpins every ruling that followed.

hiQ v. LinkedIn, Ninth Circuit (April 2022): The Ninth Circuit confirmed that scraping publicly accessible websites does not violate the CFAA. The December 2022 settlement is instructive: hiQ paid $500,000 in damages, accepted a permanent injunction, and stipulated to breach of Terms of Service. The CFAA claim failed. The contract claim stuck. The favorable CFAA precedent remains good law.

Meta v. Bright Data (January 2024): U.S. District Judge Edward Chen dismissed Meta's CFAA claims against data broker Bright Data, ruling that scraping public Facebook and Instagram data while logged out is not a CFAA violation. Meta's ToS only governs logged-in account activity. Meta dropped the suit one month later without appeal.

The pattern across all three decisions: CFAA claims against public-data scrapers are failing in US courts. The legal risk has shifted entirely to breach of contract and, in Europe, GDPR.

Is Web Scraping Legal in the United States in 2026?

In the United States, the primary question is no longer "Does the CFAA apply?" It rarely does for public-data scrapers. The questions you actually need to answer are:

• Did you create an account on this platform before scraping?
• Did the Terms of Service explicitly prohibit automated access?
• Are you collecting data about individual California residents?

If you created an account and agreed to a ToS that prohibits scraping, you have a breach-of-contract exposure. hiQ's $500,000 settlement resulted partly from workers creating fake LinkedIn profiles to access the platform. That detail strengthened the breach-of-ToS argument even after the CFAA claims were rejected by the court.

California's CCPA also matters, with new regulations effective January 1, 2026. Businesses collecting California residents' personal information must provide transparency, honor opt-out requests with equal ease to opt-in, and minimize data collection to the stated purpose. The "publicly available" exception covers government records and clearly non-restricted consumer data, but individually identifiable professional email addresses can still qualify as personal information under the current regulations.

In practice, the first document every US-focused client I onboard needs is a written ToS audit for each target domain. Without it, you cannot know whether you are operating under contract law risk.

The practical rule for US operations: keep your scraping entirely unauthenticated. Never create an account on a platform you intend to scrape. Read the full ToS before you touch a single URL.

Is Web Scraping Legal Under GDPR in Europe?

European businesses and anyone collecting data about European residents face a fundamentally different framework. GDPR treats "publicly available" as a description of where data lives, not as a license to process it.

Personal data under GDPR includes names, individual professional email addresses, photos, IP addresses, and profile details, even when publicly visible on a website. Processing any personal data requires a documented lawful basis. For competitive intelligence and business prospecting, "legitimate interest" under Article 6(1)(f) is the most viable basis. But it requires a written Legitimate Interest Assessment (LIA) that balances your business need against the individual's privacy rights. That document has to exist before you start collecting a single record.

In practice, every client I have onboarded who needed EU personal data had skipped the LIA. In every enforcement action I have tracked, it was the first thing regulators asked for.

The KASPR enforcement action is the clearest warning in the current landscape. France's CNIL fined KASPR €240,000 on December 5, 2024 for scraping approximately 160 million LinkedIn contact records, including from users who had deliberately restricted their profile visibility. The company's argument that the data was publicly available failed entirely. CNIL ordered KASPR to delete its entire database and cease LinkedIn scraping.

The Clearview AI case raised the stakes further. The Dutch DPA fined Clearview AI €30.5 million in May 2024 for scraping over 30 billion photos without any lawful basis or consent. More significant than the fine: the Dutch regulator announced it is investigating whether to hold Clearview's directors personally liable, a first under EU GDPR enforcement. GDPR non-compliance is no longer just a corporate cost. It is becoming a direct personal risk for executives.

Maximum GDPR fines reach €20 million or 4% of global annual revenue, whichever is higher. Total GDPR fines across all violations have exceeded €4 billion since the regulation took effect (DLA Piper GDPR Fines and Penalties Report, 2024). That context matters when your legal team tells you GDPR risk is "manageable."

robots.txt: Not a Legal Barrier, But Still Relevant

A 2025 court ruling settled a long-running debate. In Ziff Davis v. OpenAI, a U.S. federal court ruled that robots.txt files do not constitute "technological measures that effectively control access" under the DMCA. The court's exact language: robots.txt "does not effectively control access any more than a sign requesting that visitors keep off the grass effectively controls access to a lawn." Ignoring robots.txt carries no DMCA liability under US law.

However, EU data protection authorities explicitly treat robots.txt non-compliance as evidence of bad faith. If you are processing European personal data and your operation disregards robots.txt restrictions, regulators will cite that against your legitimate-interest defense. It is not a legal requirement in the US, but it is evidence that works against you in EU proceedings.

My recommendation: honor robots.txt restrictions by default. If a specific restricted path matters to your use case, document why your legitimate interest outweighs the restriction and get that assessment reviewed by a lawyer before proceeding.

What Data Can You Legally Scrape? A Direct Comparison

What Happens When Web Scraping Goes Wrong

The most common failure I see: a marketing team builds a pipeline to monitor competitor pricing and compile a B2B lead list from a business directory. It works for two to three weeks. Then the target site starts returning 403 errors. The scraper was sending 800 requests per minute with a default Python user-agent string and the entire IP range got blocked.

The team subscribes to a rotating residential proxy service for $200 per month, which restores access. Two months later, the site's legal team sends a cease-and-desist citing breach of Terms of Service. A team member had created a free account on the platform before the scraping started, and the ToS explicitly prohibited automated access. The CFAA does not apply (public data, no server damage), but the breach-of-contract claim is solid and defensible in court.

The fix has three steps:

1. Audit the full Terms of Service before touching any URL. Do not skim. Look explicitly for prohibitions on automated access, data aggregation, and commercial use. If the ToS prohibits automated access, do not scrape that site while logged in or with a registered session. Use only unauthenticated requests to publicly visible data. No exceptions.
2. Implement rate limiting from day one. One request per 10 to 15 seconds is a conservative baseline. Add random jitter of plus or minus 0.5 seconds to avoid pattern detection. At this rate, 800 requests takes about three hours instead of one minute. This is how you avoid the IP block that starts the whole chain of problems.
3. Keep scraping logic completely separate from any account you hold on the platform. Session cookies bleed authorization boundaries and turn a CFAA-safe unauthenticated operation into a breach-of-contract case. Never authenticate before scraping, and never use the same browser session or cookie jar for both activities.

Best Web Scraping Tools for Business in 2026: A Comparison

The choice of tool affects both your legal posture and your operational costs, often in ways the pricing pages do not make obvious. Before selecting infrastructure, see our automation services page for how these tools integrate into a compliant pipeline.

On open-source total cost of ownership: combining residential proxies with JavaScript rendering pushes effective costs to $5 to $8 per 1,000 pages, well above the advertised base rates of any proxy service. For integrating scraped data into automated workflows, the article on AI agents vs. workflow automation covers how to choose between rule-based pipelines and AI-powered processing.

A 5-Step Compliance Checklist Before You Start Any Scraping Project

In practice, the clients who ran into regulatory trouble almost always skipped step one. The Legitimate Interest Assessment is not optional if any EU personal data is involved.

1. Classify the data. Is any of it personal data under GDPR (individual emails, names, photos, profile details)? If yes, document your lawful basis and write a Legitimate Interest Assessment before collecting a single record. This document is what regulators ask for first.
2. Read the full Terms of Service. Look explicitly for prohibitions on automated access, data aggregation, and commercial use. If you hold an account on the platform, you are bound by that ToS regardless of whether the CFAA applies.
3. Check robots.txt. Honor restrictions by default. If you intend to deviate from a specific restriction, document your rationale and get it reviewed by a lawyer before proceeding.
4. Set rate limits. One request per 10 to 15 seconds as a starting point. Add jitter. Do not adjust upward without confirming the target site has no objection and that your legal basis is documented.
5. Keep documentation. Your lawful basis, your LIA if applicable, your robots.txt review, your ToS audit. EU regulators asked KASPR for documentation. The company did not have it. That made the enforcement outcome significantly worse than it would otherwise have been.

Once your data collection pipeline is compliant, see how one agency automated their weekly reporting to turn collected data into regular business intelligence without manual assembly.