AI Content Scraping & Text/Data Mining Compliance Check

Why this matters

Every UK business that publishes content online is now a potential data source for AI training — and most have never checked who’s taking it. AI crawlers such as GPTBot, CCBot, ClaudeBot and Google-Extended routinely scrape websites to build training datasets, and under the UK’s current text and data mining exception (section 29A of the Copyright, Designs and Patents Act 1988), mining copyrighted material is only lawful for non-commercial research — commercial AI training on scraped content sits in a legal grey area that rights holders are actively litigating. The Getty Images v Stability AI case, heard in the UK High Court, tested exactly this question and left significant uncertainty over when training on scraped content infringes copyright and when a business’s only real protection is a robots.txt directive that crawlers may or may not respect.

The rules businesses rely on today may not last. Following its 2024–25 consultation on AI and copyright, the government has not ruled out introducing a broader TDM exception with a rights-reservation opt-out model — similar to the EU’s approach — and the Data (Use and Access) Act 2025 commits ministers to publish a report assessing the economic impact of different options. In the meantime, publishers, image libraries and content-heavy small businesses are increasingly negotiating direct licensing deals with AI companies rather than relying on uncertain legal protection, while businesses using AI tools to mine third-party text or data commercially risk relying on an exception that doesn’t actually cover them. Getting this wrong means either giving away valuable content for free or unknowingly building a product on infringing data mining.

What you'll need

• Whether your business publishes original written content, images, or other material on a public website
• Whether you've reviewed your robots.txt file and website terms of use for AI crawler directives
• Whether your business uses AI tools to mine, scrape, or analyse third-party text or data commercially
• Whether you've been approached about, or considered, licensing your content to an AI developer

What you'll get

A personalised compliance report covering: a score out of 100, an executive summary, a list of findings ranked by severity, and a prioritised action plan with timeframes.