
Cloudflare, the internet services provider, previously announced plans to introduce a new feature aimed at addressing the unauthorized scraping of websites by AI companies. This initiative will allow websites and content publishers to charge AI firms in exchange for authorized access to their content.
The feature is now being piloted with a select group of websites. Interested publishers must manually submit an application; once approved, they can activate the feature and configure preferences such as whether to block AI crawlers entirely, impose access fees, or specify which types of content may be scraped.
Specifically:
- Websites and content publishers may choose to block all AI crawlers, permit selected ones, charge for access, or offer content freely.
- AI companies can register to scrape content based on query volume, review pricing, choose to pay, or decline access—thus obtaining quality content through a formalized mechanism.
At present, Cloudflare has not disclosed standardized pricing for paid content scraping. Given the varying value of content across sites, it is likely that future iterations will permit publishers to define their own pricing models—such as setting a fee per article.
To use this feature, the website must be hosted on Cloudflare with the “orange cloud” protection enabled. Once active, all traffic is routed through Cloudflare, allowing it to identify and manage various crawlers via a proprietary bot database.
When the paid scraping feature is enabled, AI crawlers issuing requests will receive a standard HTTP 200 response if access is permitted. If the request returns an HTTP 402 Payment Required status code, it indicates that a fee must be paid.
AI companies can determine a site’s access requirements by parsing these HTTP response codes. If willing to pay, they must register through Cloudflare’s paid scraping program. Once activated, they can begin accessing the content under the agreed terms.
Cloudflare has confirmed that several AI companies have already enrolled in this paid access initiative. However, these partnerships are contingent on publisher-defined pricing being acceptable to the AI firms. Should a publisher set a fee deemed too high, the AI company may reject the terms and cease all scraping activity.
Related Posts:
- Google’s Ultimatum: Publishers Must Share Data for AI Overviews or Lose Traffic
- Google’s New “Offerwall” Tool Helps Publishers Monetize Content Amidst AI Search Impact
- Google Requires JavaScript for Search: Bots and Crawlers Impacted