do son 
Image: CloudFlare
Cloudflare previously introduced an AI crawler detection and mitigation system designed to prevent high-frequency data scraping by AI bots, which often results in unnecessary consumption of bandwidth and server resources. These AI-driven crawlers, primarily intent on harvesting data, can also disrupt the normal operation of websites.
However, many of these AI crawlers disguise themselves as legitimate user agents, making it difficult to identify and intercept them solely based on the user agent string. In response, Cloudflare has unveiled a new feature called AI Labyrinth.
This innovative system, aptly named “AI Labyrinth,” functions by generating vast amounts of irrelevant content using artificial intelligence. When anomalous crawling behavior is detected, the suspicious bots are funneled into this AI-generated honeypot—a virtual maze of unrelated material that depletes their resources without compromising genuine site content.
Statistics underscore the growing influence of AI-generated content: by autumn 2024, four of the top 20 Facebook posts were created by AI, and nearly 47% of Medium’s published material originated from AI systems. Cloudflare itself now receives over 50 billion AI crawler requests each day.
To counter this surge and safeguard server efficiency, Cloudflare’s AI Labyrinth turns the tables—poisoning the well, so to speak. While AI crawlers seek authentic, human-authored data to fuel their training models, Cloudflare offers them nothing but AI-generated distractions.
Cloudflare utilizes Workers AI along with open-source models to pre-generate a vast collection of unique HTML pages covering a wide range of topics. These pages are stored within Cloudflare’s R2 repository via a dedicated content pipeline.
To prevent misinformation and avoid polluting the broader internet with fabricated data, the content served to AI crawlers is factual and scientifically grounded, though irrelevant to the targeted website and devoid of proprietary information.
These AI-generated pages are technically cloaked from human users and legitimate search engines. Through metadata controls and other protective measures, the pages remain hidden from organic search indexing, ensuring that site SEO remains unaffected and that real users never encounter the decoy material.
Only when abnormal scraping behavior is detected does Cloudflare activate the AI Labyrinth and redirect the bots accordingly. Verified and site-approved crawlers, including those from search engines, continue to access site content unimpeded.
The AI Labyrinth is now available to all Cloudflare users—both free and paid. It can be activated by navigating to the Cloudflare dashboard: Websites → Security → Bots → AI Labyrinth.
