Do Son AI crawlers now burden git.kernel.org with 6M daily requests, burning more CPU rendering commits for scrapers than on all legitimate access combined.
AI Crawlers
The Free Software Foundation reports ongoing DDoS attacks since Aug 2024, with much traffic from AI company crawlers. Its small team is overwhelmed by the relentless web scraping.
Cloudflare launched an AI Crawler Leaderboard ranking AI company web crawlers. ByteDance is last for ignoring robots.txt and lack of IP transparency, prompting new blocking defaults.