Do Son 
Konstantin Ryabitsev, the Linux Foundation’s head of IT infrastructure and kernel facilities administrator, has recently published actual traffic figures for git.kernel.org. Bluntly, he declares that the web crawlers of the AI era have become a lasting infrastructure burden.
It should be noted that this is not an official policy statement from the Linux Foundation. Rather, it is an observation drawn from server data by the administrator responsible for IT infrastructure. As yet, the Linux Foundation has not indicated any intention to prohibit AI crawlers from harvesting content or to enact other policies.
AI Crawlers Can Already Defeat Anti-Scraping Measures
At present, the kernel Git service receives roughly 6 million requests each day demanding random commit pages. Of these, 66% are turned away by the Anubis proof-of-work verification mechanism, which makes a visitor’s device perform a small amount of CPU computation to prove it belongs to a real human. Nonetheless, 33% of the requests now complete the challenge and proceed.
By Ryabitsev’s estimate, even under fairly generous assumptions, genuinely legitimate access may account for only about 2% of traffic. Across the 90 CPU cores distributed over five nodes, some 14 to 16 cores are perpetually devoted to rendering Git commits as HTML for these crawlers to scrape. This consumes roughly 20% of total capacity, a toll that even exceeds that of other legitimate access such as Git clones.
Cloning Is Possible, Yet They Insist on Scraping HTML
Ryabitsev observes that the Linux kernel repository is naturally suited to training models. Its historical code and commit records are public, and a vast portion of the content predates the spread of generative AI. Consequently, this material can serve as a relatively clean training corpus, free of the “digital prion disease” that arises when a model is trained on AI-generated text.
The trouble is that the data could be obtained efficiently through a direct Git clone. Instead, some crawlers choose to request the HTML pages generated by cgit one by one. The main Linux kernel repository holds about 1.48 million commits, yet git.kernel.org also hosts around 922 related fork repositories. The same set of Git objects can therefore be combined into billions of valid URLs, and mass duplicate scraping dumps the entire rendering burden onto the server.
IP Bans and Anti-Scraping Measures Are Losing Their Edge
Early crawlers still exposed their user agents or concentrated on cloud-server IPs, so the administrator could enforce efficient bans through fail2ban or by blocking IP addresses and their parent ASN networks. Later, however, the traffic began pouring in from residential and mobile-network IP proxies, each address making only a few requests before vanishing.
At the same time, the crawlers disguise themselves as ordinary browsers. Ryabitsev believes this is connected to proxy-SDK monetisation networks. Previously, Samsung and LG televisions were found to carry numerous third-party apps embedding proxy SDKs, which turned users’ networks into proxy IPs for routing traffic.
Ryabitsev subsequently deployed the Anubis mechanism, requiring visitors to complete a proof of work. Initially, this proved effective. Now, however, the crawlers can gradually solve challenges of difficulty 4 and even 5. Raising the difficulty further would force ordinary users on phones and similar devices to wait longer and bear a heavier computational load, so the difficulty cannot be increased indefinitely.
The Real Problem for Open Source
For now, the Linux infrastructure has not been overwhelmed by this traffic. Even so, the administrator is already preparing to reduce the number of scrapable URLs, disable certain high-cost features, and impose tighter limits on anonymous access. The true problem is not the harvesting of public data itself. Rather, it is that crawlers repeatedly consume the open-source community’s limited infrastructure resources in an extraordinarily inefficient manner, and the cost may ultimately be borne by all legitimate users alike.
Support Our Threat Intelligence
Find our zero-day alerts and CVE reports helpful? Support our work today and unlock a 100% ad-free reading experience!
Buy Me a Coffee
PayPal
