On August 29, Konstantin Ryabitsev — the man who runs the infrastructure behind git.kernel.org — published a post titled “Creepy crawlies” with numbers he’d clearly been sitting on for a while. The Linux kernel’s git repositories now field about six million requests a day from bots demanding to see random commits. Two-thirds get batted away by Anubis, the proof-of-work challenge his team deployed to make scraping expensive. The other third? They solve the math and get through.
Read that again. A third of the AI crawlers hitting the Linux kernel’s infrastructure are now spending real compute — real electricity, real money — to solve a deliberately expensive math problem, just to download commit data. The toll booth is working. The toll is being paid. And the Linux Foundation collects none of it.
The Toll Booth That Incinerates Its Revenue
Anubis is a proof-of-work system. It forces a client to burn CPU cycles before it can access a resource. The point is to make scraping expensive enough that it’s not worth doing. But when a third of your bot traffic is solving the challenge, the economics have flipped: the data is valuable enough that the scrapers will pay the toll.
The problem is that the toll doesn’t go to the people who run the infrastructure. It goes to the power company. Every solved Anubis challenge is a payment that is incinerated as heat. Ryabitsev’s team has built a toll booth that collects no revenue — it just makes the road more expensive for everyone, including the legitimate developers who now have to solve the same math to fetch a commit.
The Arms Race Has Already Escalated
Ryabitsev is blunt about the solutions people keep offering. He has a standing request: don’t tell him “why don’t you just” — because there is no “why don’t you just” here. The 33% figure is why. Blocking by user agent doesn’t work; the bots look like real browsers. Blocking by IP doesn’t work; the crawlers rotate through residential proxies. Rate limiting doesn’t work; the requests are distributed across thousands of addresses. Ryabitsev estimates roughly seven million requests per day are attributable to bots, and the only thing that has worked, sort of, is making the request itself expensive.
Now the crawlers have decided the expense is worth it. That’s not a failure of the defense. That’s a market clearing.
The Missing Payment Rail
The predictable response to Ryabitsev’s post is “make the AI companies pay.” But they are paying — in a currency that benefits nobody. The real problem is that there is no mechanism for a commercial entity to pay for bulk, anonymous access to public git data. The kernel is licensed under the GPL, which means anyone can take it. The infrastructure that serves it is funded by donations and corporate sponsorships. The demand is commercial, the supply is public, and the only pricing mechanism available is a proof-of-work challenge that burns value instead of transferring it.
One volunteer who runs a kernel mirror for a university CS department, reached on a Sunday afternoon while he was babysitting a failing RAID array, put it plainly: “We don’t want to block anyone. We want the crawlers to send us a check. But there’s no address to send it to.”
The Anubis challenge was supposed to be a wall. It has become a meter — one that measures demand in kilowatt-hours and collects nothing. The AI companies have told us, in the clearest way possible, that the Linux kernel’s commit history is worth paying for. The infrastructure that produces it just hasn’t figured out how to accept the payment. Until someone builds a payment rail for public code, the toll booth will keep burning its own revenue — and the people who keep the kernel’s lights on will keep paying for the privilege of being scraped.