On Saturday, Konstantin Ryabitsev — the man who runs git.kernel.org — published a blog post with a title that sounds like a pest-control invoice: “Creepy Crawlies.” The numbers inside are worse than the title suggests. The Linux kernel’s git infrastructure now receives about six million requests a day demanding to see random commits. Two-thirds get batted away by Anubis, the proof-of-work challenge Ryabitsev’s team deployed last year to slow down AI scrapers. The other third — roughly two million requests a day — are now solving the math and getting through.
That last number is the one to sit with. A third of the bots now solve the proof-of-work challenge. Not because they got smarter. Because the data is worth it.
The Toll Booth That Takes Payment in Wasted Power
Ryabitsev’s post is, on its face, an infrastructure complaint. Fourteen to sixteen CPU cores out of ninety, across five geo-distributed nodes, are constantly busy rendering git commits as HTML purely for scraper bots training large language models. That’s about 20 percent of total capacity. Legitimate human traffic, by Ryabitsev’s generous estimate, is 2 percent of total requests. The kernel’s web frontend is now, functionally, a training-data farm that happens to also serve developers.
But the more interesting number is the 33 percent. Proof-of-work is not a wall; it’s a price. The Anubis challenge forces every visitor to burn a small amount of compute before they’re allowed in. For a human developer clicking a link, that cost is invisible — a few milliseconds of CPU time. For a bot making two million requests a day, it’s a real expense: electricity, silicon, cooling. And the fact that a third of the crawlers now pay that price tells us something the “AI companies are freeloading” crowd hasn’t fully absorbed. They’re not freeloading. They’re paying — in the worst possible currency.
Freeloading, at least, is cheap. A freeloader consumes the commons without contributing, but they don’t make the commons worse by their consumption. A proof-of-work arms race is different. Every solved Anubis challenge is electricity converted into heat for no productive purpose. The kernel.org maintainers get nothing from it. The AI companies get the commit history. The grid gets the load. It’s a war of attrition fought in kilowatt-hours, and the only certain winner is the utility company.
”Make Them Pay” Is the Right Instinct, Wrong Frame
Ryabitsev gestures at the obvious solution in his post: make them pay. And he’s right that the current arrangement is unsustainable. But the frame matters. The problem isn’t that AI companies aren’t paying for the kernel’s infrastructure. The problem is that they’re already paying — in wasted compute — and that payment benefits no one. It’s a toll booth that collects its toll in burned electricity and then throws the money in a furnace.
The fix is to convert the wasted compute into actual revenue. A real toll, not a proof-of-work toll. Charge the crawlers money — per request, per gigabyte, per API key. The infrastructure to do this exists; every major cloud provider bills by the millisecond. The obstacle isn’t technical. It’s cultural. The Linux kernel’s infrastructure has been free-as-in-beer for three decades, and the people who run it are, by temperament and tradition, allergic to turning the commons into a toll road. Ryabitsev himself has spent years resisting exactly this. But the arithmetic is now forcing the question. When 98 percent of your traffic is machines that will never contribute a patch, you’re not running a public service anymore. You’re running a free data buffet.
The Commons Being Defended Is Already Mostly Abandoned
Here’s the number that should make everyone uncomfortable: 2 percent. That’s Ryabitsev’s estimate of legitimate human traffic. The kernel’s web frontend — the thing the Anubis challenge is defending — is now almost entirely a bot-facing artifact. Real kernel developers don’t browse commits in a web browser. They use git protocol, mailing lists, patchwork. The web interface is a museum, and the museum’s visitors are almost all machines.
One commenter on the LWN thread put it plainly: “I used to think my project was popular. Turns out it’s just popular with scrapers.”
That’s the quiet tragedy underneath the infrastructure complaint. The open-source community spent thirty years building public infrastructure on the assumption that the public would use it. The public did — for a while. Then the public moved on to better tools, and the machines moved in. The Anubis challenge isn’t defending a thriving commons. It’s defending the idea of a commons, long after the humans have left the building.
Which is why “make them pay” is not just a practical fix. It’s an admission. The moment you start charging for access to the kernel’s commit history, you’ve conceded that the free era is over — not because of ideology, not because of greed, but because the arithmetic stopped working. Six million requests a day, a third of them solving the math, 20 percent of your capacity burned on bots that will never contribute a line of code. At some point, the only responsible thing to do is send the crawlers an invoice.