Creepy crawlies

That’s how Konstantin Ryabitsev titles his post creepy crawlies, where he explains the scale of the problem of the “AI crawlers” in the kernel.org git repositories website. I have discussed it here a few times: the problem of the LLM crawlers, how I simplified how I self host git repos, and my attempts trying to deal with these crawlers. I haven’t reached a solution and the crawlers are still there –less so, as the blocking and some of the Javascript tricks did work–, although I haven’t tried more drastic solutions like Anubis. My problem is not at the same scale as what the people at kernel.org are experiencing: We spend more CPU cycles rendering commits for scrapers than we spend on all other kinds of legitimate access, including git clones. At any one time, across 5 geo-distributed nodes, there are 14 CPU cores doing nothing but rendering git commits as html. That’s a lot of CPU cores! The post is a good read, because it explains how they have tried several things, from blocking…