In the 24 hours before I started writing this post, Plausible told me joost.blog had 254 visitors. My own server-side bot logs told me something different. They counted 1,777 bot crawls over the same window. 536 of those came from a specific category I want to talk about: on-demand AI bots like ChatGPT-User (487) and Claude-User (49). So which number is true? Was it 254 visitors, or closer to 800? Or somewhere in between? Or neither? The honest answer is that I don’t know. And I think the reason I don’t know is the more interesting story. What -User bots actually are There are roughly three categories of AI-related bot traffic, and lumping them together hides the part that matters. Training crawlers GPTBot, ClaudeBot, Google-Extended. These run in the background, on the bot operator’s schedule, to build or refresh a model’s training corpus. Whether you want them on your site is a real question, but it’s a different question from this one. AI search crawlers OAI-SearchBot,…
No comments yet. Log in to reply on the Fediverse. Comments will appear here.