Autobanning Bad Requests

2026-10-04 Autobanning Bad Requests Like everyone, I get a lot of bots scraping my content. I'm quite intentional about what how my content can be used. I'm more than happy for people use my site, but not for commercial purposes including training AI. My robots.txt file disallows all bots except the few that I permit.1 I return 403 Forbidden responses to any unauthorized bot that happens to request a URL anyway. The request is probably from someone else with a spoofed User-Agent header. There are still many requests that are clearly bots. Some things I'm seeing are requests fishing for sensitive data like .env files and wp-admin endpoints. There are also requests with referrer headers from sources that are forbidden from crawling site. I have a static site, so any request with a POST or DELETE method are also clearly from bots. I've been working on a system to automatically discover and ban those bots. The solution I've landed on is quite simple: I scan my logs for clearly dodgy…