Nginx Rate Limiting: Taming the Meta-Crawler

Recently I noticed unusually high load on one of my servers. It turned out to be due to my somewhat new Forgejo service being absolutely hammered by bots. It wasn't a typical brute-force attack, but rather a massive, distributed crawl from the Meta (Facebook) crawler. I guess they want all that juicy code we produce.The logs were a mess. While no single IP was doing anything egregious (most were just doing 1-3 requests per minute), the sheer volume was overwhelming. We were seeing hundreds of thousands of hits a day, all from the same broad Meta range. A quick look at the User-Agent showed meta-externalagent/1.1 everywhere.The problem with traditional per-IP rate limiting is that these crawlers are distributed. If you set a limit of, say, 10 requests per second per IP, a botnet of 1,000 IPs can still hit you with 10,000 requests per second while each individual IP stays perfectly within the limits.I needed a way to rate-limit the entire Meta block as a single entity.The first step was…