Nginx Rate Limiting: Taming the Meta-Crawler 0 ▲ Things I couldn't find elsewhere 101 days ago · Tech · hide · Fediverse post Recently I noticed unusually high load on one of my servers. It turned out to be due to my somewhat new Forgejo service being absolutely hammered by bots. It wasn’t a typical brute-force attack, but rather a massive, distributed crawl from the Meta (Facebook) crawler. I guess they want all that juicy code we produce.The logs were a mess. While no single IP was doing anything egregious (most were just doing 1-3 requests per minute), the sheer volume was overwhelming. We were seeing hundreds of thousands of hits a day, all from the same broad Meta range. A quick look at the User-Agent showed meta-externalagent/1.1 everywhere.The problem with traditional per-IP rate limiting is that these crawlers are distributed. If you set a limit of, say, 10 requests per second per IP, a botnet of 1,000 IPs can still hit you with 10,000 requests per second while each individual IP stays perfectly within the limits.I needed a way to rate-limit the entire Meta block as a single entity.The first step was… Commenting on Bubbles has been removed. Why? If you want to talk about this post, write to the author, use the comments on their blog if it has them, or reply with a "RE:" post on your own blog. Suggestions? feedback@bubbles.town