The time Bluesky tried to kill my server

Ok, now that have your attention, it was actually more like “the time that the relays running indigo-relay, a codebase maintained Bluesky, tried to kill my PDS”. Because only some of them are actually operated by Bluesky, there are lots of independent ones running the same codebase! The story So I was minding my own business working on Kite when I suddenly get notifications that my server on Hetzner has been sitting at >95% total CPU usage over the last 10 minutes. I go to Grafana and find this. That’s my postgres instance sending 1GB/s of data internally on the private network. And I quickly spotted the receiver: tranquil PDS, my self-hosted atproto personal data server. A combination of postgres, tranquil, and traefik were pinning my 8 vCPU server to close to 100% saturation. My PDS dashboard gives a lot more information. I’m receiving a massive amount of subscribeRepos requests. At first I thought that it might be someone messing with my server, but digging into the logs I found…