Lego, ACME DNS-01, and negatively-cached surprises

Recently a friend was deploying Centauri, my reverse proxy service that can obtain TLS certificates using ACME DNS-01 challenges. He asked a seemingly innocent question: “When Centauri is ‘waiting for record propagation’ is there an option to increase the time?” But that’s a question with a hidden, concerning XY problem. We both use the same DNS provider, and it definitely shouldn’t take more than a minute to see a DNS record after it’s been created. Plus, I don’t have issues with it timing out! Centauri uses the excellent lego library to do the actual ACME dance, and I’ve dug into it before pretty extensively to debug other DNS issues. When I last looked, it used the local recursive DNS resolver to check for CNAMEs and find the authoritative server, then checked for record propagation there. So, naturally, I blamed Simon’s local resolver doing something weird. Turns out that was surprisingly close to correct, but not for the reasons I thought. Unfortunately for my snap judgements,…