Infrastructure Drift Detection and Remediation

If you’re not running scheduled terraform plan, you have drift. You just don’t know it yet. I learned this the hard way. A colleague made a “quick fix” in the AWS console — changed a security group rule to unblock a vendor integration. Totally reasonable in the moment. Nobody updated the Terraform code. Three weeks later, I ran a deploy that included security group changes for a different service. Terraform saw the console change as drift, reverted it, and killed the vendor connection. That vendor connection happened to feed data into our payment processing pipeline. Two hours of downtime, a war room, and a very uncomfortable post-mortem later, we had a new rule: nothing touches production infrastructure outside of code. Ever.