55 minutes ago · Tech · hide · Fediverse post

Firstly, sympathies to the sysadmins and DevOps teams managing the cloud and compute fleets during a five-alarm fire. Team members can experience white-hot stress in these situations, and it is a heroic effort to restore things to normalcy. I want to thank Microsoft for a transparent postmortem on the outage that doesn’t shy away from details. I haven’t worked with the Microsoft cloud stack in my career, but what I can say is that these systems are mind-bogglingly complex, beyond that of any one human. And also well beyond that of any capable AI. Microsoft traced the primary issue to a kernel change made three months before the incident. I find this quite impressive given that there are probably thousands of changes to these related systems during that window. As time goes by, pinpointing the source change gets harder. Three months falls outside the recall window for many developers, who have since moved on to other priorities. On top of the attribution, Microsoft also details the…

Commenting on Bubbles has been removed. Why? If you want to talk about this post, write to the author, use the comments on their blog if it has them, or reply with a "RE:" post on your own blog. Suggestions? feedback@bubbles.town