Recovering from Metastable Failures: Where is the Loop?

In light of a recent Microsoft outage that, to me, looked a lot like a metastable failure, I want to revisit metastable failure recovery. Now, I don’t really know much about the actual failure beyond the little bits of information released to the public, so I can only speculate about its metastable nature. But this snippet caught my attention and prompted me to write this post: “The servers we returned to service can become unresponsive again once they receive traffic; we’re analyzing diagnostic data to determine what’s triggering the behavior.” So a quick refresher on metastable failures. These failures start with an initial trigger that puts the systems “out of balance,” usually by overloading a critical system resource (CPU, I/O, locks, etc.). The trigger starts an amplification loop that keeps the failure persisting even after the initial fault is addressed. A typical example of a metastable failure is retry-induced metastable failure: a trigger starts the overload; once…

Commenting on Bubbles has been removed. Why? If you want to talk about this post, write to the author, use the comments on their blog if it has them, or reply with a "RE:" post on your own blog. Suggestions? feedback@bubbles.town