Animations for System Builders: Metastability and Lost Writes

Animations for System Builders: Metastability and Lost Writes Pretty pictures help us learn, maybe? One of the things I love using LLMs for is helping me understand systems better. In the Strands Decider post I wrote about how I often ask ask me questions to check I understand this. The other technique I’m enjoying right now, and am delighted to see folks around me using too, is building animations. Animations can help us build an intuition for how and why systems work, or how and why they fail. Here’s one example: did you know that a Raft cluster can lose writes if even one of the cluster members has a storage layer that lies about fsync()? Your browser does not support the video tag. When I built Stability Sim, I wanted to help engineers understand metastable failures better. That worked pretty well on an abstract level (e.g. retries and queues make metastability really easy to trigger), but I still hear doubts that these behaviors happen in real systems. After all, those real…