Tail Scale

"Our p99 is fine" and "one user in six has a bad time" are the same sentence. We rarely hear it that way, and in this post I wanted to go through why that can be.tail-scale is a small browser tool, a toy, for visualizing the tail-at-scale effect: the way a rare slow call stops being rare once a web page, a phone screen, or a service waits on enough of them. It's a single standalone HTML file running an in-browser Monte-Carlo simulation of serial and parallel fanout effects. You can download the page, open it in a browser and use the sliders to watch how good-put and bad-put numbers adjust. The model itself is captured in some reference python code that the web page is derived from.The idea isn't mine and isn't all that new. In 2013, Dean and Barroso named it in their paper, The Tail at Scale. In 2021, Marc Brooker laid out the bimodal model in a startlingly brief and imo classic blog post. Both put us in the same, uncomfortable place: a system can run inside its percentile and service…