Positive alerts in Prometheus

At work, we use Prometheus to monitor our services. Every few seconds, Prometheus scrapes a /metrics endpoint and records the data in a time series database. The metrics are then used to trigger alerts when services aren’t behaving correctly. Recently we had a flapping alert where an overloaded service was intermittently failing. We wanted to silence/adjust the alert until we could fix the service, to avoid alert fatigue, but it would be easy to forget to revert the adjustment once the service was fixed. We solved this using a “positive alert” that would tell us when the service was consistently passing the original alert – that is, when it was no longer flapping. Let’s walk through an example. One of our alerts checks that Prometheus can successfully scrape our services, which you can check with the automatically generated up time series. A successful scrape gets a 1, a failed scrape gets a 0. Here’s an alert that warns if more than 5% of scrapes fail within a half hour window:…