Prometheus Remote Write Examples

Last year, I worked on a project that sends telemetry data to Grafana Cloud. It was data from remote sensors (i.e.: internet of things) and my theory was that having the data end up in Prometheus would be ideal for data storage, software licensing and general support reasons as then it’d be possible to query the sensor data like any other application metrics. In practice this worked out pretty well, and I’m really happy with how it worked out. But, with how prometheus_exporter ends up exposing the metrics, you end up with a situation where the metric data persists. Instead of the data being a point in time, the value stays constant between scrapes. Whilst this is inherent to the design, it means that unless the whole application is restarted (or goes down), you lose any gaps in the data which is itself a helpful piece of information that you can alert on. For a specific example: if temperature data is sent every 5 minutes at 21ºC, unless that value changes, the scraped data will…