Weeknotes: 26th October 2026

Pipeline reporting One thing that I've been meaning to do for ages is look into automatically generating pipeline reports for a given run of one of the data-science pipelines I've built. These pipelines aggregate so much data and generate so much summary data that sits on disk that people only look at the final output raster typically, and only really dig into that data if there's an obvious flaw. Instead I want to make it easier to spot mistakes early, for example: did we drop half a dozen input species for no reason since the last run? how do we compare to prior work in terms of validation scores? what are the numbers of pixels of a given colour and does it reflect what we expect? In one of the first ecological data-science projects I worked on with Patrick Ferris we achieved some of this to a degree. We attempted to build a literate programming pipeline description (using a tool we built called Shark), and in the pipeline runner Patrick added some code to visualise common outputs…