Frisky and Xarray Example

Frisky and Xarray Example A couple months ago I was doing some consulting work for a finance group that used Dask and Xarray at the ~10 TiB scale. It was slow. I decided to make it faster. What resulted was three weeks of AI development on a couple historical projects I had mothballed, frisky, a Rust implementation of the Dask scheduler dask-array, query optimization for dask arrays, and now also written in Rust Both are now available on PyPI and, when combined with git main versions of dask and xarray, form a system for Xarray computations that is high performance, and usually doesn't break :) Pain If you suffer from the following, then this is for you: Large task graphs (over a million) Long wait times before things happen on the dashboard Sensitivity to chunk size Rechunking generally Solution It's hard to convey everything that goes into making something fast. I use "Rust!" above as a catch-all that people today seem to latch onto, but real performance has much more to do with…