Processing a 250 TB dataset with Coiled, Dask, and Xarray

We processed 250TB of geospatial cloud data in twenty minutes on the cloud with Xarray, Dask, and Coiled. We do this to demonstrate scale and to think about costs. Data Local Coiled Speed Up Cost 252 TB Zarr 500h+ [ 1 ] 6 mins 5,000x $25 Many people use Dask with Xarray to process large datasets for geospatial analyses, however, there is pain when operating at a large scale .