New release of hidefix (https://github.com/gauteh/hidefix) updated to work with xarrays plugin registration. pip install hidefix should do the trick, conda package in the wait: https://github.com/conda-forge/staged-recipes/pull/21742 #hidefix #rust #hdf5 #netcdf
On a machine with 64 CPUs reading with hidefix is between 10x and 16x faster than native HDF5!!
And that is _including_ generating the index.
Going from 20 seconds to 1.8 seconds is pretty significant when reading a bunch of variables! 7s to 380ms, is the difference between waiting and immediately in processing!
This is for a file on a network disk.
The ultimate goal: Make this into a #xarray backend!
#xarray #rust #hidefix #hdf5 #data #met