gauteh · @gauteh
18 followers · 26 posts · Server hachyderm.io

New release of hidefix (github.com/gauteh/hidefix) updated to work with xarrays plugin registration. pip install hidefix should do the trick, conda package in the wait: github.com/conda-forge/staged-

#hidefix #rust #hdf5 #netcdf

Last updated 1 year ago

gauteh · @gauteh
10 followers · 12 posts · Server hachyderm.io

On a machine with 64 CPUs reading with hidefix is between 10x and 16x faster than native HDF5!!

And that is _including_ generating the index.

Going from 20 seconds to 1.8 seconds is pretty significant when reading a bunch of variables! 7s to 380ms, is the difference between waiting and immediately in processing!

This is for a file on a network disk.

The ultimate goal: Make this into a backend!

#xarray #rust #hidefix #hdf5 #data #met

Last updated 2 years ago