many data #spatial providers don't even know about the benefits of well structured GeoTIFF (let alone the buzz of COG, STAC, Zarr, etc)

I reach out to providers, but more often than not there's just another individual there scratching head at my crazy email (who even reads bathymetry on the fly from a url, nuts right...)

at institution I can try collectively to feedback, but the #python and #rstats and #gdal communities could join forces here - is there something in the #xarray zone?

:python: tip

https://www.scivision.dev/xarray-to-netcdf-compression/

#xarray to_netcdf() can compress data variables to save considerable disk space, but it is not trivial until your read the docs.

#netcdf #python

A bunch of tools coming together... was revamping a soon-to-be released package that streamlines some data IO and analysis for #seismic datasets that contains pipelines to #yt (and #yt-xarray), #xarray and #geopandas. The package came out of my need to extract data for some k-means analysis, but it also makes it very easy to pop data over to yt and in this case, #yt_idv!

#python #OpenSource #DataVisualization

Pretty useful, thanks Luke Gloege https://towardsdatascience.com/xarray-recipes-for-earth-scientists-c12a10c6a293 #dataAnalysis #climatology #xarray #matplotlib #pandas #fediscience

☯️ #zen3geo v0.6.1. Quick patch release with full support for 🐍 Python 3.11! Made a couple of bug fixes 🐞 to DatashaderRasterizer (for turning points/lines/polygons to pixel masks based on a template raster). Also sneaked in a tiny enhancement ✨ to XpySTACAssetReader (which wraps around `xarray.open_dataset`) to work without xpystac 🙃. Find out more in the release notes at https://zen3geo.readthedocs.io/en/v0.6.1/changelog.html.

@pytorch #TorchData #Python #DataShader #xarray #gischat #geospatial #datapipeline #computervision

Preparing demos for next week at #jupytercon. Here is a sneak peek (turn on the audio, best with headphones). See you there!

💨💨 🔊💨💨

#sonification #xarray #ipyleaflet #ipytone #jupyter

@conorosully @lavergnetho Also, consider getting a Microsoft Planetary Computer account. Although Google's equivalent has been around longer, #PlanetaryComputer allows you to do these things easily with standard Python tools and libraries (eg #xarray )

☯️ #zen3geo v0.6.0. Scale to new heights if you dare. Open 📂 STAC Assets backed by Zarr, NetCDF or CloudOptimizedGeoTIFF files using the new XpySTACAssetReader DataPipe. Attempt super-resolution 🔭 with the walkthrough on handling multi-resolution CMIP6 climate data from CarbonPlan over the Philippines 🇵🇭 at https://zen3geo.readthedocs.io/en/v0.6.0/multi-resolution.html. Clarity, seek it, give it 🥽

@zarr #STAC #NetCDF #CloudOptimizedGeoTIFF #xarray @pytorch #DataPipe #SuperResoluton #MultiResolution #CarbonPlan #CMIP6 #ClimateModel

STAC + stackstac + xarray is a powerful combination. I put together a short post highlighting my favorite new tools for working with satellite imagery.

https://hrodmn.dev/posts/stackstac/

#python #geospatial #xarray #eo #dataengineering #stac #remotesensing

yt_xarray release day! v0.1.1 now available on pypi!

https://yt-xarray.readthedocs.io/

yt_xarray is an xarray accessor that streamlines communication between #yt and #xarray, allowing you to easily pull out a yt dataset from regularly gridded datasets that xarray can load, allowing you to more easily use yt's plotting and analysis methods. (1/N)

#python #OpenSource #geoscience

Also uses for #xarray #python #RandomDungeonGenerator

Hey #RemoteSensing enthusiasts! I've also released #spyndex v0.3.0 and now you have access to #AwesomeSpectralIndices v0.3.0 in #Python! 🐍🛰️🌿

Check it here: https://github.com/awesome-spectral-indices/spyndex

You can use it for #numpy, (geo)pandas, #xarray, #dask and #EarthEngine! 🚀😉

#GISchat #EOchat

Hey #Python package #maintainer s, here's a question - is there anything you wish you knew before you switched to date based versioning?

Asking for a friend... and a bunch of other friends... And myself... And our package :D we've been discussing the switch at https://github.com/PlasmaPy/PlasmaPy-PLEPs/pull/39 . I think we've got most of the edge cases covered, following #Dask and #xarray's example - but can't hurt to ask, right?

hidefix (https://github.com/gauteh/hidefix) can now be used as an #xarray backend: more than 6x faster on a 8 CPU machine!

This reads a full variable of about 370mb.

On a machine with 64 CPUs reading with hidefix is between 10x and 16x faster than native HDF5!!

And that is _including_ generating the index.

Going from 20 seconds to 1.8 seconds is pretty significant when reading a bunch of variables! 7s to 380ms, is the difference between waiting and immediately in processing!

This is for a file on a network disk.

The ultimate goal: Make this into a #xarray backend!

#rust #hidefix #hdf5 #data #met

Today I've been continuing some work on an experimental #xarray + #dask + #yt pipeline. It's been interesting figuring out how to align the xarray-dask array chunking with yt's domain decomposition and indexing. This meaningless but kinda fun looking image is a plot made with yt using a ~10Gb collection of netcdf files loaded via xarray and dask. yt does some initial culling of files to load then hands off the actual read to xarray and dask. #python #OpenSource

Today I've been continuing some work on an experimental #xarray + #dask + #yt pipeline. It's been interesting figuring out how to align the xarray-dask array chunking with yt's domain decomposition and indexing. This meaningless but kinda fun looking image is a plot made with yt using a ~10Gb collection of netcdf files loaded via xarray and dask. yt does some initial culling of files to load then hands off the actual read to xarray and dask. #python #OpenSource

Working with #NEMO gcm and #python?

I just released version 0.4.0 of #xnemogcm, a package that can read nemo #netcdf outputs, and adds the proper attributes (+ change dimension names) to be read by the fantastic #xgcm package.

It tries to automatically discover the T, U, V, W, etc point of variables, and sets everything for you :). For more complicated cases (dataset already loaded into #xarray, dataset with no attributes, etc) you can easily set the point by hand.

https://github.com/rcaneill/xnemogcm/releases/tag/v0.4.0

@seaice @carlosmoffat i am not so sure, Lars, but it is true that this is a legacy software that does not use #xarray. At least it is #python3 🙂

@lavergnetho @carlosmoffat With #xarray this would not have happened 😂