many data #spatial providers don't even know about the benefits of well structured GeoTIFF (let alone the buzz of COG, STAC, Zarr, etc)
I reach out to providers, but more often than not there's just another individual there scratching head at my crazy email (who even reads bathymetry on the fly from a url, nuts right...)
at institution I can try collectively to feedback, but the #python and #rstats and #gdal communities could join forces here - is there something in the #xarray zone?
#spatial #python #rstats #gdal #xarray
:python: tip
https://www.scivision.dev/xarray-to-netcdf-compression/
#xarray to_netcdf() can compress data variables to save considerable disk space, but it is not trivial until your read the docs.
A bunch of tools coming together... was revamping a soon-to-be released package that streamlines some data IO and analysis for #seismic datasets that contains pipelines to #yt (and #yt-xarray), #xarray and #geopandas. The package came out of my need to extract data for some k-means analysis, but it also makes it very easy to pop data over to yt and in this case, #yt_idv!
#seismic #yt #xarray #Geopandas #yt_idv #python #opensource #datavisualization
Pretty useful, thanks Luke Gloege https://towardsdatascience.com/xarray-recipes-for-earth-scientists-c12a10c6a293 #dataAnalysis #climatology #xarray #matplotlib #pandas #fediscience
#dataanalysis #climatology #xarray #matplotlib #pandas #Fediscience
☯️ #zen3geo v0.6.1. Quick patch release with full support for 🐍 Python 3.11! Made a couple of bug fixes 🐞 to DatashaderRasterizer (for turning points/lines/polygons to pixel masks based on a template raster). Also sneaked in a tiny enhancement ✨ to XpySTACAssetReader (which wraps around `xarray.open_dataset`) to work without xpystac 🙃. Find out more in the release notes at https://zen3geo.readthedocs.io/en/v0.6.1/changelog.html.
@pytorch #TorchData #Python #DataShader #xarray #gischat #geospatial #datapipeline #computervision
#zen3geo #torchdata #python #datashader #xarray #gischat #geospatial #datapipeline #computervision
Preparing demos for next week at #jupytercon. Here is a sneak peek (turn on the audio, best with headphones). See you there!
💨💨 🔊💨💨
#jupytercon #sonification #xarray #ipyleaflet #ipytone #jupyter
@conorosully @lavergnetho Also, consider getting a Microsoft Planetary Computer account. Although Google's equivalent has been around longer, #PlanetaryComputer allows you to do these things easily with standard Python tools and libraries (eg #xarray )
☯️ #zen3geo v0.6.0. Scale to new heights if you dare. Open 📂 STAC Assets backed by Zarr, NetCDF or CloudOptimizedGeoTIFF files using the new XpySTACAssetReader DataPipe. Attempt super-resolution 🔭 with the walkthrough on handling multi-resolution CMIP6 climate data from CarbonPlan over the Philippines 🇵🇭 at https://zen3geo.readthedocs.io/en/v0.6.0/multi-resolution.html. Clarity, seek it, give it 🥽
@zarr #STAC #NetCDF #CloudOptimizedGeoTIFF #xarray @pytorch #DataPipe #SuperResoluton #MultiResolution #CarbonPlan #CMIP6 #ClimateModel
#zen3geo #STAC #netcdf #cloudoptimizedgeotiff #xarray #DataPipe #superresoluton #multiresolution #carbonplan #cmip6 #ClimateModel
STAC + stackstac + xarray is a powerful combination. I put together a short post highlighting my favorite new tools for working with satellite imagery.
https://hrodmn.dev/posts/stackstac/
#python #geospatial #xarray #eo #dataengineering #stac #remotesensing
#remotesensing #stac #dataengineering #eo #xarray #geospatial #Python
yt_xarray release day! v0.1.1 now available on pypi!
https://yt-xarray.readthedocs.io/
yt_xarray is an xarray accessor that streamlines communication between #yt and #xarray, allowing you to easily pull out a yt dataset from regularly gridded datasets that xarray can load, allowing you to more easily use yt's plotting and analysis methods. (1/N)
#yt #xarray #python #opensource #geoscience
Also uses for #xarray #python #RandomDungeonGenerator
#xarray #python #randomdungeongenerator
Hey #RemoteSensing enthusiasts! I've also released #spyndex v0.3.0 and now you have access to #AwesomeSpectralIndices v0.3.0 in #Python! 🐍🛰️🌿
Check it here: https://github.com/awesome-spectral-indices/spyndex
You can use it for #numpy, (geo)pandas, #xarray, #dask and #EarthEngine! 🚀😉
#remotesensing #spyndex #awesomespectralindices #python #numpy #xarray #dask #EarthEngine #gischat #eochat
Hey #Python package #maintainer s, here's a question - is there anything you wish you knew before you switched to date based versioning?
Asking for a friend... and a bunch of other friends... And myself... And our package :D we've been discussing the switch at https://github.com/PlasmaPy/PlasmaPy-PLEPs/pull/39 . I think we've got most of the edge cases covered, following #Dask and #xarray's example - but can't hurt to ask, right?
#python #maintainer #dask #xarray
hidefix (https://github.com/gauteh/hidefix) can now be used as an #xarray backend: more than 6x faster on a 8 CPU machine!
This reads a full variable of about 370mb.
On a machine with 64 CPUs reading with hidefix is between 10x and 16x faster than native HDF5!!
And that is _including_ generating the index.
Going from 20 seconds to 1.8 seconds is pretty significant when reading a bunch of variables! 7s to 380ms, is the difference between waiting and immediately in processing!
This is for a file on a network disk.
The ultimate goal: Make this into a #xarray backend!
#xarray #rust #hidefix #hdf5 #data #met
Today I've been continuing some work on an experimental #xarray + #dask + #yt pipeline. It's been interesting figuring out how to align the xarray-dask array chunking with yt's domain decomposition and indexing. This meaningless but kinda fun looking image is a plot made with yt using a ~10Gb collection of netcdf files loaded via xarray and dask. yt does some initial culling of files to load then hands off the actual read to xarray and dask. #python #OpenSource
#xarray #dask #yt #python #opensource
Today I've been continuing some work on an experimental #xarray + #dask + #yt pipeline. It's been interesting figuring out how to align the xarray-dask array chunking with yt's domain decomposition and indexing. This meaningless but kinda fun looking image is a plot made with yt using a ~10Gb collection of netcdf files loaded via xarray and dask. yt does some initial culling of files to load then hands off the actual read to xarray and dask. #python #OpenSource
#xarray #dask #yt #python #opensource
Working with #NEMO gcm and #python?
I just released version 0.4.0 of #xnemogcm, a package that can read nemo #netcdf outputs, and adds the proper attributes (+ change dimension names) to be read by the fantastic #xgcm package.
It tries to automatically discover the T, U, V, W, etc point of variables, and sets everything for you :). For more complicated cases (dataset already loaded into #xarray, dataset with no attributes, etc) you can easily set the point by hand.
#xarray #xgcm #netcdf #xnemogcm #python #nemo
@seaice @carlosmoffat i am not so sure, Lars, but it is true that this is a legacy software that does not use #xarray. At least it is #python3 🙂