Not only is the product of this #DataPipeline kept a secret by "open" companies, the very nature of the pipeline is likewise cloaked in mystery, in order to obscure the exploitative labor relations it embodies (the joke that "AI" stands for "absent Indians" comes out of the South Asian clickwork industry).

30/

☯️ #zen3geo v0.6.1. Quick patch release with full support for 🐍 Python 3.11! Made a couple of bug fixes 🐞 to DatashaderRasterizer (for turning points/lines/polygons to pixel masks based on a template raster). Also sneaked in a tiny enhancement ✨ to XpySTACAssetReader (which wraps around `xarray.open_dataset`) to work without xpystac 🙃. Find out more in the release notes at https://zen3geo.readthedocs.io/en/v0.6.1/changelog.html.

@pytorch #TorchData #Python #DataShader #xarray #gischat #geospatial #datapipeline #computervision

1/ 🏗️ When building a data pipeline, focus on scalability, reliability, and maintainability to ensure long-term success. 🚀

A thread 🧵👇🏻

#DataScience #DataEngineering #DataPipeline #cloud #python

From 0 to a complete #streaming #datapipeline in ~12 minutes with @aiven_io for #ApacheFlink!

Check it out!

https://www.youtube.com/watch?v=RT5zjEWc1mg

Resonance Calendar for Week 50, 2022

There were a few posts I read this week which resulted in my sitting back and saying to myself: “Hmmmm”.

Yes, all the readings are nominally about Data Engineering, but they tend to point out the both technical issues and the pragmatics of Data Engineering, particularly the complexity of constructing a ‘modern data stack’.

#DataEngineering #DataContract #DataPipeline #DataHusbandry

https://medium.com/@rhm2k/resonance-calendar-for-50-2022-3341b2b94339

@estrenuo Seeing the word "wrangle" in this diagram made me chuckle, because I would only ever use it to describe something done to a domesticated animal 🐄

It validates the feeling I have that my lexicon is ill equipped to concisely describe a #DataPipeline 🫠

Amazon Managed Streaming for Kafka (MSK) https://aws.amazon.com/msk/ #bookmark #scale #kafa #datapipeline

(monkinetic.blog txt-ed4d2b6)