🚀 🚀 🚀Just published a comprehensive new blog post on dockerizing your #Python-based ETL pipelines!

Delve into the varied benefits and detailed instructions of Docker packaging with us.

⚡ Learn how to create a virtual environment, and package a Python pipeline.
⚡ Learn about procedure of building and running Docker images while emphasizing the importance of distinct virtual environments for each project.

🔗 https://ploomber-sql.readthedocs.io/en/latest/packaging-your-sql-project/containerizing-etl-with-docker.html

#Docker #ETLPipelines #DataEngineers #DevOps

🔎 Discover how #DeltaLake simplifies the process of building data lakehouses and data pipelines at scale. With this practical guide, #dataengineers, #datascientists, and #dataanalysts will explore key data reliability challenges and learn to apply modern data engineering and management techniques. You'll also understand how ACID transactions bring reliability to data lakehouses at scale!

Check out Delta Lake: The Definitive Guide ➡️ https://lnkd.in/g3-RBeUz

#opensource #oss #datalakes #lakehouse

another good quick comparison among data professionals 👇 💡
credit: Hamidreza haddad

#data #dataanalysis #dataengineers #graphs

It's the last day of February, so it's time for the February 2023 edition of the Iceberg Community News. Lots of great updates to #ApacheIceberg and #PyIceberg as well as adoption from StreamSets Inc., ClickHouse and Databend

#iceberg #datalakehouse #datalake #dataengineering #dataengineers #python
https://tabular.substack.com/p/february-2023-iceberg-community-news

Any #DataEngineers looking for a senior tertiary sector role in #Sydney? Look upthread for post from @kcarruthers.

Any #data #engineers #dataengineers from #google here wanna help me understand how much of the sw/eng whiteboard algo/datastructures I need to know for the #datawarehouse interview?

Do you also think that #DataEngineers are the same as data scientists or analysts? Hear from Steffen Klempau, our expert in Data Engineering, as he talks about how Data Engineer is a role on its own.
Check our Capgemini #Data and #AI community page here https://bit.ly/3gQTdKn and tap into the exciting world of Data Engineers.
#LifeAtCapgemini #GetTheFutureYouWant #DataScience

this was a really fun conversation!
---
RT @DataStackShow
Much of today’s data tooling wasn’t really built with data people in mind, so the DX kinda sucks.

Tune in to this week’s show with @meltanodata's @tayloramurphy for a great conversation on purpose-building tools for #dataengineers.

https://bit.ly/3URabq4
https://twitter.com/DataStackShow/status/1603028666509828097

@BM_DataDowntime

1. more companies moving to serverless if they aren't already
2. data contracts taking precedence just like ethics for data science folks
3. a little more education around batch vs streaming architectures
4. more jobs in #dataengineering - more specializations as well in #data teams
5. #Dataengineers being a key part of the founding team than just an engineer to support #datascience workflow