❓❓❓HOW LAKEHOUSE TABLE FORMAT WORKS❓❓❓

1. Engine reads table format metadata
2. Builds list of files with relevant data based on metadata
3. Scans those files and executes query

#DataEngineering #DataAnalytics #BigData #DataLakehouse #ApacheIceberg #ApacheHudi #DeltaLake

🔎 Discover how #DeltaLake simplifies the process of building data lakehouses and data pipelines at scale. With this practical guide, #dataengineers, #datascientists, and #dataanalysts will explore key data reliability challenges and learn to apply modern data engineering and management techniques. You'll also understand how ACID transactions bring reliability to data lakehouses at scale!

Check out Delta Lake: The Definitive Guide ➡️ https://lnkd.in/g3-RBeUz

#opensource #oss #datalakes #lakehouse

Liquid Clustering dynamically clusters data based on data patterns, which helps to avoid the over- or under-partitioning problems that can occur with Hive partitioning.

Liquid Clustering resulted in 2.5x faster clustering relative to Z-order. In the same trial, traditional Hive-style partitioning was an order of magnitude slower due to the expensive shuffle required for writing out many partitions.

Learn more 👉 https://lnkd.in/gZ9AvE8X

#deltalake #opensource #oss #dataengineering

#DeltaLake 3.0 features automatic support for competing Apache Iceberg and Hudi table formats allowing enterprise users to eliminate complicated integration work and focus on building truly open data lakehouses. 🚀

We are excited to announce the *preview* release of Delta Lake 3.0.0. Check it out today 👉 https://lnkd.in/eeYs44H4

#opensource #dataaisummit #data

In this blog post, Shingo OKAWA delves into the #Rust ecosystem and examines its characteristics. Additionally, Shingo explores how the PyO3 crate offers a straightforward example of managing #Python/Rust FFI functionality and demonstrates how to examine the generated “glue code” produced by the PyO3 crate.

Read delta-rs as a Python/Rust FFI example Part 2 👉 https://lnkd.in/eH5w6493

#deltalake #opensource #rustlang

#DataAISummit Session Spotlight ⭐ Tune into DoorDash's journey to migrate from a flaky #ETL system with 24-hour data delays, to standardizing a CDC streaming pattern across more than 150 databases to produce near real-time data in a scalable, configurable, and reliable manner.

Register for DAIS ➡️ https://dbricks.co/3lvO1hz
View the session catalog ➡️ https://bit.ly/3MLJ7Xa

Use code ETLINUX400 to save $400 off the regular price of the full conference pass!

#doordash #deltalake #opensource

Online Meetup ⭐ TOMORROW, June 13 at 9:00 AM PDT
Learn more about Databricks Connect and Spark Connect so you can use Spark from anywhere! 🌐

Come join the awesome Simon Whiteley, CTO of Advancing Analytics to discuss with a panel including Martin Grund, Stefania Leone, and his partner in crime Denny Lee.

RSVP ➡️ https://www.meetup.com/data-ai-online/events/293994300/

#deltalake #spark #dataengineering

We are excited to share that #deltars #python bindings v0.10.0 is here! This release includes optimize #zorder, #datafusion storage catalog, concurrent file compaction, and so much more. 🦀🐍

Check it out today! ➡ https://lnkd.in/e-BhV5qa

#deltalake #rust #dataengineering #opensource #linuxfoundation #oss

I attended the #AWSSummitLondon this week. I started with AWS ~6 months ago. It was gratifying to realise that I have learned a lot since then. I talked to a few experts and they told me I was in the right path and that the struggles I have with #AWSGlue are not only mine (they simply don’t support well #deltaLake ). My perfectionist self was relieved 😌 I didn’t solve any of my problems but sometimes it helps to realise you are not as stupid as your programming struggles make u feel sometimes… 😅

ONE WEEK from today! ⭐ Join Robert Pack, Sr. Digital Expert Cloud Native Machine Learning Platform and Technology Principal at BASF as he discusses the relationship between process engineering and data engineering alongside D3L2 host, Denny Lee.

🦀 D3L2: How BASF achieves global sustainability with #DeltaLake w/ Robert Pack
🗓️ Thursday, June 15
🕗 9:00 - 10:00 AM PDT

RSVP: https://community.linuxfoundation.org/events/details/lfhq-delta-lake-presents-d3l2-how-basf-achieves-global-sustainability-with-delta-lake-w-robert-pack/

In this edition of 𝙻𝚊𝚜𝚝 𝚆𝚎𝚎𝚔 𝚒𝚗 𝚊 𝙱𝚢𝚝𝚎...

✅ Hear how Kubit uses Delta Sharing to power their product analytics platform
✅ Learn about an exciting new contribution to the Dask community
✅ Plus, 2 𝙣𝙚𝙬 𝙧𝙚𝙡𝙚𝙖𝙨𝙚𝙨 from the Delta Lake and Delta Sharing projects!

Read along or watch us on YT! https://lnkd.in/e-UuBFPa

#deltalake #opensource #oss #linuxfoundation

There are a variety of ways to create #deltalake tables. You can create a Delta table by writing out a DataFrame with the Delta format, you can create an empty Delta table with #sql, or you can convert an existing Parquet table to the Delta format. Very easy to jump in and start using Delta Lake.

cc Matthew Powers, CFA

#opensource #oss #linuxfoundation

Databeans has a handbook written by their engineers that includes advice and recipes on data, particularly #deltalake. For a while, this handbook was kept a secret, but they've recently chosen to share certain pages with you!

cc Houssem Eddine Dalhoumi, DataBeans

#opensource #dataengineering #datalakes #databeans #linuxfoundation

🦀 Watch D3L2: Discussing Rust, Ballista, Ray SQL, DataFusion with Andy Grove on YouTube: https://www.youtube.com/watch?v=NEL6DluUxgw

#datafusion #raysql #ballista #opensource #deltalake

📣 We are excited to announce the release of Delta Lake 2.4.0 on Apache Spark 3.4. Similar to Apache Spark™, we have released Maven artifacts for both Scala 2.12 and Scala 2.13! 🎉

Documentation: https://lnkd.in/eTD9ua_6
Python artifacts: https://lnkd.in/e65AeChW

⭐ View the release notes: https://lnkd.in/er2PDhjJ

#deltalake #opensource #oss #data #apachespark

Join us on Thursday, May 25th for D3L2: Discussing Rust, Ballista, Ray SQL, DataFusion with Andy Grove! 🦀

Andy Grove has been specializing in query engines and distributed systems. Among many of his accolades, he started the DataFusion and Ballista query engine projects and donated both to the Apache Software Foundation as part of the Apache Arrow project. He also donated the initial Rust implementation of Apache Arrow and recently created Ray-SQL.

RSVP: https://community.linuxfoundation.org/events/details/lfhq-delta-lake-presents-d3l2-discussing-rust-ballista-ray-sql-data-fusion-with-andy-grove/

#deltalake

We are excited to announce the *preview release* of Delta Lake 2.4.0 on Apache Spark 3.4! 🎊 Similar to Apache Spark™, we have released Maven artifacts for both Scala 2.12 and Scala 2.13.

🌟Documentation: https://lnkd.in/eBwUGV-2
🌟Maven artifacts: https://lnkd.in/eex2VxMm
🌟Python artifacts: https://lnkd.in/eQf_B4eM

View the key features in this release: https://lnkd.in/eYkMGpTd

#deltalake #opensource #spark #oss #dataengineering

Wondering if you should hop on the #Rust bandwagon? This video covers why Rust is exciting, especially for Python (data) developers! 😁

https://lnkd.in/ewVq84r2

#opensource #python #dataengineering #deltalake #oss #developers

Learn about the latest innovations with #LLMs like #Dolly and other open source Data + AI technologies such as Apache Spark™, #DeltaLake, #MLflow & Delta Sharing at #DataAISummit!

📍 San Francisco, CA
🗓️ June 26 - 29, 2023

🌟 Save $400 off the regular price of the full conference pass using code ETLINUX400 (expires 6/2).

Register here: dbricks.co/3lvO1hz

#data #oss #ai

Get a detailed overview of #DeltaLake, #ApacheHudi, and #ApacheIceberg as we discuss their data storage, processing capabilities, and deployment options https://dzone.com/articles/delta-hudi-and-iceberg-the-data-lakehouse-trifecta

#analytics #spark