We are excited for the #deltars 0.10.0 release which includes:
⭐ Support #optimize on non-append-only tables
⭐ Upgrade to #apachearrow 37 and #datafusion 23
⭐ Fixed various #datafusion bugs and fixed querying partitioned tables and other fixes

View the release notes here: https://github.com/delta-io/delta-rs/releases/tag/rust-v0.10.0

#deltalake #rust #linuxfoundation #dataengineering

⚡ 12 Open Source Projects to Watch in 2023

The technologies and communities defining the way we work with data, analytics, and machine learning.

— Voltron Data
#OpenSource
#DataScience
#Analytics
#MachineLearning
#Python
#ApacheArrow
https://voltrondata.com/resources/12-open-source-projects-to-watch-2023

#ChatGPT attempting some #Perl bindings for #ApacheArrow #BigData #ML

RT @rustjobs_dev
🚨 Remote #Rust Job 🦀

@liquidanalytics is building next-generation analytics engines with Apache Arrow and Rust.

They are #hiring #RustLang engineers.

Location: Remote Globally

Tech: #RustLang #ApacheArrow #ApacheDatafusion

Find out more 👇
https://rustjobs.dev/featured-jobs/Liquid-Analytics-Senior-Rust-Platform-Engineer-JSdo5cqeoUhX7cNjYpq6

End of an era as we ran our last ever #BlazingSQL production job this morning. It was a great product of its time, #GPU accelerated #SQL on top of #RAPIDSai #CUDF, but it hasn't been supported for over a year. The job will be migrated to #DuckDB and #ApacheArrow for its next scheduled run in December.

With data from the UK #Census2021 now being published we decided to build a new app from scratch, rather then modify our 2011 code. A lot of things have changed in 10 years and we didn't have #ApacheArrow back then. This seemed like a good opportunity to test out #CICD on #Kubernetes with code updates connected to our private GitHub repository. It's just so easy, code changes are automatically integrated.

New 'Census2021 bulk datasets are being regularly published so I've automated acquisition & ETL on our 'Sorites' #Kubernetes cluster. Processing is completely #ApacheArrow native. Tables are converted to Parquet, metadata stored as JSON & queries performed in #duckdb. Unlike when I did this in 2011, there's no traditional RDBMS involved.