In the Pipeline: September 2023 edition! 🔶

This month: a roundup of the summer’s Kedro news (including Python 3.11 support in *all the things*), some release updates, and our top picks from recent articles.

https://kedro.org/blog/in-the-pipeline-kedro-news-09-23

#kedro #python #pydata #datascience

More exciting news today!

`kedro-pandera` is a new community plugin that brings data validation to your Kedro projects 🔶

With it, you can

📝 declare data schemas to your kedro datasets
🧪 add data tests
🤡 run test pipeline with fake data

and more!

Install it with `pip install kedro-pandera` and give the repository a star ⭐️

https://github.com/Galileo-Galilei/kedro-pandera

Thanks @Galileo-Galilei for creating it!

#kedro #pandera #pydata #python #data #datascience

And one more release! Kedro-Viz 6.5.0 is available 🔶 and adds official Python 3.11 support as well, along with some small bug fixes.

Install Kedro-Viz now with pip or conda:

```
pip install kedro-viz==6.5.0
conda install -c conda-forge kedro-viz=6.5.0
```

#kedro #kedroviz #dataviz #pydata #datascience

Kedro 0.18.13 and kedro-datasets 1.6.0 are out! 🔶

These releases bring official support for Python 3.11 (finally!), as well as some new `OmegaConfigLoader` features, a new `kedro catalog resolve` command, and a leaner project structure, as well as significant documentation improvements. Preparations for 0.19.0 are underway!

Install Kedro now with pip or conda:

```
pip install kedro==0.18.13
conda install -c conda-forge kedro=0.18.13
```

#kedro #python #pydata #datascience #machinelearning

It's release day again! 🔶

Kedro-Viz 6.4.0 is out, with hint cards that highlight key features of the application and support for displaying dataset statistics in the metadata panel for further investigation, aside from lots of bug fixes.

kedro-datasets 1.5.3 fixes some problems with optional dependencies, and includes other bug fixes and improvements.

Install them with `pip install kedro-viz==6.4.0` and `pip install kedro-datasets[{desired-extra}]==1.5.3` respectively!

#python #kedro #pydata

New blog post: How to integrate Kedro and Databricks Connect 🔶

In this blog post, our colleague Diego Lira explains how to use Databricks Connect with Kedro for a development experience that works completely inside an IDE.

https://kedro.org/blog/how-to-integrate-kedro-and-databricks-connect

Install it with

```
pip install databricks-connect
```

#kedro #python #pydata #datascience #databricks #dbx #spark #pyspark

Last day of summer vacations. Had so much fun and rested a lot.

Tomorrow, more #python, more #kedro, and more of everything 💗

It's release week 🔥

🔶 kedro 0.18.12 with dataset factories
🔶 kedro-datasets 1.5.1 with lazy loading and `pandas.DeltaTableDataSet`
🔶 kedro-airflow 0.6.0 with an option to configure DAG kwargs using `airflow.yml`
🔶 kedro-telemetry 0.2.5 paving the way for Python 3.11 support

Thanks to our various community contributors! 💛 Check out the complete release notes in our GitHub organisation https://github.com/kedro-org/

#kedro #python #pydata #datascience #datapipelines

Our colleague @astrojuanlu will be at #PyConEstonia2023 in September leading a workshop titled "Refactor your Jupyter notebooks into maintainable data science code with Kedro"

Don't miss it! https://pycon.ee/

#python #kedro #pydata #datascience #jupyter

Ever wondered how to use Kedro on Snowflake? Check out "From 0 to MLOps with ❄️ Snowflake Data Cloud in 3 steps with the Kedro-Snowflake plugin" by our colleagues from GetInData | Part of Xebia

https://getindata.com/blog/from-0-to-mlops-with-snowflake-data-cloud-3-steps-kedro-snowflake-plugin/

#kedro #python #snowflake #mlops

New blog post: A new Kedro dataset for Spark Structured Streaming 🔶

This post illustrates the extensibility of Kedro with a new dataset for real-time data processing using Spark Structured Streaming. It was written by our colleagues Tingting Wan, Tom Kurian, and Haris Michailidis all Data Engineers at QuantumBlack.

https://kedro.org/blog/kedro-dataset-for-spark-structured-streaming

Install it with

```
pip install "kedro-datasets[spark]~=1.4"
```

#kedro #datascience #python #pydata #spark #streaming

In the Pipeline: July 2023 edition! 🔶

In the past couple of months we celebrated our four year anniversary as an open source project, unveiled our new branding, made eleven releases (framework, Viz and datasets), and celebrated in person ❤️

https://kedro.org/blog/in-the-pipeline-kedro-news-07-23

#kedro #python #pydata #datascience

New blog post: How to use Databricks managed Delta tables in a Kedro project 🔶

In this post our colleague Jannic Holzer explains how to use a newly-released dataset for managed Delta tables in Databricks within your Kedro project.

https://kedro.org/blog/managed-delta-tables-kedro-dataset

Install it with

```
pip install "kedro-datasets[databricks.ManagedTableDataSet]"
```

#kedro #machinelearning #datascience #databricks #spark #python #pydata

Kedro-Viz 6.3.2 is out! 🔶

We added support for dataset previews and validation for layers in transcoding datasets. In addition, Viz now shows original node input and output names in the metadata panel, and we fixed a number of small issues.

Install it now for Python or JavaScript:

```
pip install kedro-viz==6.3.2
npm install @quantumblack/kedro-viz@latest
```

And check out the online demo! https://demo.kedro.org/

#kedro #kedroviz #pydata #python #datascience #machinelearning #dataviz

Kedro 0.18.11 is out! 🔶

We added added a new `databricks-iris` official starter and significantly improved the documentation around both Databricks and Prefect 2.0. We also deprecated some class names.

Install it now with pip or conda:

```
pip install kedro==0.18.11
conda install -c conda-forge kedro=0.18.11
```

And read the complete release notes online: https://github.com/kedro-org/kedro/releases/tag/0.18.11

#kedro #python #pydata #datascience #machinelearning

On my way to London to meet the wonderful @kedro team 💖🔶 #kedro

New blog post: How to build a custom Kedro runner 🔶

In this post our colleague Nok Lam Chan explains how to write an efficient custom Kedro pipeline runner that continues executing a set of nodes after encountering non-catastrophic failure points.

https://kedro.org/blog/build-a-custom-kedro-runner

#kedro #kedroviz #MachineLearning #datapipelines #python #PyData

Testing Kedro on Jupyter (Classic) Notebook 7.0.0rc, released mere minutes ago 😍 Looks like it's working nicely!

#python #kedro #jupyter #notebook #pydata

Some pictures from yesterday's talk about Kedro 😍 And remember to follow @kedro on Mastodon!

#kedro #pydata #python

Nos vemos esta tarde en el último concierto de la gira 🤠 "Kedro: la herramienta que une a Data Scientists y Developers" en las oficinas de Avoris en Palma de Mallorca

https://www.meetup.com/bigdata-ml-ai-mallorca/events/293790799/

#kedro #python #pydata #meetup