At least there are couple of options when we are working with Multi-index data that can be particularly helpful for slicing or filtering data:

#python #pandas #datascience #dataframe

- pandas.DataFrame.xs
- pandas.IndexSlice

https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.xs.html

https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.IndexSlice.html?highlight=indexslice#pandas-indexslice

During a peculiar discussion in #PyDataLondon2023 (mostly among #DataFrame users - some decent #Python performance experts among them), CSV was mentioned as the worst file format for loading data. Yet, I can't stop thinking about this #Snowflake document which says otherwise.

Are we sure the issue isn't with DataFrame libraries?!

https://community.snowflake.com/s/article/How-to-Load-Terabytes-Into-Snowflake-Speeds-Feeds-and-Techniques

Polars released this week a new version (0.16.17). The Polars library is a multiple languages (Python, Rust, Node.js) framework for working with a tabular data frame in memory using the Apache Arrow engine🚀.

Resources 📚
Source code: https://github.com/pola-rs/polars
Release notes: https://github.com/pola-rs/polars/releases/tag/py-0.16.17
Documentation: https://www.pola.rs/

#python #rust #dataframe #data #DataScience #dataengineering

Hoy traigo una librería de #Python que desconocía, sidetable, tal vez sea conocido pero yo no me lo había encontrado.

:python: https://pypi.org/project/sidetable/

Más información:

🔗🇬🇧 https://pbpython.com/sidetable.html

🔗🇬🇧 https://deepnote.com/@deepnote/Sidetable-Gives-You-the-Pandas-Methods-You-Didnt-Know-You-Needed-a41e53f6-2719-45fe-9678-5b47f68abb3f

🔗🇪🇸 https://www.analyticslane.com/2021/12/13/creacion-de-tablas-resumen-en-python-con-sidetable/

#DataScience #Pandas #Dataframe #Tables #DataScientist #analysis #análisis #Librería #research #Investigación #Library #openscience #OpenAccess

Today I got stuck in automation hell 🔥 . Wanted a script that performs #simulations, read the results, push them into a #DataFrame and write that frame to a #CSV. While that sounds reasonable I quickly encountered the fact that I didn't use a coherent naming scheme for my simulations 😩

In one hour I will be giving a talk about #Polars, my new favourite #Python #dataframe library! Join me at #PyDataGlobal:

https://pydata.org/global2022

#JuliaBeginners #JuliaTipOfTheDay​
2022-11-26

When you see `foo.bar` in your #JuliaLang code, Julia sees that as the call `getproperty(foo, :bar)`.

Let's say you load data into a #DataFrame `df`. When you access `df.Species`, that calls `getproperty(df, :Species)`.
DataFrames.jl has a matching method for that call, which returns `df[!, :Species]` i.e. makes it a column access.

You can do that for your custom types too! (Useful to understand "fields" before that, a post on those is upcoming.)

🦘 Kangas
Explore multimedia datasets at scale
#machinelearning #dataanalysis #datagrid #dataexploration #dataframe #dataset

https://github.com/comet-ml/kangas

Rule number 1 of #r : never use #tibble or #dataframe if you don't have to. 🤐