Taras Novak 🇺🇦 · @dataSamurai
143 followers · 155 posts · Server vis.social
Devs and data scientists really like our #ChicagoCrimes EDA public scripts, notebooks 📚 and data snapshots repository I created last October. That sample data/demo repository covers many different tools, libraries and notebooks to parse #LargeData:
⭐️ ⤑ https://github.com/RandomFractals/chicago-crimes
📜 ⤑ https://twitter.com/search?q=(%23ChicagoCrimes)%20(from%3ATarasNovak)&src=typed_query
#DataTools 🛠️ ...
#datatools #largedata #ChicagoCrimes
Taras Novak 🇺🇦 · @dataSamurai
128 followers · 120 posts · Server vis.social
Quick demo of our new #DuckDBSqlTools vscode extension loading and querying 7,687,725 #ChicagoCrimes recorded in 2001 through the end of November 2022 from a large 1.68 GB CSV data file in seconds ... See demo gif at:
📰 https://github.com/RandomFractals/chicago-crimes#with-duckdb-sql-tools
#datatools #vscode #sqltools #duckdb #ChicagoCrimes #duckdbsqltools
Taras Novak 🇺🇦 · @dataSamurai
123 followers · 115 posts · Server vis.social
Our new #DuckDBSQLTools VSCode extension is almost ready for prime time.
You'll be able to load remote CSV and #parquet data files via httpfs extension and create in-memory #DuckDB instances too.
See demo gif of loading #ChicagoCrimes parquet data from a GitHub repository into memory, creating a CrimeReports table, and querying it on twitter:
https://twitter.com/TarasNovak/status/1617542770184577024
#VSCode #SQLTools / #DataTools 🔬💎💎💎...
#datatools #sqltools #vscode #ChicagoCrimes #duckdb #parquet #duckdbsqltools
Taras Novak 🇺🇦 · @dataSamurai
94 followers · 99 posts · Server vis.social
📢 New 2022 #ChicagoCrimes #MalloyData #Fiddle app you can run in a browser to learn Malloy model and query semantics, nesting, and simple charting 📊📈 options:
📰 https://randomfractals.github.io/chicago-crimes/apps/malloy-fiddle
📓 https://github.com/RandomFractals/chicago-crimes#with-malloy-fiddle
#DAX #dataApps #dataTools 🔬 ...
#datatools #dataapps #dax #fiddle #MalloyData #ChicagoCrimes
Taras Novak 🇺🇦 · @dataSamurai
88 followers · 98 posts · Server vis.socialUpdated #ChicagoCrimes #PyScript #dataApp with gzipped CSV (~3.25MB). The app now loads 215,551 crime reports with #pyodide in a browser in about 8 seconds total for the #Python runtime, data transformation with #pandas 🐼 & charting with #Altair 📊📈
https://randomfractals.github.io/chicago-crimes/apps/pyscript/
#altair #pandas #python #pyodide #dataapp #Pyscript #ChicagoCrimes
Taras Novak 🇺🇦 · @dataSamurai
88 followers · 97 posts · Server vis.social
Running some quick data summary queries with #Malloy on a 2001-2022 #ChicagoCrimes parquet data file that is 533MB, created form a larger 1.66GB CSV data, without any compression. Very responsive and fast query execution thanks to #DuckDB and Malloy #VSCode extension.
View those queries in action in this GIF: https://twitter.com/TarasNovak/status/1601650935402725376
#dataTools 🛠️ ...
#datatools #vscode #duckdb #ChicagoCrimes #malloy
Taras Novak 🇺🇦 · @dataSamurai
79 followers · 93 posts · Server vis.socialOur #DataPreview 🈸 for #vscode now has over 350,000 installs. You can load large CSV files, sort & graph results with aggregate functions, and much more.
See an example of loading 48MB of #ChicagoCrimes CSV data: https://twitter.com/TarasNovak/status/1600439658810585088
Note: change data.preview.theme to light. See: https://github.com/RandomFractals/vscode-data-preview#configuration
📥 https://marketplace.visualstudio.com/items?itemName=RandomFractalsInc.vscode-data-preview
#dataViz 📊📈 #dataTools 🛠️ for #dataScientists ...
#datascientists #datatools #dataviz #ChicagoCrimes #vscode #datapreview
Taras Novak 🇺🇦 · @dataSamurai
79 followers · 91 posts · Server vis.social📢 New 2022 #ChicagoCrimes #dataApp with #Malloy #Composer is out. See:
code -> https://github.com/RandomFractals/chicago-crimes/tree/main/apps/malloy-composer
app -> https://randomfractals.github.io/chicago-crimes/apps/malloy-composer
Try it in your browser. Have fun!
#dataTools/#dataApps 🔬 ...
#datatools #composer #malloy #dataapp #ChicagoCrimes
mariana meireles ♡ · @marimeireles
151 followers · 35 posts · Server kolektiva.socialso cool! :)
---
RT @TarasNovak
Created a web page with #PyScript loading 2022 #ChicagoCrimes CSV data with #pandas and visualizing that data with #Altair charting lib:
https://github.com/RandomFractals/chicago-crimes/blob/main/apps/pyscript/index.html
#dataApps ...
https://twitter.com/TarasNovak/status/1597379498756173824
#Pyscript #ChicagoCrimes #pandas #altair #dataapps
Taras Novak 🇺🇦 · @dataSamurai
77 followers · 89 posts · Server vis.social2001-2022 #ChicagoCrimes data loaded from a #parquet file and summarized with #pandas #dataFrames and simple #Altair charts 📊📈 in a #JupyterNotebook:
📓 https://github.com/RandomFractals/chicago-crimes/blob/main/notebooks/chicago-crimes-altair.ipynb
#dataNotebooks 📚 ...
#datanotebooks #jupyternotebook #altair #DataFrames #pandas #parquet #ChicagoCrimes
Taras Novak 🇺🇦 · @dataSamurai
76 followers · 88 posts · Server vis.socialCreated a web page with #PyScript loading 2022 #ChicagoCrimes CSV data with #pandas and visualizing that data with #Altair 📉📊 charting lib:
🧐 https://github.com/RandomFractals/chicago-crimes/blob/main/apps/pyscript/index.html
#dataApps ...
#dataapps #altair #pandas #ChicagoCrimes #Pyscript
Taras Novak 🇺🇦 · @dataSamurai
74 followers · 87 posts · Server vis.social
So, I ported my old #ChicagoCrimes #pandas data wrestling with #matplotlib plots to new repo. See this #JupyterNotebook with many crime data summaries and plots from 2001 to present:
🧐 https://github.com/RandomFractals/chicago-crimes/blob/main/notebooks/chicago-crimes-pandas.ipynb
#dataNotebooks 📚 ...
#datanotebooks #jupyternotebook #matplotlib #pandas #ChicagoCrimes
Taras Novak 🇺🇦 · @dataSamurai
64 followers · 86 posts · Server vis.social
Hey #dataNerds 🤓, good news:
#DuckDB v0.6.0 brings reading #CSV data on par with #PyArrow & #Polars and loads 1.66 GB of #ChicagoCrimes data in 1.9s with 12 cores/24 threads when experimental parallel CSV reader & unordered insertion are enabled.
🧐 https://github.com/RandomFractals/chicago-crimes#with-duckdb
#dataTools 🔬 ...
#datatools #ChicagoCrimes #polars #pyarrow #csv #duckdb #datanerds
Taras Novak 🇺🇦 · @dataSamurai
63 followers · 84 posts · Server vis.social
Displaying #ChicagoCrimes parquet data with #Malloy charts, imported table data source, measures, reusable queries, limits, nested grouping and bar chart renderer:
🔬 https://github.com/RandomFractals/chicago-crimes#with-malloy-charts
#VSCode #DataVis 📊 #DataTools ...
#datatools #datavis #vscode #malloy #ChicagoCrimes
Taras Novak 🇺🇦 · @dataSamurai
52 followers · 83 posts · Server vis.social
Updated #ChicagoCrimes with #MalloyData tools example & more info in docs:
🧐 https://github.com/RandomFractals/chicago-crimes#with-malloy-data
Clone that data repo & install Malloy #vscode extension to try it out:
📥 https://marketplace.visualstudio.com/items?itemName=malloydata.malloy-vscode
#dataTools 🔬 ...
#datatools #vscode #MalloyData #ChicagoCrimes
Taras Novak 🇺🇦 · @dataSamurai
47 followers · 82 posts · Server vis.social
I've decided to try #MalloyData today.
Here is a quick example of loading #ChicagoCrimes 2022 #parquetData with #DuckDB and Malloy queries for some rough counts and data summaries:
https://github.com/RandomFractals/chicago-crimes/issues/24
#dataTools 🛠️ ...
#datatools #duckdb #parquetdata #ChicagoCrimes #MalloyData
Taras Novak 🇺🇦 · @dataSamurai
47 followers · 82 posts · Server vis.social
so, you want to parse #parquet data with #Polars like a #Jedi?
🔭 this #ChicagoCrimes data loading repo: https://github.com/RandomFractals/chicago-crimes
Pertinent thread on the #DeathStar ✴️: https://twitter.com/TarasNovak/status/1584592756168863745
#dataTools 🛠️ ...
#datatools #deathstar #ChicagoCrimes #jedi #polars #parquet
Taras Novak 🇺🇦 · @dataSamurai
47 followers · 82 posts · Server vis.social
I see some Observable fans here and others wanting to learn how to construct #JSNotebooks. You can explore our old #ChicagoCrimes 📚: https://observablehq.com/@randomfractals/leaflet-pixi-overlay?collection=@randomfractals/chicago-homicides