Maybe it just because I've worked with a lot of data that's not customer-centric, but I'm struggling to see a lot of value in Activity Schema. Has anyone on #Datadon used it?

Part 2 of my Simplifying Data Orchestration seires is out on Shipyard's blog!

A no-nonsesne guide on *WHAT* is #DataOrchestration?

Let's simplify together! 😊🤸‍♀️

Special thanks to: Bill Inmon, Monica Miller, Elena Dyachkova, Rez Khan, Zach Hendlin, & Steven J. Pope for the community contributions in this post.

#dataengineering #datadon

https://www.shipyardapp.com/blog/what-is-data-orchestration/

🤔 Hmmm I wonder how many times this GitHub action ran in total each week recently?

🦸🏻The lovely #GitHub gh CLI and #DuckDB to the rescue! 🪄

Composability and Pipes FTW 💪🏻

#datadon #openSource

I'm leading a live product demo tomorrow (Thursday, I mean). Please join me! Come see what I do at work!

I'll help you get started with data-diff, and I'll sneak preview Datafold's VS Code extension.

data-diff helps you catch data changes that other testing frameworks will miss, by comparing every single value between dev and prod, with one simple command: data-diff --dbt.

Register here: https://www.datafold.com/virtual-hands-on-lab

#datadon

@scott_bot

Probably want some # for that.

#Fedijobs #DataScience #DataScientist #PublicCommunications #Statistics #DataVisualisations #SocSci #Humanities #USJobs #Datadon

Let's see if there are boost groups:

@datadon @datascience

OK that's all I can find for you.

Good luck!

@al_merose Saw your toot because of the data hashtag 👌... so +1 for hashtags as a method of curation.

I find the data hashtag itself is a bit 'messy', but at the same time not well used, at least on the Fosstodon federated timeline.

Personally I find the #datadon and #dataengineering hashtags more engineering-relevant 👀

✍🏻 Blogged: Using @lakeFS with #ApacheIceberg https://lakefs.io/blog/using-lakefs-with-apache-iceberg/

#datadon #opensource #dataengineering @apacheicebergdevs @tabular

Hey, #python and #rstats #datadon analysts and data journalists, there is a remote for the right candidate data scientist vacancy with Carbon Brief, developing data based stories.

I'm not connected in any way, but was asked to pass it on.

https://www.carbonbrief.org/vacancy-data-scientist/

I'm super excited to announce I have joined the BNG sports media team to provide analytical content on a weekly basis!

You can continue to follow me on Substack for my broader hockey musings, too.

#HNOM #Hockey #HockeyDon #NHLBruins #Analytics #datadon @bostonbruinsgameday @hockey @hnom

https://blackngoldhockey.com/2023/06/welcome-to-the-bruins-stats-corner/

Booked flights to London for #dbtCoalesce. Last year was such a great kickoff for my data career, and I’m excited to go back with some real experience under my belt this time #datadon

There comes a time when every open core project will disappoint you, when they will put an exciting feature behind their paid wall.

Sad to see dbt labs get there, and switching in real time: https://github.com/dbt-labs/dbt-core/discussions/6725#discussioncomment-6099860

#data #dataDon

While sin is a made-up concept, the thing Christians say about sin is 100% true when applied to working with dates in #datascience:

"Sin will take you farther than you want to go, keep you longer than you want to stay, and cost you more than you want to pay."

#rstats #data #datadon #KillingMeSoftlyWithDates

'My normative lesson is, “Heed Marginalized People.” Fundamentally and foundationally. And, like, don’t include them necessarily in your training data, but include them in the questions that you ask at the outset, and who you think to ask about what you ought to do.
[…]
So to ask that question, “Who have we not thought about; whose harms, whose needs, whose voice has been, perhaps speaking, but unheeded, for a very long time? And how do we ensure that the things that they have called out as potential sites of failure, don’t go unremarked, don’t go unaddressed.'
⸺
Dr. Damien P. Williams, @Wolven https://afutureworththinkingabout.com/?p=5442 #ableism #algorithms #bias #biotech #ethics #disability #facialRecognition #feminisms #Google #homophobia #machineLearning #misogyny #neurodiversity #phenomenology #racism #science #neutrality #sexism #cognition #surveillance #dataDon @ethics @dataGovernance @data #historyOfScience

I'll be writing about team topologies and data work this week. #Datadon, are you familiar with the Team Topologies book/concepts?

#Streamlit's experimental_data_editor is dead! Long live #data_editor!

Also, now you can hide the annoying index column and define sort criteria. #datadon

https://discuss.streamlit.io/t/version-1-23-0/44490

Hey, #datadon... any thoughts on Predibase? From the folks who made Ludwig, the "declarative" approach to predictive model specification.

https://predibase.com/

#datascience

Best part of my job is hosting a seminar where engineers share cool things they’re working

* CI checks for data ingestion
* MetaDiff: Database Metadata version control
* duckdb, dbplyr, and synthea for synthetic clinical data
* instant EKS cluster deployments for changes to our Airflow pipeline

#data #dataDon

✍🏻 Final part of my blog series on Write-Audit-Publish (WAP), in which I show in detail how to implement it using #apacheSpark, @deltalakeoss, #Minio, and @lakeFS

https://lakefs.io/blog/write-audit-publish-with-lakefs/?utm_campaign=Social%20media%20activity&utm_source=Mastodon&utm_medium=social&utm_content=blog_rm-wap3

---

📝part 1: 🙋🏻What is WAP? https://lakefs.io/blog/data-engineering-patterns-write-audit-publish/?utm_campaign=Social%20media%20activity&utm_source=Mastodon&utm_medium=social&utm_content=blog_rm-wap1

📝part 2: 🛠️ Comparing how different tools implement WAP https://lakefs.io/blog/how-to-implement-write-audit-publish/?utm_campaign=Social%20media%20activity&utm_source=Mastodon&utm_medium=social&utm_content=blog_rm-wap2

#writeAuditPublish #dataEngineering #datadon #opensource

🔧🔨 How to Implement Write-Audit-Publish (WAP) - an exploration of how WAP can be done on @apacheicebergdevs, #apacheHudi, @deltalakeoss, @lakeFS , or #projectNessie

https://lakefs.io/blog/how-to-implement-write-audit-publish/?utm_campaign=Social%20media%20activity&utm_source=Mastodon&utm_medium=social&utm_content=blog_rm-wap2

🙋🏻 What is WAP?
👉🏻 Check out yesterday's blog: https://lakefs.io/blog/data-engineering-patterns-write-audit-publish/?utm_campaign=Social%20media%20activity&utm_source=Mastodon&utm_medium=social&utm_content=blog_rm-wap1

#dataEngineering #data #writeAuditPublish #datadon

Data engineers, how many hours per week do you typically spend in meetings?
#DataEngineering #Datadon