"[#DataAnalysts]..should know how the data was born, with all details of measurement... Few things have more devastating consequences ... than someone in the audience pointing out...measurement issues the analyst didn't consider." Békés and Kézdi, 2021: Data Analysis for Business, Economics, and Policy

If you're having trouble helping your org understand the value of #datalineage and #metadata, share this with them and ask if they know how all the data they're using was gathered and measured.

Q. What's the best thing about open-source software?

A. You can modify it and add your own features❤️

That's exactly what Alex, a senior data engineer at domain.com.au, did with PipeRider.

Alex added features specific to his data migration use-case of comparing two datasets on the row-level, with tolerance

https://www.youtube.com/watch?v=SSoFyWTAYGw

#OpenSource #DataQuality #DataReliability #DataLineage #DataOps #DataTools #DataViz

The lineage graph in dbt docs is great, but it’s static and represents only one state of your data.

Lineage Diff in PipeRider shows you exactly which nodes have changed in your branch code compared to main.

Lineage Diff shows:

🆕 New nodes
♻️ Changed nodes
👊 Impacted nodes
📊 Full lineage graph and change-only views
📋 Extra stats: row count and execution time

Read more about how to get Lineage Diff:

https://medium.com/inthepipeline/dbt-data-lineage-diff-impact-analysis-visualized-bec9927b0c4e

#DataQuality #DataViz #DataReliability #DataLineage #DataEngineering

Octopai: Meta-level intelligence for reducing data chaos - @OctopaiData offers a platform for continuously discovering and #cataloging #data sources across an extended enterprise environment. @bluefug #datalineage #dataprovenance #datagovernance #lineage