Q. What's the best thing about open-source software?

A. You can modify it and add your own features❤️

That's exactly what Alex, a senior data engineer at domain.com.au, did with PipeRider.

Alex added features specific to his data migration use-case of comparing two datasets on the row-level, with tolerance

https://www.youtube.com/watch?v=SSoFyWTAYGw

#OpenSource #DataQuality #DataReliability #DataLineage #DataOps #DataTools #DataViz

The lineage graph in dbt docs is great, but it’s static and represents only one state of your data.

Lineage Diff in PipeRider shows you exactly which nodes have changed in your branch code compared to main.

Lineage Diff shows:

🆕 New nodes
♻️ Changed nodes
👊 Impacted nodes
📊 Full lineage graph and change-only views
📋 Extra stats: row count and execution time

Read more about how to get Lineage Diff:

https://medium.com/inthepipeline/dbt-data-lineage-diff-impact-analysis-visualized-bec9927b0c4e

#DataQuality #DataViz #DataReliability #DataLineage #DataEngineering

We've rebooted our Discord community ♻️

If you're looking for:

🛟 Help using PipeRider
📰 Info on new releases
💬 General chat about data

Join the community here:

https://piperider.io/discord

#DataQuality #DataReliability #dbt #CodeReviewForData #DataCommunity #Discord

The latest issue of 'In the Pipeline', the PipeRider newsletter, is coming very soon.

Make sure you don't miss out by signing up!

https://app.us1.list-manage.com/subscribe?u=2b18366427f11835f05f68aeb&id=b82bfd845e

Featuring info about:

- New PipeRider features
- Data best practices
- Guest interviews
- Interesting data-related content

#DataTools. #DataOps #DataQuality #DataReliability #DataNewsletter #dbt #PipeRider

The PipeRider Community Office Hours for June 20th is now online:

https://www.youtube.com/watch?v=6scbaNMfXSc

- New features in PipeRider 0.27

- Lineage Diff in PipeRider Cloud!

- Special guest Spencer Ellinor from Sudo Labs

Timestamps and links in the video description

#DataQuality #DataReliability #PipeRider #DataViz #dbt #DataOps #LineageDiff #CodeReviewForData

PipeRider now has a dedicated tools channel on the dbt Slack!

Join the dbt Slack here:
https://getdbt.com/community/join-the-community/

Then, either search for the #tools-piperider channel, or follow this link:
https://getdbt.slack.com/archives/C05C28V7CPP

See you there 💪

#dbt #DataQuality #DataOps #OpenSource #DataReliability #DataTools #CodeReviewForData #PipeRider

How do you track down and explore data changes in your dbt project?

One way is to explore rows that fall outside previous boundaries

Check out this upcoming PipeRider feature that highlights a change, and shows the SQL you need to find the affected rows

#dbt #DataQuality #eda #DatatViz #sql #DataOps #DataReliability #DataEngineering #AnalyticsEngineering

You already use dbt to empower your data modelling workflow, now you need a way to understand how your code changes affect data

That's where "code review for data" comes in

Zero-config dbt integration and merge with confidence 💯

Get started today at
https://piperider.io

#dbt #DataQuality #DataReliability #OpenSource #DataOps #DataViz #DataProfile #DataTesting #PipeRider

PipeRider 0.26.0 has just been released 📢

'PipeRider compare' is a powerful command, so we've included two options to guide you through using it:

--dry-run
will show each command that will be run

--interactive
will guide you step by step

More info:
https://github.com/InfuseAI/piperider/releases/tag/v0.26.0

To update just run:
pip install -U piperider

#DataQuality #PipeRider #DataReliability #DataOps #DataDiff #dbt

Hi all!

PipeRider 0.25.0 has just been release with an increased focus on dbt integration.

You no longer need to initialize PipeRider in dbt projects (piperider init).

To get started in a new project, all you need to do is:

1. Install PipeRider 👩‍💻
2. Tag your models 🏷️
3. Run PipeRider 🏃‍♂️
4. Enjoy rich data profiling reports and improve your code review process 📊

Find out more:
https://medium.com/inthepipeline/zero-config-code-review-and-data-profiling-tool-for-dbt-projects-8b6de40964b4

#DataQuality #DataEngineering #DataOps #DataReliability #AnalyticsEngineering

PipeRider 0.21.0 it out now with the following main updates:

- The compare command now uses 'three-dot' compare (to compare against with main at the point at which your branch was made)

- PipeRider Cloud supports multiple workspaces

Get the latest version:

https://github.com/InfuseAI/piperider

#dbt #DataQuality #DataReliability #DataProfile #DataViz #DataOps #DataEngineer #DataEngineering #AnalyticsEngineering #AnalyticsEngineer

Are you using dbt in production and managing deployment with CI?

We ran a workshop last week as part of the Data Engineering Zoomcamp:

Understand the impact of data model changes in dbt with PipeRider

https://www.youtube.com/watch?v=O-tyUOQccSs

#DataEngineering #AnalyticsEngineering #dbt #DataQuality #DataReliability

📢 PipeRider 0.18.0 is out now and our #dbt support is even better!

- dbt defined metrics in HTML reports

- Visualize metric differences between data profiles

- Metric comparison summary in Markdown to paste into your pull request comment

Start your "code review for data projects" now:

https://github.com/InfuseAI/piperider

#DataQuality #DataReliability #DataOps #DataObservability #OpenSource #dbt #snowflake #DataWarehouse #DataEngineering

It was Groundhog day today, so it seems the perfect time to share this article with a gif I made from one of my fav movies:

How to detect schema changes in Snowflake:

https://medium.com/infuseai/how-to-detect-schema-change-in-snowflake-6ffcd28c3f15

So you won't get caught off guard by the same issues again and again :)

#GroundhogDay #Snowflake #database #DataQuality #DataReliability

PipeRider 0.16.0 is out now with the following improvements and new features:

- BigQuery repeated fields are now supported!

- PipeRider will now profile database 'views' (easily enable in your project's config.yml)

- Automatically open reports in your default browser when running PipeRider CLI

Easily upgrade with:

pip install -U piperider

Read more:

https://github.com/InfuseAI/piperider/releases/tag/v0.16.0

#DataQuality #DataReliability #DataProfiler #DataViz #DataVisualization #DataReport #Database #DataOps #EDA

dbt state is now supported from PipeRider 0.14

This means you can profile and run data assertions on only modified models

Read more, or check out the video below

Article:
https://blog.piperider.io/data-reliability-dbt-state-piperider.html

Video demo:
https://www.youtube.com/watch?v=2J2Cu84HonU

#OpenSource #DataQuality #DataReliability #DataEngineering #DataEngineer #DataObservability #dbt

Watch out for that schema change, it's a doozy!

You probably don't control upstream tables, so having some way to alert you when a table schema changes can save you time and effort.

Using Snowflake for an example, I made some major changes to a table and showed how they can be detected with #PipeRider:

https://blog.infuseai.io/how-to-detect-schema-change-in-snowflake-6ffcd28c3f15

#snowflake #DataEngineering #DataQuality #DataReliability #DataOps #DataObservability #DataWarehouse #ELT

I've been playing with schema change detection in PipeRider the last few days making an article for #Snowflake users

If you're dealing with tables updating and want to keep track of changes and other #dataquality #datareliability issues then check it out!

🔓 It's open-source and ready to go

⏩ Quick Start:
https://docs.piperider.io/cli/quick-start

⭐ Star us on GitHub if you love Data Quality :ablobcatheart:
https://github.com/infuseai/piperider

Supports #Snowflake #BigQuery #Redshift #DuckDB #CSV #SQLite #Parquet

Have you ever been bitten by a schema change?

Here are 5 schema changes that you should look out for when maintaining data pipelines:

https://blog.infuseai.io/5-database-schema-changes-data-engineers-need-to-beware-of-831aeb144749

#dataengineering #dataops #datamonitoring #dataobservability #datareliability

Reading through PipeRider documentation. https://docs.piperider.io

These #data profiling reports based on your #dbt test results are very cool. 🧪

Plus there’s an interactive sample included too: https://piperider-github-readme.s3.ap-northeast-1.amazonaws.com/run-0.13.0/index.html#/assertions

#DataProfiling #DataReliability