PipeRider 0.33 is out now, with the main feature being:

- Support for dbt's new Semantic Layer that comes with dbt 1.6.

More info:
https://docs.piperider.io/get-started/run/metrics

PipeRider Cloud now also has powered-up pull request integration with Impact Report history.

View the Impact Reports for all your pull requests in one easy dropdown.

A welcome improvement for those managing pull request review for their team.

https://medium.com/inthepipeline/managing-multiple-pull-request-reviews-in-piperider-cloud-539ec3a93119

#PipeRider #dbt #AnalyticsEngineering #DataEngineering #DataViz

I hosted the PipeRider Community Office Hours yesterday and one cool feature we demoed was this online dbt manifest viewer.

Generate a Lineage Diff from just your dbt project's manifest files.

Try it out for yourself here:

https://staging.cloud.piperider.io/online-viewer#/?utm_source=mstdn

Watch it in action on Youtube:

https://www.youtube.com/watch?v=LHuTb3e_4O8

#DataEngineering #DataViz #dbt #DataTools #AnalyticsEngineering #OpenSource #PipeRider

databricks is now a supported by PipeRider for dbt projects!

Now databricks users can get:

- Data Impact Reports
- Data profile comparisons
- Pull Request Impact summaries
- Lineage Diff

Check the PipeRider docs for a full list of supported data sources:

https://docs.piperider.io/reference/supported-data-sources

#dbt #piperider #databricks #datyaquality #dataimpact #dataengineering #analyticsengineering #opensource

How do you track down and explore data changes in your dbt project?

One way is to explore rows that fall outside previous boundaries

Check out this upcoming PipeRider feature that highlights a change, and shows the SQL you need to find the affected rows

#dbt #DataQuality #eda #DatatViz #sql #DataOps #DataReliability #DataEngineering #AnalyticsEngineering

🔎 🗺️ The State of Data Engineering 2023 - a nice round up and review from Einat Orr of #dataengineering in 2023
https://lakefs.io/blog/the-state-of-data-engineering-2023/?utm_campaign=Social%20media%20activity&utm_source=mastodon&utm_medium=social&utm_content=blog_rm-SoDE23_230524
#datadon #analyticsengineering #opensource #data

Hi all!

PipeRider 0.25.0 has just been release with an increased focus on dbt integration.

You no longer need to initialize PipeRider in dbt projects (piperider init).

To get started in a new project, all you need to do is:

1. Install PipeRider 👩‍💻
2. Tag your models 🏷️
3. Run PipeRider 🏃‍♂️
4. Enjoy rich data profiling reports and improve your code review process 📊

Find out more:
https://medium.com/inthepipeline/zero-config-code-review-and-data-profiling-tool-for-dbt-projects-8b6de40964b4

#DataQuality #DataEngineering #DataOps #DataReliability #AnalyticsEngineering

Do you have a proper code review process for pushing changes to data projects?

With ELT and dbt, small changes to code can have a big impact on downstream models. The last thing you want is to push a breaking change to prod 😱

Why you need code review for you data projects
by Balu Rama Chandra

https://medium.com/inthepipeline/why-you-need-code-review-for-you-data-projects-95e039083df0

#DataEngineering #DataQuality #DataOps #CodeReview #dbt #AnalyticsEngineering

dbt users - Do you consider data warehouse costs when building and maintaining projects?

If so, what steps you taking to keep costs down, or keep an eye on costs?

#dbt #DataQuality #DataBuildTool #DataEngineering #AnalyticsEngineering #DataOps

There's a lot of blogs and tweets recently about the number of commands that data engineers to run during development.

Command fatigue is real!

In this post we demo the new PipeRider Compare command and how it automates data profile comparison to easily see code change impact on dbt projects

https://medium.com/inthepipeline/reduce-command-fatigue-working-on-dbt-data-projects-76ce0cf0ee61

#DataEngineering #AnalyticsEngineering #dbt #DataQuality #OpenSource

PipeRider 0.21.0 it out now with the following main updates:

- The compare command now uses 'three-dot' compare (to compare against with main at the point at which your branch was made)

- PipeRider Cloud supports multiple workspaces

Get the latest version:

https://github.com/InfuseAI/piperider

#dbt #DataQuality #DataReliability #DataProfile #DataViz #DataOps #DataEngineer #DataEngineering #AnalyticsEngineering #AnalyticsEngineer

The essential component of any data toolkit: Duck[DB] Tape!

#datadon #duckdb #dataengineering #analyticsengineering

Enjoying the back catalogue of The Analytics Engineering #Podcast with @jthandy and @schottj - this episode from August 2021 with Erik Bernhardsson was really interesting.

🎧 https://overcast.fm/+w94WXfYA0
✍🏻 https://roundup.getdbt.com/p/erik-bernhardsson

#datadon #dataEngineering #analyticsEngineering

Are you using dbt in production and managing deployment with CI?

We ran a workshop last week as part of the Data Engineering Zoomcamp:

Understand the impact of data model changes in dbt with PipeRider

https://www.youtube.com/watch?v=O-tyUOQccSs

#DataEngineering #AnalyticsEngineering #dbt #DataQuality #DataReliability

Exciting news!

Join us at #GoDataFest, a 3-day conference in Amsterdam that covers the latest and greatest in data-driven organizations, analytics engineering, data storytelling and visualization, and much more!

This is a great opportunity to learn from experts, network with peers, and deepen your understanding of the data industry.

And the best part? It's completely free.

Register now at https://godatafest.com.

#AnalyticsEngineering #DataStorytelling #DataVisualization

For those who might find me by hashtag (get ready for hashtag soup): I do lots of #dbt on #snowflake #data #dataengineering and #analytics #analyticsengineering. I enjoy #python and #pandas and #sql and #dax. I’ve been lucky to work on #azure and #aws, and I prefer #powerbi as my viz Swiss Army knife.

Worst part of being laid off "effective immediately" by far is the toll wrought by a dim model I hadn't quite cracked the code for yet being left completely unfinished.

#DataDon #AnalyticsEngineering #DataModeling

Hey #datadon, what are the main conferences that someone interested in#DataEngineering or #AnalyticsEngineering should be planning to attend next year? I'm assuming #dbt's #CoalesceConf, but what else?

Do you materialize as table by default? View? How much do you use ephemeral materialization? When is a view better than a table?

My team has pretty small data and we use tables for everything except event streams (incremental) and some intermediate (ephemeral) models.

I'm really curious what other teams' materialization patterns are like.

#datadon #dbt #DataModeling #AnalyticsEngineering #DataWarehousing

This thread from @criccomini and ensuing discussion is fascinating and well worth the read: https://twitter.com/criccomini/status/1590374049733283840?s=46&t=TpqDxqQaDXlVrJ6lfNlvlg

(Maybe the next one will be over here on data-folks mastodon :))

#dataengineering #analyticsengineering #noxp

A new blog, in which I explore #dbt, #FiveTran, #Airbyte, and try to generally make sense of the ELT landscape: https://rmoff.net/2022/11/08/data-engineering-in-2022-elt-tools/

#dataengineering #data #analyticsengineering