New Ventana Research analyst perspective: How Data Observability and Data Quality Achieve Data Management https://mattaslett.ventanaresearch.com/how-data-observability-and-data-quality-achieve-data-management Where does #dataquality end and #dataobservability begin?

Generative AI is disrupting data engineering, providing easier access to data, and enabling natural language queries about data. The technology will likely become more ubiquitous, and data leaders predict that it will change the way data is used. https://www.montecarlodata.com/blog-generative-ai-data-engineering/ #dataobservability #generativeAI #softcorpremium

Referenced link: https://hackernoon.com/data-observability-the-first-step-towards-being-data-driven
Discuss on https://discu.eu/q/https://hackernoon.com/data-observability-the-first-step-towards-being-data-driven

Originally posted by HackerNoon | Learn Any Technology / @hackernoon: http://nitter.platypush.tech/hackernoon/status/1652900711498170369#m

In a nutshell, data reliability is a BIG challenge and there is a need for a solution that is easy to use, understand, and deploy, and also not hea - https://hackernoon.com/data-observability-the-first-step-towards-being-data-driven #dataobservability #dataengineering

Ventana Research's forthcoming 2023 DataOps Value Index study will consist of three parallel evaluations: #dataorchestration, #dataobservability, and overall #DataOps.

In this Analyst Perspective I discuss the key assessment criteria, as well as what defines and differentiates DataOps from traditional #datamanagement

https://mattaslett.ventanaresearch.com/dataops-understanding-the-definition-and-differentiation

RT biconnections: Data observability and orchestration are the key to unlocking the full potential of your data stack. Learn more at our upcoming webinar. Register now! #dataobservability #dataorchestration #webinar #observability #platform #dataorchestration \@thebloorgrou https://bit.ly/3keu91o

📢 PipeRider 0.18.0 is out now and our #dbt support is even better!

- dbt defined metrics in HTML reports

- Visualize metric differences between data profiles

- Metric comparison summary in Markdown to paste into your pull request comment

Start your "code review for data projects" now:

https://github.com/InfuseAI/piperider

#DataQuality #DataReliability #DataOps #DataObservability #OpenSource #dbt #snowflake #DataWarehouse #DataEngineering

dbt state is now supported from PipeRider 0.14

This means you can profile and run data assertions on only modified models

Read more, or check out the video below

Article:
https://blog.piperider.io/data-reliability-dbt-state-piperider.html

Video demo:
https://www.youtube.com/watch?v=2J2Cu84HonU

#OpenSource #DataQuality #DataReliability #DataEngineering #DataEngineer #DataObservability #dbt

Watch out for that schema change, it's a doozy!

You probably don't control upstream tables, so having some way to alert you when a table schema changes can save you time and effort.

Using Snowflake for an example, I made some major changes to a table and showed how they can be detected with #PipeRider:

https://blog.infuseai.io/how-to-detect-schema-change-in-snowflake-6ffcd28c3f15

#snowflake #DataEngineering #DataQuality #DataReliability #DataOps #DataObservability #DataWarehouse #ELT

Just "resolved" our first massive outage on a core data pipeline today.

If only we had been online during thanksgiving holiday (thurs/fri) we wouldn't have lost ~3 days of data... *shakes fists to gods of max 7-day retention policies*

Yes, it was because we (ok, a recently departed colleague) deleted a freshness test on this source in June for over-alerting on false positives. My Coalesce talk comes full circle.

#Datadon #DataTestHygiene #DataObservability

Have you ever been bitten by a schema change?

Here are 5 schema changes that you should look out for when maintaining data pipelines:

https://blog.infuseai.io/5-database-schema-changes-data-engineers-need-to-beware-of-831aeb144749

#dataengineering #dataops #datamonitoring #dataobservability #datareliability

PipeRider 0.13.0 is now available 🥳

Updates include:

📊New Markdown summary of data profile changes when using the 'compare-reports' feature

✅Generate assertions on a per-table basis

Release notes:
https://github.com/InfuseAI/piperider/releases/tag/v0.13.0

Documentation:
https://docs.piperider.io/

#datareliability #dataquality #dataobservability #opensource

Referenced link: https://hackernoon.com/what-should-i-do-after-the-data-observability-tool-alerts-me
Discuss on https://discu.eu/q/https://hackernoon.com/what-should-i-do-after-the-data-observability-tool-alerts-me

Originally posted by HackerNoon | Learn Any Technology / @hackernoon@twitter.com: https://twitter.com/hackernoon/status/1571212428343648256#m

"What Should I Do After the Data Observability Tool Alerts Me" https://hackernoon.com/what-should-i-do-after-the-data-observability-tool-alerts-me #dataobservability #datascience

Referenced link: https://hackernoon.com/data-observability-that-fits-any-data-teams-structure
Discuss on https://discu.eu/q/https://hackernoon.com/data-observability-that-fits-any-data-teams-structure

Originally posted by HackerNoon | Learn Any Technology / @hackernoon@twitter.com: https://twitter.com/Bigeyedata/status/1552303081941082113#m

RT by @hackernoon: Your #data team is either:
1️⃣ Centralized
2️⃣ Embedded
3️⃣ A hybrid

Which team structure does your data org fall under, and which observability rollout fits you best? Find out in our piece in @hackernoon ➡️ https://hackernoon.com/data-observability-that-fits-any-data-teams-structure

#blog #team #dataobservability #dataquality