Just "resolved" our first massive outage on a core data pipeline today.

If only we had been online during thanksgiving holiday (thurs/fri) we wouldn't have lost ~3 days of data... *shakes fists to gods of max 7-day retention policies*

Yes, it was because we (ok, a recently departed colleague) deleted a freshness test on this source in June for over-alerting on false positives. My Coalesce talk comes full circle.

#Datadon #DataTestHygiene #DataObservability