Dataops, the discipline of managing data integration and cleansing operations, is becoming increasingly important for businesses as they strive to leverage analytics and gain a competitive advantage. Dataops teams can now leverage AI and machine learning techniques to improve efficiency, data quality, and reliability. By reducing data... https://www.infoworld.com/article/3706090/5-ways-to-use-ai-and-machine-learning-in-dataops.html#tk.rss_all #DataOps #DataManagement #AIRevolution #softcorpremium

So if you've got #prometheus monitoring $thing as well as a continuously predicted value for $thing, you can get a rolling MAPE value visualised on a gauge in #grafana, just using PromQL! All it takes is literally the ugliest query I've ever seen! #dataops #datascience

This MIT Press Data Intelligence Special Issue on Metadata as Data Intelligence is a trove of knowledge around #metadata, #FAIR, #AI, #ML, #DataOps
https://direct.mit.edu/dint/issue/5/1

Kicking off this year with a lot of storytelling around those next #TechJobs. The ones that are dedicated to taking down more silos and building on the promise of #DevOps and #SRE.

This one is about the data reliability engineer or SRE for data or database reliability engineer (DBRE), on the front lines as “data is the new oil” becomes “data is a river delta.”

#DataOps @thenewstack

https://thenewstack.io/why-a-dataops-team-needs-a-database-reliability-engineer/

I'm not sure when the title of #dataengineer also added full on expert in #devsecops but I guess I have some learning to do.

For the record, I've always taken security very seriously and understanding of general DevOps practices and philosophy are important to #dataengineering and #dataops but I'm having to crash course in some much deeper stuff and its... a lot to take in at once.

Part of me wants to look at software managing movie libraries or photo libraries but not sure how much will apply to my use case (mass spectrometry raw files).

I looked at data pantries / warehouses / lakes etc. but they seem to be dealing with much smaller data. Also I am not interested in anything cloud or internet related.

If anyone has some keywords to help my search that would be helpful.

#dataOps #programming

Part of me wants to look at software managing movie libraries or photo libraries but not sure how much will apply to my use case (mass spectrometry raw files).

I looked at data pantries / warehouses / lakes etc. but they seem to be dealing with much smaller data. Also I am not interested in anything cloud or internet related.

If anyone has some keywords to help my search that would be helpful.

#dataOps #programming

If someone knows they are likely to be here

I am looking for a software solution (or pointers to build a solution) able to organise and manage BIG data. Not billions of small text records, dozens of thousand of ~1Gb binary files.

Something better than a basic file system. So maybe a database for the metadata (and something to extract metadata). We want to keep the raw files as source of truth but also track files converted to friendlier formats (with loss).
#programming #devops #dataOps