More info on this new capability 👉 Simplify external object access in Amazon Redshift using automatic mounting of the AWS Glue Data Catalog https://aws.amazon.com/blogs/big-data/simplify-external-object-access-in-amazon-redshift-using-automatic-mounting-of-the-aws-glue-data-catalog/ #AWS #Analytics #DataLake

This month, Muse™ Grenoble in partnership with @AlexandreMartin, looks at the place of #ethics in #ArtificialIntelligence, and more specifically in #algorithms and #data.

https://www.linkedin.com/posts/ia-muses_ethics-artificialintelligence-algorithms-activity-7078982744445206528-3zVn

Enjoy your reading 🙂

#gender #sexism #misogynistic #stereotype #discrimination #bias #racism #racial #CodeBias #EquitableAI #AccountableAI #InclusiveAI #algoethics #responsibleai #dataethics #aiethics #ethicalai #ethicaldesign #techethics #blackbox #algorithmicbias #bigdata #datalake #dataprivacy

Amazon Redshift announces automatic mounting of AWS Glue Data Catalog 👉 You no longer have to create an external schema to use data lake tables in AWS Glue Data Catalog https://aws.amazon.com/about-aws/whats-new/2023/07/amazon-redshift-automatic-mounting-aws-glue-data-catalog/ #AWS #Analytics #DataLake

There's lots of fancy file formats to choose from when building a #datalake but we still went with gzipped JSON. Why? Because we prioritize moving data into purpose-built systems rather than querying it directly. This basic shift in approach has made a world of difference! Here's a thing I wrote about that:

https://opendatascience.com/choosing-a-data-lake-format-what-to-actually-look-for/

Nouveau projet #datalake

Très heureux de co-porter avec Genoveva Vargas-Solar le projet pluridisciplinaire et international LETITIA (Lac de donnéEs, expérimenTation, vIe, Terre, curatIon, explorAtion), financé par la FIL.

https://eric.univ-lyon2.fr/jdarmont/?p=8269

📝 "3 Data Engineering Pitfalls"

👤 Valery C. Briz (@valerybriz)

🔗 https://dev.to/valerybriz/5-data-engineering-pitfalls-517m

#pyladies #python #dataengineering #datagovernance #datalake #data

A quick quide to help you shift to the new-age #data #lakehouse without disrupting business applications. #datalakehouse #datawarehouse #datalake https://venturebeat.com/data-infrastructure/how-enterprises-can-move-to-a-data-lakehouse-without-disrupting-their-business/ #press

This just arrived in the mail and I am very excited to dive in! Unlike previous books on the topic, this digs into the real meat of building a contextual score using a risk based approach and leveraging vulnerability and exploit data sets.

I am looking forward to further exploring the topic and using everything I can to improve my own working models and approach to this ever growing topic.

#VulnerabilityManagement #RiskModeling #DataLake

RT @ventanaresearch
Data lakes are supporting multiple data sources, formats, analytics workloads and business functions, @ventanaresearch’s Data Lakes Dynamics Insights research shows. https://bit.ly/3SWquSQ #Data #DataLake

Green #SitecoreLunch today! Discussed:

🏦 Bank stability
👰 Marital advice
👁️ Eye drop recall
🍀 Green pinchers
🖥️ #Microservices
👓 Headache glasses
⚡ Internet bandwidth
🏖️ #SpringBreak plans
🏊 Go jump in a #DataLake
🦸 #SitecoreXP is not dead
⏹️ #GartnerMagicQuadrant
🏆 #SitecoreMVP certification
🔍 #AzureSearch support discontinued
🌐 #Sitecore #ExperienceEdge licensing

See you same time next week! 🥪🥗

Automate schema evolution at scale with Apache Hudi in AWS Glue 👉 In this post, we show how to build a transactional data lake using Apache Hudi support for ACID transactions and CRUD operations https://aws.amazon.com/blogs/big-data/automate-schema-evolution-at-scale-with-apache-hudi-in-aws-glue/ #AWS #Analytics #OpenSource #DataLake

New blog: extracting data from a #Soap #Api into an #Azure #datalake with #AzureDataFactory. Or #Powershell. #data #zekerweten

http://sqlreitse.com/2023/01/02/azure-data-factory-and-soap-an-opera/

Matano is live on the front page of HackerNews!! 🔥

Come join the discussion on OSS, SIEM, and why we are helping orgs build on top of vendor-agnostic Security Data Lakes instead 🙂

https://news.ycombinator.com

#cybersecurity #security #oss #hackernews #cloudsecurity #detectionandresponse #threathunting #threatdetection #datalake #awssecurity #aws #datalake #siem #securitydatalake

🌐 Announcing Matano + Suricata!

Suricata is a popular open source NIDS/NIPS engine used for network analysis and threat detection.

We just shipped out a new integration that allows you to easily push Suricata logs & alerts into a Matano Security Lake in your AWS account for realtime detection-as-code with Python and analysis using AWS Athena + SQL! 🚀

Interested in how to build your own Security Data Lake using Suricata logs?

Check out our blog post: https://www.matano.dev/blog/2023/01/12/suricata-support 🔎

#opensource #infosec #networksecurity #suricata #oisf #intrustiondetection #intrusionprevention #ids #ips #nids #nips #cloudnative #cloudsecurity #rust #datalake #aws #awssecurity #apacheiceberg #secops #security #siem #threatdetection #threathunting #detectionandresponse

03-04/23 – Offre de stage : Conception et implémentation d’un lac de données de robotique agricole #datalake #agriculture

Pour accompagner la transition agroécologique, les robots ont un rôle essentiel à jouer dans le domaine de l'agriculture intelligente. Ils sont capables d'effectuer des opérations agricole

https://eric.msh-lse.fr/03-04-23-offre-de-stage-conception-et-implementation-dun-lac-de-donnees-de-robotique-agricole-datalake-agriculture/

#Offresd'emploi/thèse/stage

The latest The SAP Mentors Daily! https://paper.li/sapmentors/sapmentors?edition_id=aa87e3f0-915a-11ed-a075-fa163e1a70d7 Thanks to @janmusil@twitter.com @blackbox_europe@twitter.com @cichuck@twitter.com #techstuff #datalake

I'm excited to announce that Matano is joining YCombinator's W23 Batch! 🚀

SIEM today is broken -- it's too expensive, doesn't scale, has poor support for correlation, causes vendor lock-in, is inflexible for detection engineering, the list goes on...

My brother Shaeq and I quit our jobs at AWS to solve this problem and build a better solution for security operations and analytics that fully utilizes the power of cloud and big data tech available today.

While the cybersecurity industry has been held back by legacy architectures tied to age-old vendor products, the data analytics industry has seen a ton of innovation through open source initiatives such as Apache Iceberg, Parquet, and Arrow delivering massive cost savings and performance breakthroughs.

We started Matano to close the gap between these two worlds by building an OSS platform to help security teams leverage the modern data stack (e.g. Spark, Athena, Snowflake) to efficiently analyze security data from all the disparate sources across an organization (Cloud/SaaS, Endpoint, Network, etc.).

Matano helps Detection & Response teams break free from their SIEM by deploying a vendor-agnostic Security Data Lake into their AWS account and giving them a platform to build detection-as-code using Python and SQL!

This is just the beginning in our mission to build the first open platform for threat hunting, detection & response, and cybersecurity analytics at petabyte scale.

I am super grateful to all of our early supporters for the help & joining in on this journey to reinvent SIEM. Let's goo!

https://www.ycombinator.com/launches/Hl0-matano-open-source-siem-alternative-for-aws

#startup #ycombinator #opensource #cybersecurity #cloudsecurity #awssecurity #siem #threatdetection #secops #devsecops #aws #infosec #dfir #detectionandresponse #soc #apacheiceberg #security #datalake #blueteam

The Shape of Modern Data Architecture 👇
🔹 shows the end-to-end data process inc. data analytics lifecycle
🔹shows the data catalog at the center of the architecture and connected with every other component

#dataengineering #datalake

https://www.alation.com/blog/a-data-architects-guide-to-the-data-catalog/

RT @dez_blanchfield
"Public Cloud Continues to Grow but Not Without Challenges", by Craig Mullins @craigmullins via http://elnion.com https://bit.ly/3Z2RbZ7

#data #cloud #saas #paas #iaas #xaas #dbaas #public #hybrid #storeage #batabase #business #compliance #governance #datalake #bigdata

Woof, file compaction with #DeltaLake 1.x is the only way to make it usable. 50-100x performance improvements depending on data and partition sizes. The default merge and write operations are incredibly inefficient. I understand its been greatly improved in 2.x. We're a couple months from upgrading the platform though. #datawarehouse #datalake #spark #pyspark #aws #awsglue