New post: "Interesting Stuff - Week 18, 2023".

This week:

* AI improving their programming abilities using self-play!
* Build ML-driven segmentation using chatbots, ChatGPT and LangChain.
* Use diffusion models to mimic human behaviour.
* Inferring rewards through interaction. Use of IGL-P!
* Confluent Platform 7.4 released. Cool!
* Niels is busy with @azuredataexplorer post

Read all about it:

https://nielsberglund.com/post/2023-05-07-interesting-stuff---week-18-2023/

#AI #ML #kafka #azuredataexplorer #openai

New post: "Interesting Stuff - Week 17, 2023".

This week:

* The gambler that beat roulette!
* I finally created a repo covering how to stream data from Kafka to @azuredataexplorer.
* Real-time ML pipelines.
* Databricks and Hugging Face, love is in the air.
* Very interesting about the real-time streaming ecosystem by Hubert.

Read all about it:

https://nielsberglund.com/post/2023-04-30-interesting-stuff---week-17-2023/

#AI #ML #streaming #kafka #databricks #azuredataexplorer

New post: "Interesting Stuff - Week 16, 2023".

This week:

* How to optimize the Azure Databricks Delta Engine.
* Link to my post about the multi-platform Docker Image for @azuredataexplorer Kusto Sink and Kafka Connect.
* Create Synthetic Data using Generative AI.
* Databricks SQL and the ability to call LLMs!

Read all about it:

https://nielsberglund.com/post/2023-04-23-interesting-stuff---week-16-2023/

#AI #ML #openai #databricks #azuredataexplorer

New post: "Cross-Platform Compatibility Made Easy with Multi-Platform Docker Images: Azure Data Explorer Sink Connector & Kafka Connect"

Docker has revolutionized the way we package and deploy software applications.

However, it's challenging to maintain a single image that can work across multiple platforms.

This blog post explores the steps required to build and deploy multi-platform Docker images.

https://nielsberglund.com/post/2023-04-17-cross-platform-compatibility-made-easy-with-multi-platform-docker-images-azure-data-explorer-sink-connector--kafka-connect/

#kafka #azuredataexplorer #kusto #kafkaconnect

New post: "Interesting Stuff - Week 15, 2023".

This week:

* Generative AI and focus shift from tech-driven dev to product-driven dev.
* Game analytics solution for Game Studios!
* Late arriving events in Flink and ways to handle them.
* Very cool around materialized views.
* Me plugging an upcoming post about how to build multi-platform Docker images.

Read all about it:

https://nielsberglund.com/post/2023-04-16-interesting-stuff---week-15-2023/

#kafka #flink #AI #ML #openai #databricks #azuredataexplorer

New post: "Develop a Real-Time Leaderboard Using Kafka and Azure Data Explorer - II"

This is the third post about using Kafka and @azuredataexplorer (ADX) to develop a real-time leaderboard for online gaming.

In this post, you will see more about ingesting data into ADX and some "cool stuff" related to that. Specifically, what to do if you need to update/transform the data you ingest.

https://nielsberglund.com/post/2023-04-03-develop-a-real-time-leaderboard-using-kafka-and-azure-data-explorer---ii/

#kafka #azuredataexplorer #kql #kusto

New post: "Interesting Stuff - Week 13, 2023".

This week:

* Cool post about Polars vs Pandas.
* How to ingest data into @azuredataexplorer. Must read!
* How do you do canary releases for data streaming?
* Data streaming and AI/ML - very interesting

Read all about it:

https://nielsberglund.com/post/2023-04-02-interesting-stuff---week-13-2023/

#kafka #python #azuredataexplorer

New post: "Develop a Real-Time Leaderboard Using Kafka and Azure Data Explorer - I"

The second post in a series about using Kafka and @azuredataexplorer
(ADX) to develop a real-time leaderboard for online gaming.

In this post, you will learn more about ADX and why it is an excellent choice for near real-time applications, how to "hook up" Kafka to ADX and how to ingest data into an ADX database using streaming ingestion.

https://nielsberglund.com/post/2023-03-19-develop-a-real-time-leaderboard-using-kafka-and-azure-data-explorer---i/

#kafka #python #docker #azuredataexplorer

New post: "Interesting Stuff - Week 11, 2023".

This week:

* Learn all about @azuredataexplorer and #Kusto, thanks to Akshay Dixit.
* I finally finished the second post in the Leaderboard, Kafka, and ADX series
* ChatGPT and dialogue prompting. Very interesting
* An old colleague of mine -Ted Neward - ponders the impact of AI on developers.
* Where has all the Big Data gone?

Read all about it:

https://nielsberglund.com/post/2023-03-19-interesting-stuff---week-11-2023/

#AzureDataExplorer #KQL #kafka #kafkaconnect #AI #ml #openai

Data Analysis in @azuredataexplorer using #KQL

In this learning path, you will learn through 6 training modules, how to analyze data in #AzureDataExplorer (#ADX) using the #Kusto Query Language (#KQL).

https://learn.microsoft.com/en-us/training/paths/data-analysis-data-explorer-kusto-query-language/

(Reshared from another site 😀)

New post: "Interesting Stuff - Week 9, 2023".

Lots of streaming stuff this week:

* Prompt engineering (again). This one looks at different "frameworks" for prompt engineering.
* Article about stream-processing vs real-time OLAP databases.
* Build data lakes using open-source tools.
* How to combine Spark Structured Streaming with REST APIs.
* Niels is still working on an Azure Data Explorer post.

Read all about it:

https://nielsberglund.com/post/2023-03-05-interesting-stuff---week-9-2023/

#openai #kafka #azuredataexplorer #spark

Ingest data from #Logstash to @AzDataExplorer
#365daysofADX Day60

#AzureDataExplorer (#ADX) Logstash plugin enables you to process events from Logstash into an Azure Data Explorer database for later analysis.

Learn More
[1] https://learn.microsoft.com/en-us/azure/data-explorer/ingest-data-logstash
[2] https://learn.microsoft.com/en-us/azure/data-explorer/connector-overview

Ingest data in @AzDataExplorer using #Telegraf Output Plugin
#365daysofADX Day58

#AzureDataExplorer output supports both streaming and queued ingestion of data from many types of telegraf input plugins into #ADX.

[1] https://lnkd.in/dSPGMmRJ
[2] https://learn.microsoft.com/en-us/azure/data-explorer/connector-overview

New post: "Interesting Stuff - Week 8, 2023".

This week there is a fair chunk of Kafka (plus other stuff):

* Linux containers on Apple Silicone.
* Deep-dive into Microsoft Research LMOps
* Yours truly: Leaderboards, Kafka & @azuredataexplorer - Prelude
* Cool stuff with Mastodon, Kafka and OpenSearch.
* Kafka and Kubernetes.
* Niels, Mike Mallet and Derivco graduates

Read all about it:

https://nielsberglund.com/post/2023-02-26-interesting-stuff---week-8-2023/

#openai #kafka #azuredataexplorer #derivco

New post: "Develop a Real-Time Leaderboard Using Kafka and Azure Data Explorer - Prelude"

This post is the first in a series about developing online gaming leaderboards. The post sets the foundation: you see how to spin up a Kafka cluster in Docker and how to run a Python app in a container generating and publishing events.

In future posts, you will read about how to hook this up to @azuredataexplorer using Kafka Connect.

https://nielsberglund.com/post/2023-02-26-develop-a-real-time-leaderboard-using-kafka-and-azure-data-explorer---prelude/

#kafka #python #docker #azuredataexplorer

Right Anti Join in @AzDataExplorer
#365daysofADX Day50

Right anti-join returns all records from the right side that don't match any record from the left side.

Try: https://lnkd.in/d98BhDNS
Learn More: https://lnkd.in/dKCZpyEe

#ADX #KQL #AzureDataExplorer #Kusto #Azure #Data

Left Anti join in @AzDataExplorer
#365daysofADX Day49

Left anti-join returns all records from the left side that don't match any record from the right side.

Try: https://lnkd.in/dmCi_th4
Learn More: https://lnkd.in/dXg9xTB7

#Kusto #KQL #AzureDataExplorer #Joins #SQL #NoSQL #ADX

Hey, do you need help with #azuredataexplorer / #kql (kusto). In that case have a look at "Tinder" for #adx

https://azure.microsoft.com/en-us/updates/adx-find-my-partner

Create a dashboard from a query in @AzDataExplorer
#365daysofADX Day43

Dashboards are an excellent way to communicate data science insights visually. You can pin a query as a visual tile in ADX KWE

Learn More: https://lnkd.in/dH4r9NWJ

#KQL #Kusto #AzureDataExplorer #ADX #Azure

Join operator in @AzDataExplorer
#365daysofADX Day42

Merge the rows of two tables to form a new table by matching values of the specified columns from each table.

Run the sample - https://dataexplorer.azure.com/clusters/help/databases/ContosoSales?query=H4sIAAAAAAAAA3MuLS7Jz00tKuaqUcjKz8xTyM7MS7HNzMtLLVIITsxJLXZLTC5RyM9TcIYq9E6tBCotScxOVTA0AABvXAKoPgAAAA==

Learn more: https://learn.microsoft.com/en-us/azure/data-explorer/kusto/query/joinoperator?pivots=azuredataexplorer

#Kusto #KQL #AzureDataExplorer #ADX