nielsb · @nielsb
80 followers · 129 posts · Server fosstodon.org

New post: "Interesting Stuff - Week 18, 2023".

This week:

* AI improving their programming abilities using self-play!
* Build ML-driven segmentation using chatbots, ChatGPT and LangChain.
* Use diffusion models to mimic human behaviour.
* Inferring rewards through interaction. Use of IGL-P!
* Confluent Platform 7.4 released. Cool!
* Niels is busy with @azuredataexplorer post

Read all about it:

nielsberglund.com/post/2023-05

#ai #ml #kafka #azuredataexplorer #openai

Last updated 1 year ago

nielsb · @nielsb
80 followers · 128 posts · Server fosstodon.org

New post: "Interesting Stuff - Week 17, 2023".

This week:

* The gambler that beat roulette!
* I finally created a repo covering how to stream data from Kafka to @azuredataexplorer.
* Real-time ML pipelines.
* Databricks and Hugging Face, love is in the air.
* Very interesting about the real-time streaming ecosystem by Hubert.

Read all about it:

nielsberglund.com/post/2023-04

#ai #ml #streaming #kafka #databricks #azuredataexplorer

Last updated 2 years ago

nielsb · @nielsb
79 followers · 127 posts · Server fosstodon.org

New post: "Interesting Stuff - Week 16, 2023".

This week:

* How to optimize the Azure Databricks Delta Engine.
* Link to my post about the multi-platform Docker Image for @azuredataexplorer Kusto Sink and Kafka Connect.
* Create Synthetic Data using Generative AI.
* Databricks SQL and the ability to call LLMs!

Read all about it:

nielsberglund.com/post/2023-04

#ai #ml #openai #databricks #azuredataexplorer

Last updated 2 years ago

nielsb · @nielsb
79 followers · 126 posts · Server fosstodon.org

New post: "Cross-Platform Compatibility Made Easy with Multi-Platform Docker Images: Azure Data Explorer Sink Connector & Kafka Connect"

Docker has revolutionized the way we package and deploy software applications.

However, it's challenging to maintain a single image that can work across multiple platforms.

This blog post explores the steps required to build and deploy multi-platform Docker images.

nielsberglund.com/post/2023-04

#kafka #azuredataexplorer #kusto #KafkaConnect

Last updated 2 years ago

nielsb · @nielsb
78 followers · 125 posts · Server fosstodon.org

New post: "Interesting Stuff - Week 15, 2023".

This week:

* Generative AI and focus shift from tech-driven dev to product-driven dev.
* Game analytics solution for Game Studios!
* Late arriving events in Flink and ways to handle them.
* Very cool around materialized views.
* Me plugging an upcoming post about how to build multi-platform Docker images.

Read all about it:

nielsberglund.com/post/2023-04

#kafka #flink #ai #ml #openai #databricks #azuredataexplorer

Last updated 2 years ago

nielsb · @nielsb
78 followers · 123 posts · Server fosstodon.org

New post: "Develop a Real-Time Leaderboard Using Kafka and Azure Data Explorer - II"

This is the third post about using Kafka and @azuredataexplorer (ADX) to develop a real-time leaderboard for online gaming.

In this post, you will see more about ingesting data into ADX and some "cool stuff" related to that. Specifically, what to do if you need to update/transform the data you ingest.

nielsberglund.com/post/2023-04

#kafka #azuredataexplorer #KQL #kusto

Last updated 2 years ago

nielsb · @nielsb
78 followers · 122 posts · Server fosstodon.org

New post: "Interesting Stuff - Week 13, 2023".

This week:

* Cool post about Polars vs Pandas.
* How to ingest data into @azuredataexplorer. Must read!
* How do you do canary releases for data streaming?
* Data streaming and AI/ML - very interesting

Read all about it:

nielsberglund.com/post/2023-04

#kafka #python #azuredataexplorer

Last updated 2 years ago

nielsb · @nielsb
76 followers · 120 posts · Server fosstodon.org

New post: "Develop a Real-Time Leaderboard Using Kafka and Azure Data Explorer - I"

The second post in a series about using Kafka and @azuredataexplorer
(ADX) to develop a real-time leaderboard for online gaming.

In this post, you will learn more about ADX and why it is an excellent choice for near real-time applications, how to "hook up" Kafka to ADX and how to ingest data into an ADX database using streaming ingestion.

nielsberglund.com/post/2023-03

#kafka #python #docker #azuredataexplorer

Last updated 2 years ago

nielsb · @nielsb
76 followers · 119 posts · Server fosstodon.org

New post: "Interesting Stuff - Week 11, 2023".

This week:

* Learn all about @azuredataexplorer and , thanks to Akshay Dixit.
* I finally finished the second post in the Leaderboard, Kafka, and ADX series
* ChatGPT and dialogue prompting. Very interesting
* An old colleague of mine -Ted Neward - ponders the impact of AI on developers.
* Where has all the Big Data gone?

Read all about it:

nielsberglund.com/post/2023-03

#kusto #azuredataexplorer #KQL #kafka #KafkaConnect #ai #ml #openai

Last updated 2 years ago

nielsb · @nielsb
75 followers · 118 posts · Server fosstodon.org

Data Analysis in @azuredataexplorer using

In this learning path, you will learn through 6 training modules, how to analyze data in () using the Query Language ().

learn.microsoft.com/en-us/trai

(Reshared from another site 😀)

#KQL #azuredataexplorer #adx #kusto

Last updated 2 years ago

nielsb · @nielsb
75 followers · 115 posts · Server fosstodon.org

New post: "Interesting Stuff - Week 9, 2023".

Lots of streaming stuff this week:

* Prompt engineering (again). This one looks at different "frameworks" for prompt engineering.
* Article about stream-processing vs real-time OLAP databases.
* Build data lakes using open-source tools.
* How to combine Spark Structured Streaming with REST APIs.
* Niels is still working on an Azure Data Explorer post.

Read all about it:

nielsberglund.com/post/2023-03

#openai #kafka #azuredataexplorer #spark

Last updated 2 years ago

Ingest data from to @AzDataExplorer
Day60

() Logstash plugin enables you to process events from Logstash into an Azure Data Explorer database for later analysis.

Learn More
[1] learn.microsoft.com/en-us/azur
[2] learn.microsoft.com/en-us/azur

#logstash #365daysofadx #azuredataexplorer #adx

Last updated 2 years ago

Ingest data in @AzDataExplorer using Output Plugin
Day58

output supports both streaming and queued ingestion of data from many types of telegraf input plugins into .

[1] lnkd.in/dSPGMmRJ
[2] learn.microsoft.com/en-us/azur

#telegraf #365daysofadx #azuredataexplorer #adx

Last updated 2 years ago

nielsb · @nielsb
73 followers · 114 posts · Server fosstodon.org

New post: "Interesting Stuff - Week 8, 2023".

This week there is a fair chunk of Kafka (plus other stuff):

* Linux containers on Apple Silicone.
* Deep-dive into Microsoft Research LMOps
* Yours truly: Leaderboards, Kafka & @azuredataexplorer - Prelude
* Cool stuff with Mastodon, Kafka and OpenSearch.
* Kafka and Kubernetes.
* Niels, Mike Mallet and Derivco graduates

Read all about it:

nielsberglund.com/post/2023-02

#openai #kafka #azuredataexplorer #derivco

Last updated 2 years ago

nielsb · @nielsb
73 followers · 113 posts · Server fosstodon.org

New post: "Develop a Real-Time Leaderboard Using Kafka and Azure Data Explorer - Prelude"

This post is the first in a series about developing online gaming leaderboards. The post sets the foundation: you see how to spin up a Kafka cluster in Docker and how to run a Python app in a container generating and publishing events.

In future posts, you will read about how to hook this up to @azuredataexplorer using Kafka Connect.

nielsberglund.com/post/2023-02

#kafka #python #docker #azuredataexplorer

Last updated 2 years ago

Right Anti Join in @AzDataExplorer
Day50

Right anti-join returns all records from the right side that don't match any record from the left side.

Try: lnkd.in/d98BhDNS
Learn More: lnkd.in/dKCZpyEe

#365daysofadx #adx #KQL #azuredataexplorer #kusto #azure #data

Last updated 2 years ago

Left Anti join in @AzDataExplorer
Day49

Left anti-join returns all records from the left side that don't match any record from the right side.

Try: lnkd.in/dmCi_th4
Learn More: lnkd.in/dXg9xTB7

#365daysofadx #kusto #KQL #azuredataexplorer #joins #sql #nosql #adx

Last updated 2 years ago

nielsb · @nielsb
70 followers · 108 posts · Server fosstodon.org

Hey, do you need help with / (kusto). In that case have a look at "Tinder" for

azure.microsoft.com/en-us/upda

#azuredataexplorer #KQL #adx

Last updated 2 years ago

Create a dashboard from a query in @AzDataExplorer
Day43

Dashboards are an excellent way to communicate data science insights visually. You can pin a query as a visual tile in ADX KWE

Learn More: lnkd.in/dH4r9NWJ

#365daysofadx #KQL #kusto #azuredataexplorer #adx #azure

Last updated 2 years ago