Was experimenting today with #Debezium and the #PostgreSQL logical decoding messages.

The message is encoded in #ApacheKafka based on the binary.handling.mode parameter, with base64 being the default

Glad that #ApacheFlink has a native SQL function to decode from base64: FROM_BASE64

More info:

- Debezium logical decoding messages https://debezium.io/documentation/reference/stable/connectors/postgresql.html#postgresql-message-events
- ApacheFlink FROM_BASE64 function https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/functions/systemfunctions/#string-functions

Need to build/parse #JSON payloads using #SQL? Check #ApacheFlink functions!

- JSON_BUILD_OBJECT to build an object based on a series of keys and values
- JSON_ARRAYAGG to aggregate values into an array
- JSON_VALUE to retrieve a single field from a JSON object
- JSON_QUERY to retrieve multiple nested records from a JSON object

More info https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/functions/systemfunctions/#json-functions

Test your #ApacheFlink SQL with Docker!

https://github.com/Aiven-Open/sql-cli-for-apache-flink-docker

SQL Client for #ApacheFlink on Docker now available for Apache Flink 1.17.1!

Thanks to Sergey Nuyanzin for the speedy review and the help in fixing some problems!

https://github.com/Aiven-Open/sql-cli-for-apache-flink-docker/releases/tag/1.17.1

#CaseStudy – Instacart made the platform easier to use & reduced their operational / infrastructure costs.

How? Building an #ApacheFlink self-serve platform on #Kubernetes at scale.

Check out #InfoQ to learn how they did it: https://bit.ly/44l5tGP

#SoftwareArchitecture

Curious to hear from folks, besides #apacheKafka, what are your favorite #streaming technologies? 🤔

#streamingdata #eventstreaming #streamingtechnology #apacheFlink #apacheSpark #apachePulsar

The end of May brings the May edition of the #ApacheIceberg Community News. There is a lot of great content from the community once again with the release of Iceberg 1.3, and significant updates to PyIceberg, with the 0.4.0 release right around the corner. Important support was added for #ApacheSpark version 3.4 and #ApacheFlink version 1.17. There is also significant news from the vendor community and great blog posts from folks like Anuj Syal and Marin Aglić Čuvić.
https://tabular.io/blog/iceberg-202305/

Quick video on #ApacheFlink Windowing options

https://www.youtube.com/watch?v=hxsdibISgAs

@ftisiot I'm obviously biassed, because I work at Aiven and @ftisiot is a colleague, but I loved this, and I do love the data dances one can do with #ApacheKafka and #ApacheFlink

Need to convert #ApacheKafka topics from JSON to AVRO (bonus points for using #SchemaRegistry)?

Check how you can do it with #ApacheFlink

https://www.youtube.com/watch?v=FED4u6er9DE

Need to covert #ApacheKafka topics from JSON to AVRO (bonus points for using #SchemaRegistry)?

Check how you can do it with #ApacheFlink

https://www.youtube.com/watch?v=FED4u6er9DE

The @ApacheIceberg newsletter for March is here. Lots of big news with version 1.2, #ApacheSpark, #ApacheFlink and vendor support.
https://tabular.substack.com/p/iceberg-newsletter-mar-2023?sd=pf

New blog is out!

One of #ApacheKafka’s mantra is that it preserves message ordering, but, is it true?

Check out the edge cases and how to solve them (also using #ApacheFlink)

https://aiven.io/blog/kafka-real-ordering

#datastreaming

RT @snntrable
📢The CfP for Current22 is closing soon. Me, @alpinegizmo and the rest of program committee are looking forward to more last minute, #ApacheFlink-related submissions.

Happy to give feedback on an abstract upfront via DM.

https://sessionize.com/current-2023/

#Current23 is the next generation of #KafkaSummit, so expect some #apacheKafka talks. But there's room for everyone! #apacheDruid, #apacheFlink, #ETL, #dataMesh, #machineLearning, #IoT, #databases...

We have room for talks on the unique #devops challenges of monitoring and managing any of these systems; talks on #sociotechnical impact of rearchitecting your business for streaming; #streaming in academia; and just plain fun and geeky sessions.

The list goes on and on!

Had the pleasure of being interviewed by @alex_casalboni for the @awscloud podcast in Italian.

Easy part: share my passion about #datastreaming with #ApacheKafka and #ApacheFlink

Complex part: talk about work in Italian after 15 years.

Check it out! https://podcasts.google.com/feed/aHR0cHM6Ly9hbmNob3IuZm0vcy8xNTM4OTBiOC9wb2RjYXN0L3Jzcw/episode/MzQ2MWE2OWQtYmJjOC00NzY5LWEwMjAtYjFhMmI3ZmRjNjk5?sa=X&ved=0CAUQkfYCahcKEwiA1cvTptn9AhUAAAAAHQAAAAAQAQ

From 0 to a complete #streaming #datapipeline in ~12 minutes with @aiven_io for #ApacheFlink!

Check it out!

https://www.youtube.com/watch?v=RT5zjEWc1mg

You can then use #LogicalDecoding - #Postgres’ change data capture capability - to retrieve the messages from write-ahead log (WAL), process them, and relay them to external consumers.

In this #InfoQ article, Gunnar Morling explores how to take advantage of this feature for implementing three different use cases:
✅ Propagating data between microservices via the #OutboxPattern
✅ #ApplicationLogging
✅ Enriching #AuditLogs with metadata

Learn more: http://bit.ly/3ZiMZnH

#Debezium #ApacheFlink #Microservices

It's incredible to see how a product is born and help shaping it.

After more than a decade in consulting, I joined @aiven_io a couple years ago and got to experience how a new product like Aiven for #ApacheFlink is defined, developed and taken to market.

🧵

Today, 4PM CET, come and learn about #streamprocessing, #ApacheFlink and how you can start creating streaming data pipelines in minutes with @aiven_io

https://landing.aiven.io/webinar/apache-flink-dev-experience/