Was experimenting today with #Debezium and the #PostgreSQL logical decoding messages.

The message is encoded in #ApacheKafka based on the binary.handling.mode parameter, with base64 being the default

Glad that #ApacheFlink has a native SQL function to decode from base64: FROM_BASE64

More info:

- Debezium logical decoding messages https://debezium.io/documentation/reference/stable/connectors/postgresql.html#postgresql-message-events
- ApacheFlink FROM_BASE64 function https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/functions/systemfunctions/#string-functions

Join us for a live demo on replicating data from various databases to CrateDB using #Debezium and #Kafka 🚀

🎤 We are thrilled to have Danica Fine, Senior Developer Advocate at Confluent, as our guest speaker to introduce you to Kafka.

🔍 Our #CrateDB experts will show you step-by-step how to leverage Debezium and Kafka to replicate data from different databases into CrateDB.

Don't miss out! Register now 👉 https://hubs.ly/Q01Wm2jJ0

#data #db #database #DataReplication

Orgulloso de aportar mi granito de arena en el proyecto open source #Debezium https://issues.redhat.com/browse/DBZ-6112

🚀 Join our FREE live #demo on July 13th and discover how to replicate data from other databases to #CrateDB using #Debezium and #Kafka

Senior Developer Advocate at Confluent, Danica Fine, will join us to introduce you to Kafka 👩🏼‍💻

Our experts will guide you through the process of setting up connections for a popular database such as #SQL server and implementing data synchronization with CrateDB without having to write any custom code. 🙌

Don't miss out! 👉 https://hubs.ly/Q01WhP4z0

Learn how to replicate data from other databases to CrateDB with Debezium and Kafka 👨🏻‍💻

This step-by-step tutorial will show you an example of replicating changes on a table from MSSQL to CrateDB 💻🔍 

Check out our latest post by Hernan Cianfagna, CrateDB Solution Engineer ➡️ https://crate.io/blog/replicating-data-from-other-databases-to-cratedb-with-debezium-and-kafka

#cratedb #db #database #Debezium #Kafka #DataAnalytics #DatabaseReplication #TechTutorial #tutorial #data

You can then use #LogicalDecoding - #Postgres’ change data capture capability - to retrieve the messages from write-ahead log (WAL), process them, and relay them to external consumers.

In this #InfoQ article, Gunnar Morling explores how to take advantage of this feature for implementing three different use cases:
✅ Propagating data between microservices via the #OutboxPattern
✅ #ApplicationLogging
✅ Enriching #AuditLogs with metadata

Learn more: http://bit.ly/3ZiMZnH

#Debezium #ApacheFlink #Microservices

Wondering how to track the changes in a #PostgreSQL database to #ApacheKafka?

Join @saltnburnem and me today, at 7PM CET on Twitch http://twitch.tv/split_software to find it out!

Hint: #CDC with #Debezium FTW!
#ChangeDataCapture

📢 Blogged: "Postgres 15: Logical Decoding Row Filters With #Debezium"

Discussing how to fine-tune the contents of change data streams right within #Postgres itself. My first one for the @decodable blog 🥳!

https://www.decodable.co/blog/postgres-15-logical-decoding-row-filters-with-debezium

During #PulsarSummit San Francisco earlier this year, I did a hands-on session about using KoP (Kafka-On-Pulsar) with these uses cases:

✅ Microservices with #ApacheKafka
✅ CDC with #Debezium and #MySQL
✅ Stream processing with #ksqlDB

Here is the recording. 👇🏻

https://www.youtube.com/watch?v=Rk_5XBciYuk

🗣️ "It raises the question of how we can automatically update the database and publish events"

Loved reading this *excellent* post by Yingying Tang of Brex about their experiences with implementing the outbox pattern using #Debezium 👍.

https://medium.com/brexeng/transactional-events-publishing-at-brex-66a5984f0726

RT @nilshartmann@twitter.com

I am happy and honoured (and a little nervous 😬) to be part of #GraphQLGalaxy conference with that great line up of #GraphQL experts (and me...). 😊

Looking forward to share some ideas about implementing subscriptions with #Kafka and #Debezium in my lightning talk. ⚡️ https://twitter.com/GraphQLGalaxy/status/1595777033728102400

🐦🔗: https://twitter.com/nilshartmann/status/1595815880876851201

📢 Happy to introduce pgoutput-cli, a thin wrapper around the pypgoutput library for examining #Postgres CDC events in the pgoutput binary format. It's on Docker Hub, so just do `docker run`, and you can e.g. examine your #Debezium pgoutput streams.

Big kudos to Daniel Geals, the author of pypgoutput, which does all the heavy-lifting here. Make sure to 🌟 it on GitHub. OSS FTW 🚀!

https://github.com/gunnarmorling/pgoutput-cli
https://github.com/dgea005/pypgoutput

I am happy and honoured (and a little nervous 😬) to be part of #GraphQLGalaxy conference with that great line up of #GraphQL experts (and me...). 😊

Looking forward to share some ideas about implementing subscriptions with #Kafka and #Debezium in my lightning talk. ⚡️

Dive is such a useful tool for identifying unnecessary bloat in the image layers of container images. I just could shave off ~170 MB from #Debezium's image for Kafka by combining several separate RUN steps into one 🤯.

https://github.com/wagoodman/dive

Good weekend read 🤓

RT @gunnarmorling@twitter.com

"Squirreling Away $640 Billion: How Stripe Leverages Flink for Change Data Capture"

Excellent deck by @thejeffchao@twitter.com about a really cool @ApacheFlink@twitter.com use case: aggregating events per source transaction, based on #Debezium's TX marker events 🔥.

https://www.slideshare.net/FlinkForward/squirreling-away-640-billion-how-stripe-leverages-flink-for-change-data-capture

🐦🔗: https://twitter.com/gunnarmorling/status/1593620793740738562

"Squirreling Away $640 Billion: How Stripe Leverages Flink for Change Data Capture"

Excellent deck by Jeff Chao about a really cool Apache Flink use case: aggregating events per source transaction, based on #Debezium's TX marker events 🔥.

https://www.slideshare.net/FlinkForward/squirreling-away-640-billion-how-stripe-leverages-flink-for-change-data-capture

📺 The recording of Sharon Xie's and my session about change stream processing with Apache Flink and #Decodable is up on YouTube now! Thank you so much to everyone who joined and for all the great questions, this was fun 🤓!

https://www.youtube.com/watch?v=LKX76Hnwtis

#streamprocessing #changedatacapture #debezium #apacheflink #dataengineering

RT @gunnarmorling
📺 "Keep Your Cache Always Fresh With #Debezium!"

The recording from my #Current22 session is online now! Queryable denormalized data views, close to your users -- One of the most powerful use cases enabled by change data capture. #CDC #Kafka #Infinispan

https://www.confluent.io/events/current-2022/keep-your-cache-always-fresh-with-debezium-1/

📺 "Keep Your Cache Always Fresh With #Debezium!"

The recording from my #Current22 session is online now! Queryable denormalized data views, close to your users -- One of the most powerful use cases enabled by change data capture.

https://www.confluent.io/events/current-2022/keep-your-cache-always-fresh-with-debezium-1/

#CDC #Kafka #Infinispan #DataEngineering

Do you want to CDC out of Oracle RDS? Here’s and easy walk through. https://github.com/decodableco/examples/blob/main/oracle_cdc/README.md. #cdc #oracle #rds #debezium