I attended the #AWSSummitLondon this week. I started with AWS ~6 months ago. It was gratifying to realise that I have learned a lot since then. I talked to a few experts and they told me I was in the right path and that the struggles I have with #AWSGlue are not only mine (they simply don’t support well #deltaLake ). My perfectionist self was relieved 😌 I didn’t solve any of my problems but sometimes it helps to realise you are not as stupid as your programming struggles make u feel sometimes… 😅

Has anyone had problems importing their own (meaning self made) module into #AWSglue notebooks? The package works perfectly fine in a normal glue job but if I add it to a notebook, I all the time get an Import Error! #AWS is so frustrating sometimes!! Could it be that my job uses glue 4 and the notebook 3 (since 4 is not supported)?!!!

Woof, file compaction with #DeltaLake 1.x is the only way to make it usable. 50-100x performance improvements depending on data and partition sizes. The default merge and write operations are incredibly inefficient. I understand its been greatly improved in 2.x. We're a couple months from upgrading the platform though. #datawarehouse #datalake #spark #pyspark #aws #awsglue

An #Introduction

Former strategy consultant turned data person living in the Highlands of Scotland and #RemoteWorking

I work primarily with a #SQLServer #DataWarehouse but am in the midst of a project to migrate these data pipelines and analytics to #AWS leveraging #AWSAthena and #AWSGlue with #Tableau

I'd like to learn about other visualization tools like #D3js that present complex yet understandable

Outside work I enjoy being in #Nature, #DogWalking, #Coffee, and #Reading