After a good three months of #dezoomcamp, I can only recommend the free course to everyone. The team puts together a fantastic program and the community is great. Everyone is so helpful so that everyone can get the best out of this course. THANK YOU

Additional advantages of peer reviewing:
- opportunity for networking and collaboration with others in the same field or discipline
- stay current with the latest research and developments in the field
- learn about new approaches and techniques
#dezoomcamp

Peer reviewing other people's projects can be a valuable learning experience for several reasons:
- identify strengths and weaknesses in your own work by comparing
- exposes you to new perspectives and ideas that you may not have considered before.
- develop critical thinking skills by evaluating someone else's work.
#dezoomcamp

The peer review at #dezoomcamp is a great opportunity to find inspiration for future projects. Read more about it here:
https://github.com/DataTalksClub/data-engineering-zoomcamp/blob/main/cohorts/2023/project.md

I got three really exciting projects: The first one, which I'm supposed to grade, had a student do a sentiment analysis with tweets about the Corona crisis.
#dezoomcamp

Yesterday was the deadline for our projects for #dezoomcamp. Today the peer review finally starts: Everyone gets assigned three repositories and has to rate them according to given criteria.

For orchestrating my prefect flow I decided to use another technology. Github Actions is a CI/CD service that automates workflows, like building, testing, and deploying code. You can create custom workflows using YAML files or choose from community-shared workflows. This is a great step-by-step guide:
https://medium.com/the-prefect-blog/scheduled-data-pipelines-in-5-minutes-with-prefect-and-github-actions-39a5e4ab03f4
#dezoomcamp

The last part of my project for #dezoomcamp is the visualization. Since my tables are in BigQuery, I decided to use Google Looker Studio. You can also use it with a bunch of other data sources. https://lookerstudio.google.com/

In dbt, staging and production are two separate environments that serve different purposes. Staging is where you prepare data for production, with testing and validation. Production is where end-users access and interact with data. Staging is for data preparation, production is for delivering value to users. #dezoomcamp

I also found that you can use dbt to partition tables. The help to improve query performance and manageability by dividing large tables into smaller, more manageable parts. They enable faster queries by allowing data to be accessed and processed more efficiently, reducing the amount of data scanned by queries. #dezoomcamp

After loading my data into Google BigQuery I do some transformations with dbt. For beginners I would recommend the resources provided on the dbt website:
https://courses.getdbt.com/courses/fundamentals
#dezoomcamp

A first draft for my architecture for my project for #dezoomcamp :

It took me a while to fully grasp the concept of scheduling: If you schedule a flow in prefect cloud you still have to keep the agent running. Possible setup solution can be found here: https://medium.com/@danilo.drobac/7-a-complete-google-cloud-deployment-of-prefect-2-0-32b8e3c2febe
#dezoomcamp

You can work with prefect locally or in the cloud. I went with the cloud version, since I sometimes have problems accessing the Orion UI.
https://docs.prefect.io/ui/cloud/
#dezoomcamp

For orchestration I decided to go with prefect. Every day at 6, my flow loads the data from the .csv into Google Cloud Storage. 15 minutes later it loads the data from Cloud Storage to BigQuery. I used Blocks for that:
https://docs.prefect.io/concepts/blocks/
#dezoomcamp

I want to design a clear diagram of the pipeline of my capstone project for #dezoomcamp. For this I have decided to use miro: https://miro.com/de/

The first step of my capstone project for #dezoomcamp will be to create a new virtual machine on google cloud platform and document this step already - so I can make sure that everything is reproducible.

Today is the first day that I am working on my capstone project for #dezoomcamp . I decided to work with covid data by our world in data: https://github.com/owid/covid-19-data/

An important factor in data quality are tests: I learned in the workshop that Piperider allows you to test aspects of your data such as:
- do certain columns exist?
- are the unique?
- is the datatype correct?
- are there null value?
- what is the range of the column?
- is the schema consistent?
#dezoomcamp

PipeRider is open-source which is a big plus. You can connect it to your data source at any point in your pipeline, generate a profile and test your profile against your assertions.
#dezoomcamp
https://blog.piperider.io/data-reliability-automated-with-piperider.html