The AWS Data Engineering Project Every Engineer Needs
⬇️ Click here to learn how to land a high paying data engineering role NOW ⬇️ https://dataengineerinterviews.com/op...
If you’re new to my channel, my name is Christopher Garzon. I run the top Data Engineering Academy in the country, where we help students transition into data engineering from other data professions to increase their compensation.
How I got here…
At 18 years old, I started at Boston College.
At 20, I was sneaking into graduate-level classes to take machine learning and data science courses.
At 21, I invested in a data science course from a mentor and wired him $3,000 without ever meeting him.
At 22, I landed my first job as a data analyst at Amazon, making $60,000 per year.
At 24, I became a data engineer at Amazon, increasing my salary to $100,000 and started angel investing in a couple of data companies.
At 25, I moved to a startup as a data engineer and doubled my income to $200,000 per year.
At 26, I was making about $350,000 at Lyft.
At 27, Lyft stocks went up, and my total compensation reached around $450,000. That same year, I launched the Data Engineering Academy.
For the last two and a half years, I’ve been running the Data Engineering Academy full-time, helping thousands of people transition into data engineering and significantly increase their earning potential.
To all the data professionals grinding—your journey is still being written. The bigger the obstacles, the greater the story.
Remember, don’t settle for your next job. Go for a better one.
Chris
More Resources:
Learn Snowflake in 2 Hours: • Learn Snowflake in 2 Hours| High Paying Sk...
How to Ace the Data Modeling Interview: • The Easiest Way to Ace the Data Modeling I...
Don't Get Replaced by AI: • Don’t Get Replaced By AI - Do This Instead...
(00:00:00) Introduction to Data Engineering Career and Project Focus
(00:01:14) Yahoo Finance End-to-End Data Engineering Project
(00:01:25) Project Prerequisites: AWS CLI, Docker, and AWS Account
(00:01:59) Project Architecture Overview (Fetching and Transforming Data with AWS Batch)
(00:03:47) Implementing a Failure Notification Mechanism (SNS and CloudWatch)
(00:04:32) Setting Up the Yahoo Finance API Key
(00:06:29) Storing the API Key in AWS Secrets Manager
(00:10:00) Creating S3 Buckets and Folders (Raw and Transformed)
(00:11:22) Setting Up AWS CLI and Configuring User Access Key
(00:14:38) Writing the Python Script to Fetch and Load Data (Imports)
(00:16:49) Defining a Function to Get AWS Credentials (debug_credentials)
(00:21:21) Defining a Function to Fetch Secrets from Secrets Manager (fetch_secret)
(00:25:21) Defining a Function to Fetch Stock Details from Yahoo Finance API (fetch_stock_details)
(00:30:29) Defining a Function to Write Data to S3 (write_to_s3)
(00:35:28) The Main Function of the Python Script
(00:41:33) Overview of the Transform Script (Reading, Transforming, and Writing to S3)
(00:46:18) Building a Docker File for the Scripts
(00:49:55) Connecting Docker to AWS ECR and Building/Pushing the Image
(00:55:58) Setting Up the AWS Batch Job (Four Steps: Compute Environment, Job Queue, Job Definition, Job)
(00:57:48) Creating a Compute Environment (Fargate Option Chosen)
(00:59:15) Creating a Job Definition
(01:00:01) Inspecting the Docker Image Architecture (ARM 64, Linux)
(01:01:41) Setting up the IAM Role for Job Definition (Permissions Needed: ECR, S3, Batch, Secrets Manager)
(01:02:40) Specifying the Docker Image URI in Job Definition
(01:03:30) Adding Environmental Variables to the Job Definition
(01:08:27) Creating the Job Queue and Attaching the Compute Environment
(01:09:57) Submitting the AWS Batch Job
(01:13:00) Validating Job Success, Checking Logs, and Data in S3
(01:14:57) Setting Up Failure Notification with SNS and CloudWatch
(01:15:32) Creating an SNS Topic and Subscription
(01:17:39) Creating a CloudWatch Event Rule for Batch Job Failure
⬇️ Click here to learn how to land a high paying data engineering role NOW ⬇️ https://dataengineerinterviews.com/op...