Managed Apache Airflow With Astro! Goodbye MWAA and GCC
Manage Airflow With Astro! Goodbye, MWAA and GCC
Astro is the new way of managing Apache Airflow in the cloud!
It supercharges Airflow by providing all you need to author, run, manage, and monitor your data pipelines at scale without the usual hassles such as no local development environments, no auto-scaling or in-place upgrades, poor CI/CD experience, being stuck in a proprietary ecosystem, etc.
At the end of this video, you will be able to:
1️⃣ Create Airflow deployments for your dev and prod environments
2️⃣ Set up a local development environment with the Astro CLI
3️⃣ Easily and quickly create data pipelines and tasks with Cloud IDE
4️⃣ Define a CI/CD pipeline to deploy in dev and prod automatically
5️⃣ Monitor your pipelines and get alerted if something goes wrong
Try Astro for FREE: https://bit.ly/43YD1tS
00:00 - Intro
00:37 - Why Astro?
00:55 - Sign up
01:17 - Welcome to Astro
01:19 - Create the DEV Airflow deployment
02:48 - Astro Deployment Features
03:25 - The Data Pipeline you will create
03:52 - Prerequisites
04:15 - Set up your local development environment with the Astro CLI
05:52 - Create the DEV branch
06:03 - Create your data pipeline with Cloud IDE
06:45 - Task 1: Fetch API data
07:14 - Create an Airflow Variable
07:29 - Install Python dependencies
07:47 - Make global imports
07:57 - Test the task
08:10 - Task 2: Print the random activity
08:59 - Commit your DAG with Cloud IDE
10:03 - Create and configure the CI/CD pipeline
10:30 - Create the DEV API key
11:26 - Create the PROD Airflow deployment
11:59 - Create the PROD API key
12:31 - Deploy on DEV with the CICD pipeline
13:17 - Deploy on PROD with the CICD pipeline
13:54 - Create the API variable in the deployments
14:18 - Trigger the DAG
14:26 - Monitoring your deployments and pipelines
14:50 - Get alerted if a DAG run fails
15:19 - Congratulations!
15:47 - User management and more
Enjoy ❤️
#dataengineering #data #airflow