How to manage Airflow Dags in Production | Dags versioning & Deployment
#apacheairflow #airflowatjagex #jagex #apacheairflowforbeginners #airflowinproduction
In this video, I'll discussing on how we (at Jagex) manage and operate more than 100+ ETL pipelines in Airflow efficiently.
00:00 Intro
00:27 Issues we faced
02:08 Airflow & DAGs versioning
05:47 Airflow Deployment Process
06:51 Automated CICD deployment
08:53 Airflow Dag versioning proposal AIP-36
09:11 Outro
Learn Something New Below:
https://maxcotec.com/blog/chip-chick-...
https://maxcotec.com/blog/what-is-tra...
👍 If this video was helpful to you, please don't forget to thumbs-up,
⌨️ leave a comment and
🔗 share with your friends.
⭐ Support our work by hitting that subscribe button.
🔔 hit bell icon to stay alert on more upcoming interesting stuff.
DAG versioning proposal
https://cwiki.apache.org/confluence/d...