Apache Airflow Event Based Trigger: Is It Actually Real-Time?
Apache Airflow event based trigger is one of the coolest new features in Airflow 3. In this video, we gona test how Airflow reacts to real events using Kafka and measure the latency between a message being produced and a DAG being triggered.
This experiment runs on Airflow standalone using SQLite and SequentialExecutor, which is the typical local development setup.
Code: https://github.com/maxcotec/apache-ai...
Video outline:
00:00 intro
01:00 setup kafka + topic
02:33 setup airflow
04:55 write event-trigger dag
09:38 spinup airflow + UI overview
10:50 test event-trigger dag
13:00 diagnose trigger delay
15:50 outro