Apache Airflow | Build Custom Sensor | KubernetesPodSensor
In this video we will build a custom sensor for Apache Airflow known as KubernetesPodSensor. This Sensor is inspired from KubernetesPodOperator, which lets you run a docker image in a dedicated pod on each poke interval.
UPDATE 2025: Watch Airflow 3 tutorials here • Apache Airflow 3 | Deploy on Kubernetes us...
Prerequsites:
1. Airflow Sensors: • Airflow sensors | Create custom sensor to ...
2. Airflow on Kubernetes using rancher: • Run Airflow on kubernetes using Rancher De...
3. Airflow KubernetesPodOperators: • Deep dive into Airflow Kubernetes Pod Oper...
4. UPDATE 2025: Watch Airflow 3 tutorials here • Apache Airflow 3 | Deploy on Kubernetes us...
Get the code:
dynamoDB-readiness-check: https://github.com/maxcotec/dynamodb-...
dag and sensor code: https://github.com/maxcotec/airflow-k...
Timeline:
00:00 intro & outline
00:38 prerequisites
01:31 Airflow Sensor recap
02:08 why KubernetesPodSensor
02:43 How KubernetesPodSensor will work
04:13 write code for sensor
06:28 write dag code
09:23 build image
11:06 deploy airflow
12:07 test the dag
15:02 conclusion
15:53 outro
#apacheairflow #maxcoteclearning #apacheairflowsensor