The ExternalPythonOperator: No more dependency conflicts in Apache Airflow
The ExternalPythonOperator: No more dependency conflicts in Apache Airflow
🏆 BECOME A PRO WITH AIRFLOW: https://www.udemy.com/course/the-comp...
Airflow has many options to avoid dependency conflicts.
If you have a Kubernetes cluster, you can use the KubernetesPodOperator to run a task in a Kubernetes Pod.
If you have Docker, then the DockerOperator helps to run a task in a Docker Container.
An easier, though, is the ExternalPythonOperator which requires a pre-created Python virtual environment.