Airflow DockerOperator: The Basics (and more 🤫)
Airflow DockerOperator: The Basics (and more 🤫)
👍 Smash the like button to become an Airflow Super Hero!
❤️ Subscribe to my channel to become a master of Airflow
🏆 BECOME A PRO: https://www.udemy.com/course/the-comp...
🚨 My Patreon: / marclamberti
The Airflow DockerOperator is a very powerful operator.
It executes your task within a docker container. There are multiple advantages of using the DockerOperator such as:
Easier way to test your task
Control over the resources needed by your task
Avoid dependencies conflicts
and more.
Even if you shouldn't use ONLY the DockerOperator, knowing how it works and what you can with it will truly help to make more reliable data pipelines.
Ready?
Lets go!