Jakub Adamczyk - Pipeline orchestration tools - what to look out for
🔹Pipeline orchestration tools - what to look out for
🔹Jakub Adamczyk
Modern ML and data engineering pipelines are complex distributed systems. Pipeline (workflow) orchestration systems aim to bring order and efficiency to chaos, but come with their own host of problems, and the first one is to pick your poison from the vast number of possibilities: Apache Airflow, Prefect, Dagster, Metaflow, Kubeflow Pipelines, AWS Step Functions...
There are many advantages, problems and tradeoffs with every tool. In this talk, I will explain concepts and ideas underlying pipeline orchestration, outline basic characteristics of most popular tools, and compare them for typical use cases. We will consider common challenges such as multi-repository Git setup, dependency separation, versioning, execution engines etc.
Challenges and problems will also be illustrated by my personal experiences and horror stories, coming from over a dozen projects in ML and data engineering pipelines.