Delay Modeling and DAG Connectivity: Optimizing Airflow performance in large organizations
Presented by Ahuitz Rojas in Airflow Summit 2023.
In large organizations, data workflows can be complex and interconnected, with multiple dependencies and varied runtime requirements. To ensure efficient and timely execution of workflows, it is important to understand the factors that affect the performance of the system, such as network congestion, resource availability, and DAG structure. In this talk, we will explore how delay modeling and DAG connectivity analysis can be used to optimize Airflow performance in large organizations.