Data Platform Week Day 2 - Core Orchestration: Event-driven automation and native partitioning
Best-in-class data engineering requires a comprehensive orchestration solution with flexible tools for partitioning and event-based scheduling.
Dagster Engineer Owen Kephart discusses the key new features in declarative automation to simplify complex workflow dependency management and custom automation rules.
Director of Engineering at Enigma Tian Xie discusses how his team uses declarative automation to orchestrate complex interweaving pipelines of different SMB data into one cohesive data.
Data Engineer at Kahoot! Stig-Martin Liavåg shares how Kahoot! orchestrates jobs at different levels of granularity. Due to their volume of data, needing to be able to customize automation conditions is critical to get the desired performance without breaking the bank
Timeline:
00:00 Introduction
01:29 Declarative Automation with Owen
15:14 Orchestrating SMB Data with Tian
26:08 Big Data Orchestration with Stig-Martin
40:13 Q&A