Scale By The Bay 2020: Yves Parès, Reproducible Data Pipelines Using Controlled Effects
As the data science and machine learning fields have grown over the past decade, so has the number of data pipelining frameworks which allow users to chain together a DAG of tasks. The types of tasks that one can define are nearly endless. They can range from performing pure computations like calculating the average over a window in a stream of data to performing impure actions such as writing files or executing a database query. While existing data pipelining libraries are powerful, they often suffer from a lack of reproducibility. How can one guarantee that a pipeline is reproducible when it is able to execute arbitrary side effects?
In this talk, we present a new architecture for data pipelining frameworks which promotes reproducibility along with an implementation in Haskell, kernmantle. We discuss how this architecture allows you to explicitly control which effects a pipeline is allowed to execute and how they are executed. Such effects may include file and network access, random number generation, parallel execution, and more. This framework also opens the door for config-time interpretation, which allows a pipeline to be analyzed at load-time, prior to its execution.
Yves Parès
Tweag
Software Engineer
I'm Yves Parès, a 30 y.o. software engineer, and I've been on a strict FP-only diet for about 10 years, to the point I now have trouble processing imperative programming.
My go-to meal is some closures as a starter, a big stack of monad as the main course and usually a monoid for desert (And sometimes, I indulge myself to a little mutable reference, as a cheat meal).
Lately I've been spicing it all up with arrows and algebraic effects. At first it's hot and takes some getting used to, but then it's great.
I obtained a PhD in knowledge engineering in health sciences 4 years ago, after which I started working for Tweag as a Haskell dev, mainly on scientific computing and data engineering projects.
I'm the main developer of the Porcupine and Kernmantle Haskell libraries for building configurable and portable task workflows in Haskell, the former being used by one of our clients in production.
I'd like to present to you why I and other very sound of mind people at Tweag think that this kind of high-abstraction diet is promising for software engineers in the fields of data science and simulation.
When I'm not working, I ponder over the great questions of the universe, like are we all Boltzman brains which just fluctuated out of a high-entropy soup, does the arrow of time originate from the information loss induced by quantum wavefunctions collapsing, or why are people still playing Catan when there are much better games around. Also I like improv theater.