sfspark.org: Stefan Ewen, Introducing Apache Flink
Scale By the Bay 2019 is held on November 13-15 in sunny Oakland, California, on the shores of Lake Merritt: https://scale.bythebay.io. Join us!
-----
ALl APache Flink cofounders were present at this inaugural SF Spark and Friends talk, and Stefan did an amazing and thorough overview of Apache Flink, a streaming-first fast data framework.
Apache Flink (flink.apache.org) is an open-source framework for batch- and streaming data analysis on top of a streaming dataflows, with high-level APIs and libraries for diverse use cases. Flink joined the Apache Incubator in 2014, and graduated as a top level project in December 2014. Since entering the Apache family, Flink has grown a lot both in terms of features and community.
At the heart of Apache Flink is a flexible dataflow engine that supports diverse features and workloads without compromising on performance or usability: The engine executes data streaming programs directly as streams (with low latency and flexible user-defined state), and models batch programs as streaming programs on finite data streams. Iterative programs are supported though feedback in the dataflow, graph analysis via "delta-iterations". Through elaborate memory management inside the JVM, Flink scales beyond main memory resident data sets.
On top of the dataflow engine, the Flink community has added fluent programming APIs for batch-, and stream processing, as well as a set of libraries, such as the Table API (relational queries), FlinkML (Machine Learning library), and Gelly (API and library for graph analysis).
This talk will present the architecture of Flink and discusses the design choices and tradeoffs that come with building a versatile analysis engine on top of a data streaming abstraction. We show examples and use cases, and give an outlook of the current developments in the Flink project.
Stephan Ewen is Flink committer and co-founder and CTO of data Artisans. Before founding data Artisans, Stephan was leading the development of Flink since the early days of the project (then called Stratosphere). Stephan has a PhD in Computer Science from TU Berlin and has worked with IBM and Microsoft.