Beam on Flink: How does it actually work? - Maximilian Michels
Apache Beam is a data processing model built with focus on portability. Beam jobs can be written in the language of your choice: Java, Python, Go, or SQL. Once written, they can be executed using various execution engines including Apache Flink, Apache Spark, Google Cloud Dataflow, and many more.
In order for Beam to support multiple execution engines, the Beam API needs to be translated to the API of the execution engine (e.g. Flink's). In Beam, this is the responsibility of the ""Runner"".
The Flink Runner has come a long way from an early stage Runner to a fully-featured Runner. Its latest addition is the integration of Beam's language portability layer which enabled to run jobs written in other languages than Java.
In this talk, we will dissect the Flink Runner and show how Beam's components tie together with Flink. If you have ever wondered how the Flink Runner or Beam works, this is your chance to find out.