Machine Learning Inference in Flink with ONNX
What is the best way to run inference on a machine learning model in your streaming application? We will unpack this question, and explore the ways to leverage machine learning models in Flink. Starting from a PyTorch model in a Python training environment, we will leverage ONNX to be able to serve the model through a Scala Flink application. By using the Java version of the onnxruntime library, we will show that the computation can be done in the same TaskManager and packaged in the same JAR as a resource. This was inspired by a previous Flink Forward talk on ONNX that leveraged Py4J to run inference from Python. Our technique simplifies the deployment and management for production inference by using Scala directly and packaging the model with the code. We will dive into the pros and cons of this technique, and examine other methods for harnessing the power of machine learning in Flink.
About the Speaker: Colin Jermain is a Data Science Team Lead at Vectra AI in Boston, managing a team of Data Scientists to build advanced algorithms and machine learning models for cyberattack detection. Colin is an avid open-source developer, and has contributed to keras2onnx and PyTorch to improve the ONNX ecosystem. He holds a PhD in Physics from Cornell University and has been an active Python developer for over 14 years. He enjoys working across technical disciplines, from DevOps in the cloud to exploring new AI techniques and how to run them in production settings.