Fighting phishing and spam with online machine learning on data streams - Wojtek Ptak
Machine Learning on streaming data is becoming truly important for our products. Our systems grow in size producing more and more data with users expecting to get their results in real-time. On the other hand, our real-life business scenarios are changing even faster - we’re even experiencing critical patterns changing within hours. Great examples where online machine learning on data streams can be applied are problems like spam or fraud detection, online recommendations, feeds personalisation and others.
In this presentation, I will go through our approach and lessons we learnt when building and applying Machine Learning solutions on data streams using Apache Flink. In our FreshMail case we are working on using this approach to build a new generation anti-abuse engine - SendGuard - working on thousands of messages per second, over 70M messages a day using FlinkML and Flink CEP. We use it to protect recipients from getting abuse messages like spam, spoofing or phishing.