How to Avoid Suffering in MLOps/Data Engineering Role // Igor Lushchyk // MLOps Meetup #55
MLOps community meetup #55! Last Wednesday we talked to Igor Lushchyk, Data Engineer, Adyen.
// Abstract:
Building Data Science and Machine Learning platforms at a scale-up. Having the main difficulty in finding correct processes and basically being a toddler who learns how to walk on a steep staircase. The transition from homegrown platform to open source solutions, supporting old solutions and maturing them with making data scientists happy.
// Bio:
Igor is a software engineer with more than 10 years of experience. With a background in bioinformatics, he even started Ph.D. but didn't finish it.
As a data engineer, Igor has been working for the last 6 or 7 years, or maybe more - because he was doing almost the same data engineering stuff but his position was named differently.
Igor has been doing a lot of MLOps in 4-5 years now. He doesn't know what he was doing more then - Data Engineering or MLOps. And that’s how this topic came about.
---------- ✌️Connect With Us ✌️------------
Join our Slack community: https://go.mlops.community/slack
Follow us on Twitter: @mlopscommunity
Sign up for the next meetup: https://go.mlops.community/register
Catch all episodes, Feature Store, Machine Learning Monitoring and Blogs: https://mlops.community/
Connect with Demetrios on LinkedIn: / dpbrinkm
Connect with Igor on LinkedIn: / igor-lushchyk
Timestamps:
[00:00] Introduction to Igor Lushchyk
[02:05] Igor's background in tech
[07:42] Tips you can pass on
[08:25] "Don't use huge frameworks when you don't understand how they work. Try to understand everything from smaller pieces."
[08:52] "It's better to use more frameworks and to build everything by pieces."
[09:58] "Work with small pieces and look inside."
[11:05] How do these tools work and how do they play together and what is underneath?
[12:16] "If I don't know something, it's better for me to say 'I'll know.'"
[13:18] Dedicated MLOps team
[13:55] Central Data Infrastructure Section
[16:06] "We need to share our pain because we need to solve the pain."
[16:57] Transfer over to open-source
[18:46] "You cannot start thinking about POC. You need to think about being production-ready."
[20:24] If you don't plan for production from the beginning, then it's going to be painful trying to go from POC to production.
[20:37] "In the data world, you need to forget about POC. Period. You need to think about the product."
[22:08] Ho do you handle data lineage?
[25:09] You chose that back in the day but you're regretting it.
[25:47] "It's better for me to work supported by the community open-source and try to enrich it somehow."
[26:21] "You're not the only one who has this problem. We have a lot of companies who try to solve the same problem so try to reach the community."
[26:34] "Try to use tools which solve 80% of your use cases and maybe 20% you'll have the suffering but at least it's not 100% suffering."
[27:27] Friction points
[28:07] "It's no constraints. I just need to understand their pain and help."
[28:53] Interaction with Data Scientists
[29:21] "We have alignment sessions. We have different levels of representations. We share our progress."
[32:42] Build verse by decisions
[33:36] "Companies are pushed to deliver something for a good reason but I beg them to check their quality, etc."
[34:04] When to build or grab an open-source tool
[35:14] "When you don't have any legal regulatory restrictions, go with open-source."
[35:51] Build your own or buy open-source?
[36:57] "You can make very stupid mistakes which can be spotted by another person."
[37:11] Certain maturity and a certain number of engineers
[38:11] Startup to go with open-source
[38:32] "You need to understand what you need to buy."
[39:53] "Paying for something - you need to have use case as a tool, outgrow your own resources and own skills."
[40:14] Correct transition process
[40:56] "There are no other ways but to communicate with data scientists. Your team needs to have a close loop for future priorities, what to take with you and what to leave behind."
[44:51] What to use in monitoring piece
[45:36] Prometheus and Grafana
[48:07] Do you have automatic retriggering monitoring of Models set up?
[50:01] "If I'm building something, I cannot own its deployment decision which we need to obey. For us, it's something we need to discover and understand how to do it correctly."
[50:23] "Everyone wants to have just one button, click it and model it. Unfortunately, we cannot because of some restrictions at it."
[51:55] Hardware for on Prim model training
[52:38] "Machine Learning model prediction is a spear bomb."
[53:55] War or horror stories
[54:15] "Guys, don't do context switching!"
[55:54] "I won't say that Adyen is a company that allows you to make mistakes but you can make mistakes."