Lecture 11B: Monitoring ML Models (Full Stack Deep Learning - Spring 2021)
New course announcement ✨
We're teaching an in-person LLM bootcamp in the SF Bay Area on November 14, 2023. Come join us if you want to see the most up-to-date materials building LLM-powered products and learn in a hands-on environment.
https://www.scale.bythebay.io/llm-wor...
Hope to see some of you there!
--------------------------------------------------------------------------------------------- In this lecture, you will learn how to monitor models (aka, keeping them healthy in production).
00:00 - Introduction
01:10 - Model Performance Degrades Post-Deployment
04:00 - Data Drift
07:25 - What To Monitor?
10:42 - How To Measure When Things Change
24:07 - How To Tell If A Change Is Bad
26:19 - Tools For Monitoring
27:46 - Monitoring And Your Broader ML System
34:26 - Takeaways