How to monitor Spark and Python data pipelines with DataOps
Monitoring your data pipelines can be fairly complex. In this video I am showing you how easy it is to monitor your spark and python jobs with Kensu.io I could have used this many times before in my work as Data Engineer.
Link to Kensu website: https://www.kensu.io/
0:44 Kensu.io
1:11 Typical approaches to monitoring
3:04 Example of data pipelines
6:00 Typical problems with monitoring data pipelines
7:30 Data Observerability driven development
8:05 Hands-on monitoring the pipeline and finding the root cause
14:27 TLDR the future of Data Engineering
►Learn Data Engineering with my Data Engineering Academy:
https://learndataengineering.com
►LEARN MORE ABOUT DATA ENGINEERING:
-Check out my free 100+ page data engineering cookbook on GitHub:
https://github.com/andkret/Cookbook
►PLEASE SUPPORT WHAT YOU LIKE:
Just use this link whenever you buy something from Amazon:
https://www.amazon.com/shop/plumberso...
(As an Amazon Associate I earn from qualifying purchases from Amazon)
This is free of charge for you but super helpful for supporting this channel.
#dataengineering #kensu #dataobserverability