End to End Monitoring with Prometheus, Grafana, Apache Kafka and Spark - A Data Engineering Project
In this hands-on project, learn how to set up real-time monitoring for a high-performance data pipeline processing 1.2 billion records per hour! We cover everything from configuring Grafana and Prometheus to creating a comprehensive Kafka and Spark monitoring dashboard. Watch to see how modern data engineering tools come together for effective system monitoring.
PART 1: • End to End Monitoring of High Performance ...
Timestamps:
0:00 Introduction
1:08 Setting up Grafana
5:36 Linking Prometheus and Grafana
9:52 Grafana Dashboard Setup
13:50 Grafana, Kafka Realtime Monitoring Dashboard
19:42 Outro
Like this video? Support us: / @codewithyu
👀 Don't just watch it, build it! 🚧
👍 Like, Comment, & Subscribe for more cutting-edge data engineering content!
Resources:
Full Source Code:
https://buymeacoffee.com/yusuf.ganiyu...
Kafka Documentation: https://kafka.apache.org/documentation/
Apache Spark Documentation: https://spark.apache.org/documentatio...
JMX Exporter Agent - https://github.com/prometheus/jmx_exp...
#ApacheKafka, #Grafana, #Prometheus, #DataEngineering, #BigData, #RealTimeProcessing, #BigDataAnalytics, #DataPipeline, #StreamingData, #KafkaMonitoring, #SparkStreaming, #DataArchitecture, #HighPerformanceComputing