Apache Spark Dashboard V2 Demo
This is a video on how to get started with 👉 Spark-Dashboard ➜ https://github.com/cerndb/spark-dashb... - A custom monitoring pipeline and dashboard for Apache Spark.
Navigating the complexities of Apache Spark jobs and resource management can be daunting. This video introduces the Spark-Dashboard, a custom solution designed to monitor and visualize vital metrics from Apache Spark's metrics system. Learn how to effortlessly track time series data for metrics, aiding in the analysis of memory usage, job delays, and more. This tool is invaluable for pinpointing performance bottlenecks and understanding the underlying causes of issues in Spark applications.
Links:
👉 Blog: Building an Apache Spark Performance Lab: Tools and Techniques for Spark Optimization https://db-blog.web.cern.ch/node/195
👉 sparkMeasure ➜ https://github.com/LucaCanali/sparkMe... a tool for performance troubleshooting of Apache Spark jobs
👉TPCDS PySpark https://github.com/LucaCanali/Miscell... ➜ A tool you can use run TPCDS with PySpark, instrumented with sparkMeasure
👉 Introductory course on Apache Spark ➜ https://sparktraining.web.cern.ch/
👉 CERN DB Blog ➜ https://db-blog.web.cern.ch/
👉 Luca's Home ➜ https://cern.ch/canali
Acknowledgements: A special acknowledgment goes out to the teams behind the CERN data analytics, monitoring, and web notebook services, as well as the dedicated members of the ATLAS database group.